To add to this slide, yes parallelism and hardware specialization are essential, but cross-stack co-design is another key way we'll see higher performance realize in practice. Instead of having folks who build algorithms, folks who implement them efficiently, and folks who build hardware all work separately, expertise across this entire stack (i.e., algorithm--software--hardware co-design) is already being used to find more "optimal" ways of doing things. The slides effectively gave an example sort-of like that with MobileNet. But an excellent source on this is Stanford's very own Foundation Models paper, written by ~100 authors from several departments, particularly Section 4.5 on "Systems". https://arxiv.org/pdf/2108.07258.pdf

^ This is quite similar to the discussion on DSLs and hardware specialization, but perhaps more extreme. We've talked in CS149 about the importance of abstractions (vs. implementations) but co-design emphasizes blurring these lines momentarily and re-designing these abstractions to accommodate solutions that take efficiency and domain-specific considerations (e.g., ML quality) into consideration from the start.

It seems like Moore's narrowly defined will stop soon, but if it is broadened to say that the amount of 'work' (flops, size of problem solved, etc.) we can do towards the most computational important problems doubles every two years, it might continue for a bit longer.