900 Gb/s is an mind-melting figure. I never would have imagined that our hardware is capable of reaching such a number.

@matteosantamaria the V100 is from several years ago, and the newer A100 has over 1500 GB/s bandwidth. Things just get faster and faster...

Appreciated how clear Kayvon made the distinction between latency and bandwidth, and especially the implications of each (i.e., when low latency matters and when low bandwidth matters).

I'm curious to know how much these distances of the memories to the processor truly make a difference in memory throughput. I'm not an electrical engineer, but I am curious how engineers go about designing the layout of such systems. Is it working within a set of physical constraints or just choosing from a subset of combinatorial possibilites