“Google's TPU pods feature torus-shaped interconnects and low-level software collectives like all-reduce, which were derived from supercomputing and proved suitable for distributed deep learning.”