“Low-level software collectives like all-reduce, originating from supercomputing, proved to be the right foundation for building distributed deep learning systems.”