Stanza: Distributed Deep Learning with Small Communication Footprint

This work designs a new distributed training system called Stanza, which exploits the fact that in many models such as convolution neural networks, most data exchange is attributed to the fully connected layers, while most computation is carried out in convolutional layers.