Preserved central model for faster bidirectional compression in distributed settings

Author name not available (Why is that?)

Publication date: 24 February 2021

Abstract: We develop a new approach to tackle communication constraints in a distributed learning problem with a central server. We propose and analyze a new algorithm that performs bidirectional compression and achieves the same convergence rate as algorithms using only uplink (from the local workers to the central server) compression. To obtain this improvement, we design MCM, an algorithm such that the downlink compression only impacts local models, while the global model is preserved. As a result, and contrary to previous works, the gradients on local servers are computed on perturbed models. Consequently, convergence proofs are more challenging and require a precise control of this perturbation. To ensure it, MCM additionally combines model compression with a memory mechanism. This analysis opens new doors, e.g. incorporating worker dependent randomized-models and partial participation.

Has companion code repository: https://github.com/philipco/artemis-bidirectional-compression

This page was built for publication: Preserved central model for faster bidirectional compression in distributed settings

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6361490)