DoCoM: Compressed Decentralized Optimization with Near-Optimal Sample Complexity

Chung-Yiu Yau, Hoi-To Wai

Publication date: 1 February 2022

Abstract: This paper proposes the Doubly Compressed Momentum-assisted Stochastic Gradient Tracking algorithm (DoCoM-SGT) for communication efficient decentralized learning. DoCoM-SGT utilizes two compression steps per communication round as the algorithm tracks simultaneously the averaged iterate and stochastic gradient. Furthermore, DoCoM-SGT incorporates a momentum based technique for reducing variances in the gradient estimates. We show that DoCoM-SGT finds a solution

in

T

iterations satisfying

for non-convex objective functions; and we provide competitive convergence rate guarantees for other function classes. Numerical experiments on synthetic and real datasets validate the efficacy of our algorithm.

Has companion code repository: https://github.com/OscarYau525/docom

This page was built for publication: DoCoM: Compressed Decentralized Optimization with Near-Optimal Sample Complexity

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6389805)