Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD

arXiv2002.09539MaRDI QIDQ6335340

Author name not available (Why is that?)

Publication date: 21 February 2020

Abstract: Distributed stochastic gradient descent (SGD) is essential for scaling the machine learning algorithms to a large number of computing nodes. However, the infrastructures variability such as high communication delay or random node slowdown greatly impedes the performance of distributed SGD algorithm, especially in a wireless system or sensor networks. In this paper, we propose an algorithmic approach named Overlap-Local-SGD (and its momentum variant) to overlap the communication and computation so as to speedup the distributed training procedure. The approach can help to mitigate the straggler effects as well. We achieve this by adding an anchor model on each node. After multiple local updates, locally trained models will be pulled back towards the synchronized anchor model rather than communicating with others. Experimental results of training a deep neural network on CIFAR-10 dataset demonstrate the effectiveness of Overlap-Local-SGD. We also provide a convergence guarantee for the proposed algorithm under non-convex objective functions.

Has companion code repository: https://github.com/JYWa/Overlap_Local_SGD

This page was built for publication: Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6335340)