ACMo: Angle-Calibrated Moment Methods for Stochastic Optimization
From MaRDI portal
Publication:6342752
arXiv2006.07065MaRDI QIDQ6342752
Author name not available (Why is that?)
Publication date: 12 June 2020
Abstract: Due to its simplicity and outstanding ability to generalize, stochastic gradient descent (SGD) is still the most widely used optimization method despite its slow convergence. Meanwhile, adaptive methods have attracted rising attention of optimization and machine learning communities, both for the leverage of life-long information and for the profound and fundamental mathematical theory. Taking the best of both worlds is the most exciting and challenging question in the field of optimization for machine learning. Along this line, we revisited existing adaptive gradient methods from a novel perspective, refreshing understanding of second moments. Our new perspective empowers us to attach the properties of second moments to the first moment iteration, and to propose a novel first moment optimizer, emph{Angle-Calibrated Moment method} (method). Our theoretical results show that method is able to achieve the same convergence rate as mainstream adaptive methods. Furthermore, extensive experiments on CV and NLP tasks demonstrate that method has a comparable convergence to SOTA Adam-type optimizers, and gains a better generalization performance in most cases.
Has companion code repository: https://github.com/Xunpeng746/ACMo
No records found.
This page was built for publication: ACMo: Angle-Calibrated Moment Methods for Stochastic Optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6342752)
