A Diffusion Approximation Theory of Momentum Stochastic Gradient Descent in Nonconvex Optimization

DOI10.1287/stsy.2021.0083zbMath1489.90097arXiv1802.05155OpenAlexW3210625965MaRDI QIDQ5084492

Tuo Zhao, Enlu Zhou, Tianyi Liu, Zhehui Chen

Publication date: 24 June 2022

Published in: Stochastic Systems (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1802.05155

zbMATH Keywords

diffusion approximationnonconvex optimizationsaddle points and optimaSGD with momentum

Mathematics Subject Classification ID

Quadratic programming (90C20) Stochastic programming (90C15) Methods of reduced gradient type (90C52)

Uses Software

• TensorFlow
• PyTorch
• Adam

Cites Work

• Unnamed Item
• Accelerated gradient methods for nonconvex nonlinear and stochastic programming
• Handbook of simulation optimization
• Stochastic approximation with two time scales
• The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning
• A mean field view of the landscape of two-layer neural networks
• Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
• Some methods of speeding up the convergence of iteration methods
• A Stochastic Approximation Method
• Stochastic differential equations. An introduction with applications.