Convergence and Dynamical Behavior of the ADAM Algorithm for Nonconvex Stochastic Optimization

DOI10.1137/19M1263443zbMath1465.90050arXiv1810.02263MaRDI QIDQ5147028

Anas Barakat, Pascal Bianchi

Publication date: 2 February 2021

Published in: SIAM Journal on Optimization (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1810.02263

zbMATH Keywords

dynamical systemsstochastic approximationadaptive gradient methodsconditional central limit theorem.

Mathematics Subject Classification ID

Central limit and other weak theorems (60F05) Numerical mathematical programming methods (65K05) Nonconvex programming, global optimization (90C26) Stochastic programming (90C15) Initial value problems, existence, uniqueness, continuous dependence and continuation of solutions to ordinary differential equations (34A12) Stochastic approximation (62L20) Nonautonomous smooth dynamical systems (37C60)

Related Items (7)

Deterministic neural networks optimization from a continuous and energy point of view ⋮ A control theoretic framework for adaptive gradient optimizers ⋮ Taming Neural Networks with TUSLA: Nonconvex Learning via Adaptive Stochastic Gradient Langevin Algorithms ⋮ Stochastic optimization with momentum: convergence, fluctuations, and traps avoidance ⋮ Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning ⋮ Incremental without replacement sampling in nonconvex optimization ⋮ An Inertial Newton Algorithm for Deep Learning

Uses Software

• AdaGrad
• RMSprop
• Adam

Cites Work

• Proximal alternating linearized minimization for nonconvex and nonsmooth problems
• On the convergence of the proximal algorithm for nonsmooth functions involving analytic features
• Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing
• Stochastic heavy ball
• Stochastic subgradient method converges on tame functions
• The Convergence Problem for Dissipative Autonomous Systems
• On the long time behavior of second order differential equations with asymptotically small dissipation
• THE HEAVY BALL WITH FRICTION METHOD, I. THE CONTINUOUS DYNAMICAL SYSTEM: GLOBAL EXPLORATION OF THE LOCAL MINIMA OF A REAL-VALUED FUNCTION BY ASYMPTOTIC ANALYSIS OF A DISSIPATIVE DYNAMICAL SYSTEM
• Asymptotic Behavior of a Markovian Stochastic Algorithm with Constant Step
• Constant step stochastic approximations involving differential inclusions: stability, long-run convergence and applications
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item

This page was built for publication: Convergence and Dynamical Behavior of the ADAM Algorithm for Nonconvex Stochastic Optimization