An Inertial Newton Algorithm for Deep Learning

zbMath1482.68211arXiv1905.12278MaRDI QIDQ5159400

Camille Castera, Cédric Févotte, Edouard Pauwels, Jérôme Bolte

Publication date: 27 October 2021

Full work available at URL: https://arxiv.org/abs/1905.12278

zbMATH Keywords

stochastic optimizationnonconvex optimizationdynamical systemsecond-order methodsdeep learning

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Nonconvex programming, global optimization (90C26) Stochastic programming (90C15)

Related Items

Asymptotic behavior of Newton-like inertial dynamics involving the sum of potential and nonpotential terms ⋮ The rate of convergence of optimization algorithms obtained via discretizations of heavy ball dynamical systems for convex optimization problems ⋮ Newton-type inertial algorithms for solving monotone equations Governed by sums of potential and nonpotential operators ⋮ Convergence of iterates for first-order optimization algorithms with inertia and Hessian driven damping ⋮ Subgradient Sampling for Nonsmooth Nonconvex Minimization ⋮ An Improved Unconstrained Approach for Bilevel Optimization ⋮ A fast and simple modification of Newton's method avoiding saddle points ⋮ Inertial Newton algorithms avoiding strict saddle points ⋮ First order inertial optimization algorithms with threshold effects associated with dry friction ⋮ Conservative parametric optimality and the ridge method for tame min-max problems ⋮ Convergence of inertial dynamics driven by sums of potential and nonpotential operators with implicit Newton-like damping ⋮ Fast optimization via inertial dynamics with closed-loop damping ⋮ Continuous Newton-like Methods Featuring Inertia and Variable Mass ⋮ Unnamed Item ⋮ On the effect of perturbations in first-order optimization methods with inertia and Hessian driven damping

Uses Software

• NumPy
• Matplotlib
• Python
• TensorFlow
• CIFAR
• PyTorch
• Adam

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• A Stochastic Quasi-Newton Method for Large-Scale Optimization
• Proximal alternating linearized minimization for nonconvex and nonsmooth problems
• A dynamical system associated with Newton's method for parametric approximations of convex minimization problems
• On gradients of functions definable in o-minimal structures
• A second-order gradient-like dissipative dynamical system with Hessian-driven damping. Application to optimization and mechanics.
• Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning
• Convergence of constant step stochastic gradient descent for non-smooth non-convex functions
• Convergence rate of inertial proximal algorithms with general extrapolation and proximal coefficients
• Newton-type methods for non-convex optimization under inexact Hessian information
• Stochastic subgradient method converges on tame functions
• On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning
• Proximal Alternating Minimization and Projection Methods for Nonconvex Problems: An Approach Based on the Kurdyka-Łojasiewicz Inequality
• Clarke Subgradients of Stratifiable Functions
• Characterizations of Łojasiewicz inequalities: Subgradient flows, talweg, convexity
• Analysis of recursive stochastic algorithms
• Stochastic Methods for Composite and Weakly Convex Optimization Problems
• Optimization Methods for Large-Scale Machine Learning
• An investigation of Newton-Sketch and subsampled Newton methods
• Convergence and Dynamical Behavior of the ADAM Algorithm for Nonconvex Stochastic Optimization
• Stochastic Approximations and Differential Inclusions
• Learning representations by back-propagating errors
• The Łojasiewicz Inequality for Nonsmooth Subanalytic Functions with Applications to Subgradient Dynamical Systems
• Some methods of speeding up the convergence of iteration methods
• A Stochastic Approximation Method