Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning

DOI10.1007/s10107-020-01501-5zbMath1471.65057arXiv1909.10300OpenAlexW3016321495MaRDI QIDQ2039229

Jérôme Bolte, Edouard Pauwels

Publication date: 2 July 2021

Published in: Mathematical Programming. Series A. Series B (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1909.10300

zbMATH Keywords

automatic differentiationClarke subdifferentialbackpropagation algorithmo-minimal structuresstochastic gradientdeep learningdefinable setsfirst order methodsnonsmooth stochastic optimization

Mathematics Subject Classification ID

Large-scale problems in mathematical programming (90C06) Numerical optimization and variational techniques (65K10) Learning and adaptive systems in artificial intelligence (68T05) Set-valued and variational analysis (49J53) Decomposition methods (49M27) Neural nets and related approaches to inference from stochastic processes (62M45)

Related Items (20)

Convergence of constant step stochastic gradient descent for non-smooth non-convex functions ⋮ Convergence of a Piggyback-Style Method for the Differentiation of Solutions of Standard Saddle-Point Problems ⋮ Modeling design and control problems involving neural network surrogates ⋮ Global convergence of the gradient method for functions definable in o-minimal structures ⋮ Lyapunov stability of the subgradient method with constant step size ⋮ Subgradient Sampling for Nonsmooth Nonconvex Minimization ⋮ An Improved Unconstrained Approach for Bilevel Optimization ⋮ Stochastic approximation with discontinuous dynamics, differential inclusions, and applications ⋮ Certifying the Absence of Spurious Local Minima at Infinity ⋮ Sufficient Conditions for Instability of the Subgradient Method with Constant Step Size ⋮ Differentiating Nonsmooth Solutions to Parametric Monotone Inclusion Problems ⋮ Conservative parametric optimality and the ridge method for tame min-max problems ⋮ Densities of almost surely terminating probabilistic programs are differentiable almost everywhere ⋮ Incremental without replacement sampling in nonconvex optimization ⋮ An Inertial Newton Algorithm for Deep Learning ⋮ Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation ⋮ Perturbed iterate SGD for Lipschitz continuous loss functions ⋮ The Structure of Conservative Gradient Fields ⋮ Learning Maximally Monotone Operators for Image Recovery ⋮ Examples of Pathological Dynamics of the Subgradient Method for Lipschitz Path-Differentiable Functions

Uses Software

• TensorFlow
• DiffSharp
• PyTorch

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• On Lipschitz optimization based on gray-box piecewise linearization
• Proximal alternating linearized minimization for nonconvex and nonsmooth problems
• Tangencially continuous directional derivatives in nonsmooth analysis
• On gradients of functions definable in o-minimal structures
• Essentially smooth Lipschitz functions
• Integration of subdifferentials of lower semicontinuous functions on Banach spaces
• Geometric categories and o-minimal structures
• Stochastic subgradient method converges on tame functions
• On the maximal monotonicity of subdifferential mappings
• Generalized subdifferentials: a Baire categorical approach
• On stable piecewise linearization and generalized algorithmic differentiation
• Clarke Subgradients of Stratifiable Functions
• Evaluating Derivatives
• Optimization and nonsmooth analysis
• Nonsmooth Analysis: Differential Calculus of Nondifferentiable Mappings
• Analysis of recursive stochastic algorithms
• Variational Analysis
• A Chain Rule for Essentially Smooth Lipschitz Functions
• THE HEAVY BALL WITH FRICTION METHOD, I. THE CONTINUOUS DYNAMICAL SYSTEM: GLOBAL EXPLORATION OF THE LOCAL MINIMA OF A REAL-VALUED FUNCTION BY ASYMPTOTIC ANALYSIS OF A DISSIPATIVE DYNAMICAL SYSTEM
• Optimization Methods for Large-Scale Machine Learning
• Variational Analysis of Regular Mappings
• Constant step stochastic approximations involving differential inclusions: stability, long-run convergence and applications
• Convergence and Dynamical Behavior of the ADAM Algorithm for Nonconvex Stochastic Optimization
• Stochastic Approximations and Differential Inclusions
• Learning representations by back-propagating errors
• Integrability of subdifferentials of directionally Lipschitz functions
• A Stochastic Approximation Method
• Proof of the gradient conjecture of R. Thom.

This page was built for publication: Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning