Full error analysis for the training of deep neural networks

DOI10.1142/S021902572150020XzbMath1492.65132arXiv1910.00121OpenAlexW2977465327MaRDI QIDQ5083408

Christian Beck, Arnulf Jentzen, Benno Kuckuck

Publication date: 20 June 2022

Published in: Infinite Dimensional Analysis, Quantum Probability and Related Topics (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1910.00121

zbMATH Keywords

convergence analysisartificial neural networksmachine learningapproximation errorgeneralization erroroptimization errorfull error analysis

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Error analysis and interval analysis (65G99)

Related Items

A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions ⋮ A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions ⋮ Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation ⋮ Lower bounds for artificial neural network approximations: a proof that shallow neural networks fail to overcome the curse of dimensionality ⋮ Learning the random variables in Monte Carlo simulations with stochastic gradient descent: Machine learning for parametric PDEs and financial derivative pricing

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Tractability of multivariate problems. Volume I: Linear information
• Tractability of multivariate problems. Volume II: Standard information for functionals.
• Approximation and estimation bounds for artificial neural networks
• Multilayer feedforward networks are universal approximators
• General multilevel adaptations for stochastic approximation algorithms of Robbins-Monro and Polyak-Ruppert type
• Provable approximation properties for deep neural networks
• A distribution-free theory of nonparametric regression
• Degree of approximation by neural and translation networks with a single hidden layer
• Approximation of functions and their derivatives: A neural network implementation with applications
• Exponential convergence of the deep neural network approximation for analytic functions
• Topological properties of the set of functions generated by neural networks of fixed size
• Solving the Kolmogorov PDE by means of deep learning
• Proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients
• DNN expression rate analysis of high-dimensional PDEs: application to option pricing
• A theoretical analysis of deep neural networks and parametric PDEs
• Approximation spaces of deep neural networks
• Universal approximations of invariant maps by neural networks
• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• On the approximation by single hidden layer feedforward neural networks with fixed weights
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Gradient descent optimizes over-parameterized deep ReLU networks
• Nonlinear approximation via compositions
• A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics
• A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations
• Error bounds for approximations with deep ReLU networks
• Lower error bounds for the stochastic gradient descent optimization algorithm: sharp convergence rates for slowly and fast decaying learning rates
• Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
• Local Rademacher complexities
• On the mathematical foundations of learning
• Deep vs. shallow networks: An approximation theory perspective
• Sets of Finite Perimeter and Geometric Variational Problems
• Universal approximation bounds for superpositions of a sigmoidal function
• Neural Networks for Localized Approximation
• Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ
• Strong error analysis for stochastic gradient descent optimization algorithms
• Deep Neural Network Approximation Theory
• Optimal Approximation with Sparsely Connected Deep Neural Networks
• Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black--Scholes Partial Differential Equations
• Error bounds for approximations with deep ReLU neural networks in Ws,p norms
• Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems
• Deep Neural Networks Algorithms for Stochastic Control Problems on Finite Horizon: Convergence Analysis
• Deep Network Approximation Characterized by Number of Neurons
• On Stochastic Gradient Langevin Dynamics with Dependent Data Streams: The Fully Nonconvex Case
• Equivalence of approximation by convolutional neural networks and fully-connected networks
• Probability Inequalities for Sums of Bounded Random Variables
• Breaking the Curse of Dimensionality with Convex Neural Networks
• Understanding Machine Learning
• Approximation by superpositions of a sigmoidal function
• Error bounds for approximation with neural networks