Limitations of neural network training due to numerical instability of backpropagation

DOI10.1007/s10444-024-10106-xarXiv2210.00805MaRDI QIDQ6122651

Clemens Karner, Philipp Petersen, Vladimir A. Kazeev

Publication date: 1 March 2024

Published in: Advances in Computational Mathematics (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/2210.00805

zbMATH Keywords

numerical stabilitygradient descentround-off errorsdeep neural networks

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Wave front sets in context of PDEs (35A18) Error analysis and interval analysis (65G99)

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Quantized tensor-structured finite elements for second-order elliptic PDEs in two dimensions
• Provable approximation properties for deep neural networks
• The Deep Ritz Method: a deep learning-based numerical algorithm for solving variational problems
• Topological properties of the set of functions generated by neural networks of fixed size
• A theoretical analysis of deep neural networks and parametric PDEs
• Tensor rank bounds for point singularities in \(\mathbb{R}^3\)
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Stability of low-rank tensor representations and structured multilevel preconditioning for elliptic PDEs
• Error bounds for approximations with deep ReLU networks
• Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• QTT-finite-element approximation for multiscale problems. I: Model problems in one dimension
• Robust Stochastic Approximation Approach to Stochastic Programming
• Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ
• Deep Convolutional Neural Network for Inverse Problems in Imaging
• Accuracy and Stability of Numerical Algorithms
• Solving high-dimensional partial differential equations using deep learning
• Optimal Approximation with Sparsely Connected Deep Neural Networks
• Quantized Tensor FEM for Multiscale Problems: Diffusion Problems in Two and Three Dimensions
• NETT: solving inverse problems with deep neural networks
• Solving inverse problems using data-driven models
• Understanding Machine Learning

This page was built for publication: Limitations of neural network training due to numerical instability of backpropagation