Optimal approximation of piecewise smooth functions using deep ReLU neural networks

DOI10.1016/j.neunet.2018.08.019zbMath1434.68516arXiv1709.05289OpenAlexW2963146412WikidataQ91665574 ScholiaQ91665574MaRDI QIDQ2182898

Philipp Petersen, Felix Voigtlaender

Publication date: 26 May 2020

Published in: Neural Networks (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1709.05289

zbMATH Keywords

metric entropypiecewise smooth functionsfunction approximationcurse of dimensiondeep neural networkssparse connectivity

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Algorithms for approximation of functions (65D15)

Related Items (only showing first 100 items - show all)

Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black--Scholes Partial Differential Equations ⋮ Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space ⋮ Neural network approximation ⋮ A Proof that Artificial Neural Networks Overcome the Curse of Dimensionality in the Numerical Approximation of Black–Scholes Partial Differential Equations ⋮ SelectNet: self-paced learning for high-dimensional partial differential equations ⋮ Discretization of parameter identification in PDEs using neural networks ⋮ A deep learning approach to Reduced Order Modelling of parameter dependent partial differential equations ⋮ Multivariate neural network interpolation operators ⋮ Solving eigenvalue PDEs of metastable diffusion processes using artificial neural networks ⋮ Theoretical issues in deep networks ⋮ Quantitative Approximation Results for Complex-Valued Neural Networks ⋮ Approximations with deep neural networks in Sobolev time-space ⋮ Deep ReLU Networks Overcome the Curse of Dimensionality for Generalized Bandlimited Functions ⋮ Full error analysis for the training of deep neural networks ⋮ New Function Spaces Associated to Representations of Nilpotent Lie Groups and Generalized Time-Frequency Analysis ⋮ Neural Parametric Fokker--Planck Equation ⋮ Deep Neural Network Surrogates for Nonsmooth Quantities of Interest in Shape Uncertainty Quantification ⋮ Nonlinear approximation via compositions ⋮ On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces ⋮ Theory of deep convolutional neural networks: downsampling ⋮ Approximation bounds for norm constrained neural networks with applications to regression and GANs ⋮ Solving second-order nonlinear evolution partial differential equations using deep learning ⋮ Simultaneous neural network approximation for smooth functions ⋮ Deep Neural Networks for Solving Large Linear Systems Arising from High-Dimensional Problems ⋮ Neural network approximation: three hidden layers are enough ⋮ A deep network construction that adapts to intrinsic dimensionality beyond the domain ⋮ Theory of deep convolutional neural networks. III: Approximating radial functions ⋮ Approximation capabilities of neural networks on unbounded domains ⋮ On the capacity of deep generative networks for approximating distributions ⋮ De Rham compatible deep neural network FEM ⋮ Deep ReLU neural network approximation in Bochner spaces and applications to parametric PDEs ⋮ Approximating smooth and sparse functions by deep neural networks: optimal approximation rates and saturation ⋮ Rates of approximation by ReLU shallow neural networks ⋮ Neural Network Approximation of Coarse-Scale Surrogates in Numerical Homogenization ⋮ Three ways to solve partial differential equations with neural networks — A review ⋮ Deep learning methods for partial differential equations and related parameter identification problems ⋮ Fast convergence rates of deep neural networks for classification ⋮ On decision regions of narrow deep neural networks ⋮ Deep ReLU neural networks in high-dimensional approximation ⋮ A New Certified Hierarchical and Adaptive RB-ML-ROM Surrogate Model for Parametrized PDEs ⋮ Neural network-based limiter with transfer learning ⋮ Exponential ReLU neural network approximation rates for point and edge singularities ⋮ Space-time error estimates for deep neural network approximations for differential equations ⋮ A convenient infinite dimensional framework for generative adversarial learning ⋮ Phase transitions in rate distortion theory and deep learning ⋮ The universal approximation theorem for complex-valued neural networks ⋮ Unnamed Item ⋮ Computation and learning in high dimensions. Abstracts from the workshop held August 1--7, 2021 (hybrid meeting) ⋮ Approximation properties of residual neural networks for Kolmogorov PDEs ⋮ Learning elliptic partial differential equations with randomized linear algebra ⋮ Affine symmetries and neural network identifiability ⋮ Deep ReLU networks and high-order finite element methods ⋮ Error bounds for approximations with deep ReLU neural networks in Ws,p norms ⋮ Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems ⋮ Deep neural network expression of posterior expectations in Bayesian PDE inversion ⋮ Applied harmonic analysis and data processing. Abstracts from the workshop held March 25--31, 2018 ⋮ Deep ReLU network expression rates for option prices in high-dimensional, exponential Lévy models ⋮ Deep Network Approximation for Smooth Functions ⋮ Unnamed Item ⋮ Plateau Phenomenon in Gradient Descent Training of RELU Networks: Explanation, Quantification, and Avoidance ⋮ Better Approximations of High Dimensional Smooth Functions by Deep Neural Networks with Rectified Power Units ⋮ Dying ReLU and Initialization: Theory and Numerical Examples ⋮ Deep Network Approximation Characterized by Number of Neurons ⋮ Randomized Optimal Stopping Algorithms and Their Convergence Analysis ⋮ Asymptotic expansion for neural network operators of the Kantorovich type and high order of approximation ⋮ Efficient approximation of solutions of parametric linear transport equations by ReLU DNNs ⋮ Topological properties of the set of functions generated by neural networks of fixed size ⋮ Numerical solution of the parametric diffusion equation by deep neural networks ⋮ Universality of deep convolutional neural networks ⋮ Equivalence of approximation by convolutional neural networks and fully-connected networks ⋮ Error bounds for deep ReLU networks using the Kolmogorov-Arnold superposition theorem ⋮ Fast generalization error bound of deep learning without scale invariance of activation functions ⋮ Approximation rates for neural networks with encodable weights in smoothness spaces ⋮ Theory of deep convolutional neural networks. II: Spherical analysis ⋮ Optimal approximation rate of ReLU networks in terms of width and depth ⋮ Constructive deep ReLU neural network approximation ⋮ Sparsest piecewise-linear regression of one-dimensional data ⋮ Deep learning observables in computational fluid dynamics ⋮ Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations ⋮ The Gap between Theory and Practice in Function Approximation with Deep Neural Networks ⋮ Unnamed Item ⋮ Discontinuous neural networks and discontinuity learning ⋮ On the approximation of rough functions with deep neural networks ⋮ Deep Network With Approximation Error Being Reciprocal of Width to Power of Square Root of Depth ⋮ Adaptive machine learning-based surrogate modeling to accelerate PDE-constrained optimization in enhanced oil recovery ⋮ CAS4DL: Christoffel adaptive sampling for function approximation via deep learning ⋮ Stable recovery of entangled weights: towards robust identification of deep neural networks from minimal samples ⋮ A measure theoretical approach to the mean-field maximum principle for training NeurODEs ⋮ Approximation of Smoothness Classes by Deep Rectifier Networks ⋮ Optimal Approximation with Sparsely Connected Deep Neural Networks ⋮ New Error Bounds for Deep ReLU Networks Using Sparse Grids ⋮ Deep ReLU neural networks overcome the curse of dimensionality for partial integrodifferential equations ⋮ Deep neural networks can stably solve high-dimensional, noisy, non-linear inverse problems ⋮ Spline representation and redundancies of one-dimensional ReLU neural network models ⋮ DNN expression rate analysis of high-dimensional PDEs: application to option pricing ⋮ A theoretical analysis of deep neural networks and parametric PDEs ⋮ Neural network identifiability for a family of sigmoidal nonlinearities ⋮ Approximation spaces of deep neural networks ⋮ Exponential ReLU DNN expression of holomorphic maps in high dimension ⋮ Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation

Uses Software

• ImageNet
• AlexNet

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Compactly supported shearlets are optimally sparse
• Lower bounds for approximation by MLP neural networks
• Approximation and estimation bounds for artificial neural networks
• Unconditional bases are optimal bases for data compression and for statistical estimation
• Multilayer feedforward networks are universal approximators
• Error bounds for approximations with deep ReLU networks
• Entropies of several sets of real valued functions
• Introduction to Smooth Manifolds
• Introduction to Shearlets
• Group Invariant Scattering
• Optimally Sparse Data Representations
• An Introduction to Banach Space Theory
• Universal approximation bounds for superpositions of a sigmoidal function
• New tight frames of curvelets and optimal representations of objects with piecewise C² singularities
• A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction
• Real Analysis and Probability
• Representation and Compression of Multidimensional Piecewise Functions Using Surflets
• Optimally Sparse Multidimensional Representation Using Shearlets
• On the extension of a vector function so as to preserve a Lipschitz condition
• A logical calculus of the ideas immanent in nervous activity
• Approximation by superpositions of a sigmoidal function
• Sparse components of images and optimal atomic decompositions

This page was built for publication: Optimal approximation of piecewise smooth functions using deep ReLU neural networks