Dying ReLU and Initialization: Theory and Numerical Examples

DOI10.4208/cicp.OA-2020-0165OpenAlexW3101398262MaRDI QIDQ5162356

Yeonjong Shin, Lu Lu, Yan-Hui Su, George Em. Karniadakis

Publication date: 2 November 2021

Published in: Communications in Computational Physics (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1903.06733

zbMATH Keywords

neural networkdying ReLUrandomized asymmetric initializationvanishing/exploding gradient

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Learning and adaptive systems in artificial intelligence (68T05) Neural nets and related approaches to inference from stochastic processes (62M45)

Related Items (11)

Artificial-neural-network-based nonlinear algebraic models for large-eddy simulation of compressible wall-bounded turbulence ⋮ A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data ⋮ A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions ⋮ A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions ⋮ Training thinner and deeper neural networks: jumpstart regularization ⋮ Active Neuron Least Squares: A Training Method for Multivariate Rectified Neural Networks ⋮ Estimating permeability of 3D micro-CT images by physics-informed CNNs based on DNS ⋮ Learning Specialized Activation Functions for Physics-Informed Neural Networks ⋮ Convex and concave envelopes of artificial neural network activation functions for deterministic global optimization ⋮ A weight initialization based on the linear product structure for neural networks ⋮ PDE-Aware Deep Learning for Inverse Problems in Cardiac Electrophysiology

Uses Software

• ImageNet
• AdaGrad
• Adam
• ADADELTA

Cites Work

• Introductory lectures on convex optimization. A basic course.
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Error bounds for approximations with deep ReLU networks
• Relu Deep Neural Networks and Linear Finite Elements
• A Limited Memory Algorithm for Bound Constrained Optimization
• Singularities Affect Dynamics of Learning in Neuromanifolds
• Unnamed Item
• Unnamed Item
• Unnamed Item

This page was built for publication: Dying ReLU and Initialization: Theory and Numerical Examples