Activation function design for deep networks: linearity and effective initialisation

DOI10.1016/j.acha.2021.12.010OpenAlexW4225674648MaRDI QIDQ2134109

Yanyan Li

Publication date: 6 May 2022

Published in: Applied and Computational Harmonic Analysis (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/2105.07741

zbMATH Keywords

random networksdeep learninginitialisationactivation function design

Mathematics Subject Classification ID

Biology and other natural sciences (92-XX) Computer science (68-XX)

Related Items

A decoupled physics-informed neural network for recovering a space-dependent force function in the wave equation from integral overdetermination data

Cites Work

• Unnamed Item
• Unnamed Item
• High dimensional robust M-estimation: asymptotic variance via approximate message passing
• Statistical decision theory and Bayesian analysis. 2nd ed
• One-sided inference about functionals of a density
• On the Lambert \(w\) function
• Bayesian learning for neural networks
• Accurate Prediction of Phase Transitions in Compressed Sensing via a Connection to Minimax Denoising
• Distributions
• Variation Diminishing Transformations: A Direct Approach to Total Positivity and its Statistical Applications
• Stable architectures for deep neural networks
• The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
• Learning representations by back-propagating errors
• Robust Estimation of a Location Parameter
• Robust Statistics