The loss surfaces of neural networks with general activation functions

DOI10.1088/1742-5468/abfa1eOpenAlexW3015328986MaRDI QIDQ3382363

Francesco Mezzadri, Nicholas P. Baskerville, Joseph Najnudel, Jonathan P. Keating

Publication date: 21 September 2021

Published in: Journal of Statistical Mechanics: Theory and Experiment (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/2004.03959

zbMATH Keywords

spin glassesmachine learningdeep learningrandom matrix theory and extensions

Mathematics Subject Classification ID

Statistical mechanics, structure of matter (82-XX)

Related Items (9)

Superposition of random plane waves in high spatial dimensions: Random matrix approach to landscape complexity ⋮ Optimization landscape in the simplest constrained random least-square problem ⋮ Counting equilibria in a random non-gradient dynamics with heterogeneous relaxation rates ⋮ Landscape complexity beyond invariance and the elastic manifold ⋮ On random matrices arising in deep neural networks: General I.I.D. case ⋮ Local convexity of the TAP free energy and AMP convergence for \(\mathbb{Z}_2\)-synchronization ⋮ Exponential growth of random determinants beyond invariance ⋮ A spin glass model for the loss surfaces of generative adversarial networks ⋮ Universal characteristics of deep neural network loss surfaces from random matrix theory

Uses Software

• MNIST
• PyTorch
• BERT
• StyleGAN

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• A Fourier view on the \(R\)-transform and related asymptotics of spherical integrals
• StyleGAN
• Large deviations for Wigner's law and Voiculescu's non-commutative entropy
• Characteristic polynomials of random Hermitian matrices and Duistermaat-Heckman localisation on non-compact Kähler manifolds
• Global asymptotics for multiple integrals with boundaries.
• Large deviations of the extreme eigenvalues of random deformations of matrices
• On random matrix averages involving half-integer powers of GOE characteristic polynomials
• Replica symmetry breaking condition exposed by random matrix calculation of landscape complexity
• Statistical Mechanics of Learning
• Complexity of Random Energy Landscapes, Glass Transition, and Absolute Value of the Spectral Determinant of Random Matrices
• A shortcut through the Coulomb gas method for spectral linear statistics on random matrices
• Quenched complexity of the mean-fieldp-spin spherical model with external magnetic field
• Statistical Physics of Spin Glasses and Information Processing
• Random Matrices and Complexity of Spin Glasses
• The Landscape of the Spiked Tensor Model
• The space of interactions in neural network models
• Comparing dynamics: deep neural networks versus glassy systems
• Aging of spherical spin glasses

This page was built for publication: The loss surfaces of neural networks with general activation functions