The interpolation phase transition in neural networks: memorization and generalization under lazy training

DOI10.1214/22-AOS2211MaRDI QIDQ2105197

Yanyan Li

Publication date: 8 December 2022

Published in: The Annals of Statistics (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/2007.12826

zbMATH Keywords

overparametrizationkernel ridge regressionoverfittingmemorizationneural tangent kernel

Mathematics Subject Classification ID

Ridge regression; shrinkage estimators (Lasso) (62J07) Linear regression; mixed models (62J05)

Related Items (6)

Deep learning: a statistical viewpoint ⋮ Adversarial examples in random neural networks with general activations ⋮ Dense Hebbian neural networks: a replica symmetric picture of supervised learning ⋮ High-Dimensional Analysis of Double Descent for Linear Regression with Random Projections ⋮ Controlled descent training ⋮ The interpolation phase transition in neural networks: memorization and generalization under lazy training

Cites Work

• Unnamed Item
• Unnamed Item
• Spherical harmonics and approximations on the unit sphere. An introduction
• On the capabilities of multilayer perceptrons
• Linearized two-layers neural networks in high dimension
• The interpolation phase transition in neural networks: memorization and generalization under lazy training
• Surprises in high-dimensional ridgeless least squares interpolation
• Generalization error of random feature and kernel methods: hypercontractivity and kernel matrix concentration
• Gradient descent optimizes over-parameterized deep ReLU networks
• Just interpolate: kernel ``ridgeless regression can generalize
• A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics
• Positive definite functions on spheres
• Theoretical Insights Into the Optimization Landscape of Over-Parameterized Shallow Neural Networks
• High-Dimensional Probability
• When do neural networks outperform kernel methods?*
• Two Models of Double Descent for Weak Features
• Memory Capacity of Neural Networks with Threshold and Rectified Linear Unit Activations
• The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve
• Benign overfitting in linear regression
• Reconciling modern machine-learning practice and the classical bias–variance trade-off
• Breaking the Curse of Dimensionality with Convex Neural Networks
• Spherical Harmonics in p Dimensions
• Enumeration of Seven-Argument Threshold Functions
• Wide neural networks of any depth evolve as linear models under gradient descent ^*
• Deep learning: a statistical viewpoint

This page was built for publication: The interpolation phase transition in neural networks: memorization and generalization under lazy training