Universality Laws for High-Dimensional Learning With Random Features
From MaRDI portal
Publication:6194015
DOI10.1109/TIT.2022.3217698arXiv2009.07669OpenAlexW4312426305MaRDI QIDQ6194015
No author found.
Publication date: 19 March 2024
Published in: IEEE Transactions on Information Theory (Search for Journal in Brave)
Abstract: We prove a universality theorem for learning with random features. Our result shows that, in terms of training and generalization errors, a random feature model with a nonlinear activation function is asymptotically equivalent to a surrogate linear Gaussian model with a matching covariance matrix. This settles a so-called Gaussian equivalence conjecture based on which several recent papers develop their results. Our method for proving the universality theorem builds on the classical Lindeberg approach. Major ingredients of the proof include a leave-one-out analysis for the optimization problem associated with the training process and a central limit theorem, obtained via Stein's method, for weakly correlated random variables.
Full work available at URL: https://arxiv.org/abs/2009.07669
universalityexact asymptoticsrandom feature modeloverparameterized neural networkGaussian equivalence
Related Items (3)
Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks ⋮ The curse of overparametrization in adversarial training: precise analysis of robust generalization for random features regression ⋮ Phase diagram of stochastic gradient descent in high-dimensional two-layer neural networks
This page was built for publication: Universality Laws for High-Dimensional Learning With Random Features
