Mathematical Models of Overparameterized Neural Networks
From MaRDI portal
Publication:6357022
arXiv2012.13982MaRDI QIDQ6357022
Author name not available (Why is that?)
Publication date: 27 December 2020
Abstract: Deep learning has received considerable empirical successes in recent years. However, while many ad hoc tricks have been discovered by practitioners, until recently, there has been a lack of theoretical understanding for tricks invented in the deep learning literature. Known by practitioners that overparameterized neural networks are easy to learn, in the past few years there have been important theoretical developments in the analysis of overparameterized neural networks. In particular, it was shown that such systems behave like convex systems under various restricted settings, such as for two-layer NNs, and when learning is restricted locally in the so-called neural tangent kernel space around specialized initializations. This paper discusses some of these recent progresses leading to significant better understanding of neural networks. We will focus on the analysis of two-layer neural networks, and explain the key mathematical models, with their algorithmic implications. We will then discuss challenges in understanding deep neural networks and some current research directions.
Has companion code repository: https://github.com/hendrydong/NTK-and-MF-examples
This page was built for publication: Mathematical Models of Overparameterized Neural Networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6357022)
