Understanding approximate Fisher information for fast convergence of natural gradient descent in wide neural networks*

DOI10.1088/1742-5468/ac3ae3OpenAlexW3098534458MaRDI QIDQ5020052

Kazuki Osawa, Ryo Karakida

Publication date: 3 January 2022

Published in: Journal of Statistical Mechanics: Theory and Experiment (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/2010.00879

zbMATH Keywords

analysis of algorithmsdeep learning

Mathematics Subject Classification ID

Statistical mechanics, structure of matter (82-XX)

Related Items (2)

Sketch-based empirical natural gradient methods for deep learning ⋮ SCORE: approximating curvature information under self-concordant regularization

Cites Work

• Unnamed Item
• On the discrete analogues of some generalizations of Gronwall's inequality
• Riemannian metrics for neural networks I: feedforward networks
• Tridiagonal Toeplitz matrices: properties and novel applications
• Wide neural networks of any depth evolve as linear models under gradient descent ^*
• Explicit inverses of some tridiagonal matrices

This page was built for publication: Understanding approximate Fisher information for fast convergence of natural gradient descent in wide neural networks*