Neural Collapse with Cross-Entropy Loss

DOI10.1016/J.ACHA.2021.12.011arXiv2012.08465MaRDI QIDQ6356198

Jianfeng Lu, Stefan Steinerberger

Publication date: 15 December 2020

Abstract: We consider the variational problem of cross-entropy loss with $n$ feature vectors on a unit hypersphere in $mathbb{R}^d$. We prove that when $dgeqn-1$, the global minimum is given by the simplex equiangular tight frame, which justifies the neural collapse behavior. We also prove that as $nightarrowinfty$ with fixed $d$, the minimizing points will distribute uniformly on the hypersphere and show a connection with the frame potential of Benedetto & Fickus.

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Nonconvex programming, global optimization (90C26) General harmonic expansions, frames (42C15)

This page was built for publication: Neural Collapse with Cross-Entropy Loss

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6356198)