ISAAC Newton: Input-based Approximate Curvature for Newton's Method

Author name not available (Why is that?)

Publication date: 30 April 2023

Abstract: We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size smaller than the number of neurons. We show that it is possible to compute a good conditioner based on only the input to a respective layer without a substantial computational overhead. The proposed method allows effective training even in small-batch stochastic regimes, which makes it competitive to first-order as well as second-order methods.

Has companion code repository: https://github.com/felix-petersen/isaac

This page was built for publication: ISAAC Newton: Input-based Approximate Curvature for Newton's Method

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6434854)