HesScale: Scalable Computation of Hessian Diagonals

Author name not available (Why is that?)

Publication date: 20 October 2022

Abstract: Second-order optimization uses curvature information about the objective function, which can help in faster convergence. However, such methods typically require expensive computation of the Hessian matrix, preventing their usage in a scalable way. The absence of efficient ways of computation drove the most widely used methods to focus on first-order approximations that do not capture the curvature information. In this paper, we develop HesScale, a scalable approach to approximating the diagonal of the Hessian matrix, to incorporate second-order information in a computationally efficient manner. We show that HesScale has the same computational complexity as backpropagation. Our results on supervised classification show that HesScale achieves high approximation accuracy, allowing for scalable and efficient second-order optimization.

Has companion code repository: https://github.com/mohmdelsayed/hesscale

This page was built for publication: HesScale: Scalable Computation of Hessian Diagonals

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6414622)