Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach

arXiv2210.05177MaRDI QIDQ6413523

Author name not available (Why is that?)

Publication date: 11 October 2022

Abstract: Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes. One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight. However, we find the indiscriminate perturbation of SAM on all parameters is suboptimal, which also results in excessive computation, i.e., double the overhead of common optimizers like Stochastic Gradient Descent (SGD). In this paper, we propose an efficient and effective training scheme coined as Sparse SAM (SSAM), which achieves sparse perturbation by a binary mask. To obtain the sparse mask, we provide two solutions which are based onFisher information and dynamic sparse training, respectively. In addition, we theoretically prove that SSAM can converge at the same rate as SAM, i.e., $O(logT/sqrtT)$. Sparse SAM not only has the potential for training acceleration but also smooths the loss landscape effectively. Extensive experimental results on CIFAR10, CIFAR100, and ImageNet-1K confirm the superior efficiency of our method to SAM, and the performance is preserved or even better with a perturbation of merely 50% sparsity. Code is availiable at https://github.com/Mi-Peng/Sparse-Sharpness-Aware-Minimization.

Has companion code repository: https://github.com/dbahri/universal_sam

This page was built for publication: Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6413523)