VeLO: Training Versatile Learned Optimizers by Scaling Up

Author name not available (Why is that?)

Publication date: 17 November 2022

Abstract: While deep learning models have replaced hand-designed features across many domains, these models are still trained with hand-designed optimizers. In this work, we leverage the same scaling approach behind the success of deep learning to learn versatile optimizers. We train an optimizer for deep learning which is itself a small neural network that ingests gradients and outputs parameter updates. Meta-trained with approximately four thousand TPU-months of compute on a wide variety of optimization tasks, our optimizer not only exhibits compelling performance, but optimizes in interesting and unexpected ways. It requires no hyperparameter tuning, instead automatically adapting to the specifics of the problem being optimized. We open source our learned optimizer, meta-training code, the associated train and test data, and an extensive optimizer benchmark suite with baselines at velo-code.github.io.

Has companion code repository: https://github.com/google/learned_optimization/tree/main/learned_optimization/research/general_lopt

This page was built for publication: VeLO: Training Versatile Learned Optimizers by Scaling Up

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6417618)