A robust multi-batch L-BFGS method for machine learning

DOI10.1080/10556788.2019.1658107zbMath1430.90523arXiv1707.08552OpenAlexW2970227300WikidataQ127324376 ScholiaQ127324376MaRDI QIDQ4972551

Albert S. Berahas, Martin Takáč

Publication date: 25 November 2019

Published in: Optimization Methods and Software (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1707.08552

zbMATH Keywords

samplingoverlapconsistencyL-BFGSfault-tolerantmulti-batch

Mathematics Subject Classification ID

Numerical mathematical programming methods (65K05) Large-scale problems in mathematical programming (90C06) Nonlinear programming (90C30) Methods of quasi-Newton type (90C53) Computational aspects of data analysis and big data (68T09)

Related Items (10)

Quasi-Newton methods for machine learning: forget the past, just sample ⋮ Limited-memory BFGS with displacement aggregation ⋮ Nonmonotone diagonally scaled limited-memory BFGS methods with application to compressive sensing based on a penalty model ⋮ An overview of stochastic quasi-Newton methods for large-scale machine learning ⋮ Globally Convergent Multilevel Training of Deep Residual Networks ⋮ Adaptive sampling quasi-Newton methods for zeroth-order stochastic optimization ⋮ Trust-region algorithms for training responses: machine learning methods using indefinite Hessian approximations ⋮ A robust multi-batch L-BFGS method for machine learning ⋮ Diagonally scaled memoryless quasi-Newton methods with application to compressed sensing ⋮ LSOS: Line-search second-order stochastic optimization methods for nonconvex finite sums

Uses Software

• L-BFGS
• SGD-QN
• PyTorch
• GitHub

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• A Stochastic Quasi-Newton Method for Large-Scale Optimization
• Sample size selection in optimization methods for machine learning
• On the limited memory BFGS method for large scale optimization
• The BFGS method with exact line searches fails for non-convex objective functions
• Single-machine and parallel-machine serial-batching scheduling problems with position-based learning effect and linear setup time
• On the Global Convergence of the BFGS Method for Nonconvex Unconstrained Optimization Problems
• Global Convergence of Online Limited Memory BFGS
• Hybrid Deterministic-Stochastic Methods for Data Fitting
• On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning
• Distributed asynchronous deterministic and stochastic gradient optimization algorithms
• Updating Quasi-Newton Matrices with Limited Storage
• Algorithms for nonlinear constraints that use lagrangian functions
• Numerical Optimization
• Adaptive Sampling Strategies for Stochastic Optimization
• Perturbed Iterate Analysis for Asynchronous Stochastic Optimization
• Optimization Methods for Large-Scale Machine Learning
• Convergence Properties of the BFGS Algoritm
• A robust multi-batch L-BFGS method for machine learning
• Quasi-Newton methods for machine learning: forget the past, just sample
• An investigation of Newton-Sketch and subsampled Newton methods
• Redundancy Techniques for Straggler Mitigation in Distributed Optimization and Learning
• Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
• Quasi-Newton Methods and their Application to Function Minimisation
• A Family of Variable-Metric Methods Derived by Variational Means
• A new approach to variable metric algorithms
• Conditioning of Quasi-Newton Methods for Function Minimization
• Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization
• A Stochastic Approximation Method
• Exact and inexact subsampled Newton methods for optimization
• A modified BFGS method and its global convergence in nonconvex minimization

This page was built for publication: A robust multi-batch L-BFGS method for machine learning