Search Direction Correction with Normalized Gradient Makes First-Order Methods Faster

DOI10.1137/20M1335480zbMath1477.90107arXiv1905.06507OpenAlexW3201216965MaRDI QIDQ3382800

ZaiWen Wen, Yi-Fei Wang, Zeyu Jia

Publication date: 22 September 2021

Published in: SIAM Journal on Scientific Computing (Search for Journal in Brave)

Full work available at URL: https://arxiv.org/abs/1905.06507

zbMATH Keywords

stochastic optimizationLyapunov functionfirst-order methodscomposite optimizationsearch direction correction

Mathematics Subject Classification ID

Numerical mathematical programming methods (65K05) Nonlinear programming (90C30) Stochastic programming (90C15)

Uses Software

• L-BFGS
• LIBSVM
• FPC_AS
• CIFAR
• AdaGrad
• Adam
• Saga

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
• On the limited memory BFGS method for large scale optimization
• Introductory lectures on convex optimization. A basic course.
• A regularized semi-smooth Newton method with projection steps for composite convex programs
• First-order optimization algorithms via inertial systems with Hessian driven damping
• Adaptive restart for accelerated gradient schemes
• A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
• A Fast Algorithm for Sparse Reconstruction Based on Shrinkage, Subspace Optimization, and Continuation
• Analysis and Design of Optimization Algorithms via Integral Quadratic Constraints
• Two-Point Step Size Gradient Methods
• Sparse Reconstruction by Separable Approximation
• A variational perspective on accelerated methods in optimization
• A Nonmonotone Line Search Technique and Its Application to Unconstrained Optimization
• A Semismooth Newton Method with Multidimensional Filter Globalization for $l_1$-Optimization
• A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization
• A Proximal Stochastic Gradient Method with Progressive Variance Reduction

This page was built for publication: Search Direction Correction with Normalized Gradient Makes First-Order Methods Faster