Stochastic Zeroth Order Gradient and Hessian Estimators: Variance Reduction and Refined Bias Bounds

DOI10.1093/IMAIAI/IAAD014arXiv2205.14737MaRDI QIDQ6400483

Yasong Feng, Tianyu Wang

Publication date: 29 May 2022

Abstract: We study stochastic zeroth order gradient and Hessian estimators for real-valued functions in $mathbb{R}^n$. We show that, via taking finite difference along random orthogonal directions, the variance of the stochastic finite difference estimators can be significantly reduced. In particular, we design estimators for smooth functions such that, if one uses $Thetaleft(kight)$ random directions sampled from the Stiefel's manifold $extSt(n,k)$ and finite-difference granularity $delta$, the variance of the gradient estimator is bounded by $mathcalOleft(left(fracnk-1ight)+left(frac{n}^{2}k-night)delt{a}^2+frac{n}^{2}delt{a}^{4}kight)$, and the variance of the Hessian estimator is bounded by $mathcalOleft(left(frac{n}^2{k}^2-1ight)+left(frac{n}^4{k}^2-n^{2}ight)delt{a}^2+frac{n}^{4}delt{a}^4{k}^{2}ight)$. When $k=n$, the variances become negligibly small. In addition, we provide improved bias bounds for the estimators. The bias of both gradient and Hessian estimators for smooth function $f$ is of order $mathcalOleft(delt{a}^{2}Gammaight)$, where $delta$ is the finite-difference granularity, and $Gamma$ depends on high order derivatives of $f$. Our results are evidenced by empirical observations.

Mathematics Subject Classification ID

Statistics (62-XX)

Related Items (1)

On the global complexity of a derivative-free Levenberg-Marquardt algorithm via orthogonal spherical smoothing

This page was built for publication: Stochastic Zeroth Order Gradient and Hessian Estimators: Variance Reduction and Refined Bias Bounds

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6400483)