Statistical Inference on Multi-armed Bandits with Delayed Feedback

Jingshen Wang, Lei Shi, Tianhao Wu

Publication date: 3 July 2023

Abstract: Multi armed bandit (MAB) algorithms have been increasingly used to complement or integrate with A/B tests and randomized clinical trials in e-commerce, healthcare, and policymaking. Recent developments incorporate possible delayed feedback. While existing MAB literature often focuses on maximizing the expected cumulative reward outcomes (or, equivalently, regret minimization), few efforts have been devoted to establish valid statistical inference approaches to quantify the uncertainty of learned policies. We attempt to fill this gap by providing a unified statistical inference framework for policy evaluation where a target policy is allowed to differ from the data collecting policy, and our framework allows delay to be associated with the treatment arms. We present an adaptively weighted estimator that on one hand incorporates the arm-dependent delaying mechanism to achieve consistency, and on the other hand mitigates the variance inflation across stages due to vanishing sampling probability. In particular, our estimator does not critically depend on the ability to estimate the unknown delay mechanism. Under appropriate conditions, we prove that our estimator converges to a normal distribution as the number of time points goes to infinity, which provides guarantees for large-sample statistical inference. We illustrate the finite-sample performance of our approach through Monte Carlo experiments.

Has companion code repository: https://github.com/leishi-rocks/delaybandits

Mathematics Subject Classification ID

Asymptotic distribution theory in statistics (62E20) Theory of statistical experiments (62B15) Learning and adaptive systems in artificial intelligence (68T05)

This page was built for publication: Statistical Inference on Multi-armed Bandits with Delayed Feedback

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6442337)