Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration

Author name not available (Why is that?)

Publication date: 29 May 2023

Abstract: In online reinforcement learning (online RL), balancing exploration and exploitation is crucial for finding an optimal policy in a sample-efficient way. To achieve this, existing sample-efficient online RL algorithms typically consist of three components: estimation, planning, and exploration. However, in order to cope with general function approximators, most of them involve impractical algorithmic components to incentivize exploration, such as optimization within data-dependent level-sets or complicated sampling procedures. To address this challenge, we propose an easy-to-implement RL framework called extit{Maximize to Explore} ( exttt{MEX}), which only needs to optimize emph{unconstrainedly} a single objective that integrates the estimation and planning components while balancing exploration and exploitation automatically. Theoretically, we prove that exttt{MEX} achieves a sublinear regret with general function approximations for Markov decision processes (MDP) and is further extendable to two-player zero-sum Markov games (MG). Meanwhile, we adapt deep RL baselines to design practical versions of exttt{MEX}, in both model-free and model-based manners, which can outperform baselines by a stable margin in various MuJoCo environments with sparse rewards. Compared with existing sample-efficient online RL algorithms with general function approximations, exttt{MEX} achieves similar sample efficiency while enjoying a lower computational cost and is more compatible with modern deep RL methods.

Has companion code repository: https://github.com/agentification/MEX

This page was built for publication: Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6438368)