Random Latent Exploration for Deep Reinforcement Learning

Srinath Mahankali, Zhang-Wei Hong, Ayush Sekhari, Alexander Rakhlin, Pulkit Agrawal

Improbable AI Lab

Computer Science and Artificial Intelligence Laboratory (CSAIL)

Massachusetts Institute of Technology (MIT)

International Conference on Machine Learning (ICML), 2024

Exploration is a longstanding challenge in reinforcement learning (RL), which is typically addressed through either noise-based or bonus-based exploration strategies. Noise-based methods are easy to implement but struggle with deep exploration, while bonus-based methods, though more complex, excel in tasks requiring deep exploration. Since both approaches perform similarly on average, noise-based exploration is commonly used in deep RL algorithms due to its simplicity. In this paper, we introduce Random Latent Exploration (RLE), a simple yet effective exploration strategy that outperforms both methods on average. The core idea is to motivate the agent to explore different parts of the environment by pursuing randomly sampled goals. The agent's policy is conditioned on randomly sampled vectors that serve as goals, and these vectors provide varying rewards at each state, encouraging the agent to explore different areas of the environment. RLE is as simple as noise-based methods, as it avoids complex bonus calculations. Our experiments show that RLE improves performance on average in both discrete (e.g., Atari) and continuous control tasks (e.g., Isaac Gym), enhancing exploration while remaining a simple and general plug-in for existing RL algorithms.