Planning from Pixels in Atari with Learned Symbolic Representations

Abstract

Width-based planning methods have been shown to yield state-of-the-art performance in the Atari 2600 domain using pixel input. One successful approach, RolloutIW, represents states with the B-PROST boolean feature set. An augmented version of RolloutIW, pi-IW, shows that learned features can be competitive with handcrafted ones for width-based search. In this paper, we leverage variational autoencoders (VAEs) to learn features directly from pixels in a principled manner, and without supervision. The inference model of the trained VAEs extracts boolean features from pixels, and RolloutIW plans with these features. The resulting combination outperforms the original RolloutIW and human professional play on Atari 2600 and drastically reduces the size of the feature set.

Is Policy Learning Overrated?: Width-Based Planning and Active Learning for Atari

“…VAE-IW (Dittadi, Drachmann, and Bolander 2021) extends RIW to learn encodings from screen images. In a training stage, the game is run using RIW until a fixed number of screens are encountered and saved.…”

“…Learning the features from data permits the encoding to be tailored to a specific game, but generating a faithful encoding of a game is non-trivial, and using a static dataset is typically insufficient. For example, Dittadi, Drachmann, and Bolander (2021) save screens that are reached by a Rollout-IW agent using a hand-coded B-PROST feature set, and use those screens to train a Binary-Concrete VAE to produce a game-specific encoding. However, in order for the encoding to be representative of a game, the dataset must include screens from all visually distinct parts, such as separate levels.…”

“…For every configuration, we trained the system with 5 different random seeds, then evaluated the result for 10 episodes (playthroughs) for 50 evaluations total. Following existing work (Junyent, Jonsson, and Gómez 2019;Junyent, Gómez, and Jonsson 2021;Dittadi, Drachmann, and Bolander 2021), we use a discount factor of γ = 0.99 (rather than 0.995 in (Bandres, Bonet, and Geffner 2018)) in line 28. However, note that the reported final scores are undiscounted sums of rewards.…”

“…Resource Constraints: We repeated the setup of previous work (Lipovetzky and Geffner 2012;Bandres, Bonet, and Geffner 2018;Dittadi, Drachmann, and Bolander 2021) where each action is held for 15 frames, which we refer to as taking a single action. We define a single simulator call as an update to the simulator when taking a single action.…”

“…π-IW discretizes the last hidden layer of a policy function as the input. Recently, VAE-IW (Dittadi, Drachmann, and Bolander 2021) obtains a compact binary representation by training a Binary-Concrete VAE offline.…”