MuZero with Self-competition for Rate Control in VP9 Video Compression

Mandhane, Amol; Zhernov, Anton; Rauh, Maribeth; Gu, Chenjie; Wang, Miaosen; Xue, Flora; Shang, Wendy; Pang, Derek; Claus, Rene A.; Chiang, Ching-Han; Chen, Cheng; Han, Jingning; Chen, Angie; Mankowitz, Daniel J.; Broshear, Jackson; Schrittwieser, Julian; Hübert, Thomas; Vinyals, Oriol; Mann, Timothy

doi:10.48550/arxiv.2202.06626

Cited by 8 publications

(9 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It provided impressive results e.g. in automated theorem proving [27], games of chess and Go [30], Atari benchmark [29] and video compression [19].…”

Section: Related Workmentioning

confidence: 99%

Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search

Michał¹,

Tyrolski²,

Czechowski³

et al. 2022

Preprint

View full text Add to dashboard Cite

Complex reasoning problems contain states that vary in the computational cost required to determine a good action plan. Taking advantage of this property, we propose Adaptive Subgoal Search (AdaSubS), a search method that adaptively adjusts the planning horizon. To this end, AdaSubS generates diverse sets of subgoals at different distances. A verification mechanism is employed to filter out unreachable subgoals swiftly and thus allowing to focus on feasible further subgoals. In this way, AdaSubS benefits from the efficiency of planning with longer subgoals and the fine control with the shorter ones. We show that AdaSubS significantly surpasses hierarchical planning algorithms on three complex reasoning tasks: Sokoban, the Rubik's Cube, and inequality proving benchmark INT, setting new state-of-the-art on INT. * equal contribution, MZ and MT implemented most of the experiments, KC led the project

show abstract

“…It provided impressive results e.g. in automated theorem proving [27], games of chess and Go [30], Atari benchmark [29] and video compression [19].…”

Section: Related Workmentioning

confidence: 99%

Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search

Michał¹,

Tyrolski²,

Czechowski³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…An average-case performance metric R that represents good performance on the normal task. 3 Given an "in-distribution" input distribution over X and a per-example scoring function r ∶ X × Y → R, the performance objective is to maximize the overall score:…”

Section: General Setting: High-stakes Reliabilitymentioning

confidence: 99%

“…Advances in deep learning have led to increasingly powerful AI systems, for example in sequential decision making [1,2,3,4], robotics [5,6], and language modeling and text-based reasoning [7,8,9,10,11]. Steering powerful AI systems may be challenging [12,13], and we expect this to become more difficult as AI systems become more powerful [14,15,16].…”

Section: Introductionmentioning

confidence: 99%

Adversarial Training for High-Stakes Reliability

Ziegler¹,

Nix²,

Chan³

et al. 2022

Preprint

View full text Add to dashboard Cite

In the future, powerful AI systems may be deployed in high-stakes settings, where a single failure could be catastrophic. One technique for improving AI safety in high-stakes settings is adversarial training, which uses an adversary to generate examples to train on in order to achieve better worst-case performance. In this work, we used a language generation task as a testbed for achieving high reliability through adversarial training. We created a series of adversarial training techniques-including a tool that assists human adversaries-to find and eliminate failures in a classifier that filters text completions suggested by a generator. In our simple "avoid injuries" task, we determined that we can set very conservative classifier thresholds without significantly impacting the quality of the filtered outputs. With our chosen thresholds, filtering with our baseline classifier decreases the rate of unsafe completions from about 2.4% to 0.003% on in-distribution data, which is near the limit of our ability to measure. We found that adversarial training significantly increased robustness to the adversarial attacks that we trained on, without affecting in-distribution performance. We hope to see further work in the high-stakes reliability setting, including more powerful tools for enhancing human adversaries and better ways to measure high levels of reliability, until we can confidently rule out the possibility of catastrophic deployment-time failures of powerful models.

show abstract

“…MuZero has demonstrated state-of-the-art capabilities over a variety of deterministic or neardeterministic environments, such as Go, Chess, Shogi and Atari, and has been successfully applied to real-world domains such as video compression (Mandhane et al, 2022). Although here we focus on MuZero for deterministic environments, we note that extensions to stochastic environments also exist (Antonoglou et al, 2021) and are an interesting target for future work.…”

Section: Introductionmentioning

confidence: 99%

Equivariant MuZero

Deac¹,

Weber²,

Papamakarios³

2023

Preprint

View full text Add to dashboard Cite

Deep reinforcement learning repeatedly succeeds in closed, well-defined domains such as games (Chess, Go, StarCraft). The next frontier is real-world scenarios, where setups are numerous and varied. For this, agents need to learn the underlying rules governing the environment, so as to robustly generalise to conditions that differ from those they were trained on. Model-based reinforcement learning algorithms, such as the highly successful MuZero, aim to accomplish this by learning a world model. However, leveraging a world model has not consistently shown greater generalisation capabilities compared to model-free alternatives. In this work, we propose improving the data efficiency and generalisation capabilities of MuZero by explicitly incorporating the symmetries of the environment in its world-model architecture. We prove that, so long as the neural networks used by MuZero are equivariant to a particular symmetry group acting on the environment, the entirety of MuZero's action-selection algorithm will also be equivariant to that group. We evaluate Equivariant MuZero on procedurally-generated MiniPacman and on Chaser from the ProcGen suite: training on a set of mazes, and then testing on unseen rotated versions, demonstrating the benefits of equivariance. Further, we verify that our performance improvements hold even when only some of the components of Equivariant MuZero obey strict equivariance, which highlights the robustness of our construction.

show abstract

MuZero with Self-competition for Rate Control in VP9 Video Compression

Cited by 8 publications

References 22 publications

Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search

Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search

Adversarial Training for High-Stakes Reliability

Equivariant MuZero

Contact Info

Product

Resources

About