Debugging a Policy: Automatic Action-Policy Testing in AI Planning

Abstract

Testing is a promising way to gain trust in neural action policies π. Previous work on policy testing in sequential decision making targeted environment behavior leading to failure conditions. But if the failure is unavoidable given that behavior, then π is not actually to blame. For a situation to qualify as a "bug" in π, there must be an alternative policy π' that does better. We introduce a generic policy testing framework based on that intuition. This raises the bug confirmation problem, deciding whether or not a state is a bug. We analyze the use of optimistic and pessimistic bounds for the design of test oracles approximating that problem. We contribute an implementation of our framework in classical planning, experimenting with several test oracles and with random-walk methods generating test states biased to poor policy performance and/or state novelty. We evaluate these techniques on policies π learned with ASNets. We find that they are able to effectively identify bugs in these π, and that our random-walk biases improve over uninformed baselines.

New Fuzzing Biases for Action Policy Testing

“…For our experiments, we extend the testing framework by Eisenhut et al (2023), which builds on the one by Steinmetz et al (2022). As in this previous work, we test AS-Net policies (Toyer et al 2018(Toyer et al , 2020.…”

“…Recent work (Steinmetz et al 2022) addresses this in the context of classical planning, defining test cases as states s, and policy bugs as states on which the policy is sub-optimal (a better policy exists for s, e.g., avoiding failure). Given a test case s, test oracles are used to detect whether s is a bug, by evaluating sufficient criteria that avoid the need for a full optimal planning process (Eisenhut et al 2023).…”

“…Prior work (Steinmetz et al 2022) only explored a simple bias function, mapping a state s to the summed-up action cost of the policy trace in s. Here, we investigate this topic more deeply. We introduce three new fuzzing biases that aim at analyzing the policy-trace shape.…”

“…We adopt the testing framework of Steinmetz et al (2022): Given a task Π and a policy π, a state t is called a bug in π if cost π (t) > h ⋆ (t). 1 To find bugs, policy testing (1) generates a pool P ⊆ S of test states using a fuzzing process, and (2) runs test oracles on each pool state.…”

“…Steinmetz et al (2022), see Algorithm 1. Within a runtime limit T , we iteratively generate a pool P of maximal size N by random walks of maximal length L. Random walks start from a randomly selected pool state s 0 and are steered by a bias function B, influencing which successor state s ′ is selected in each step of the random walk.…”