Controller Compilation and Compression for Resource Constrained Applications

Abstract

Abstract. Recent advances in planning techniques for partially observable Markov decision processes have focused on online search techniques and offline point-based value iteration. While these techniques allow practitioners to obtain policies for fairly large problems, they assume that a non-negligible amount of computation can be done between each decision point. In contrast, the recent proliferation of mobile and embedded devices has lead to a surge of applications that could benefit from state of the art planning techniques if they can operate under severe constraints on computational resources. To that effect, we describe two techniques to compile policies into controllers that can be executed by a mere table lookup at each decision point. The first approach compiles policies induced by a set of alpha vectors (such as those obtained by point-based techniques) into approximately equivalent controllers, while the second approach performs a simulation to compile arbitrary policies into approximately equivalent controllers. We also describe an approach to compress controllers by removing redundant and dominated nodes, often yielding smaller and yet better controllers. The compilation and compression techniques are demonstrated on benchmark problems as well as a mobile application to help Alzheimer patients to way-find.

Approximate Linear Programming for Constrained Partially Observable Markov Decision Processes

“…It outperforms the state of the art in terms of both solution quality and time. The policies found are finite state controllers which are also advantageous for deployment in resource constrained applications such as embedded systems as well as smartphones (Grześ, Poupart, and Hoey 2013b). In future work, it would be interesting to extend this work to constrained decentralized POMDPs (Wu, Jennings, and Chen 2012) and to explore reinforcement learning techniques for CPOMDPs.…”

Stochastic Shortest Path with Energy Constraints in POMDPs

“…One crucial difference between previous approaches to policy succinctness in both POMDP [22] and other settings [25] is that in previous work they concurrently optimize both the performance of a policy and its size, which requires dedicated algorithms, while we separate these tasks: first we search for a well-performing, though possibly "ugly" policy, and then learn its succinct representation (similar approach was used in [23], where policies computed by point-based methods were "compiled" into FSCs). Thus, we present a framework for obtaining succinct representations in which various state-of-the art algorithms for POMDP solving and DT learning can be used.…”

Probabilistic Inference Techniques for Scalable Multiagent Decision Making

“…There are other approaches to compute policies for infinite-horizon Dec-POMDPs that are not based on a controller representation of the joint-policy (MacDermed & Isbell, 2013). However, a key advantage of policies based on finite-state controllers is their ease of execution in resource constrained environments (Grzes, Poupart, & Hoey, 2013;Grześ, Poupart, Yang, & Hoey, 2015), without any expensive belief update operations required in other approaches. Furthermore, policies represented as finite-state controllers can carry more semantic information, where each controller node summarizes some relevant aspects of the observation history.…”