State-Based Recurrent SPMNs for Decision-Theoretic Planning under Partial Observability

Abstract

The sum-product network (SPN) has been extended to model sequence data with the recurrent SPN (RSPN), and to decision-making problems with sum-product-max networks (SPMN). In this paper, we build on the concepts introduced by these extensions and present state-based recurrent SPMNs (S-RSPMNs) as a generalization of SPMNs to sequential decision-making problems where the state may not be perfectly observed. As with recurrent SPNs, S-RSPMNs utilize a repeatable template network to model sequences of arbitrary lengths. We present an algorithm for learning compact template structures by identifying unique belief states and the transitions between them through a state matching process that utilizes augmented data. In our knowledge, this is the first data-driven approach that learns graphical models for planning under partial observability, which can be solved efficiently. S-RSPMNs retain the linear solution complexity of SPMNs, and we demonstrate significant improvements in compactness of representation and the run time of structure learning and inference in sequential domains.

Inference and Learning in Dynamic Decision Networks Using Knowledge Compilation

“…Similar to mapl-cirup, SPUDD (Hoey et al 1999) is a variation of the classic value iteration algorithm using Although SPUDD was introduced over two decades ago, it is still considered the state-of-the-art approach for solving factored MDPs exactly, as mentioned in multiple recent publications on approximate methods (Hayes et al 2021;Heß, Sundermann, and Thüm 2021;Moreira et al 2021;Dudek, Shrotri, and Vardi 2022;Tan and Nejat 2022). Vlasselaer et al (2016) investigate how dynamic Bayesian networks (Murphy 2002) benefit from knowledge compilation techniques.…”