Model-Based Offline Planning with Trajectory Pruning

Abstract

Graph neural network (GNN) is popular now to solve the tasks in non-Euclidean space and most of them learn deep embeddings by aggregating the neighboring nodes. However, these methods are prone to some problems such as over-smoothing because of the single-scale perspective field and the nature of low-pass filter. To address these limitations, we introduce diffusion scattering network (DSN) to exploit high-order patterns. With observing the complementary relationship between multi-layer GNN and DSN, we propose Hierarchical Diffusion Scattering Graph Neural Network (HDS-GNN) to efficiently bridge DSN and GNN layer by layer to supplement GNN with multi-scale information and band-pass signals. Our model extracts node-level scattering representations by intercepting the low-pass filtering, and adaptively tunes the different scales to regularize multi-scale information. Then we apply hierarchical representation enhancement to improve GNN with the scattering features. We benchmark our model on nine real-world networks on the transductive semi-supervised node classification task. The experimental results demonstrate the effectiveness of our method.

Beijing Language and Culture University, China

“…Value regularization methods [8,14,32] regularize the value function to assign low values on OOD actions. Uncertainty-based and model-based methods [1,30,[36][37][38] estimate the epistemic uncertainty from value functions or learned models to penalize OOD data. Finally, in-sample learning methods [3,9,31] learn the value function entirely within data to avoid directly querying the Q-function on OOD actions produced by policies.…”

Uncertainty-Driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learning

“…We, instead, aim at offering higherquality synthetic data for offline training via an uncertainty-based trajectory truncation method. The most relevant to our work are M2AC [33], MOReL [17], and MOPP [45]. MOReL requires the generated single sample (instead of the trajectory) at each step to lie in a safe region.…”

“…However, MBPO fails to resolve the issue of extrapolation error in the offline setting. Modern model-based offline RL methods usually leverage methods like uncertainty quantification [44,17], penalizing value function to enforce conservatism [43], planning [45], for learning meaningful policies from static logged data. Utilizing the learned dynamics model for offline data augmentation is also explored recently [38,27].…”

Discriminator-Guided Model-Based Offline Imitation Learning

“…Incorporating the Dynamics Model. Model-based approaches have been widely adopted in RL to improve sample efficiency and shows good performance and generalization ability in recent offline RL studies [20,53,56]. In our work, we introduce a probabilistic dynamics model implemented using a neural network that outputs a Gaussian distribution over the difference between the current and next state, i.e., f (s |s, a) = N (s + µ θ f (s, a), Σ θ f (s, a)), where µ θ f (s, a) and Σ θ f (s, a) are the parameterized mean and diagonal covariance matrix.…”

“…The sample efficiency requirement for offline IL methods reminds us of the success of model-based approaches in the online and offline RL domains [21,20,53,56]. Dynamics models learned from the data can greatly supplement the limited expert data to improve state-action space coverage, leading to potentially improved policy performance and generalizability [20,56,15,5]. However, adopting a model-based approach in offline IL is still an underexplored area [15,5,37].…”