Trajectory balance: Improved credit assignment in GFlowNets

Malkin, Nikolay; Moksh, Jain,; Bengio, Emmanuel; Sun, Cheng; Bengio, Yoshua

doi:10.48550/arxiv.2201.13259

Cited by 6 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(22) PB(s | s ′ ; θ)F (s; θ) Several alternative learning objectives for GFlowNets have been proposed, especially for longer trajectories to sample the object y (Malkin et al, 2022;Madan et al, 2022). Trajectory Balance (Malkin et al, 2022, TB) is a prominent learning objective for training GFlowNets.…”

Section: Multiple Parent Statesmentioning

confidence: 99%

GFlowNets for AI-driven scientific discovery

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Multiple Parent Statesmentioning

confidence: 99%

GFlowNets for AI-driven scientific discovery

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

“…Since in the PRL framework, with the update of the 𝑄 soft , the energy-based policy distribution is also constantly changing. DPO adopts a joint training framework where the EBM and the GFlowNet are optimized alternately, similar with [99]: The energy function serves as the negative log-reward function for the GFlowNet, which is trained with the trajectory balance [54] objective to sample from the evolving energy-based policies. In contrast, the energy function is trained with soft Bellman backup, where the GFlowNet provides diverse samples.…”

Section: Diverse Policy Optimizationmentioning

confidence: 99%

“…To train the parameters 𝜃 𝐹 and 𝜃 𝐵 of the reward-conditional GFlowNet, we use the trajectory balance objective [54] that optimizes the following objective along complete trajectories 𝜏 = (𝑠…”

Section: Reward-conditional Gflownet Trainingmentioning

confidence: 99%

“…Since 𝑍 represents the total flow in the entire flow network, many samples are required for an accurate estimation. Unlike the original work of trajectory balance [54], in our setting, the scalar function 𝑍 needs to condition on the external environmental state 𝑠 𝑒 thus has higher sample complexity. Interestingly, since the target EBM of GFlowNets is derived from the PRL framework in our method, 𝑍 has an additional physical meaning, i.e., the soft value function 𝑉 * soft (•) in §2.1.…”

Section: Reward-conditional Gflownet Trainingmentioning

confidence: 99%

“…𝜃 𝐵 , 𝜃 𝑍 }. The scalar function 𝑍 (•) is parametrized in the log domain, as suggested by Malkin et al[54]. With the trajectory balance objective, we train the reward-conditional GFlowNet with stochastic gradient E 𝜏∼𝜋 Θ (𝜏 |𝑠 𝑒 ) [∇ Θ L Θ (𝜏 |𝑠 𝑒 )] with some training trajectory distribution 𝜋 Θ (𝜏).…”

mentioning

confidence: 99%

See 2 more Smart Citations

Diverse Policy Optimization for Structured Action Space

Li¹,

Wang²,

Yang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Enhancing the diversity of policies is beneficial for robustness, exploration, and transfer in reinforcement learning (RL). In this paper, we aim to seek diverse policies in an under-explored setting, namely RL tasks with structured action spaces with the two properties of composability and local dependencies. The complex action structure, non-uniform reward landscape, and subtle hyperparameter tuning due to the properties of structured actions prevent existing approaches from scaling well. We propose a simple and effective RL method, Diverse Policy Optimization (DPO), to model the policies in structured action space as the energy-based models (EBM) by following the probabilistic RL framework. A recently proposed novel and powerful generative model, GFlowNet, is introduced as the efficient, diverse EBM-based policy sampler. DPO follows a joint optimization framework: the outer layer uses the diverse policies sampled by the GFlowNet to update the EBM-based policies, which supports the GFlowNet training in the inner layer. Experiments on ATSC and Battle benchmarks demonstrate that DPO can efficiently discover surprisingly diverse policies in challenging scenarios and substantially outperform existing state-of-the-art methods.

show abstract