Tianshou: a Highly Modularized Deep Reinforcement Learning Library

Weng, Jiayi; Chen, Huayu; Yan, Dong; You, Kaichao; Duburcq, Alexis; Zhang, Minghao; Su, Hang; Zhu, Jun

doi:10.48550/arxiv.2107.14171

Cited by 17 publications

(20 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Training and Evaluation We used the Tianshou reinforcement learning framework [37] to learn cryoRL. Each model was trained with 20 epochs, using the Adam optimizer and an initial learning rate of 0.01.…”

Section: Methodsmentioning

confidence: 99%

CryoRL: Reinforcement Learning Enables Efficient Cryo-EM Data Collection

Fan¹,

Li²,

Yao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Single-particle cryo-electron microscopy (cryo-EM) has become one of the mainstream structural biology techniques because of its ability to determine high-resolution structures of dynamic bio-molecules. However, cryo-EM data acquisition remains expensive and labor-intensive, requiring substantial expertise. Structural biologists need a more efficient and objective method to collect the best data in a limited time frame. We formulate the cryo-EM data collection task as an optimization problem in this work. The goal is to maximize the total number of good images taken within a specified period. We show that reinforcement learning offers an effective way to plan cryo-EM data collection, successfully navigating heterogenous cryo-EM grids. The approach we developed, cryoRL, demonstrates better performance than average users for data collection under similar settings.

show abstract

Section: Methodsmentioning

confidence: 99%

CryoRL: Reinforcement Learning Enables Efficient Cryo-EM Data Collection

Fan¹,

Li²,

Yao³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We evaluate the performance of PPO (Schulman et al 2017) and Dueling DQN (Wang et al 2016) on the costed version of the Open AI gym environments Cartpole, Acrobot and Lunar lander (Brockman et al 2016). The DRL algorithms are implemented using Pytorch and the Tianshou deep learning packages (Weng et al 2021), and the results were record with Weights and Biases (Biewald 2020). The experiments were executed on Ubuntu 18.04 desktop running a GeForce RTX 2080 Ti GPU.…”

Section: Methodsmentioning

confidence: 99%

Scientific Discovery and the Cost of Measurement -- Balancing Information and Cost in Reinforcement Learning

Bellinger¹,

Drozdyuk²,

Crowley³

et al. 2021

Preprint

View full text Add to dashboard Cite

The use of reinforcement learning (RL) in scientific applications, such as materials design and automated chemistry, is increasing. A major challenge, however, lies in fact that measuring the state of the system is often costly and time consuming in scientific applications, whereas policy learning with RL requires a measurement after each time step. In this work, we make the measurement costs explicit in the form of a costed reward and propose a framework that enables offthe-shelf deep RL algorithms to learn a policy for both selecting actions and determining whether or not to measure the current state of the system at each time step. In this way, the agents learn to balance the need for information with the cost of information. Our results show that when trained under this regime, the Dueling DQN and PPO agents can learn optimal action policies whilst making up to 50% fewer state measurements, and recurrent neural networks can produce a greater than 50% reduction in measurements. We postulate the these reduction can help to lower the barrier to applying RL to real-world scientific applications.

show abstract

“…Most of them are developed based on Python, with PyTorch and TensorFlow as automatic gradient solvers. Tensorforce [Kuhnle et al, 2017] and Tianshou [Weng et al, 2021] packages are used as an alternative ready-to-use backend to provide a wide range of reinforcement learning algorithms in the DRLinFluids platform.…”

Section: Functionalitymentioning

confidence: 99%

“…Currently, there is no general and mature platform for simplifying the application of DRL in OpenFOAM simulations. This paper proposes an open-source python platform, DRLinFluids, for coupling DRL and OpenFOAM based on reliable DRL packages, including Tensorforce [Kuhnle et al, 2017] and Tianshou [Weng et al, 2021], and OpenFOAM [Jasak et al, 2007]. DRLinFluids is a flexible and scalable platform to utilize DRL in the field of computational fluid mechanics, even in continuum mechanics.…”

Section: Introductionmentioning

confidence: 99%

DRLinFluids -- An open-source python platform of coupling Deep Reinforcement Learning and OpenFOAM

Qiulei¹,

Liu²,

Hu³

et al. 2022

Preprint

View full text Add to dashboard Cite

We propose an open-source python platform for applications of Deep Reinforcement Learning (DRL) in fluid mechanics. DRL has been widely used in optimizing decision-making in nonlinear and high-dimensional problems. Here, an agent maximizes a cumulative reward with learning a feedback policy by acting in an environment. In control theory terms, the cumulative reward would correspond to the cost function, the agent to the actuator, the environment to the measured signals and the learned policy to the feedback law. Thus, DRL assumes an interactive environment or, equivalently, control plant. The setup of a numerical simulation plant with DRL is challenging and time-consuming. In this work, a novel python platform, named DRLinFluids is developed for this purpose, with DRL for flow control and optimization problems in fluid mechanics. The simulations employ OpenFOAM as popular, flexible Navier-Stokes solver in industry and academia, and Tensorforce or Tianshou as widely used versatile DRL packages. The reliability and efficiency of DRLinFluids are demonstrated for two wake stabilization benchmark problems. DRLinFluids significantly reduces the application effort of DRL in fluid mechanics and is expected to greatly accelerates academic and industrial applications.

show abstract

Tianshou: a Highly Modularized Deep Reinforcement Learning Library

Cited by 17 publications

References 16 publications

CryoRL: Reinforcement Learning Enables Efficient Cryo-EM Data Collection

CryoRL: Reinforcement Learning Enables Efficient Cryo-EM Data Collection

Scientific Discovery and the Cost of Measurement -- Balancing Information and Cost in Reinforcement Learning

DRLinFluids -- An open-source python platform of coupling Deep Reinforcement Learning and OpenFOAM

Contact Info

Product

Resources

About