Opportunities of Hybrid Model-based Reinforcement Learning for Cell Therapy Manufacturing Process Control

Abstract

Driven by the key challenges of cell therapy manufacturing, including high complexity, high uncertainty, and very limited process observations, we propose a hybrid model-based reinforcement learning (RL) to efficiently guide process control. We first create a probabilistic knowledge graph (KG) hybrid model characterizing the risk-and science-based understanding of biomanufacturing process mechanisms and quantifying inherent stochasticity, e.g., batch-to-batch variation. It can capture the key features, including nonlinear reactions, nonstationary dynamics, and partially observed state. This hybrid model can leverage existing mechanistic models and facilitate learning from heterogeneous process data. A computational sampling approach is used to generate posterior samples quantifying model uncertainty. Then, we introduce hybrid model-based Bayesian RL, accounting for both inherent stochasticity and model uncertainty, to guide optimal, robust, and interpretable dynamic decision making. Cell therapy manufacturing examples are used to empirically demonstrate that the proposed framework can outperform the classical deterministic mechanistic model assisted process optimization.

Integrate Bioprocess Mechanisms into Modeling, Analytics, and Control Strategies to Advance Biopharmaceutical Manufacturing and Delivery Processes

“…KG hybrid model for Biomanufacturing SDP. The probabilistic KG hybrid model proposed in [77] and [78] can provide the risk-and science-based understanding of underlying stochastic decision process (SDP) mechanisms. The input-output relationship in each step is modeled by a hybrid (mechanistic/statistical) model that can leverage the prior knowledge on biophysicochemical mechanisms from existing mechanistic models and further advance scientific learning from process data.…”

“…x θ θ θ where p(θ θ θ ) represents the prior distribution. Since the likelihood evaluation of KG hybrid model with high fidelity, that can faithfully capture the critical properties of bioprocess, is intractable, i.e., p(τ τ τ x |θ θ θ ) = • • • p(τ τ τ|θ θ θ )dz z z 1 • • • dz z z H+1 , approximate Bayesian computation sampling with Sequential Monte Carlo (ABC-SMC) is developed to approximate the posterior distribution [77,80]. In the naive ABC implementation, we draw a candidate sample from the prior θ θ θ ∼ p(θ θ θ ) and then generate a simulation dataset D from the hybrid model.…”

Sequential Importance Sampling for Hybrid Model Bayesian Inference to Support Bioprocess Mechanism Learning and Robust Control

“…Driven by the critical challenges of biomanufacturing and limitations of existing process modeling methods, we developed probabilistic knowledge graph (KG) hybrid model characterizing the risk-and science-based understanding of bioprocess spatiotemporal causal interdependiences [2,3,4]. It can leverage the information from existing mechanistic models between and within each operation unit, as well as facilitating mechanism learning from heterogeneous data.…”

“…Since the proposed model-based RL scheme on the Bayesian KG can provide an insightful prediction on how the effect of inputs propagates through mechanism pathways, impacting on the output trajectory dynamics and variations, it can find process control policies that are interpretable and robust against model risk, and overcome the key challenges of biopharmaceutical manufacturing. [4] further generalized this KG hybrid model to capture the important properties of integrated biomanufacturing processes, including nonlinear reactions, partially observed state, and nonstationary dynamics. It can faithfully represent and advance the understanding of underlying bioprocessing mechanisms; for example enabling the inference of metabolic states and cell response to environmental perturbations.…”

Sample-efficient reinforcement learning and its applications

“…In Part I of this thesis, I present a collection of my research [19,20,21], focusing on the use of Bayesian Networks to enhance policy optimization algorithms, a method we refer to as Bayesian network-based policy optimization or policy-augmented Bayesian network (PABN). This line of work delves into both theoretical frameworks and practical applications of these methods, illustrating these breakthroughs, with a particular emphasis on their implications for improving process control of the biopharmaceutical manufacturing processes.…”