A general deep hybrid model for bioreactor systems: combining first Principles equations with deep neural networks

Abstract

Numerous studies have reported the use of hybrid semiparametric systems that combine shallow neural networks with mechanistic models for bioprocess modeling. Here we revisit the general bioreactor hybrid modeling problem and introduce some of the most recent deep learning techniques. The single layer networks were extended to multi-layer networks with varying depths and combined with First Principles equations in the form of deep hybrid models. Deep learning techniques, namely the adaptive moment estimation method (ADAM), stochastic regularization and depth-dependent weights initialization were evaluated. Modified sensitivity equations are proposed for the computation of gradients in order to reduce CPU time for the training of deep hybrid models. The methods are illustrated with applications to a synthetic dataset and a pilot 50 L MUT+ Pichia pastoris process expressing a single chain antibody fragment. All in all, the results point to a systematic generalization improvement of deep hybrid models over its shallow counterpart. Moreover, the CPU cost to train the deep hybrid models is shown to be lower than for the shallow counterpart. In the pilot 50L MUT+ Pichia pastoris data set, the prediction accuracy was increased by 18.4% and the CPU decreased by 43.4%.HighlightsShallow hybrid models have been widely used for bioprocess modeling and optimizationNon-deep training using e.g. the Levenberg – Marquardt method, cross-validation and indirect sensitivity equations have been the methods of choiceDeep learning with ADAM, stochastic regularization and indirect sensitivity significantly reduces the training CPUThe generalization capacity of deep hybrid models systematically outperforms that of shallow hybrid models

Hybrid modeling for biopharmaceutical processes: advantages, opportunities, and implementation

“…The first axis, more commonly exploited in the literature, is the type of machine learning/statistical algorithm used in data-based modeling. Several options presented in the literature include Artificial Neural networks or multi-layer perceptron (ANNs or MLPs) (Psichogios and Ungar, 1992;Schubert et al, 1994b;Schubert et al, 1994b;Feyo de Azevedo et al, 1997;Chen et al, 2000;Teixeira et al, 2006;Von Stosch et al, 2012a;Narayanan et al, 2019;Bayer et al, 2021;Narayanan et al, 2021b;Narayanan et al, 2021c;Narayanan et al, 2022b), Principal Component Regression (PCR) (Okamura et al, 2022), Partial Least Square regression (PLSR) (Von Stosch et al, 2011;Von Stosch et al, 2012b;Carvalho et al, 2022), Tree-based models (Hutter et al, 2017), Gaussian processes (GPs) (Hutter et al, 2021;Vega-Ramon et al, 2021;Cruz-Bournazou et al, 2022), and Deep Neural Networks (DNNs) (Pinto et al, 2022). Subsequently, efforts have also been devoted to using techniques such as symbolic regression and customized neural networks as data-based modeling approaches to have enhanced interpretability compared to the traditional ML approaches (Narayanan et al, 2022a;Doyle et al, 2023).…”

Artificial intelligence-driven systems engineering for next-generation plant-derived biopharmaceuticals

“…Deep learning techniques in a hybrid semi metric modelling contest, such as deep feed forward neural network with varying depths, the rectified linear unit (ReLU) activation function, dropout regularization of network weights, and stochastic training with the ADAM method were explored (Mestre et al, 2022). Performance of ML algorithms was analyzed to predict n-caproate and n-caprylate productivities in bacteria using 16S rRNA amplicons in a bioreactor.…”