Understanding the contribution of operator measurement variability within Flow Cytometry data analysis for Quality Control of Cell and Gene Therapy manufacturing

Grant, Rachel; Coopman, Karen; Medcalf, Nick; Silva‐Gomes, Sandro; Campbell, Jonathan J.; Kara, Bo; Braybrook, Julian; Petzing, Jon N.

doi:10.1016/j.measurement.2019.106998

Cited by 13 publications

(10 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, since the absolute counts and frequencies of each cell population was known in the synthetic dataset, the evaluation of the software was based on robust absolute traceable figures, and did not rely on comparison with a manually gated reference subpopulation count, which has already been shown to be operator dependent and potentially biased (36,37).…”

Section: Application Of Synthetic Datasets To a Cell Population Identification Softwarementioning

confidence: 99%

Systematic Design, Generation, and Application of Synthetic Datasets for Flow Cytometry

Cheung

Campbell

Thomas

et al. 2022

PDA Journal of Pharmaceutical Science and Technology

Self Cite

View full text Add to dashboard Cite

Application of synthetic datasets in training and validation of analysis tools have led to improvements in many decision-making tasks in a range of domains from computer vision to digital pathology. Synthetic datasets overcome the constraints of real-world datasets, namely difficulties in collection and labelling, expense, time and privacy concerns.In flow cytometry, real cell-based datasets are limited by properties such as size, number of parameters, distance between cell populations and distributions, and are often focused on a narrow range of disease or cell types. Researchers in some cases have designed these desired properties into synthetic datasets, however operators have implemented them in inconsistent approaches and there is a scarcity of publicly available, high-quality synthetic datasets.In this research, we propose a method to systematically design and generate flow cytometry synthetic datasets with highly controlled characteristics. We demonstrate the generation of two-cluster synthetic datasets with specific degrees of separation between cell populations, and of non-normal distributions with increasing levels of skewness and orientations of skew pairs. We apply our synthetic datasets to test the performance of a popular automated cell populations identification software, SPADE3, and define the region where the software performance decreases as the clusters get closer together.Application of the synthetic skewed dataset suggests the software is capable of processing non-normal data. We calculate the classification accuracy of SPADE3 with robustness not achievable with real-world datasets. Our approach aims to advance research towards generation of high-quality synthetic flow cytometry datasets, and to increase their awareness among the community.The synthetic datasets can be utilised in benchmarking studies that critically evaluate cell population identification tools and help illustrate potential digital platform inconsistencies.

show abstract

Section: Application Of Synthetic Datasets To a Cell Population Identification Softwarementioning

confidence: 99%

Systematic Design, Generation, and Application of Synthetic Datasets for Flow Cytometry

Cheung

Campbell

Thomas

et al. 2022

PDA Journal of Pharmaceutical Science and Technology

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, little research exists to quantify the impact of post-analytical Flow Cytometry, focusing specifically on the data analysis (gating) procedure, commonly performed by subjective manual population segmentation analysis. Our recent study has shown the variation of multiple operators analysing the same Flow Cytometry data across a 3-step gating procedure, showing the variability that opinion and equipment optimisation can have on downstream analysis and variability [5].…”

Section: Introductionmentioning

confidence: 96%

“…Variability within Cell and Gene Therapy (CGT) manufacturing arises from numerous sources and it has been well documented that starting material [1,2], inter-site comparability [3] and final product comparability [4] to name a few examples cause variability. However, operators are a key function of the manufacturing processes and cause inherent variation due to numerous factors, such as experience with a specific instrument, assay or specific cell lines [5]. Other manufacturing and safety critical industries have reported the influence of the operator within "normal" and stressful scenarios, as well as decisionmaking and treatment options in medical scenarios [6][7][8][9][10].…”

Section: Introductionmentioning

confidence: 99%

Assessment of Protocol Impact on Subjectivity Uncertainty When Analyzing Peripheral Blood Mononuclear Cell Flow Cytometry Data Files

Grant

Coopman

Silva‐Gomes

et al. 2021

MPs

Self Cite

View full text Add to dashboard Cite

Measured variability of product within Cell and Gene Therapy (CGT) manufacturing arises from numerous sources across pre-analytical to post-analytical phases of testing. Operators are a function of the manufacturing process and are an important source of variability as a result of personal differences impacted by numerous factors. This research uses measurement uncertainty in comparison to Coefficient of Variation to quantify variation of participants when they complete Flow Cytometry data analysis through a 5-step gating sequence. Two study stages captured participants applying gates using their own judgement, and then following a diagrammatical protocol, respectively. Measurement uncertainty was quantified for each participant (and analysis phase) by following Guide to the Expression of Uncertainty in Measurement protocols, combining their standard deviations in quadrature from each gating step in the respective protocols. When participants followed a diagrammatical protocol, variation between participants reduced by 57%, increasing confidence in a more uniform reported cell count percentage. Measurement uncertainty provided greater resolution to the analysis processes, identifying that most variability contributed in the Flow Cytometry gating process is from the very first gate, where isolating target cells from dead or dying cells is required. This work has demonstrated the potential for greater usage of measurement uncertainty within CGT manufacturing scenarios, due to the resolution it provides for root cause analysis and continuous improvement.

show abstract

“…The lattermost factor, data analysis (manual gating), is a significant source of variation, and its removal from the analytical process has been shown to reduce inter-laboratory variation by as much as 5–20% [ 6 ]. Manual setting of gates is difficult to reproduce, subjective [ 7 , 8 ], and potentially biased in favour of a value that aids product release and a successful manufacturing run.…”

Section: Introductionmentioning

confidence: 99%

Assessment of Automated Flow Cytometry Data Analysis Tools within Cell and Gene Therapy Manufacturing

Cheung

Campbell

Thomas

et al. 2022

IJMS

Self Cite

View full text Add to dashboard Cite

Flow cytometry is widely used within the manufacturing of cell and gene therapies to measure and characterise cells. Conventional manual data analysis relies heavily on operator judgement, presenting a major source of variation that can adversely impact the quality and predictive potential of therapies given to patients. Computational tools have the capacity to minimise operator variation and bias in flow cytometry data analysis; however, in many cases, confidence in these technologies has yet to be fully established mirrored by aspects of regulatory concern. Here, we employed synthetic flow cytometry datasets containing controlled population characteristics of separation, and normal/skew distributions to investigate the accuracy and reproducibility of six cell population identification tools, each of which implement different unsupervised clustering algorithms: Flock2, flowMeans, FlowSOM, PhenoGraph, SPADE3 and SWIFT (density-based, k-means, self-organising map, k-nearest neighbour, deterministic k-means, and model-based clustering, respectively). We found that outputs from software analysing the same reference synthetic dataset vary considerably and accuracy deteriorates as the cluster separation index falls below zero. Consequently, as clusters begin to merge, the flowMeans and Flock2 software platforms struggle to identify target clusters more than other platforms. Moreover, the presence of skewed cell populations resulted in poor performance from SWIFT, though FlowSOM, PhenoGraph and SPADE3 were relatively unaffected in comparison. These findings illustrate how novel flow cytometry synthetic datasets can be utilised to validate a range of automated cell identification methods, leading to enhanced confidence in the data quality of automated cell characterisations and enumerations.

show abstract

Understanding the contribution of operator measurement variability within Flow Cytometry data analysis for Quality Control of Cell and Gene Therapy manufacturing

Cited by 13 publications

References 31 publications

Systematic Design, Generation, and Application of Synthetic Datasets for Flow Cytometry

Systematic Design, Generation, and Application of Synthetic Datasets for Flow Cytometry

Assessment of Protocol Impact on Subjectivity Uncertainty When Analyzing Peripheral Blood Mononuclear Cell Flow Cytometry Data Files

Assessment of Automated Flow Cytometry Data Analysis Tools within Cell and Gene Therapy Manufacturing

Contact Info

Product

Resources

About