Improving the Rigor and Reproducibility of Flow Cytometry‐Based Clinical Research and Trials Through Automated Data Analysis

Abstract

The steps that characterize data analysis for flow cytometrybased clinical trials can be grouped into quality assessment, compensation, normalization, transformation, cell population identification, cross-sample comparison (population mapping or matching), feature extraction, visualization, and interpretation. Of these steps, cell population identification (i.e., gating), which generates reportables such as cell population counts/percentages and MFI, is the focus of significant efforts to improve the rigor and reproducibility of data analysis. Here, rigor is the application of the scientific method to ensure unbiased and well-controlled analysis, interpretation, and reporting of results. Reproducibility is important as analyses are only validated when they can be duplicated by multiple scientists. It is especially important for clinical studies due to what has been deemed a reproducibility crisis in medicine, as only 11% of a series of preclinical cancer studies could be replicated (1).While manual gating is the gold standard and current practice for cell population identification, assessments of its reproducibility have recognized it as a significant contributor of variation in flow cytometry studies, with interlaboratory C.V.s up to 30% (2). While having a single operator analyze all the data can significantly reduce variability, the chosen individual is still influenced by the same personal biases that result in inconsistencies within and across individuals, centers, and time. Resorting to a single operator to improve reproducibility does not scale for large studies as manual gating can take from 45 to 90 min for one clinical sample (3). With clinical trials now involving thousands of patients assayed with 18 parameters, the challenges associated with the rigor and reproducibility of manual analysis of flow cytometry-based clinical data have only become more apparent and pressing.The first algorithm for automated flow cytometry cell population identification was published in 1985, where it was noted that: "Unfortunately, the use of three or more independent fluorescent parameters complicates the analysis of the resulting data significantly" (4). While data complexity has increased significantly since then, automated analysis approaches for flow cytometry data have also matured. The state of the art is at a stage where results from automated data analysis algorithms have reached a level of maturity that enables them to match, and in many cases exceed, the results produced by human experts. The maturity and acceptance has gone as far as being included as part of the documentation for the FDA approval of a first in class CAR-T therapy (5-7).The overwhelming majority of automated gating methods are unsupervised. Unsupervised algorithms are those approaches that work on data that do not come with predefined labels. With respect to FCM data, these algorithms

Concordance between flow cytometry CLL scores

“…(a) Abnormal patterns of expression (ie, lack of expression or expression of antigens that are expressed or not expression in standard‐phenotype CLL) are well‐recognized 14‐16 . Therefore, the inclusion or not of a selected antigen/s in a score can easily change the classification of a LPD; (b) the use of dichotomous (awarding 1 or 0 points per antigen), and relatively randomly set (eg, ROC curve‐determined or antigenic expression in ≥20% of cells, which do not necessarily capture the underlying biology of the disorder), cutoffs, rather than relying on expression patterns (eg, complete, bright, dim, heterogeneous, etc), is likely to accentuate the differences between observers; (c) manual gating is highly variable 17‐20 and is likely to be a major source of inter‐observer discrepancies in borderline dim or partial antigenic expressions. For instance, the MS has been reported to perform quite differently when used by different investigators as their gold standard (Figure ) and the percentage of patients awarded 3 points (often regarded as the gray zone) varies widely, despite supposedly being applied to similar populations 1,2,21 ; and (d), and more fundamentally, the existence of numerous scores reveals, in line with a recent landmark consensus report, 5 the lack of agreement over what CLL is or can be from a phenotypic standpoint.…”

Integration, exploration, and analysis of high-dimensional single-cell cytometry data using Spectre

“…This is in part due to the number of gates required to fully parse the dataset. As such, a variety of computational approaches have been adopted by the cytometry community to help analyze HD datasets, including automated gating [3], clustering (such as PhenoGraph [4], FlowSOM [5], X-Shift [6]), dimensionality reduction (such as t-SNE [7,8], UMAP [9], trajectory inference (such as Wanderlust [1], Wishbone [10]), and automated cell classification [11] [12,13]. Many of these tools have been brought together into 'toolboxes', providing either code-or GUI-based analysis workflows, such as Cytofkit [14], CITRUS [15], CATALYST [16], Cytofworkflow [17], or diffcyt [18].…”

Comparison of Automated and Manual Gating of Lymphocyte Subsets in Hematopoietic Stem Cell Transplantation Recipients

“…Gating in ow cytometry analysis refers to generation of reportable data such as cell counts, percentages, and mean uorescence 조혈모세포 이식 환자의 림프구 아형 검사에서 자동 게이팅과 수동 게이팅 비교 intensity (MFI), which is essential for maintaining the data consistency and improving reproducibility of ow cytometry data [6].…”