No abstract

Blinded independent central review is recommended by the US FDA for registration of oncology trials as it provides bias-free image assessment and avoidance of potential unblinding of patient data. Double read with adjudication is a highly advocated review model used in such trials. Disagreement between readers is natural and inevitable. Radiological disagreement rates or Adjudication Rate (AR) of 30–65% are reported by several papers since 1959 for different oncologic indications. The aim of the study is to develop and use an algorithm to identify reader pair with predicted high AR and investigate if overall study AR can be kept constant or improved by assigning less cases to a reader pair with high AR. A retrospective analysis was performed of 285 subjects with 3351 post-baseline timepoints reviewed by board-certified radiologist reviewers using Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria in a BICR set up. The reader adjudication rate was calculated and analyzed throughout the duration of review. The distribution of cases per reader and distribution of cases per reader pair was calculated and overall study AR, and each reader pair AR were calculated. Data was prepared and analyzed using linear regression with MS Excel and R programming script (R version 4.1.2 (2021-11-01) -- "Bird Hippie" augmented by RStudio 2022.07.0+548 "Spotted Wakerobin". Using the data at completion of 50% reads, predicted AR per reader pair was found to correlate well for five out of six reader pairs on the study. This predicted AR can then be used to assign more cases to reader pair with low AR and less cases to reader pair with high AR to keep study level variability low. Predictive modeling of AR using linear regression can provide an insight into variability of each reader pair, which in turn determines the AR of that reader pair and collectively determines study AR. However, it is still not clear to what level of prioritization in case assignment to specific pairs can be considered acceptable and not artificial.

Variability in observer performance BICR is common but not well understood and various measures like AR, AAR, RDI help quantify it which leads to multiple complex data points. Network analysis uses mathematically based algorithms to characterize the components of a network of entities and identifying, visualizing, and analysing their relationships. In a network, variables are represented by nodes, the relationships represented by edges between these nodes. The visualization technique involves mapping relationships among entities based on the symmetry or asymmetry of data. Maps from data points generated during double read adjudication study can provide the performance of each reader pair primarily based on AAR. Adjudication data from four oncology clinical trials with 2163 subjects, 16937 post-baseline responses was analyzed. Performance metrics included number of cases, adjudication rate, adjudication agreement rate for each read and reader pair. The data were aggregated and prepared for network analysis in Python-a high-level, cross-platform, and open-sourced programming language released under a GPL-compatible license. Python Software Foundation (PSF), a non-profit organization, holds the copyright. Url-https://www.python.org Version 3.9.0 This graphic visualization provides simplistic organization of a complicated data analysis and supports the quality monitoring process of independent reviews. The tool provides a snapshot of the review performance of all the readers in the trial allowing the study team to investigate and intervene in a timely manner with the intent of supporting robust and accurate data analysis. Network analysis plots for reader performance metrics in BICR provide excellent visual mapping to interpret multiple critical metrics in a single plot which would otherwise require multiple plots and tables. Timely review of these plots during the trial can help demonstrate the effectiveness of interventions as well.