A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis

Abstract

IMPORTANCE An automated, accurate method is needed for unbiased assessment quantifying accrual of joint space narrowing and erosions on radiographic images of the hands and wrists, and feet for clinical trials, monitoring of joint damage over time, assisting rheumatologists with treatment decisions. Such a method has the potential to be directly integrated into electronic health records. OBJECTIVESTo design and implement an international crowdsourcing competition to catalyze the development of machine learning methods to quantify radiographic damage in rheumatoid arthritis (RA). DESIGN, SETTING, AND PARTICIPANTSThis diagnostic/prognostic study describes the Rheumatoid Arthritis 2-Dialogue for Reverse Engineering Assessment and Methods (RA2-DREAM Challenge), which used existing radiographic images and expert-curated Sharp-van der Heijde (SvH) scores from 2 clinical studies (674 radiographic sets from 562 patients) for training (367 sets), leaderboard (119 sets), and final evaluation (188 sets). Challenge participants were tasked with developing methods to automatically quantify overall damage (subchallenge 1), joint space narrowing (subchallenge 2), and erosions (subchallenge 3). The challenge was finished on June 30, 2020. MAIN OUTCOMES AND MEASURESScores derived from submitted algorithms were compared with the expert-curated SvH scores, and a baseline model was created for benchmark comparison.Performances were ranked using weighted root mean square error (RMSE). The performance and reproductivity of each algorithm was assessed using Bayes factor from bootstrapped data, and further evaluated with a postchallenge independent validation data set. RESULTSThe RA2-DREAM Challenge received a total of 173 submissions from 26 participants or teams in 7 countries for the leaderboard round, and 13 submissions were included in the final evaluation. The weighted RMSEs metric showed that the winning algorithms produced scores that were very close to the expert-curated SvH scores. Top teams included Team Shirin for subchallenge 1 (weighted RMSE, 0.44), HYL-YFG (Hongyang Li and Yuanfang Guan) subchallenge 2 (weighted RMSE, 0.38), and Gold Therapy for subchallenge 3 (weighted RMSE, 0.43). Bootstrapping/Bayes factor approach and the postchallenge independent validation confirmed the reproducibility and the estimation concordance indices between final evaluation and postchallenge independent validation data set were 0.71 for subchallenge 1, 0.78 for subchallenge 2, and 0.82 for subchallenge 3. CONCLUSIONS AND RELEVANCEThe RA2-DREAM Challenge resulted in the development of algorithms that provide feasible, quick, and accurate methods to quantify joint damage in RA. (continued) Key Points Question Can a worldwide collaborative effort develop machine learning algorithms to quantify joint space narrowing and erosions automatically to improve the current visual inspection approach to radiography in rheumatoid arthritis (RA)? Findings This prognostic study assesses an international, crowdsourcing competition using scored radiographs of...

Community assessment of methods to deconvolve cellular composition from bulk gene expression

“…This is consistent with findings from previous DREAM Challenges showing the superiority of such approaches. [36][37][38][39][40][41][42] As such, the ensemble may prove to be a robust strategy across cell types.…”

Advancing precision rheumatology: applications of machine learning for rheumatoid arthritis management

“…Despite these challenges, high-caliber investigations are somewhat limited and the dependability and transferability of pertinent ML methods remain largely undetermined, rendering periodic evaluation of algorithm performance imperative. The current research trend involves the utilization of thousands of digitally annotated images obtained from large-scale observational studies, clinical trials, and electronic medical records, along with clinical data, to automatically classify and quantify the extent of joint damage and activity scores in RA using ML algorithms ( 100 – 102 ).…”

"Automated Computer Vision Technology Algorithm for Wrist Joint Detection for Rheumatoid Arthritis"

“…Its symbolic symptoms include inflammation in the synovial membrane (the lining of the joint), which can cause pain, stiffness, and swelling in the affected area. Over time, RA can lead to damage to the joints characterized by joint space narrowing, erosions in subchondral bone, and joint deformity [2], which may eventually result in irreversible abnormality and loss of function. Patients at early stage of RA often develop disease-related or therapy-induced osteopenia and joint space narrowing caused by the dissolution of cartilage tissue.…”