A Crowdsourcing Approach to Develop Machine Learning Models to Quantify Radiographic Joint Damage in Rheumatoid Arthritis

Abstract

To develop machine learning methods to quantify joint damage in patients with rheumatoid arthritis (RA), we developed the RA2 DREAM Challenge, a crowdsourced competition that utilized existing radiographic images and "gold-standard" scores on 674 sets of films from 562 patients. Training and leaderboard sets were provided to participants to develop methods to quantify joint space narrowing and erosions. In the final round, participants submitted containerized codes on a test set; algorithms were evaluated using weighted root mean square error (RMSE). In the leaderboard round, there were 173 submissions from 26 teams in 7 countries. Of the 13 submissions in the final round, four top-performing teams were identified. Robustness of results was assessed using Bayes factor and validated using an independent set of radiographs. The top-performing algorithms, which consisted of different styles of deep learning models, provided accurate and robust quantification of joint damage in RA. Ultimately, these methods lay the groundwork to accelerate research and help clinicians to optimize treatments to minimize joint damage.

Multilevel Modeling of Joint Damage in Rheumatoid Arthritis

“…Although many deep learning models have been developed for image-based joint damage detection, there is still room for improvement. Top-performing methods in the DREAM Rheumatoid Arthritis Challenge, including ours, consist of multiple steps [41]. Multi-step methods require more human designing of multiple modules, whereas end-to-end methods have more simplified workflows and are easier to deploy without external priors and constraints in clinical settings [42].…”

“…Two types of joint damages were investigated: joint space narrowing and bone erosion. The ground truth label is the Sharp/van der Heijde (SvH) score generated by human experts through manual inspection of images [41]. Typically targeted joints by RA were examined by the SvH scoring system, including multiple joints in wrists, proximal interphalangeal (PIP), and metacarpal phalangeal (MCP) of the fingers, PIP and metatarsal phalangeal (MTP) of the toes.…”

Multilevel Modeling of Joint Damage in Rheumatoid Arthritis

“…Top‐performing methods in the DREAM Rheumatoid Arthritis Challenge, including ours, consist of multiple steps. [ 39 ] Multistep methods require more human designing of multiple modules, whereas end‐to‐end methods have more simplified workflows and are easier to deploy without external priors and constraints in clinical settings. [ 40 ] Ideally, end‐to‐end deep learning algorithms should be designed to simultaneously output damages scores of joint space narrowing as well as bone erosion.…”

“…The ground truth label is the Sharp/van der Heijde (SvH) score generated by human experts through manual inspection of images. [ 39 ] Typically targeted joints by RA were examined by the SvH scoring system, including multiple joints in wrists, proximal interphalangeal (PIP), and metacarpal phalangeal (MCP) of the fingers, and PIP and metatarsal phalangeal (MTP) of the toes. For joint space narrowing, 15 joints from each hand and 6 joints from each foot were assessed, with the score ranging from 0 to 5.…”