Crowdsourcing Morphology Assessments in Oculoplastic Surgery: Reliability and Validity of Lay People Relative to Professional Image Analysts and Experts

Abstract

Purpose: To determine if crowdsourced ratings of oculoplastic surgical outcomes provide reliable information compared to professional graders and oculoplastic experts. Methods: In this prospective psychometric evaluation, a scale for the rating of postoperative eyelid swelling was constructed using randomly selected images and topic experts. This scale was presented adjacent to 205 test images, including 10% duplicates. Graders were instructed to match the test image to the reference image it most closely resembles. Three sets of graders were solicited: crowdsourced lay people from Amazon Mechanical Turk marketplace, professional graders from the Doheny Image Reading Center (DIRC), and American Society of Ophthalmic Plastic and Reconstructive Surgery surgeons. Performance was assessed by classical correlational analysis and generalizability theory. Results: The correlation between scores on the first rating and the second rating for the 19 repeated occurrences was 0.60 for lay observers, 0.80 for DIRC graders and 0.84 for oculoplastic experts. In terms of inter-group rating reliability for all photos, the scores provided by lay observers were correlated with DIRC graders at a level of r = 0.88 and to experts at r = 0.79. The pictures themselves accounted for the greatest amount of variation among all groups. The amount of variation in the scores due to the rater was highest in the lay group at 25%, and was 20% and 21% for DIRC graders and experts, respectively. Conclusions: Crowdsourced observers are insufficiently precise to replicate the results of experts in grading postoperative eyelid swelling. DIRC graders performed similarly to experts and present a less resource-intensive option.

Müller Muscle Conjunctival Resection: A Multicentered Prospective Analysis of Surgical Success

“…The authors note that the following variables were considered potentially important predictors of MMCR outcomes, but did not achieve statistical significance on bivariate analysis: intraoperative events that increased tissue trauma (slippage of the ptosis clamp requiring repositioning, suture transection requiring re-suturing, and hematoma formation), the degree of edema 26 at the 1-week postoperative visit, the use of a nightly protective eye shield for 1 month after surgery, and the use of prostaglandin drops 27 (eSupplement 3, Supplemental Digital Content, available at http://links.lww.com/IOP/A343).…”

A Virtual Reading Center Model Using Crowdsourcing to Grade Photographs for Trachoma: Validation Study

“…Crowdsourcing using unskilled internet users has been used to identify pathological features of diabetic retinopathy [ 19 , 20 , 29 ] and glaucoma [ 18 , 30 ] on fundus photographs as well as to classify surgical outcomes of ophthalmic surgery [ 31 - 33 ]. Aggregating the scores of a “crowd” of skilled graders has also been used in ophthalmology to generate robust consensus annotations of images for artificial intelligence (AI) algorithm training [ 34 , 35 ].…”

“…Crowdsourcing using unskilled internet users has been used to identify pathological features of diabetic retinopathy [19,20,29] and glaucoma [18,30] on fundus photographs as well as to classify surgical outcomes of ophthalmic surgery [31][32][33].…”

Crowdsourced Assessment of Surgical Skill Proficiency in Cataract Surgery

“…A recent study showed that reading center graders were nearly as reliable and accurate as experts for assessment of oculoplastic morphological outcomes, whereas crowdsourced lay observers were less accurate and consistent despite a strong overall correlation between the three groups. 29 Reading centers may be more time- and cost-effective than experts while more reliable than naïve crowdworkers, and could represent a feasible middle ground.…”