OBJECTIVE: To test whether crowdsourced lay raters can accurately assess cataract surgical skills. DESIGN: Two-armed study: independent cross-sectional and longitudinal cohorts. SETTING: Washington University Department of Ophthalmology. PARTICIPANTS AND METHODS: Sixteen cataract surgeons with varying experience levels submitted cataract surgery videos to be graded by 5 experts and 300+ crowdworkers masked to surgeon experience. Cross-sectional study : 50 videos from surgeons ranging from first-year resident to attending physician, pooled by years of training. Longitudinal study : 28 videos obtained at regular intervals as residents progressed through 180 cases. Surgical skill was graded using the modified Objective Structured Assessment of Technical Skill (mOSATS). Main outcome measures were overall technical performance, reliability indices, and correlation between expert and crowd mean scores. RESULTS: Experts demonstrated high interrater reliability and accurately predicted training level, establishing construct validity for the modified OSATS. Crowd scores were correlated with (r = 0.865, p < 0.0001) but consistently higher than expert scores for first, second, and third-year residents (p < 0.0001, paired t-test). Longer surgery duration negatively correlated with training level (r = −0.855, p < 0.0001) and expert score (r = −0.927, p < 0.0001). The longitudinal dataset reproduced cross-sectional study findings for crowd and expert comparisons. A regression equation transforming crowd score plus video length into expert score was derived from the cross-sectional dataset (r 2 = 0.92) and demonstrated excellent predictive modeling when applied to the independent longitudinal dataset (r 2 = 0.80). A group of student raters who had edited the cataract videos also graded them, producing scores that more closely approximated experts than the crowd. CONCLUSIONS: Crowdsourced rankings correlated with expert scores, but were not equivalent; crowd scores overestimated technical competency, especially for novice surgeons. A novel approach of adjusting crowd scores with surgery duration generated a more accurate predictive model for surgical skill. More studies are needed before crowdsourcing can be reliably used for assessing surgical proficiency.

No abstract

PURPOSE To elicit end-user and stakeholder perceptions regarding design and implementation of an inpatient clinical deterioration early warning system (EWS) for oncology patients to better fit routine clinical practices and enhance clinical impact. METHODS In an explanatory-sequential mixed-methods study, we evaluated a stakeholder-informed oncology early warning system (OncEWS) using surveys and semistructured interviews. Stakeholders were physicians, advanced practice providers (APPs), and nurses. For qualitative data, we used grounded theory and thematic content analysis via the constant comparative method to identify determinants of OncEWS implementation. RESULTS Survey respondents generally agreed that an oncology-focused EWS could add value beyond clinical judgment, with nurses endorsing this notion significantly more strongly than other clinicians (nurse: median 5 on a 6-point scale [6 = strongly agree], interquartile range 4-5; doctors/advanced practice providers: 4 [4-5]; P = .005). However, some respondents would not trust an EWS to identify risk accurately (n = 36 [42%] somewhat or very concerned), while others were concerned that institutional culture would not embrace such an EWS (n = 17 [28%]). Interviews highlighted important aspects of the EWS and the local context that might facilitate implementation, including (1) a model tailored to the subtleties of oncology patients, (2) transparent model information, and (3) nursing-centric workflows. Interviewees raised the importance of sepsis as a common and high-risk deterioration syndrome. CONCLUSION Stakeholders prioritized maximizing the degree to which the OncEWS is understandable, informative, actionable, and workflow-complementary, and perceived these factors to be key for translation into clinical benefit.

No abstract

In their article, Mowitz et al investigated the burden of comorbidities and healthcare resource utilization among extremely premature infants enrolled in Medicaid, laying a foundation for further policy action.