A Methodological Critique of the ProPublica Surgeon Scorecard

Reporting and evaluating wait times for urgent hip fracture surgery in Ontario, Canada

“…As such, initiatives for quality improvement may target hospitallevel processes preferentially rather than individual physician practices. In contrast, and contrary to calls for physician-level reporting, [50][51][52] the finding that wait times, clinical outcomes and costs were similar between physicians after accounting for patient and hospital factors suggests such reporting may be less informative than hospital-level information. Examples of hospital-level interventions include medical and surgical comanagement models, 53 and policies for preoperative consultations, echocardiography and anticoagulant reversal, 10 which may ensure that coordinated care does not compromise the provision of timely surgery.…”

Association Between Board Certification, Maintenance of Certification, and Surgical Complications in the United States

“…Although the ProPublica data use up-to-date case-mix adjustment techniques, these data have been found to have poor correlation with registry data and may not represent overall surgeon quality, only risk-adjusted readmission and mortality rates. 11,35-37 Claims data also may not capture all instances of reoperation and surgical site infections. Nevertheless, the Scorecard uses national Medicare claims, which currently provides the broadest view of surgeon performance in Medicare patients at a national level.…”

A Standard-Setting Body for US Health Care Quality Measurement

“…Recent publicized examples of measures that have divided the health care community include CMS's Overall Hospital Quality Star Ratings, which systemically gave higher ratings to smaller community hospitals and much lower ratings to major academic medical centers, 24 and ProPublica's surgeon quality ratings, which were criticized for methodology choices that included failure to adjust for hospital-level performance, failing to include complications occurring during the index hospitalization, and nonstandard practices in measuring surgical complications. 8 (See online Supplementary Table S1 for a more in-depth discussion of these measures.) Both these measures diverted providers' attention away from quality improvement work and toward attacking the fairness of the measures.…”