Setting a Fair Performance Standard for Physicians’ Quality of Patient Care

Abstract

BACKGROUND: Assessing physicians' clinical performance using statistically sound, evidence-based measures is challenging. Little research has focused on methodological approaches to setting performance standards to which physicians are being held accountable. OBJECTIVE: Determine if a rigorous approach for setting an objective, credible standard of minimally-acceptable performance could be used for practicing physicians caring for diabetic patients. DESIGN: Retrospective cohort study. PARTICIPANTS: Nine hundred and fifty-seven physicians from the United States with time-limited certification in internal medicine or a subspecialty. MAIN MEASURES: The ABIM Diabetes Practice Improvement Module was used to collect data on ten clinical and two patient experience measures. A panel of eight internists/subspecialists representing essential perspectives of clinical practice applied an adaptation of the Angoff method to judge how physicians who provide minimallyacceptable care would perform on individual measures to establish performance thresholds. Panelists then rated each measure's relative importance and the Dunn-Rankin method was applied to establish scoring weights for the composite measure. Physician characteristics were used to support the standard-setting outcome. KEY RESULTS: Physicians abstracted 20,131 patient charts and 18,974 patient surveys were completed. The panel established reasonable performance thresholds and importance weights, yielding a standard of 48.51 (out of 100 possible points) on the composite measure with high classification accuracy (0.98). The 38 (4%) outlier physicians who did not meet the standard had lower ratings of overall clinical competence and professional behavior/attitude from former residency program directors (p=0.01 and p=0.006, respectively), lower Internal Medicine certification and maintenance of certification examination scores (p= 0.005 and p< 0.001, respectively), and primarily worked as solo practitioners (p=0.02). CONCLUSIONS: The standard-setting method yielded a credible, defensible performance standard for diabetes care based on informed judgment that resulted in a reasonable, reproducible outcome. Our method represents one approach to identifying outlier physicians for intervention to protect patients.

Assessing the Quality of Osteoporosis Care in Practice

“…The methodology has been previously described [9][10][11] and patented. 16 For each measure, we established a threshold for delivering both competent and excellent care using an expert panel and an adaptation of the Angoff standard-setting method.…”

“…2,6 Recent research on composites drawn from evidence-based measures has advanced the science of quality assessment. [7][8][9][10][11] Composites are comprehensive measure sets that provide a more reliable assessment than individual measures alone. 8,9 Using a composite to make a judgment about an individual physician's performance in practice requires a defensible absolute standard of performance on the composite.…”

“…12 We have previously shown that reasonable absolute standards can be set for composites in diabetes and preventive cardiology care. 10,11 Multiple individual osteoporosis measures are currently available in National Quality Measures Clearinghouse; 13 we believe this is the first study to address the development and evaluation of a composite for osteoporosis care.…”

Community Size and Organization of Practice Predict Family Physician Recertification Success

“…As one component of MOC, the examination serves as an objective measure of cognitive expertise. 12,13 We previously demonstrated that family physicians who are international medical graduates (IMGs), are in solo practice, or who practice in poor areas of the country are less likely to meet their MOC requirements, which includes passing the MOC examination.…”

Protecting the Public: Is It Time for a Paradigm Shift in Expected Practice Standards?

“…Methods to ensure safe and effective practitioners have traditionally focused on training and assessment of competence before entering practice. However, an increasing number of visionary credentialing bodies such as the American Board of Internal Medicine have created systematic methods of measuring a practitioner's ability to deliver care in practice 2 .…”