Was Rodney Ledward a statistical outlier? Retrospective analysis using routine hospital data to identify gynaecologists' performance

Abstract

Objectives To investigate whether routinely collected data from hospital episode statistics could be used to identify the gynaecologist Rodney Ledward, who was suspended in 1966 and was the subject of the Ritchie inquiry into quality and practice within the NHS. Design A mixed scanning approach was used to identify seven variables from hospital episode statistics that were likely to be associated with potentially poor performance. A blinded multivariate analysis was undertaken to determine the distance (known as the Mahalanobis distance) in the seven indicator multidimensional space that each consultant was from the average consultant in each year. The change in Mahalanobis distance over time was also investigated by using a mixed effects model. Setting NHS hospital trusts in two English regions, in the five years from 1991-2 to 1995-6. Population Gynaecology consultants (n = 143) and their hospital episode statistics data. Main outcome measure Whether Ledward was a statistical outlier at the 95% level. Results The proportion of consultants who were outliers in any one year (at the 95% significance level) ranged from 9% to 20%. Ledward appeared as an outlier in three of the five years. Our mixed effects (multi-year) model identified nine high outlier consultants, including Ledward. Conclusion It was possible to identify Ledward as an outlier by using hospital episode statistics data. Although our method found other outlier consultants, we strongly caution that these outliers should not be overinterpreted as indicative of "poor" performance. Instead, a scientific search for a credible explanation should be undertaken, but this was outside the remit of our study. The set of indicators used means that cancer specialists, for example, are likely to have high values for several indicators, and the approach needs to be refined to deal with case mix variation. Even after allowing for that, the interpretation of outlier status is still as yet unclear. Further prospective evaluation of our method is warranted, but our overall approach may be potentially useful in other settings, especially where performance entails several indicator variables.

The use of quality control performance charts to analyze cesarean delivery rates nationally

“…The annual publication of national benchmarks enables individuals to compare their own performances with those of their peers before any external audit and, if necessary, to review their practices. Hospital episode statistics data can also identify individuals whose clinical practices may be hazardous and, thus, can be used as an early warning management system [20]. Deviations from the normal range for elective cesarean deliveries may reflect individual obstetric practices; however, deviation from the normal range for emergency cesareans may reflect collective practice.…”

Professionalism

“…21 Harley et al demonstrated that HES data could have been used to identify Ledward. 22 Worryingly, they also identified 11 other unidentified statistical outliers. Using HES data can, however, only identify problems after they have been happening for a number of years.…”

Performance Evaluation Based on the Robust Mahalanobis Distance and Multilevel Modeling Using Two New Strategies

“…Among the most recent studies on performance are those of Harley et al (2005) in the area of health and Behrenz and Althin (2005) in the area of labor market and unemployment. In Harley et al (2005), the authors have used a two-stage statistical method for evaluating health indicators using routine hospital data to identify gynecologists' performance.…”

“…They applied the so-called Malmquist productivity index, which measures the distance from every office to the production frontier and the shifting of the frontier over time. Note that, in Behrenz and Althin (2005), the analysis has been done at the micro level and no consideration have been taken for the possible existence of any hierarchy structure in the data, while in Harley et al (2005), possible hierarchy is taken into account.…”

“…In Harley et al (2005), the authors have used a two-stage statistical method for evaluating health indicators using routine hospital data to identify gynecologists' performance. The robust Mahalanobis distance is first used to reduce the dimension of the indicators, after which the robust residuals from the level-two estimation were investigated to indicate the outliers.…”