In the eye of the beholder? An examination of the inter-rater reliability of the LSI-R and YLS/CMI in a correctional agency

Effects of Automating Recidivism Risk Assessment on Reliability, Predictive Validity, and Return on Investment (ROI)

“…When we focus more closely on the bottom quintile, we see the ICC values ranged from a low of 0.583 for general recidivism to a high of 0.648 for violent recidivism. The average ICC value for the bottom quintile was 0.615, which is similar to the 0.65 ICC value reported by Rocque and Plummer‐Beale () in their IRR study of the LSI‐R. The difference in the AUC between the automated and manual data ranged from a low of 0.019 (nonviolent recidivism) to a high of 0.069 (felony recidivism).…”

“…What is notable about this finding is that caseworkers who scored the MnSTARR underwent minimal training, which points to the importance of using objective items to achieve adequate reliability. As Rocque and Plummer‐Beale () observed in their IRR study on the LSI‐R, the inclusion of subjective items (e.g., is the offender a “social isolate?”) can make it more difficult to attain consistency.…”

“…As we reported, so far researchers have found the difference between the best and worst classification methods to be at least 0.07 for the H measure and 0.05 for the AUC (Duwe and Kim, ). Although IRR research within the recidivism risk assessment literature is almost nonexistent, the results of one recent study of a commonly used tool—the LSI‐R—found that it had an ICC of 0.65 (Rocque and Plummer‐Beale, ). The results presented here suggest inter‐rater disagreement within this range may drop the H by 0.04 and the AUC by approximately 0.05.…”

“…In general, the results of the little research that has been done on common risk assessment tools in corrections indicate these tools have good-to-excellent reliability (e.g., Canales, Campbell, Wei, and Totten, 2014;Duwe, 2014;Lowenkamp et al, 2004;Onifade et al, 2008;Schmidt, Hoge, and Gomes, 2005;van der Knaap et al, 2012;Viljoen, Elkovitch, Scalora, and Ullman, 2009). The results of other research have indicated lower IRR for these tools (see Austin, Coleman, Peyton, and Johnson, 2003;Baird et al, 2013;Rocque and Plummer-Beale, 2014). In part, the variation across IRR findings is a result of variations in methodology and data analyses used.…”

Automating Risk Assessment Instruments and Reliability

“…As discussed by Grant Duwe and Michael Rocque (, this issue), however, the aim of much of this literature has been focused on issues of validity or on the predictive power of these instruments (Gendreau, Goggin, and Smith, ; Gendreau, Little, and Goggin, ; Smith, Cullen, and Latessa, ). There have been fewer efforts to examine inter‐rater reliability or the extent to which raters generate consistent scores across assessments (Desmarais and Singh, ; Rocque and Plummer‐Beale, ) . Moreover, no studies to date have been conducted to examine the relationship between reliability and validity and the capacity of automated machine scoring, where risk instruments are populated through electronic data extraction methods, to minimize reliability problems and enhance prediction (Duwe and Rocque, ).…”