Reliable and valid: A procedure for establishing item-level interrater reliability for child maltreatment risk and safety assessments

Orsi, Rebecca; Drury, Ida; Mackert, Marc J.

doi:10.1016/j.childyouth.2014.04.016

Cited by 14 publications

(14 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One possible explanation for this surprising finding is that the risk assessment tool itself was not being implemented and administered correctly. Other scholars have also noted this potential concern with how standardized risk assessment tools, including problems with fidelity and consistency (Gillingham & Humphreys, ; Orsi et al, ; (Semanchin Jones & Logan‐Greene, ), staff simply checking off boxes on a checklist (Gillingham ; Hughes & Rycus, ), insufficient training for workers on the use of risk assessments, and staff resistance (Lee et al, ). Many current risk assessments include factors that have been found in research to be protective in reducing risk of child maltreatment, such as parent knowledge of child development and parenting styles (Li, Godinet, & Arnsberger, ); however, these were not included on the risk assessment in this study.…”

Section: Discussionmentioning

confidence: 99%

Predicting chronic neglect: Understanding risk and protective factors for CPS‐involved families

Logan-Greene

Jones

2017

Child & Family Social Work

View full text Add to dashboard Cite

Child neglect continues to receive less attention in research literature compared to other forms of maltreatment, despite accumulating evidence of serious negative impacts on child well‐being. Chronic neglect is also poorly understood. Although such cases comprise a relatively small percentage of workers' caseloads, they represent accumulation of harm that impacts the cognitive and social development of children. These cases can also disproportionately utilize protective service resources. This study contributes to the literature by examining risk and protective factors of chronic neglect. We utilized administrative data (N = 2,074) from a midsize city in the Northeast to examine the use of existing risk assessment tools to distinguish families with and without chronic neglect, including an analysis of the predictive capacity of risk and protective factors. We found that families with chronic neglect were younger, had more children, were more likely to have children under age one, and had higher rates of domestic violence, mental health problems, and cognitive impairment. None of the assessed protective factors differed significantly. The overall predictive value of the assessment was low. Implications include the need to expand risk assessment tools to incorporate patterns over time and identify early indicators specific to chronic neglect.

show abstract

Section: Discussionmentioning

confidence: 99%

Predicting chronic neglect: Understanding risk and protective factors for CPS‐involved families

Logan-Greene

Jones

2017

Child & Family Social Work

View full text Add to dashboard Cite

show abstract

“…However, the results of both Bartelink et al (2017) and Orsi et al (2014) should be interpreted cautiously, as the results seemed to have been negatively influenced by the "prevalence problem". For this reason, Orsi et al (2014) do no draw conclusions on which items of the safety assessment instrument they examined are most reliable. This problem (also known as "kappa paradox" or "paradox of high agreement but low reliability"), entails that reliability measures (e.g., Kappa, Krippendorff's alpha) are underestimated in case of a low or high item prevalence (Cicchetti & Feinstein, 1990;Hallgren, 2012;Lantz & Nebenzahl, 1996;Zhao et al, 2018).…”

mentioning

confidence: 96%

“…One study performed in the Netherlands showed a slight to fair interrater reliability of the items of a Dutch safety assessment instrument, and a moderate interrater reliability of the safety outcome of that same instrument (Bartelink et al, 2017). Orsi et al (2014) examined the interrater reliability of the items of several safety assessment instruments, and found mixed interrater reliability of the items, varying from slight to substantial. However, the results of both Bartelink et al (2017) and Orsi et al (2014) should be interpreted cautiously, as the results seemed to have been negatively influenced by the "prevalence problem".…”

mentioning

confidence: 99%

“…Orsi et al (2014) examined the interrater reliability of the items of several safety assessment instruments, and found mixed interrater reliability of the items, varying from slight to substantial. However, the results of both Bartelink et al (2017) and Orsi et al (2014) should be interpreted cautiously, as the results seemed to have been negatively influenced by the "prevalence problem". For this reason, Orsi et al (2014) do no draw conclusions on which items of the safety assessment instrument they examined are most reliable.…”

mentioning

confidence: 99%

“…Studies on reliability of instruments used in child welfare may as well have been influenced by this problem (D'andrade, Austin, & Benton, 2008;Baird, Wagner, Healy, & Johnson, 1999;Barber, Shlonsky, Black, Goodman, & Trocmé, 2008;Cash 2001;Knoke & Trocme, 2005), since Cohen's kappa or Krippendorff's alpha were estimated in these studies. These statistics were also estimated in the studies of Bartelink et al (2017) and Orsi et al (2014).…”

mentioning

confidence: 99%

See 2 more Smart Citations

Safety and Risk Assessment in Child Welfare: A Reliability Study Using Multiple Measures

et al. 2019

View full text Add to dashboard Cite

Objectives In the Netherlands, the Actuarial Risk Assessment Instrument for Youth Protection (ARIJ) is a widely used safety and risk assessment instrument in child welfare, although little is known about its reliability. Therefore, this study aimed to determine the reliability of the ARIJ by examining the inter-and intrarater reliability. Methods For determining interrater reliability, professionals of two Dutch agencies (child and family support, n = 39 & child protection, n = 24) and master students (n = 65) each rated a random selection of 4 out of 24 vignettes. The vignettes were based on actual cases that were handled by the two agencies. For determining intrarater reliability, the professionals rated four vignettes twice with an interval of at least 3 months. Three reliability measures were calculated for each of the three samples: percent agreement, Krippendorff's alpha, and Gwet's Agreement Coefficient.Results Overall, the items and outcome of the safety assessment instrument showed a moderate or higher than moderate interrater reliability, and a substantial to almost perfect intrarater reliability. In general, the risk assessment items showed a moderate interrater and a substantial-to-high intrarater reliability. The risk assessment outcome had a near perfect interrater reliability and a substantial to almost perfect intrarater reliability. Conclusions The outcome of both the safety and risk assessment of the ARIJ proved to be reliable and justifies the use of the ARIJ in the Dutch child welfare by professionals with different levels of experience.

show abstract

Systems Measures of a Trauma‐Informed Approach: A Systematic Review

Champine

Lang

Nelson

et al. 2019

American J of Comm Psychol

View full text Add to dashboard Cite

Interest in trauma‐informed approaches has grown substantially. These approaches are characterized by integrating understanding of trauma throughout a program, organization, or system to enhance the quality, effectiveness, and delivery of services provided to individuals and groups. However, variation in definitions of trauma‐informed approaches, coupled with underdeveloped research on measurement, poses challenges for evaluating the effectiveness of models designed to support a trauma‐informed approach. This systematic review of peer‐reviewed and gray literature identified 49 systems‐based measures that were created to assess the extent to which relational, organizational, and community/system practices were trauma‐informed. Measures were included if they assessed at least one component of a trauma‐informed approach, were not screening or diagnostic instruments, were standardized, were relevant to practices addressing the psychological impacts of trauma, were printed in English, and were published between 1988 and 2018. Most (77.6%) measures assessed organizational‐level staff and climate characteristics. There remain several challenges to this emerging field, including inconsistently reported psychometric data, redundancy across measures, insufficient evidence of a link to stakeholder outcomes, and limited information about measurement development processes. We discuss these opportunities and challenges and their implications for future research and practice.

show abstract

Reliable and valid: A procedure for establishing item-level interrater reliability for child maltreatment risk and safety assessments

Cited by 14 publications

References 20 publications

Predicting chronic neglect: Understanding risk and protective factors for CPS‐involved families

Predicting chronic neglect: Understanding risk and protective factors for CPS‐involved families

Safety and Risk Assessment in Child Welfare: A Reliability Study Using Multiple Measures

Systems Measures of a Trauma‐Informed Approach: A Systematic Review

Contact Info

Product

Resources

About