Assessing the reliability of the Outcomes Star in research and practice

Abstract

The purpose of this paper is to describe a pilot to test an approach to measuring inter-rater reliability of the Outcomes Star suite of tools. The intention, in publishing this account, is to show transparency in ongoing development of the tool, and to invite further cooperative development. Design/Methodology/Approach Twenty four workers, trained to use the first edition Family Star, scored a tested case-study. Scoring was analysed using two metrics on the 10-point scale and the underlying 5-point Journey of Change. The case-study approach and metrics were evaluated for validity and accessibility. Findings This initial evaluation suggests this edition of the Family Star has good inter-rater reliability for the 5 point Journey of Change, reaching the accepted threshold of 0.8 for the inter-rater reliability coefficient when three outlying workers are excluded. The reliability for the full 10 point scale was moderate. Research limitations/implicaitons The sample size of twenty four raters is small, though sufficient for an initial test of the approach, which will now be applied to larger samples, using other versions of the Outcomes Star. Practical implications The findings indicate that it is important that service providers test worker understanding of the scales to ensure consistency of use. The second edition of the Family Star incorporates more precise definitions of the 10 point scales to help improve the reliability. Originality/value The case-study method and metrics provide an accessible measure of reliability, both for Star development and to enable managers to assess the reliability of an organisation's client data for internal and external purposes.

Unsubstantiated conclusions about the Family Star Plus as an outcome measure: a rebuttal to Sweet, Winter, Neeson and Connolly (2020)

“…Rather surprisingly, they did not mention the inter-rater reliability findings shared with them in manuscript form prior to the publication of Good and MacKeith (2020). Good and MacKeith (2020) showed very good inter-rater reliability for the Family Star Plus (Krippendorff’s α = 0.83), using a much larger sample than MacKeith (2014) and with a chance-corrected inter-rater reliability coefficient. This article also demonstrated that Star readings predicted hard outcomes, offering further evidence that they can be meaningful and accurately completed.…”

“…This makes it appear that there was poor inter-rater reliability, when in fact no evidence was found because the researchers chose not to test this. They also focus on pilot findings reported in the study of MacKeith (2014) from a much earlier precursor of the Family Star Plus (the first edition of the Family Star), describing agreement on the journey of change in negative terms, when it fact it was substantially higher than frequently cited thresholds (Chaturvedi and Shweta, 2015). Rather surprisingly, they did not mention the inter-rater reliability findings shared with them in manuscript form prior to the publication of Good and MacKeith (2020).…”

Assessing Family Functioning: Psychometric Evaluation of the Family Star Plus

“…As far as we are aware, the current research is the first to report on the psychometric properties of the Family Star Plus. However, initial support for interrater reliability when using the Family Star is reported by MacKeith (2014). This assessment reports outcomes from a small sample of caseworkers who assigned Journey of Change readings (using the 1–10 scale) to a written service user case study.…”

Assessing the reliability and validity of an outcomes star

“…There is a small but limited evidence base for the psychometric properties of the Outcome Stars and to date there has not been any peer-reviewed analysis of the ten-item Family Star Plus. The original eight-item Family Star has only limited support for its inter-rater reliability; an analysis of star data collected by 24 key workers found low inter-rater reliability when the eight outcome areas were examined separately, but adequate reliability when these were grouped into 'Journey of Change' categories and three outlier scores were removed: a score of 0.81 against a threshold of 0.8 (MacKeith, 2014). Limited conclusions can be drawn here due to the small dataset, while for the other psychometric qualities, only one analysis has been published -but not peer-reviewed-by Triangle.…”

“…An analysis of star data collected by 24 key workers found low inter-rater reliability when the eight outcome areas were examined separately. However when these were grouped into the “Journey of Change” categories and three outlier scores removed, adequate reliability was found (a score of 0.81 with a threshold of 0.8 noted by MacKeith, 2014). Limited conclusions can be drawn here because of the small data set, while for the other psychometric qualities, only one analysis has been published – but not peer reviewed by Triangle.…”