Training in Routine Mental Health Outcome Assessment: an evaluation of the Victorian experience

Abstract

This paper evaluates training in the use of measures of outcomes and casemix provided to four pilot agencies in Victoria, Australia. The training program is outlined along with key evaluation findings. The knowledge and skills of participants developed during training is described. Deficiencies in the training program are identified and opportunities for improvement outlined.

Clinical Indicators for Routine Use in the Evaluation of Early Psychosis Intervention: Development, Training Support and Inter-Rater Reliability

“…In this situation, observed inter-rater agreement is as much influenced by the adequacy of the training procedures, the quality of the clinical vignettes, and clinician compliance with data collection, as by indicator reliability per se . Given the elaborate training procedures developed to support HoNOS ratings [15,35], the single 2-h CPIN-EP training session may not have given trainees enough time to learn all coding rules, an effect with the potential to deflate indicator reliability estimates. Some verbal feedback from trainees suggested that there were deficiencies in the vignettes themselves, resulting mainly from the need to compress information to code five indicators into a single-page clinical narrative.…”

“…Reliability was assessed descriptively by calculating percentage agreement between coding of clinicians-in-training versus consensus coding made by the expert reference group. For the SR, SOFAS, and GARF indicators we followed Australian procedures used to assess HoNOS training [15] and calculated ‘broad range’ (consensus score plus or minus one scoring level) percentage agreement, as well as that for the exact consensus score. An index of reliability was estimated by entering for each indicator the rates of agreement on each of the four vignettes into an online kappa calculator (http://justus.randolph.name/kappa).…”

The Use of the Health of the Nation Outcome Scale in an Outreach Rehabilitation Program

“…Coombs et al also reported early results of outcome assessment training in Victoria. [11] Results indicate that evaluation of training was positive and there was support for outcome measures. The authors suggest further training, regular promotion, improving enthusiasm and providing quick and useful feedback.…”

“…The introduction of ROM in Australia has raised considerable debate about its utility, feasibility issues and attitudes towards it. The Australian literature on ROM reveals a range of reactions from strong support7–11 to concerns about validity and feasibility,[2], [12] and claims that it is “reductionistic and alienating”. [13] This paper illustrates the use of ROM in an outreach-based rehabilitation program and discusses its use in light of current literature on feasibility issues, attitudes and clinical matters that impact on the uptake of ROM in routine clinical practice in Australia.…”

Attitudes of mental health staff to routine outcome measurement

“…A questionnaire (available from the first author on request) based on the work of Stedman et al (1997), Callaly & Hallebone (2001) and Coombs et al (2002) was devised; it covered basic professional information, nine general items concerning attitudes to OM, recent exposure to and desire for feedback, and eight items relating to the usefulness of the Health of the Nation Outcome Scales (HoNOS; Wing et al, 1998), a short form of the Life Skills Profile (Rosen et al, 1989), and the Behaviour and Symptom Identification Scale (BASIS; Eisen & Dickey, 1996). Apart from some informal piloting, no prior technical work was undertaken to assess the questionnaire's psychometric properties.…”