The validation of a rating scale to assess dietitians’ use of behaviour change skills

Abstract

Background: Evidence suggests that education alone is unlikely to elicit dietary-behavioural change (Contento, 1995). Consequently, many dietitians are moving from a traditional advice-giving role to one which utilises Ôbehav-iour change skillsÕ (BCS) in dietary counselling. BCS is an umbrella term used to cover a wide range of skills and techniques drawn from the fields of counselling, motivational interviewing (MI) and cognitive behavioural therapy (CBT). In order to assess the efficacy of this approach, a means of quantifying BCS-use is required. This two-stage study aimed to validate a newly-devised scale to assess dietitiansÕ BCS-use in one-to-one dietary counselling. Methods: Items for the scale were generated by drawing on the literature, syllabi for training in BCS and its parent disciplines (counselling, MI and CBT), and specialist dietitians. The resulting scale and manual were revised following assessment of content validity by expert panel and piloting. In stage one, 21 dietetic consultations were audiotaped and rated for BCS-use by three BCS-trained dietitians. Inter-rater agreement was calculated using the kappa statistic and intra-class correlation (ICC), to give a Ôchance correctedÕ measure of agreement. Validity was tested using a psychologist's subjective assessment of BCSuse as a proxy Ôgold-standardÕ compared with the dietitiansÕ ratings, again using kappa and ICC. In stage two the scale was further revised before an additional 20 audiotaped consultations were analysed using the same procedure. Ethical approval for the study was given by the appropriate NHS and university research ethics committees. Results: At stage one, although kappas were fairly poor for agreement on individual criteria, the ICC for overall scores indicated a ÔfairÕ level of agreement, according to Shrout's (1998) classifications: ICC = 0.584 (CI 0.339-0.784). Results for validity were poor with the psychologist frequently rating higher than the dietitians. At stage two, following scale revision, results for inter-rater agreement improved with more criteria showing ÔmoderateÕ or ÔsubstantialÕ agreement. Ten out of the 21 criteria achieved levels of agreement classified as ÔfairÕ or higher for all three rater pairs. The ICC for overall scores improved to indicate ÔmoderateÕ agreement: ICC = 0.640 (CI 0.404-0.821). Validity results remained poor. Discussion: The moderate level of overall inter-rater agreement observed in the revised scale is considered acceptable (Jones, 2006) and indicates this tool is useful. This measure is more relevant to the purpose of the tool than agreement on individual criteria given it is intended to

‘If you listen to me properly, I feel good’: a qualitative examination of patient experiences of dietetic consultations

“…The present study was part of a larger investigation (Bonner et al. , 2008) with three recruitment stages.…”

Assessing communication skills in dietetic consultations: the development of the reliable and valid DIET‐COMMS tool

“…() and Bonner et al . () to design such a tool, both of which demonstrated good face and content validity, and in the case of Bonner et al . () moderate inter‐rater reliability as defined by Shrout ().…”

“…() moderate inter‐rater reliability as defined by Shrout (). However, neither scale was shown to meet standard psychometric criteria for both reliability and validity, and both study teams recognised the need for further work before their scales could be used to assess dietitians with confidence (Horacek et al ., ; Bonner et al ., ).…”

Changing dietary behaviour: the role and development of practitioner communication

“…In this case the content domain is the use of CSBC in dietetic one-to-one consultations. However all of these research teams recognised the need for further development before their tools could be used to assess the skills of dietitians with confidence (55)(56)(57) . More recently Whitehead et al (54) have developed and validated a tool, DIET-COMMS, for this purpose.…”

“…There is also a need for an appropriate validated tool to be available for use. There have been attempts to develop suitable assessment tools (55)(56)(57) all of which have demonstrated good content validity, that is the extent to which a panel of experts believe that the items included examine comprehensively, or represent a wellbalanced sample of, the content domain to be measured (58) . In this case the content domain is the use of CSBC in dietetic one-to-one consultations.…”