Abstract:Supervisors' ratings of psychology trainees' competence in field settings are a critical component of training assessment. There has been little systematic research regarding the validity of these assessments, but the available evidence suggests we have a problem! Supervisors' judgments may be affected by systemic biases that pose a serious threat to assessment credibility. The current study is part of a research collaboration among six universities that endeavors to develop and evaluate a new method the use o… Show more
“…This applied to supervisor beliefs of their own and of peer assessments. These results are consistent with recent evidence in psychology (Gonsalvez et al, ; Gonsalvez & Freestone, ) and other health disciplines (e.g., Bogo, Regehr, Hughes, Power, & Globerman, ), but is somewhat different from earlier observations (Robiner et al, ) when the central tendency bias (43%) was judged to occur slightly more frequently than leniency (39%). The presence of any form of bias merits attention, but the high incidence of leniency is of particular concern because supervisors' assessments are among the most important gate‐keeping, quality assurance tools of the profession (Gonsalvez & Freestone, ; O'Donovan et al, ).…”
Section: Discussionsupporting
confidence: 89%
“…These findings highlight the urgent need to develop and validate objective measures to assess competence and to outline reliable criteria with relevant behavioural anchors to identify performance that falls below competence thresholds, a view articulated by others (e.g., Gonsalvez & McLeod, ; Kaslow et al, , ). These research initiatives may have to include the development of new and innovative instruments less sensitive than Likert‐type scales are to common rating biases (Gonsalvez et al, ).…”
Objective
The objective of the current study was to compare current supervisory practices in Australia against those derived from pedagogic principles and/or practice guidelines recommended by experts in the field. Three core supervisor competencies, namely goal‐setting, providing formative feedback, and conducting summative assessments, were chosen for scrutiny.
Methods
One hundred thirteen accredited psychology supervisors in Australia completed an online questionnaire that had supervisors report details about their goal‐setting, feedback, and summative assessment practices in supervision.
Results
Several aspects pertaining to summative assessment require improvement. Specifically, the limited use of observational techniques was inconsistent with principles of competency‐based pedagogies and with recommendations by experts. A significant percentage of supervisors believed that summative ratings of trainee competence conducted by themselves (58%) and by their peers (66%) were compromised by leniency effects. Further, half the supervisors surveyed reported that summative assessments were made difficult by little or no guidance from training institutions about the benchmarks trainees were expected to meet at the end of placements.
Conclusions
Supervisory practices concerning goal‐setting were generally consistent with best‐practice guidelines derived from the literature. However, improvements are warranted in key supervisory practices, including more frequent use of observational techniques to inform formative feedback, more effective strategies to counter leniency in summative assessment, and better communication between training institutions and supervisors.
“…This applied to supervisor beliefs of their own and of peer assessments. These results are consistent with recent evidence in psychology (Gonsalvez et al, ; Gonsalvez & Freestone, ) and other health disciplines (e.g., Bogo, Regehr, Hughes, Power, & Globerman, ), but is somewhat different from earlier observations (Robiner et al, ) when the central tendency bias (43%) was judged to occur slightly more frequently than leniency (39%). The presence of any form of bias merits attention, but the high incidence of leniency is of particular concern because supervisors' assessments are among the most important gate‐keeping, quality assurance tools of the profession (Gonsalvez & Freestone, ; O'Donovan et al, ).…”
Section: Discussionsupporting
confidence: 89%
“…These findings highlight the urgent need to develop and validate objective measures to assess competence and to outline reliable criteria with relevant behavioural anchors to identify performance that falls below competence thresholds, a view articulated by others (e.g., Gonsalvez & McLeod, ; Kaslow et al, , ). These research initiatives may have to include the development of new and innovative instruments less sensitive than Likert‐type scales are to common rating biases (Gonsalvez et al, ).…”
Objective
The objective of the current study was to compare current supervisory practices in Australia against those derived from pedagogic principles and/or practice guidelines recommended by experts in the field. Three core supervisor competencies, namely goal‐setting, providing formative feedback, and conducting summative assessments, were chosen for scrutiny.
Methods
One hundred thirteen accredited psychology supervisors in Australia completed an online questionnaire that had supervisors report details about their goal‐setting, feedback, and summative assessment practices in supervision.
Results
Several aspects pertaining to summative assessment require improvement. Specifically, the limited use of observational techniques was inconsistent with principles of competency‐based pedagogies and with recommendations by experts. A significant percentage of supervisors believed that summative ratings of trainee competence conducted by themselves (58%) and by their peers (66%) were compromised by leniency effects. Further, half the supervisors surveyed reported that summative assessments were made difficult by little or no guidance from training institutions about the benchmarks trainees were expected to meet at the end of placements.
Conclusions
Supervisory practices concerning goal‐setting were generally consistent with best‐practice guidelines derived from the literature. However, improvements are warranted in key supervisory practices, including more frequent use of observational techniques to inform formative feedback, more effective strategies to counter leniency in summative assessment, and better communication between training institutions and supervisors.
“…A competency emphasis requires clarification of the nature and theoretical grounding of competencies, and of the most effective methods to train and assess them, including the evaluation of problem‐based learning approaches and of optimal modes of supervision. (Kavanagh, , p. 65) Psychology is in the process of turning its attention to defining and developing competency‐based training (Baillie et al, ; Gonsalvez et al, ; Pachana, Sofronoff, Scott, & Helmes, ). Drawing on insights from the medical field, the growing focus on competency within psychology training programmes has been described as an educational “culture shift” (Roberts, Borden, Christiansen, & Lopez, , p. 356).…”
Section: Introductionmentioning
confidence: 99%
“…(Nelson, ; Pachana et al, ; Roberts et al, ). The importance of experiential learning within professional psychology training environments and the fair and meaningful assessment of these skills as a measurement of competence have led to interest in case‐based training tools and structured oral examination (SOE) formats (Gonsalvez et al, ; Pachana et al, ; Scott, Pachana, & Sofronoff, ). This review will explore recent developments in competency‐based training, and consider the role of oral training and assessment tools as mechanisms for building and measuring competence of clinical psychology graduates.…”
Background
Competency‐based training and assessment is considered the best practice internationally in postgraduate clinical psychology education. In Australia, there are still some ways to go as programmes begin to integrate competency‐based pedagogical models into clinical training and assessment. Further understanding of the strengths and challenges of these models will be useful to educators interested in developing a competency‐based approach.
Methods
A structured literature review was carried out using the databases PsychInfo, PubMed, and PsychArticles. Keywords were: competency, and “training” or “assessment” or “model” or “clinical” or “psychology” or “medicine” or “allied health.” Articles had to be written in the English language and published in peer‐reviewed journals. Relevant book chapters and web references from professional accreditation bodies were also assessed for inclusion. A total of 54 references were utilised in the review.
Results
The review supports the relevance of competency‐based learning and teaching. It draws on seminal benchmarking work from the international literature and considers the strengths and limitations of competency‐based approaches to clinical health training and assessment. The review provides support for the ongoing progression towards competency‐based training models in Australian postgraduate clinical psychology.
Conclusions
Competency‐based training and assessment methods offer educators sophisticated mechanisms for ensuring that clinical psychology graduates are prepared to meet the demands of professional practice and public accountability. Further efforts at integrating competency‐based training models into Australian postgraduate curriculum, and associated research into the outcomes, are necessary to ensure a pedagogical culture of best practice in this country.
“…This becomes apparent particularly when ratings are compared to other performance indicators (e.g., positive subjective appraisal and ratings despite formal fidelity ratings of observed therapy session suggesting only average or poor performance). It is worth noting that rating biases may affect supervisor ratings as well, with research indicating that supervisors' summative assessments are vulnerable to leniency and halo biases (Gonsalvez & Freestone, 2007;Gonsalvez et al, 2013;Robiner et al, 1997), especially when supervisor evaluations occur after a long supervisor-trainee relationship (e.g., at placement end). The current study indicates that leniency biases may be exaggerated in peer evaluations.…”
Observation of counsellor skills through a one-way mirror, video or audio recording followed by supervisors and peers feedback is common in counsellor training. The nature and extent of agreement between supervisor-peer dyads is unclear. Using a standard scale, supervisors and peers rated 32 interviews by psychology trainees observed through a one-way mirror. Results indicated that peers and supervisors used similar dimensions to cluster the various competencies. Peers rated counsellor performance more positively for general counselling skills but not for specialised techniques. Analyses revealed good supervisor-peer agreement for some items and poor agreement on others, with some differences being unacceptably large. The study has important implications for how feedback involving supervisors and peers might be managed and for peer supervision models. Observation of counsellor skills through a one-way mirror, video or audio recording followed by supervisors and peers feedback is common in counsellor training. The nature and extent of agreement between supervisor-peer dyads is unclear. Using a standard scale, supervisors and peers rated 32 interviews by psychology trainees observed through a one-way mirror. Results indicated that peers and supervisors used similar dimensions to cluster the various competencies. Peers rated counsellor performance more positively for general counselling skills but not for specialised techniques. Analyses revealed good supervisor-peer agreement for some items and poor agreement on others, with some differences being unacceptably large. The study has important implications for how feedback involving supervisors and peers might be managed and for peer supervision models. (Gonsalvez & Milne, 2010). First of all, novice trainees lack both competence and confidence, and have to be supported through phases of misgivings and self-doubt as they deal with high levels of affect and difficult psychological problems (Stoltenberg, Bailey, Cruzan, Hart, & Ukuku, 2014). Secondly, the requirement for intensive supervision is mainly determined by the perceived importance of observation that may be immediate (e.g., through a one-way mirror or co-therapy) or delayed (e.g., through review of video or audio recordings). As a supervisory technique, observation is supported by expert consensus (see Reiser, 2014) and by research (e.g., Townend, Iannetta & Freeston, 2002). Conversely, an over reliance on subjective methods is not recommended, because self-report of case work may be unreliable, may miss important information, and may be vulnerable to bias particularly during early stages of counsellor development, when trainees are less capable of accurate self appraisal (Campbell, 1994;Gonsalvez & Calvert, 2014;Townend, Iannetta & Freeston, 2002). Finally, important knowledge-application (e.g., SUPERVISOR AND PEER RATINGS case conceptualization), skills (e.g., generic counseling and other specialized therapy skills) and relationship (e.g., self-awareness and transference reactions) competencies are...
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
