A Within-Program Analysis of edTPA Score Reliability, Validity, and Utility

Abstract

The edTPA is a multidimensional performance assessment that targets student teachers’ planning, instruction, and assessment proficiency as reflected in narrative commentary, lesson plans, teaching videos, and student work. The purpose of this study was to analyze the reliability, validity, and utility of edTPA scores for three academic year cohorts of music student teachers (N = 60) at an institution that adopted the edTPA for use in teacher candidate evaluation. Overall, music teacher candidates were rated highest in planning, followed by assessment and instruction. Differences in assessment task and total scores were statistically significant between Cohort 1 and Cohort 3, suggesting that faculty may have become more effective over time in preparing student teachers to complete the edTPA portfolio. Scores for all three edTPA tasks and total edTPA scores exhibited adequate internal consistency. Factor analysis produced a clear structure that corresponded to the three edTPA tasks, with evidence of adequate model fit. Criterion-related validity evidence for edTPA scores, however, was mixed, with most correlations being of modest magnitude (< .25). We discuss implications for teacher educators and future research.

Reliability and Construct Validity of the edTPA for Music Education

“…Measures of internal consistency (α) in this study for all tasks and total scores exceeded .70 and were similar to those attained by Austin and Berg (2020). However, only Task 2 (Instruction, R6-R10) met the .80 benchmark for acceptable reliability while Tasks 1 (Planning, R1-R5) and 3 (Assessment, R11-R15) did not.…”

“…Maybe these tasks are more closely related in music than in other subjects. However, the factor structure that emerged in Austin and Berg (2020) clearly aligned with the Planning, Instruction, and Assessment tasks, and does not support this assertion. Perhaps the teacher preparation program involved in this study taught assessment in such a way that caused students to view planning and assessment as being so closely associated that the scores they received for the a priori assessment rubrics did not coalesce in a meaningful way and, instead, loaded onto two different factors.…”

“…Continuous research is needed to monitor the psychometric qualities and identify weaknesses in this assessment. In the absence of publicly available data, researchers could replicate this study and others (Austin & Berg, 2020;Musselwhite & Wesolowski, 2019) by combining scores from multiple institutions to create analyses that are more robust. Future studies should involve multiple statistical procedures due to limitations and advantage of various methods.…”

“…In addition, violations of monotonicity (i.e., the assumption that variables move consistently in the same or opposite directions) raised concerns with the overall rating scale structure. Austin and Berg (2020) analyzed the reliability, validity, and utility of edTPA scores for music teacher candidates (N = 60) over a 3-year period from 2013 to 2015. Scores for all three tasks (α = .76-.81) and the 15 rubrics combined (α = .84) demonstrated adequate internal consistency.…”

A Critical Examination of Preservice Music Teachers’ Writing on the edTPA Portfolio

“…Though SCALE maintains the K–12 Performing Arts edTPA Portfolio is a performance-based measure of preservice teacher effectiveness (AACTE & SCALE, 2016), the instrument has received a considerable amount of attention and criticism regarding its validity for assessing preservice teacher readiness to enter the profession (Au, 2013; Austin & Berg, 2020; Bergstrand Othman et al, 2017; Hash, 2020; Madeloni, 2015). Researchers have indicated that university faculty feel compelled to assist preservice teachers in comprehending the numerous technical issues and portfolio criteria such as (a) page limits, (b) number of documents per task, and (c) interpretation of the handbook, rubrics, and commentaries, rather than spending more time on developing specific instructional skills (Greenblatt & O’Hara, 2015; Parkes & Powell, 2015, 2019).…”