A Review of the Valid Methodological Use of Adaptive Comparative Judgment in Technology Education Research

Buckley, Jeffrey; Seery, Niall; Kimbell, Richard

doi:10.3389/feduc.2022.787926

Cited by 5 publications

(2 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Extending the application of the ACJ process beyond that of a single cohort or cluster to form a national picture of performance brings into focus the details of the ACJ reliability statistics, parameter values and the association with validity (cf. Buckley et al, 2022) all of which are of particular interest to this research. Although reported reliability statistics of more than 0.9 give confidence (Pollitt, 2012b;Bartholomew and Jones, 2021), there are notable critiques of the adaptive process.…”

Section: Gap Analysismentioning

confidence: 97%

Using Teachers’ Judgments of Quality to Establish Performance Standards in Technology Education Across Schools, Communities, and Nations

2022

Self Cite

View full text Add to dashboard Cite

The establishment and maintenance of national examination standards remains a serious issue for teachers and learners, whilst the levers of control remain firmly in the hands of Awarding Bodies and supervising politicians. Significantly, holistic assessment presents an agility and collective approach to establishing in the minds of teachers “what is of value” when determining the comparative evidence of pupil performance. It is argued in this paper that the collation of the comparative judgment process can initially identify and subsequently maintain standards of performance that can be defined on a cluster, regional or even national level. Much comparative judgment research centers on the formative benefits for learners, but here we place the focus on teachers operating in collaborative groups to establish standards within and beyond their own schools, and ultimately across the nation. We model a proof-of-concept research project. A rank is produced by the collective consensus of the participating teachers and used to simulate a definition of standard. Extrapolations are statistically modeled to demonstrate the potential for this approach to establishing a robust definition of national standards. But central to the process is what is going on in the minds of teachers as they make their judgements of quality. The research aims to draw out teachers’ constructs of quality; to make them explicit; to share them across classrooms and schools; and to empower teachers to debate and agree their standards across schools. This research brings to the fore the symbiotic relationship between teaching, learning and assessment.

show abstract

Section: Gap Analysismentioning

confidence: 97%

Using Teachers’ Judgments of Quality to Establish Performance Standards in Technology Education Across Schools, Communities, and Nations

2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Arguably for an assessment mechanism the most important factors considered are the reliability and validity of the assessment tool. The validity of ACJ refers to the validity of the rank-order that is produced and is directly tied to the cohort of judges which is assembled [18]. The reliability of ACJ sessions is described by the Scale Separation Reliability (SSR) coefficient which, in the context of comparative judgement, has strong indications that it reflects an interrater reliability index [19].…”

Section: Figure 1: Acj Pairwise Comparison Processmentioning

confidence: 99%