Using Adaptive Comparative Judgement to Assess Student Work in an MBA Course

Metzgar, Matthew

doi:10.20533/iji.1742.4712.2016.0148

Cited by 5 publications

(5 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of CJ studies have further shown high interrater reliability scores among trained and untrained raters (Heldsinger & Humphrey, 2013; Jones & Alcock, 2014; Jones & Inglis, 2015; Jones, Swan, & Pollitt, 2015; McMahon & Jones, 2015; Metzgar, 2016). Pollitt (2012) indicated untrained raters competently assessed writing quality and achieved absolute reliability (an equivalent of Cronbach's α ) estimates of .96 of the objects rated.…”

Section: Introductionmentioning

confidence: 99%

“…Researchers have advocated for wider adoption of CJ based on its high reliability as measured in a range of assessment domains across a number of studies (Bramley, 2007;Jones & Inglis, 2015;Pollitt, 2012;Steedle & Ferrara;2016). None of these studies, however, examined CJ reliability when rating texts composed by English as a second language (ESL) writers, whose language skills and organizational patterns may make it difficult for raters to agree on relative essay quality.…”

mentioning

confidence: 99%

“…Furthermore, CJ has not been adequately compared to traditional MFRM‐adjusted scores to determine whether it has an advantage over the more commonly accepted reliability procedure. Finally, it is unknown if CJ is reliable when performed by minimally trained raters, though a number of studies involving CJ have revealed high reliability results among both trained and untrained raters in other domains (Heldinger & Humphrey, 2013; Jones & Alcock, 2014; McMahon & Jones, 2015; Metzgar, 2016). Moreover, a comprehensive comparison among rubric‐rated and CJ‐rated essays by both experienced and inexperienced raters of ESL writing has not yet been conducted.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Rubric Rating with MFRM versus Randomly Distributed Comparative Judgment: A Comparison of Two Approaches to Second‐Language Writing Assessment

Sims

Cox

Eckstein

et al. 2020

Educational Measurement

View full text Add to dashboard Cite

The purpose of this study is to explore the reliability of a potentially more practical approach to direct writing assessment in the context of ESL writing. Traditional rubric rating (RR) is a common yet resource-intensive evaluation practice when performed reliably. This study compared the traditional rubric model of ESL writing assessment and many-facet Rasch modeling (MFRM) to comparative judgment (CJ), the new approach, which shows promising results in terms of reliability. We employed two groups of raters-novice and experienced-and used essays that had been previously double-rated, analyzed with MFRM, and selected with fit statistics. We compared the results of the novice and experienced groups against the initial ratings using raw scores, MFRM, and a modern form of CJ-randomly distributed comparative judgment (RDCJ). Results showed that the CJ approach, though not appropriate for all contexts, can be as reliable as RR while showing promise as a more practical approach. Additionally, CJ is easily transferable to novel assessment tasks while still providing context-specific scores. Results from this study will not only inform future studies but can help guide ESL programs in selecting a rating model best suited to their specific needs.

show abstract

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Rubric Rating with MFRM versus Randomly Distributed Comparative Judgment: A Comparison of Two Approaches to Second‐Language Writing Assessment

Sims

Cox

Eckstein

et al. 2020

Educational Measurement

View full text Add to dashboard Cite

show abstract

“…Last, traditional assessments such as written exams and quizzes often test conceptual understanding and are simple to develop and implement. Compared to experiential learning-based assessments, less ambiguity and subjectivity are involved during evaluation and grading [ 39 ]. Some differences among them are related to frequency and content volume.…”

Section: Methodsmentioning

confidence: 99%

How to assess? Student preferences for methods to assess experiential learning: A best-worst scaling approach

Melo

Monteza²,

Colson³

et al. 2022

PLoS ONE

View full text Add to dashboard Cite

Transitioning from traditional in-person classroom formats to online instructional delivery methods and online student assessments during the COVID-19 pandemic was a significant challenge to effective teaching, learning, and evaluation. Although there is a growing literature assessing the relative efficacy of different online teaching techniques, previous literature has not analyzed, from the student perspective, what methods are preferred for evaluating performance in experiential learning courses. How students perceive assessment methods is critical because it can affect their learning experience and academic achievements. To better understand student preferences for assessment methods, the best-worst scaling approach was used in two online surveys of 218 undergraduate students enrolled in experiential learning-based programs during the COVID-19 pandemic. Analysis of student responses indicates students’ highest levels of support for assessments that emphasize the development of critical thinking skills and professional skills, such as case studies. Most students would prefer assessments that are driving (develop different skills such as creative thinking) and realistic (develop skills transferable to the real world), while only a few (< 1%) prefer assessments that are fast (involve little time), frequent, safe (has preventive measures to eliminate cheating), or strategic (high probability of getting good grades).

show abstract

“…Given the impact of psychological safety in motivating individuals to seek out feedback, we contend that further research is needed as to the influence the strategies by which leaders or instructors implement peer review and feedback processes into the teams they oversee have on the construct. ACJ is one such strategy for peer review and feedback, and though it has only once been used in business education to assist in peer evaluation (Metzgar, 2016), how ACJ influences psychological safety within teams is greatly under-researched. Therefore, a particular interest for our study was the interaction and impact of peer review using ACJ on one's ability to take interpersonal risks.…”

Section: Psychological Safetymentioning

confidence: 99%

Adaptive Comparative Judgment and Psychological Safety in Accounting Education

Bartholomew

Jones

Hawkins

et al. 2021

JET

View full text Add to dashboard Cite

Traditional Accounting curricula include instruction on the preparation of audit documentation, however, experiential instruction for students on the process of review-both as a reviewer and reviewee-is often scarce or missing. This study investigated a classroom intervention that engaged undergraduate students in peer-review activities to gauge how peer review and feedback impacted student performance and their perceptions of being able to engage in interpersonal risks. Using a case-method approach, students developed audit workpapers that were later peer-reviewed through a digital system utilizing adaptive comparative judgment (ACJ). Students’ achievement scores were collected, and students also completed a pre/post survey on psychological safety. Our results indicate that student psychological safety increased over the course of the semester; however, the peer review process through ACJ did not significantly improve student performance within the class. The students responded positively to the intervention as an engaging learning process and effective in teaching real-world skills. Thus, this intervention provides an example of how peer review activities could enhance the learning experience for students.

show abstract

Using Adaptive Comparative Judgement to Assess Student Work in an MBA Course

Abstract: Abstract

Cited by 5 publications

References 7 publications

Rubric Rating with MFRM versus Randomly Distributed Comparative Judgment: A Comparison of Two Approaches to Second‐Language Writing Assessment

Rubric Rating with MFRM versus Randomly Distributed Comparative Judgment: A Comparison of Two Approaches to Second‐Language Writing Assessment

How to assess? Student preferences for methods to assess experiential learning: A best-worst scaling approach

Adaptive Comparative Judgment and Psychological Safety in Accounting Education

Contact Info

Product

Resources

About