2023
DOI: 10.1177/01466216231174566
|View full text |Cite
|
Sign up to set email alerts
|

Modeling Rating Order Effects Under Item Response Theory Models for Rater-Mediated Assessments

Abstract: Rater effects are commonly observed in rater-mediated assessments. By using item response theory (IRT) modeling, raters can be treated as independent factors that function as instruments for measuring ratees. Most rater effects are static and can be addressed appropriately within an IRT framework, and a few models have been developed for dynamic rater effects. Operational rating projects often require human raters to continuously and repeatedly score ratees over a certain period, imposing a burden on the cogni… Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...
2
1

Citation Types

0
3
0

Year Published

2023
2023
2024
2024

Publication Types

Select...
4

Relationship

0
4

Authors

Journals

citations
Cited by 4 publications
(3 citation statements)
references
References 49 publications
0
3
0
Order By: Relevance
“…Before measuring rater severity, it is crucial to determine the item validity used as parameters for assessing students' writing. To ascertain this, the parameter utilized in the Many-Facet Rasch Measurement is the unidimensionality test (Sumintono & Widhiarso, 2013;Myford & Wolfe, 2004;Huang, 2023). The assumption test criteria used for testing the item's unidimensionality in the MFRM (Many-Facet Rasch Measurement) is the Raw Variance Explained by Measure, with a threshold value greater than 20% (≥ 20%); if it exceeds 40%, it indicates good quality, and if it surpasses 60%, it is considered exceptional (Sumintono & Widhiarso, 2013).…”
Section: Resultsmentioning
confidence: 99%
See 2 more Smart Citations
“…Before measuring rater severity, it is crucial to determine the item validity used as parameters for assessing students' writing. To ascertain this, the parameter utilized in the Many-Facet Rasch Measurement is the unidimensionality test (Sumintono & Widhiarso, 2013;Myford & Wolfe, 2004;Huang, 2023). The assumption test criteria used for testing the item's unidimensionality in the MFRM (Many-Facet Rasch Measurement) is the Raw Variance Explained by Measure, with a threshold value greater than 20% (≥ 20%); if it exceeds 40%, it indicates good quality, and if it surpasses 60%, it is considered exceptional (Sumintono & Widhiarso, 2013).…”
Section: Resultsmentioning
confidence: 99%
“…Thus, a comprehensive understanding of raters' tendencies towards being excessively severe or lenient is imperative to ensure fair and equitable evaluations of students' writing proficiency. Rater severity/leniency in students' composition refers to the degree to which raters consistently assign higher or lower scores when evaluating the written work of students (Myford & Wolfe, 2004;Huang, 2023). It is an important aspect of assessment in various educational contexts, including language education and writing courses.…”
Section: Introductionmentioning
confidence: 99%
See 1 more Smart Citation