Examining the reliability of an international Chinese proficiency standardized writing assessment: Implications for assessment policy makers

Huang, Jinyan; Zhu, Danni; Xie, Duquan; Shu, Tiantian

doi:10.1016/j.asw.2023.100693

Cited by 3 publications

(2 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Standards for educational and psychological testing (AERA et al, 2014) defines validity as the degree to which evidence and theory support the interpretations of the measurement outcomes for proposed uses; and reliability as the consistency of the measurement outcomes across instances of the measurement procedure. Both validity and reliability must be treated as cornerstones in the construction and validation process of assessments (AERA et al, 2014; Huang et al, 2023; Kamaruddin & MohdMatore, 2021; Tan et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

“…The examination of construct-irrelevant variance and reliability evidence of international assessments such as the Program for International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS) administered across multiple countries and regions has become the central task for researchers and assessment professionals (Abu-Hilal et al, 2013; Arrindell et al, 2022; Huang et al, 2023; Espejo et al, 2022; Hambleton et al, 2005; Jia et al, 2018; Kamaruddin & MohdMatore, 2021; Marsh et al, 2013; Rias et al, 2021; Tan et al, 2021). This is because construct-irrelevant variance and reliability could be different across countries and regions due to unnecessary linguistic and cultural characteristics (Hambleton et al, 2005; Tan et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluating the Language- and Culture-Related Construct-Irrelevant Variance and Reliability of the Sense of School Belonging Scale: Suggestions for Revision

Huang

Dong

Han

et al. 2023

Journal of Psychoeducational Assessment

View full text Add to dashboard Cite

Using expert reviews and item response theory (IRT), this study evaluated the language- and culture-related construct-irrelevant variance and reliability of the 2019 TIMSS sense of school belonging scale (SSBS) for grades 4 and 8. The five items of the SSBS, which were identical for both grades, were reviewed for the language- and culture-related construct-irrelevant variance evidence by 10 assessment experts who have expertise in Chinese and English languages and cultures. Further, the existing TIMSS 2019 SSBS data associated with the countries of China (Hong Kong), Singapore, Australia, and America were analyzed within the framework of IRT for estimating the item and scale information function values (i.e., reliability) of the SSBS. The experts did find language- and culture-related evidence in each of the five items that might lead to the construct variance of the SSBS across these four countries. The IRT results further indicated that the SSBS item and scale information function values varied considerably across these four countries. Suggestions for revising the SSBS are proposed for TIMSS program developers.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Evaluating the Language- and Culture-Related Construct-Irrelevant Variance and Reliability of the Sense of School Belonging Scale: Suggestions for Revision

Huang

Dong

Han

et al. 2023

Journal of Psychoeducational Assessment

View full text Add to dashboard Cite

show abstract

Comparing Chinese L2 writing performance in paper-based and computer-based modes: Perspectives from the writing product and process

Wang,

Wang

2024

Assessing Writing

View full text Add to dashboard Cite

Rater variability and reliability of constructed response questions in New York state high-stakes tests of English language arts and mathematics: implications for educational assessment policy

Huang,

Whipple

2023

Humanit Soc Sci Commun

View full text Add to dashboard Cite

Using generalizability (G-) theory as a theoretical framework and research methodology, this study examined the impact of the current one-rater holistic scoring practice on the rater variability and reliability of constructed response questions in New York State high-stakes tests of grades four and six English language arts (ELA) and grades four and five mathematics assessments. Following the New York State scoring rubrics, a total of 36 grades four and six ELA constructed response samples and 72 grades four and five mathematics constructed response samples were marked holistically by ten independent raters having current certifications as educators in the State of New York. The results indicated that the current single-rater holistic scoring practice would not be able to yield acceptable G-coefficients for the New York State grades four and six ELA and grades four and five mathematics assessments. Implications for assessment policy making at the local and state levels are discussed.

show abstract

Examining the reliability of an international Chinese proficiency standardized writing assessment: Implications for assessment policy makers

Cited by 3 publications

References 23 publications

Evaluating the Language- and Culture-Related Construct-Irrelevant Variance and Reliability of the Sense of School Belonging Scale: Suggestions for Revision

Evaluating the Language- and Culture-Related Construct-Irrelevant Variance and Reliability of the Sense of School Belonging Scale: Suggestions for Revision

Comparing Chinese L2 writing performance in paper-based and computer-based modes: Perspectives from the writing product and process

Rater variability and reliability of constructed response questions in New York state high-stakes tests of English language arts and mathematics: implications for educational assessment policy

Contact Info

Product

Resources

About