Investigating the decision-making process of standard setting participants

Papageorgiou, Spiros

doi:10.1177/0265532209349472

Cited by 34 publications

(20 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, overall band levels and overall performance descriptors might be useful for score users who need a holistic evaluation of language ability. Moreover, because external criteria such as the CEFR scales are intentionally context-free so that they can have a wide range of applications, researchers note problems when using them in specific contexts, such as the design of test specifications (Alderson et al, 2006) or the setting of cut scores (Papageorgiou, 2010a). It has also been pointed out that external proficiency scales, such as the ones from the CEFR, are intentionally underspecified and their level of abstraction is too high, resulting in a gap between the score and its meaning (Fulcher et al, 2011, p. 8).…”

Section: Developing and Validating Band Levels And Descriptors For Rementioning

confidence: 99%

Developing and Validating Band Levels and Descriptors for Reporting Overall Examinee Performance

Papageorgiou

Morgan

et al. 2015

Language Assessment Quarterly

Self Cite

View full text Add to dashboard Cite

This study presents the development and empirical validation of score levels and descriptors specifically designed for reporting purposes to provide test takers with more than just a number on a score scale. In the context of a test primarily intended for 11-to 15-year-old students learning English as a second/foreign language, the study examined the number of band levels that could be meaningfully distinguished, the reliability of the classification of students into these band levels, and the development of overall performance descriptors that would provide meaningful information to score users. The performance data from 2,931 students who took the test were used. The band level solution was determined by balancing considerations for the reliability of classification decisions and the desire for the levels to represent meaningful performance differences. To construct meaningful descriptors for the band levels, multiple sources of information were examined, including the scoring rubrics, the characteristics of test items, typical student performance profiles, and the performance of norm groups on the test. The importance of establishing the psychometric quality of band levels and the empirical basis for performance descriptors, as well as the implications for similar efforts, are discussed.

show abstract

Section: Developing and Validating Band Levels And Descriptors For Rementioning

confidence: 99%

Developing and Validating Band Levels and Descriptors for Reporting Overall Examinee Performance

Papageorgiou

Morgan

et al. 2015

Language Assessment Quarterly

Self Cite

View full text Add to dashboard Cite

show abstract

“…A qualitative approach to investigating the judges' decision-making process when setting cut scores to the CEFR was employed by Papageorgiou (2010a). The study investigated the factors reportedly affecting the panelists' decision to set a cut score and the problems faced when setting cut scores in relation to the CEFR.…”

Section: Exploring the Quality Of Alignment With The Cefrmentioning

confidence: 99%

Issues in aligning assessments to the Common European Framework of Reference

Papageorgiou

2019

Self Cite

View full text Add to dashboard Cite

One of the main aims of the Common European Framework of Reference is to help providers and users of assessments "describe the levels of proficiency required by existing standards, tests and examinations in order to facilitate comparisons between different systems of qualifications" (Council of Europe 2001: 21). Providers of language assessments both inside and outside Europe follow various methodologies to align their assessments with the CEFR levels, as several case studies show (Figueras and Noijons 2009;Martyniuk 2010). This paper discusses the use of the CEFR in the field of language assessment, focusing in particular on issues related to alignment. The paper presents the types of validity evidence that should be accumulated to support an alignment claim and concludes with directions for future research in order to further enhance our understanding of using the CEFR for the design of assessments and the interpretation of assessment results.

show abstract

“…Papageorgiou (2010), for example, identified problems with the CEFR descriptors when used for setting cut scores. Participants of his standard-setting study reported that the context-free characteristics of the framework made it less applicable to a language testing context.…”

Section: Issues With the Cefrmentioning

confidence: 99%

Developing the China Standards of English: challenges at macropolitical and micropolitical levels

et al. 2017

View full text Add to dashboard Cite

Background: A project has recently been proposed by the National Education Examinations Authority, P. R. China, to develop a national framework of reference for English language education, provisionally called China Standards of English (CSE). In this article, we argue for the necessity of developing the framework of reference in China and challenges facing such an important endeavor. Methods: We first reviewed relevant documents of English language education in China to identify the overlaps and inconsistencies in the curricular requirements at different educational stages. We then explored the need for improving the transparency in policies and practices of English language education. Finally, challenges facing the development and implementation of the CSE were discussed based on Alderson's (2009) conceptualization of politics in language education. Results: For governmental organizations, coordinating resources and negotiating views of different educational stages may prove challenging due to the segmental management structure of English language education in China. Micropolitically, the CSE project may face technical challenges due to the innovative nature of the work involved. Resistance may also be encountered from practitioners who may pay lip service to the new framework and feel reluctant to change their pedagogical and assessment practices. Conclusions: The discussion of challenges facing the development of a national framework for English, the first of its kind in China, is intended to influence and inform decisions on how the CSE can be developed, and more importantly, how the framework can be implemented so as to achieve the ultimate purpose of improving English teaching, learning and assessment in China.

show abstract

Investigating the decision-making process of standard setting participants

Cited by 34 publications

References 23 publications

Developing and Validating Band Levels and Descriptors for Reporting Overall Examinee Performance

Developing and Validating Band Levels and Descriptors for Reporting Overall Examinee Performance

Issues in aligning assessments to the Common European Framework of Reference

Developing the China Standards of English: challenges at macropolitical and micropolitical levels

Contact Info

Product

Resources

About