Effective rating scale development for speaking tests: Performance decision trees

Fulcher, Glenn; Davidson, Fred; Kemp, Jenny

doi:10.1177/0265532209359514

Cited by 116 publications

(117 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…When constructing the scale, samples of learner performance undertaking test tasks in specific language contexts need to be collected, following the transcription, and identification of key performance features by discourse analysis (Fulcher, 2003;Fulcher, Davidson, & Kemp, 2011). Descriptors for a performance data-driven approach are thus generated from the key discourse features of observed language use, such as grammar or vocabulary.…”

Section: Overviewmentioning

confidence: 99%

“…A No answer to this question would result in a score of 3; a Yes answer would yield a score of 4. Fulcher et al (2011), combining the advantages of the descriptive richness of the performance data-driven approach and the simplicity of decision making of the EBB scale, devised a 'performance decision tree' (PDT) for service encounters for L1 examinees. A PDT comprises a series of questions describing performance in service encounters and two options.…”

Section: Overviewmentioning

confidence: 99%

“…The problem of a time-consuming rating is largely eliminated by using a PDT. One problem Fulcher et al (2011) found with PDT scales was that each was developed for a particular task and a single population. Furthermore, the feature of dichotomous choices in EBB and PDT scales casts doubt on the accuracy of interpretation of learner performance, because the score interpretation is simply based on whether or not learners achieved fixed criteria, rather than on the degrees of learner skill or knowledge.…”

Section: Overviewmentioning

confidence: 99%

“…The concept of a rating checklist was adapted from EBB scales (Upshur & Turner, 1995) and the PDT (Fulcher et al, 2011). The elements in the checklist for this study were almost the same as the criteria in the rating scale (see Table 1), but with trichotomous instead of binary choices.…”

Section: -1mentioning

confidence: 99%

See 3 more Smart Citations

Teacher Interpretation of Test Scores and Feedback to Students in EFL Classrooms: A Comparison of Two Rating Methods

Chou¹

2013

HES

View full text Add to dashboard Cite

Rating scales have been used as a major assessment instrument to measure language performance on oral tasks. The present research concerned, under the same rating criteria, how far teachers interpreted students' speaking scores by using two different types of rating method, and how far students could benefit from the feedback of the description of the two rating methods. Fifteen English teachers and 300 college students in Taiwan participated in the study. Under the same rating criteria, the two types of rating method, one with the level descriptors and the other with a checklist, were used by the teachers to assess an English speaking test. It was discovered that the rating method had a noticeable impact on how teachers judged students' performance and interpreted their scores. The majority of the students considered feedback from the verbally more detailed method more helpful for improving language ability.

show abstract

Section: Overviewmentioning

confidence: 99%

Section: Overviewmentioning

confidence: 99%

Section: Overviewmentioning

confidence: 99%

Section: -1mentioning

confidence: 99%

See 2 more Smart Citations

Teacher Interpretation of Test Scores and Feedback to Students in EFL Classrooms: A Comparison of Two Rating Methods

Chou¹

2013

HES

View full text Add to dashboard Cite

show abstract

“…Regarding the chaotic rating situation in writing, many scholars have attempted to improve the subjective assessment of writing (Fulcher, 1996;Fulcher et al, 2011;Knotch, 2007;Maftoon & Akef, 2010;North & Schneider, 1998;Upshur & Turner, 2002). These lines of queries aim to create more objectivity in the writing assessment.…”

Section: The Scale That Is Used In Assessing Performance Tasks Such Amentioning

confidence: 99%

Rating Scales Revisited: EFL Writing Assessment Context of Iran under Scrutiny

Ghanbari

Barati

Moinzadeh

2012

Language Testing in Asia

View full text Add to dashboard Cite

Through addressing particular ideologies regarding language, meaning, level of proficiency and target writer and reader, rating criteria define and control the what and how of the assessment process. But a point which has been neglected or intentionally concealed due to concerns of practicality and the legitimacy of the native authority in setting assessment guidelines in EFL writing assessment contexts is the appropriateness of the scale. To raise attention to the current vague rating situation and consequently remedy the state, present study followed two lines of argument. First, drawing on the socio-cognitive framework of Weir (2005) for validating writing assessment, it is discussed that the important characteristic of scoring validity necessitates an appropriate choice of rating rubrics. Second, through posing a critical argument, deficiencies of the present practice of adopting rating scales are revealed and consequently it is discussed how assessment circles in native countries by setting rating standards control and dominate the whole process of writing assessment. To add more flesh to the argument, the ESL Composition Profile of Jacobs, et Keywords: writing assessment, academic writing, rating scale, validity, construct validity, ESL Composition Profile (Jacobs, et al., 1981). IntroductionWithin the past few decades, writing assessment has been a constant concern to the extent that any new publications on written composition have some references to the issues related on evaluating writing. Due to the ascending importance of writing among all sections of the present modern society that values written communication as an index of educational growth, pronouncing judgment on a piece of writing text has found a significant place (Gere, 1980). However, assessing writing faces challenges on two major frontiers: on the one hand, program-level decisions regarding placement in different levels of a course or admission purposes necessitates a rigorous assessment plan, and on the other hand Pandora's Box of performance assessment reveals itself in the writing (Mc Namara, 1996) as there are still vague grounds in the articulation of a sound and explicit basis in scoring writing (Gere, 1980).The ability to make sound decisions about the writing ability of individual writers is the de facto function expected from the scoring procedures involved. Therefore, any malfunctioning in the writing assessment might pop up this basic but critical question in mind: do scoring procedures work correctly to accomplish their expected purpose in providing a sound appraisal of writers' writing ability? Inspired by the above line of inquiry, the present study proceeds to give a second thought to the procedures of writing assessment. In this line, the venerable tradition of using rating scales in writing assessment is investigated. Upon contextualizing the concept of rating scale in its theoretical background and analyzing the value-laden nature of the scales involved, the writer proceeds to underscore the appropriateness of r...

show abstract

Spoken Discourse

Lazaraton

2013

The Companion to Language Assessment

View full text Add to dashboard Cite

When second or foreign language (L2) learners take a speaking proficiency test, their scores are really the only outcome in which they are interested, especially in high stakes assessment contexts—where a sufficient score is necessary for university admission, citizenship, or employment. So it has been with language testers: the emphasis on test scores (or ratings, as they are likely to be in speaking assessment) leads to the primary concern about score as well as rater reliability—the degree to which a rater is consistent over time, across language samples, and with other raters. More recently, however, attention has shifted to issues of test validity, which considers the processes by which test scores lead to inferences about their meaning. This is increasingly the case for L2 speaking tests, where investigation of the process of speaking assessment has become almost as important as questions about outcome scores. Two primary questions underlie the emphasis on test process: first, how can the language that is produced in a speaking test be characterized? Second, how can careful analyses of that language lead to improvements in test design and rating processes? To answer these questions, the analysis of spoken discourse has become a key strand of research in the language assessment community. The chapter begins with a definition of discourse and discourse analysis; subsequent sections present facts about the nature of spoken discourse in both monologic and interactive contexts and empirical findings on the co‐construction of discourse in the speaking assessment process involving test takers, interviewers, and peers. The chapter concludes with a look at the challenges facing spoken discourse analysts and future directions in L2 speaking assessment, including the impact of digital technologies and large‐scale language corpora, the systematic development of rating scales, and the importance of formative, classroom‐based assessment in understanding the nature of L2 speaking proficiency.

show abstract

Effective rating scale development for speaking tests: Performance decision trees

Cited by 116 publications

References 47 publications

Teacher Interpretation of Test Scores and Feedback to Students in EFL Classrooms: A Comparison of Two Rating Methods

Teacher Interpretation of Test Scores and Feedback to Students in EFL Classrooms: A Comparison of Two Rating Methods

Rating Scales Revisited: EFL Writing Assessment Context of Iran under Scrutiny

Spoken Discourse

Contact Info

Product

Resources

About