Addressing Consequences and Validity during Test Design and Development:

KELLY, JUSTIN; Renn, Jennifer; NORTON, JENNIFER

doi:10.2307/j.ctvvngrq.14

Cited by 3 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, there is a difference between the performative aspect of rehearsed scripts and actual language proficiency assessment. The tendency to memorize rote responses threatens the score validity based on the CAL validation framework (Kelly et al 2017) because memorized responses may not apply to other speaking situations and raters may not consistently detect or score rehearsed speech. While rehearsed speech is acknowledged as a proven strategy for acquiring vocabulary or formulaic sequences and preparing for speeches or exams, it does not reflect real-world communication.…”

Section: Resultsmentioning

confidence: 99%

“…However, we may wisely question some decisions based upon test scores vis-à-vis the CAL Validation framework (Kelly et al 2017). McNamara et al (2019) note the difference between fairness and justice in language testing.…”

Section: Resultsmentioning

confidence: 99%

“…Performance describes language that is practiced in designated contexts while proficiency refers to spontaneous language used in conditions that are unanticipated (Rubio and Hacking 2019). The Center for Applied Linguistics (CAL) validation framework notes that a test's validity can be threatened when the interpretation of the test score is impacted (Kelly et al 2017), and this could be by a raters' failure to detect or appropriately score scripted speech samples.…”

Section: Rote Learning and Language Proficiencymentioning

confidence: 99%

“…Therefore, when examinees memorize scripts for their oral responses, validity is threatened. For tests to be valid, responses need to be consistently rated, and score interpretations need to be both meaningful and impartial (Kelly et al 2017); however, when raters encounter memorized responses intermingled with the spontaneous, scoring can be impacted. Thus, the prevalence of rehearsed responses in speaking tests should be evaluated to determine the effect it might have on test validity.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Line, please? An analysis of the rehearsed speech characteristics of native Korean speakers on the English Oral Proficiency Interview—Computer (OPIc)

et al. 2020

View full text Add to dashboard Cite

Two assumptions of speaking proficiency tests are that the speech produced is spontaneous and the the scores on those tests predict what examinees can do in real-world communicative situations. Therefore, when examinees memorize scripts for their oral responses, the validity of the score interpretation is threatened. While the American Council on the Teaching of Foreign Languages (ACTFL) Proficiency Guidelines identify rehearsed content as a major hindrance to interviewees being rated above Novice High, many examinees still prepare for speaking tests by memorizing and rehearsing scripts hoping these "performances" are awarded higher scores. To investigate this phenomenon, researchers screened 300 previously rated Oral Proficiency Interview-computer (OPIc) tests and found 39 examinees who had at least one response that had been tagged as rehearsed. Each examinee’s responses were then transcribed, and the spontaneous and rehearsed tasks were compared. Temporal fluency articulation rates differed significantly between the spontaneous and rehearsed segments; however, the strongest evidence of memorization lay in the transcriptions and the patterns that emerged within and across interviews. Test developers, therefore, need to be vigilant in creating scoring guidelines for rehearsed content.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Rote Learning and Language Proficiencymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Line, please? An analysis of the rehearsed speech characteristics of native Korean speakers on the English Oral Proficiency Interview—Computer (OPIc)

et al. 2020

View full text Add to dashboard Cite

show abstract

“…In general, a test is never considered valid or invalid in a dichotomous manner but instead an argument is made for an instrument's validity within a specific context and for a specific use. The Center for Applied Linguistics Validation Framework (Kelly, Renn, & Norton, 2017) offers a useful way to examine these features and build an argument that an instrument can measure what it intends to measure. This framework builds on the Evidence-Centered Design work of Mislevy, Almond, and Lukas (2003) and connects it with Bachman and Palmer's (2010) Assessment Use Argument.…”

Section: Validity Use Argumentmentioning

confidence: 99%

A validity argument to support the ACTFL Assessment of Performance Toward Proficiency in Languages (AAPPL)

Cox

Malone

2018

Foreign Language Annals

View full text Add to dashboard Cite

This article presents evidence for a validity argument on the ACTFL Assessment of Performance Toward Proficiency in Languages (AAPPL) tests (Interpersonal Listening/Speaking, Presentational Writing, Interpretive Listening, and Interpretive Reading) by summarizing an analysis of the 2014 test data from examinations administered in three languages (Chinese, French, and Spanish) to more than 10,000 students in grades 5 to 12. The specific stages the authors evaluated included the following: (1) the design of the instrument, (2) the effectiveness of the rating scale, (3) the reliability of the instrument, and (4) the extent to which ACTFL proficiency levels were generalizable across languages. As a complete test battery, there is validity evidence that the Assessment of Performance Toward Proficiency can be used to measure the performance of upper elementary, middle, and high school students as they develop increasingly sophisticated language proficiency. The strengths of the examinations are the productive skill areas, whereas the receptive skill areas have been targeted for continuing development.

show abstract

Does text entry method make a difference on Chinese writing test scores?

Bourgerie

Cox

Riep

2022

CSL

View full text Add to dashboard Cite

Should the writing construct be assessed through handwriting or keyboarding? As the only major language entirely without a syllabary or alphabet, the Chinese writing system is unique among modern languages, thus the question of writing proficiency is complicated by character recall. Most of the testing research comparing text entry methods has been conducted in English and has found that keyboarding and handwriting can be used interchangeably. This paper reports the outcome of a study comparing the results of handwritten and typed versions of the Chinese ACTFL Writing Proficiency Test (WPT). L2 Chinese students (n = 25) with Intermediate to Superior speaking skills were randomly divided into two groups and took both WPT versions in a counterbalanced design. Keyboarding resulted in significantly higher test scores [repeated measures ANOVA F(1, 23) = 62.7, p < .001, effect size partial eta squared = .73]. Keyboarding was on average 1.69 ACTFL sublevels higher than handwriting. Finally, this paper will discuss the writing construct in Chinese along with pedagogical implications around curricular decisions on teaching and assessing handwriting vs. keyboarding.

show abstract

Addressing Consequences and Validity during Test Design and Development:

Cited by 3 publications

References 0 publications

Line, please? An analysis of the rehearsed speech characteristics of native Korean speakers on the English Oral Proficiency Interview—Computer (OPIc)

Line, please? An analysis of the rehearsed speech characteristics of native Korean speakers on the English Oral Proficiency Interview—Computer (OPIc)

A validity argument to support the ACTFL Assessment of Performance Toward Proficiency in Languages (AAPPL)

Does text entry method make a difference on Chinese writing test scores?

Contact Info

Product

Resources

About