No abstract

No abstract

A fundamental issue in validating topic-based tests of language proficiency is the effect of the topic on the test takers’ performance. Topic-based test developers must ensure that test takers are neither advantaged nor disadvantaged in terms of their test results when presented with a given test topic. We have termed this threat a ‘topic effect’ and argue that this topic effect may constitute a source of construct-irrelevant variance (Messick, 1989). We contend that investigating the possibility of a topic effect is a critical step in establishing the validity of all topic-based tests. This research investigates the potential presence of a topic effect for the Canadian Academic English Language (CAEL) Assessment using the mechanism of choice. The principal research question is to determine if test-takers given a choice of topic perform significantly differently than test-takers not given a choice. ESL university applicants ( n = 254) were randomly assigned to one of two conditions: no choice of topic or choice among five topics. Overall Proficiency Level, Reading, Lecture and Essay scores were compared for the two conditions. Ordinal level data were analysed using the Mann Whitney U, Chi-Square and Kolmogorov-Smirnov tests. While the scores for the choice groups were usually higher than the scores for the no-choice groups, the differences were not statistically significant. Despite the lack of significance, we felt that the scores warranted closer examination. For the topic where the difference between the choice and no-choice groups was largest, a textual analysis of the essays was undertaken to look for instances of the use of information not provided in the test. Again, no difference was found between the choice and no-choice groups. The results provide support for the validity of inferences drawn from this test. Because choice is an essential element of the research design, a second focus of the study is to explore the advantages and disadvantages of the use of choice in language testing settings from the perspective of both the tester and the test-taker. The potential value of choice as a testing feature is discussed and a call for further research is made.

No abstract

No abstract