Validity and topic generality of a writing performance test

Abstract

The goal of the current study was to examine the validity and topic generality of a writing performance test designed to place international students into appropriate ESL courses at a large mid-western university. Because for each test administration the test randomly rotates three academic topics integrated with listening and reading sources, it is necessary to investigate the extent to which the three topics are compatible in terms of difficulty and generality across a diverse group of examinees. ESL Placement Test (EPT) scores from more than 1,000 examinees were modeled using multinomial logistic regression. Possible explanatory variables were identified as the assigned writing topic, students' majors, and their scores on the Test of English as a Foreign Language (TOEFL). Results indicate that after controlling for general English proficiency as measured by the TOEFL, students' majors were not related to their writing performance; however, the different topics did affect performance. In light of test validity, the demonstrated topic effect argues against the comparability of the three topics. Nevertheless, the absence of effect from the interaction of essay topic and writers' majors supports the generality of each topic for examinees from a wide range of disciplinary areas. Study limitations and future research suggestions are discussed.

Topic and background knowledge effects on performance in speaking assessment

“…(a) a strong reliance on assumption-based indicators of BK as a predictor of task difficulty and on the basis of group level factors such as academic major (Bei, 2010;Lee & Anderson, 2007), gender (Lumley & O'Sullivan, 2005), choice of topic (Jennings et al, 1999;Tedick, 1990), religion (Markham & Latham, 1987) and cultural background (He & Shi, 2012) rather than measuring BK at the level of the individual. These group-level factors may be weak proxies for BK; for example, the use of academic major, particularly for pre-entry candidates, may simply reflect interest in a field of study.…”

“…Moreover, being from a specific major, cultural background or religion does not necessarily preclude having knowledge about other majors, cultures and religions. Such assumption-based approaches to estimating task difficulty have been problematized by Bachman (2002) who views difficulty as an 'artifact' of test performance and emphasizes the need to clearly delineate between (i) task-specific features (e.g., topic) which do not require making assumptions about test takers, (ii) test taker characteristics (e.g., BK of topic), (iii) interactions between (i) and (ii) and to subsequently conceptualize and model interactions as such; (b) study designs that are likely to yield significant results through the use of limited number (often two) of clearly oppositional topics (e.g., He & Shi, 2012;Tedick, 1990); (c) differences in the way in which language proficiency (as a mediating factor) has been measured in various studies from classroom groupings (Schmidt-Rinehart, 1994) to standardized English tests (Lee & Anderson, 2007); (d) the complexity of performance assessment contexts in which additional variance can be introduced by a number of different factors including the well-documented rater effect (McNamara, 1996).…”

Essay Topic Writability Examined through a Statistical Approach from the College Writer’s Perspective

“…In Wang's words, essay topics perceived easier are chosen more frequently by student writers (1999). A recent study confirms from another perspective that topics generated from specific disciplines are perceived to be more difficult by students of other majors (Lee and Anderson, 2007;also cf. Graham et al, 2006).…”

“…For the minority, an essay topic is easier if they have the interest or the necessary data or experience. That prior knowledge makes one essay topic easier for college students than another finds theoretical support in various studies which discover that a specific topic assigned in a test or examination may affect the examinee's writing performance due to various factors including subject knowledge (e.g., Lee & Anderson, 2007). The same study by Lee and Anderson points out that the matter of topic content of a writing test becomes even more problematic and important when measuring graduate students as the EPT [English placement test] does, because these students study in a variety of subject areas, with deeper knowledge in their own major (2007).…”

“…It is a well-known fact that other than writing ability, factors such as knowledge, information, and experience may affect the quality of a student's writing. Numerous studies on topic validity, rating schemes and frameworks, effects of topic interest and prior knowledge on second/foreign-language reading and listening and writing have been conducted (e.g., Lee & Anderson, 2007;Harris et al, 2006;Wing et al, 2006;Cumming et al, 2002;Mendonca & Johnson, 1994;Reaves et al, 1993;Baldwin et al, 1985). However, no published studies have sufficiently focused on essay topic writability examined from the perspective of college writers of the new millennium including both native English speakers and ESL speakers.…”