Elicited Speech From Graph Items on the Test of Spoken English™

Abstract

This research applied a cognitive model to identify item features that lead to irrelevant variance on the Test of Spoken English™ (TSE ® ). The TSE is an assessment of English oral proficiency and includes an item that elicits a description of a statistical graph. This item type sometimes appears to tap graph-reading skills-an irrelevant construct; TSE raters report that many examinees perform worse on this item type than they do on the other 11 items in the test. We adapted a cognitive theory of graph comprehension to predict the degree to which TSE graph items tap irrelevant skills such as graph reading. Through analyses of existing TSE data as well as an experiment, we show how the theory provides specific, empirically justified recommendations on the construction of graph items that minimize the influence of extraneous skills. ETS administers the TOEFL program under the general direction of a policy board that was established by, and is affiliated with, the sponsoring organizations. Members of the TOEFL Board (previously the Policy Council) represent the College Board, the GRE Board, and such institutions and agencies as graduate schools of business, junior and community colleges, nonprofit educational exchange agencies, and agencies of the United States government.A continuing program of research related to the TOEFL test is carried out under the direction of the TOEFL Committee of Examiners. Its 12 members include representatives of the TOEFL Board and distinguished English as a second language specialists from the academic community. The Committee meets twice yearly to review and approve proposals for test-related research and to set guidelines for the entire scope of the TOEFL research program. Members of the Committee of Examiners serve four-year terms at the invitation of the Board; the chair of the committee serves on the Board.Because the studies are specific to the TOEFL test and the testing program, most of the actual research is conducted by ETS staff rather than by outside researchers. Many projects require the cooperation of other institutions, however, particularly those with programs in the teaching of English as a foreign or second language and applied linguistics. Representatives of such programs who are interested in participating in or conducting TOEFL-related research are invited to contact the TOEFL program office. All TOEFL research projects must undergo appropriate ETS review to ascertain that data confidentiality will be protected. Current (2003Current ( -2004 members of the TOEFL Committee of Examiners are:

Factor analysis for fairness: Examining the impact of task type and examinee L1 background on scores of an ITA speaking test

“…Although the OEPT graph items contain statistical information, findings from the factorial invariance tests across academic disciplines do not suggest that those items measure statistical knowledge or graphic familiarity in addition to general speaking skills. At first glance, these findings may seem to contradict previous research which suggested that graph items elicit different kinds of language use (e.g., Katz et al, 2004) and that examinees more familiar with graphs perform better on those items (Xi, 2005(Xi, , 2010b. However, this contradiction might have resulted from three quality management procedures for the OEPT.…”

“…The speculation that graph items are more prone to test bias results from the assumption that prior topic or content knowledge related to the task will facilitate task performance (Skehan, 1998;Robinson, 2001), thereby posing threats to test fairness and validity. Katz, Xi, Kim and Cheng (2004) found that graph complexity affects speech production, and Xi (2005Xi ( , 2010b reported that graph complexity and familiarity affect speaking performance. Overall, these studies suggest that graph familiarity and complexity might be potential sources of test bias and that the comparability of different types of integrated speaking tasks cannot be assumed.…”

A comparative study of composing processes in reading- and graph-based writing tasks

“…Clearly, graph familiarity and comprehension ability play roles in graph-based writing performance. As indicated in a number of studies on graph-based assessment tasks (Katz, Xi, Kim, & Cheng, 2004;Xi, 2005), caution must be taken due to the potential influence of graph familiarity on writers' processes and performance.…”

“…Moreover, most of these studies have been conducted in the contexts of listening (Ginther, 2002;Gruba, 1997) and speaking assessments (Katz, Xi, Kim, & Cheng, 2004;Xi, 2005Xi, , 2010. Very little research, apart from internal IELTS validation reports (Bridges, 2010;Mickan, Slater, & Gibson, 2000), has addressed these processes involved in writing assessments.…”

Graph Structure Supports Graph Description

“…However, many graphs evidenced no such difficulties. Analyses of data from the past two years of TSE administrations confirm that graph description questions are more likely than questions with non-graph prompts to elicit such discrepant performance (Katz, Xi, Kim, & Cheng, 2002).…”