A Study of Characteristics of the Speak Test

Abstract

The purpose of this project was to investigate issues of reliability and validity arising from the local administration and scoring of the Spoken Proficiency English Assessment Kit (SPEAK®), the “off‐the‐shelf” version of the Test of Spoken English (TSE®), and to assess the validity of the SPEAK test as a predictor of classroom English ability for international teaching assistants (ITAs). The reliability issues addressed included interrater consistency within and across institutions, score comparability across institutions and Educational Testing Service (ETS®), and differential weightings by individual raters, institutions, and TSE raters of the three diagnostic language factors in assessing the overall comprehensibility of spoken English. SPEAK tests were administered to 119 international teaching assistants at three participating institutions, arranged in a counterbalanced design with 12 ETS‐scored tests of the same form, and scored by raters at each institution. High interrater reliability coefficients as measured by Pearson product‐moment correlations were obtained for the global comprehensibility score and the diagnostic subscores; other means of assessing interrater consistency, however, revealed some variation among individual raters. Scores were generally comparable across institutions and ETS in the high‐to‐mid range on the overall comprehensibility scale, although differences among institutions increased in the lower range of scores. Other interesting individual and institutional differences were found regarding correspondence with ETS scores. With regard to the weighting by different institutions of the three component subscales, regression analyses showed essentially no differences and thus indicate that the same construct is being measured by raters at each institution. For the predictive validity phase of the study, the SPEAK scores of the 59 ITAs who held classroom teaching responsibilities were used as predictive variables in multiple regression analyses. Criterion variables were student ratings of the ITA's spoken language use in the classroom and other instructional contexts as well as ratings of the ITA on several dimensions of teaching effectiveness from the Student Instructional Report (SIR), a standardized instructor/course rating instrument. Although squared multiple correlations for predicting student ratings of ITAs' spoken English proficiency from SPEAK scores were highly significant for nearly all of the student ratings, regression coefficients for the separate scales of SPEAK were consistently not significant due to the high multicollinearity of those scales, suggesting some redundancy in the information provided by the separate scales. SPEAK scales were also correlated with the more global aspects of teaching performance, as measured by student responses to relevant questions and question groupings on the SIR. These relationships were generally weak and not statistically significant. The SPEAK scales were thus effective predictors of student ratings of the classroom language...

Using Just Noticeable Differences to Interpret Test of Spoken English Scores

“…5 For standard TAs used in analyses of the equal proportions JND, the mean of the equal proportions over comparison TAs with various TSE scores ranged from .34 to .74, with a median of .66. 6 The .38 correlation in the present study was significantly different (Z [121] = 2.35, p < .05, two-tail) from the .68 correlation in the Clark and Swinton (1980) study but was not significantly different (Z [120] = .37, p > .05) from the .46 correlation in the Sarwark et al (1995) study. The correlations in these previous studies were not significantly different (Z [113] = 1.76, p > .05.…”

“…The procedures were modeled after those used in earlier studies (Clark & Swinton, 1980;Sarwark et al, 1995). A previously administered form of the current TSE test (used in July 1996) was administered with standard TSE procedures to the TAs by local study coordinators or their assistants, ESL staff responsible for screening international TAs at their institutions for English proficiency, usually with the TSE test or SPEAK.…”

“…The course evaluation form, modeled after the Student Instructional Report (Student Instructional Report, 1993) and supplementary ratings used in previous studies of the TSE's validity for TAs (Clark & Swinton, 1980;Sarwark et al, 1995), consisted of four ratings of the TA and his or her course: value of the course, value of the course's laboratory or recitation/discussion section, quality of the TA's teaching, and the TA's ability to communicate in English (all ratings used a four-point scale, Excellent = 4 to Poor = 1) and a question about whether the student's native language was English. (A copy of the form appears in the Appendix.)…”

“…This approach can readily be extended to the situation where different panels rate different individuals (e.g., college students rating their teaching assistants, TAs) by assuming that the panels are comparable. Accordingly, the main purpose of the present study was to explore the value of obtaining a JND for the current TSE test, using college students' ratings of their TAs' English proficiency, given the manifest relevance of these assessments to the test and the well-established relationship between scores on the oriJinal version of the TSE (or its Speaking Proficiency English Assessment Kit, SPEAK, counterpart) and such ratings in previous studies (Clark & Swinton, 1980;Sarwark, Smith, MacCallum, & Cascallar, 1995). Specifically, the aims were to determine the size of the JND, the correspondence between different estimates of it, and their associations with the TSE scores in a homogeneous set of departments and universities.…”

Mitigating U.S. Undergraduates’ Attitudes Toward International Teaching Assistants

“…The instructional competence measure included nine semantic differential scales. It was adopted from an earlier scale of ITA proficiency and instructional quality (Sarwark, Smith, MacCallum, & Cascallar, 1995). For example, one of the items was effective teacher-ineffective teacher.…”