Predicting Variations in Math Performancein Four Countries Using TIMSS

Abstract

Although international comparisons of average student performance are a staple of U.S. educational debate, little attention has been paid to cross-national differences in the variability of performance. It is often assumed that the performance of U.S. students is unusually variable or that the distribution of U.S. scores is left-skewed – that is, that it has an unusually long ‘tail' of low-scoring students – but data from international studies are rarely brought to bear on these questions. This study used data from the Third International Mathematics and Science Study (TIMSS) to compare the variability of performance in the U.S., Australia, France, Germany, Hong Kong, Korea, and Japan; investigate how this performance variation is distributed within and between classrooms; and explore how well background variables predict performance at both levels. TIMSS shows that the U.S. is not anomalous in terms of the amount, distribution, or prediction of performance variation. Nonetheless, some striking differences appear between countries that are potentially important for both research and policy. In the U.S., Germany, Hong Kong, and Australia, between 42 and 47 percent of score variance was between classrooms. At the other extreme, Japan and Korea both had less than 10 percent of score variance between classrooms. Two-level models (student and classroom) were used to explore the prediction of performance by social background variables in four of these countries (the U.S., Hong Kong, France, and Korea). The final models included only a few variables; TIMSS lacked some important background variables, such as income, and other variables were dropped either because of problems revealed by exploratory data analysis or because of a lack of significance in the models. In all four countries, these sparse models predicted most of the between-classroom score variance (from 59 to 94 percent) but very little of the within-classroom variance. Korea was the only country in which the models predicted more than 5 percent of the within-classroom variance in scores. In the U.S. and Hong Kong, the models predicted about one-third of the total score variance, and almost all of this prediction was attributable to between-classroom differences in background variables. In Korea, only 19 percent of total score variance was predicted by the model, and most of this most of this was attributable to within-classroom variables. Thus, in some instances, countries differ more in terms of the structure and prediction of performance variance than in the simple amount of variance. TIMSS does not provide a clear explanation of these differences, but this paper suggests hypotheses that warrant further investigation.

A Study of the Effectiveness of the Louisiana Algebra I Online Course

“…The types of interactions among the students in the treatment classrooms appear to contradict the commonly held concern that students in K-12 online classes will not remain on task. As studies in higher education have shown, online programs can often isolate students during the learning process (e.g., Bernard et al 2004;Lally & Barrett, 1999). While the Louisiana online Algebra I program is unusual in that it provides students with the benefit of a regular class schedule where they interact with their peers as well as an online teacher, other emerging K-12 online models may not provide such opportunities for important socio-academic interactions.…”

“…There is currently a lack of research on the mediating or moderating effect of student satisfaction and confidence levels on learning outcomes in K-12 settings. Given the importance of student attitudes and beliefs for moderating learning outcomes in traditional, face-to-face learning environments (Koretz, McCaffrey, & Sullivan, 2001;o'Dwyer, 2005), future research into the effectiveness of online or distance learning initiatives in K-12 settings should pay particular attention to student satisfaction and other affective measures.…”

“…Looking to the research that has been conducted in online learning environments in higher education settings, several studies have found that online learning programs provide students and teachers with quite different learning and teaching environments than those provided in traditional, face-to-face settings (Bernard et al, 2004;Peters, 2003;Stodel, Thompson, & MacDonald, 2006). The distinguishing characteristics include class structure (Paloff & Pratt, 2001), the level and types of peer-to-peer and student-teacher interactions that take place (Bernard et al 2004;Lally & Barrett, 1999), and the lack of "social presence" during the learning process (Aragon, 2003;Bibeau, 2001;Garrison & Cleveland-Innes, 2005). While many studies of higher education learning outcomes have found no statistical differences in achievement for online courses compared to traditional, face-to-face courses, there remains concern that the quality and depth of learning may be affected in online courses (Allen, Bourhis, Burrell, & Mabry, 2002;Lockyer, Patterson, & harper, 2001;Neuhauser, 2002;Thirunarayanan & Peres-Prado, 2001-2002.…”

“…The distinguishing characteristics include class structure (Paloff & Pratt, 2001), the level and types of peer-to-peer and student-teacher interactions that take place (Bernard et al 2004;Lally & Barrett, 1999), and the lack of "social presence" during the learning process (Aragon, 2003;Bibeau, 2001;Garrison & Cleveland-Innes, 2005). While many studies of higher education learning outcomes have found no statistical differences in achievement for online courses compared to traditional, face-to-face courses, there remains concern that the quality and depth of learning may be affected in online courses (Allen, Bourhis, Burrell, & Mabry, 2002;Lockyer, Patterson, & harper, 2001;Neuhauser, 2002;Thirunarayanan & Peres-Prado, 2001-2002. Mediating variables that may promote or hinder learning success in an online environment have been isolated in several studies and these include students' attitudes towards the online environment (Sherry, fulford, & Zhang, 1998) and their beliefs about the efficacy of online learning (Cooper, 2001;Allen et al, 2002;Muilenburg & Berge, 2001;Navarro & Shoemaker, 2000;Smith & Dillon, 1999).…”

Evaluating the risk of nonresponse bias in educational large-scale assessments with school nonresponse questionnaires: a theoretical study

“…There is extensive evidence on the literature that the main outcome variables in IEA assessments (usually achievement scores in specific subject domains) are highly associated with background characteristics of the participants (Caldas and Bankston 1997;Fuller 1987;Grace and Thompson 2003), suggesting that school context explains an important portion of the variability of student achievement scores (e.g., Koretz et al 2001;Lamb and Fullarton 2001;Baker et al 2002;Mullis et al 2012a, b).…”