A content analysis of research published in the Journal of Research in Science Teaching from 1985 through 1989

Abstract

To determine if actual practice was consistent with commonly recommended research methods and procedures, this study examined 130 studies reported over a 5‐year period in three volumes of the Journal of Research in Science Teaching (JRST). The results were consistent with similar previous analyses (Shaver & Norton, 1980a, 1980b; Wallen & Fraenkel, 1988a) and indicate that appropriate generalizations beyond the confines of the reported studies may be impossible for most (64%) of the JRST studies surveyed. The findings also show that replication studies, which could be employed to offset deficiencies in generalizability, were not commonly encountered (3%) in these 130 reports. In addition, the study results indicate that many researchers (48%) do not properly restrict their conclusions based on the limits imposed by the accessible populations and samples used; nor do they typically provide possible alternative explanations for the outcomes obtained (76%). These findings prompt the following recommendations: 1. A greater awareness and use of replication as a check on generalizability should be encouraged by the science education community. 2. Clearly defined populations (target and accessible) and fully described samples warrant increased attention as report components from authors, reviewers, and editorial board members of JRST. 3. In light of the difficulties inherent in effecting random selection in educational settings, a greater emphasis should be placed on recognizing the limits that the underlying assumptions of inferential statistics place on research conclusions. The results of this study indicate that the methodological quality of published science education research should remain a concern for both practitioners and readers.

Exploring Authenticity Through an Engineering-Based Context in a Project-Based Learning Mathematics Activity

“…Inherent in all classroom-based studies is a certain amount of variability in teaching practices between treatment groups. Differences in subject characteristics is an internal validity concern to all studies, and nearly half of studies do not address it properly (Horton et al, 1993). However, given that the Teacher of Record was the same for each class in this instance, the threat was controlled for as best as possible.…”

“…The Hawthorne effect is a concern for all field experiments, present in 48% (Horton et al, 1993); it can be defined as ''the problem in field experiments that subjects' knowledge that they are in an experiment modifies their behavior from what it would have been without the knowledge'' (Adair, 1984, p. 334). When applied to teaching situations like those in this study, the Hawthorne effect can have positive implications.…”

Replication in criminology: A necessary practice

“…In addition, the publication rate of replications in marketing journals had declined since the 1970s. According to Horton and colleagues (1993), of the 130 studies published in a five-year period in the Journal of Research in Science Teaching , only 3 percent were intended by the author to be a direct replication of an earlier study. Finally, Kelly’s examination of the three top journals in behavioral ecology (2006) found no true replications, with most studies categorized as quasi-replications.…”

Analysis of South African graduate degrees in science education: 1930-2000

“…), and particular languages (e.g., French [Martinand & Giordan, 1989]), but no reviews of this kind are based specifically on higher degrees awarded in science education. While summaries and analytical reviews highlighting trends in the content of research studies in science education have, for example, been conducted on particular research domains (e.g., students' conceptions [Duit, 1993]), on research reported in particular journals (e.g., [White;cited in Rennie, Fraser, & Treagust, 1999]), and on the methods and procedures employed in research studies (e.g., Horton et al, 1993), these analyses have not focused on research conducted for higher degrees. Summaries of research published in particular years (e.g., Lederman, Gess-Newsome, & Zeidler, 1993) have referenced graduate degrees (and then only doctorates) but not summarized or reviewed them.…”