Comparison of a Randomized and Two Quasi-Experimental Designs in a Single Outcome Evaluation

Aiken, Leona S.; West, Stephen G.; Schwalm, David E.; Carroll, James L.; Hsiung, Shenghwa

doi:10.1177/0193841x9802200203

Cited by 84 publications

(85 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In analogy, two also significantly more improved dimensions in our study that have to do with bullying prevention-namely control of verbal and physical aggression and control of victimization, turned out to constitute equally widely pursued and met targets by other researchers (Aiken et al, 1998;Catalano et al, 2004;McMahon et al, 2000;Powell & Dunlap, 2009), indicating that socio-emotional enhancement school approaches can make a significant difference regarding violence and aggression.…”

Section: Discussionsupporting

confidence: 65%

Evaluation of the “Steps for Life” Personal and Social Skills Greek K-Curriculum

Kourmousi¹,

Salagianni²,

Merakou³

et al. 2017

View full text Add to dashboard Cite

Background: The past few decades have seen a significant growth of research on socio-emotional skills development programs and on mental health education in schools since their key-importance has been well documented. This study presents the evaluation of "Steps for life", an open, question-based, annual, teacher-taught, mental health Greek K-curriculum, designed to improve personal and social skills in young students aged 4 -6. The curriculum uses classroom-appropriate methods such as dialoguing, story-telling, role-playing and diffusion in formal and hidden curriculum, while incorporating a considerable degree of parental involvement. Methods: The study was conducted in Greece during [2009][2010]. The sample consisted of 998 kindergartners, with 518 of them forming the experimental group and 480 the control group. The questionnaire designed for the study consisted of items regarding demographics, of the Behavioural Academic Self-Esteem scale and of 8 more subscales, namely concentration of attention, participation and cooperation in the class, emotions' identification and management, physical and verbal aggressiveness control, victimization control, self-esteem, empathy, friendship skills, and problem solving. Its 96-in total-items examined students' behaviors as exhibited in the school environment and as observed and rated by their teachers. It was administered pre and post intervention to both groups' educators. Results: The results suggest that both groups exhibited significant improvement in all investigated skills. As indicated by the regression analysis however, the intervention students showed significantly higher improvement in all targeted abilities except friendship skills. Conclusions: Our intervention improved more significantly almost all the taught skills, corroborating the findings of other researchers who found that similar approaches can improve

show abstract

Section: Discussionsupporting

confidence: 65%

Evaluation of the “Steps for Life” Personal and Social Skills Greek K-Curriculum

Kourmousi¹,

Salagianni²,

Merakou³

et al. 2017

View full text Add to dashboard Cite

show abstract

“…Critics may contend that they will lead to comparison studies that can only be done in laboratory settings (e.g., Shadish, Clark & Steiner, 2008). However, we have reviewed recent field-study comparisons of this type (Cook et al, 2008) and found a number that took place in field settings and still were able to remedy many of the problems in PMO (e.g., Aiken, West, Schwalm, Carroll & Hsuing, 1998;Diaz & Handa, 2006). It is challenging, of course, to meet all or most of them, and many practical situations do not allow much more than PMO were able to do.…”

Section: Discussionmentioning

confidence: 99%

A Case Study About Why It Can Be Difficult To Test Whether Propensity Score Analysis Works in Field Experiments

Shadish

Steiner

Cook³

2013

JMMSS

View full text Add to dashboard Cite

Peikes, Moreno and Orzol (2008) sensibly caution researchers that propensity score analysis may not lead to valid causal inference in field applications. But at the same time, they made the far stronger claim to have performed an ideal test of whether propensity score matching in quasi-experimental data is capable of approximating the results of a randomized experiment in their dataset, and that this ideal test showed that such matching could not do so. In this article we show that their study does not support that conclusion because it failed to meet a number of basic criteria for an ideal test. By implication, many other purported tests of the effectiveness of propensity score analysis probably also fail to meet these criteria, and are therefore questionable contributions to the literature on the effects of propensity score analysis. Keywords: Propensity Scores, Strong Ignorability, Quasi-Experiments, Within-Study ComparisonIn 2008, Peikes, Moreno and Orzol (henceforth PMO) published a case study in The American Statistician that cautioned social program evaluators that propensity score analysis may yield quite different results than those from a randomized experiment. We join them in that caution because we doubt that many applications of propensity scores meet some of the most basic conditions for their valid use (Shadish, 2012;Steiner, Cook & Shadish, 2011;Shadish, Cook, Steiner & Clark, 2010). In that sense, we applaud the PMO article. Field researchers need to appreciate how difficult it can be to use propensity score analysis in a way that yields confidence in the results.At the same time, however, PMO made a second claim, that they performed an "ideal" (pp. 222, 223, 230) test of whether propensity score matching in quasi-experimental data could approximate the results of a randomized experiment. In fact, a careful analysis of PMO suggests just the opposite conclusion, that they neither implemented an ideal propensity score analysis nor an ideal comparison of results from a propensity score analysis to results from a randomized experiment. In this article, we show why the PMO study was not ideal in both respects. Just as practitioners need to appreciate how difficult it can be to use propensity

show abstract

“…That is, in each study, the control group was in the same locale (students in the same grade level and at the same school) as the treatment group, and possessed similar characteristics on variables that were important to the study. Previous research has found that effects from randomized experiments were identical when a focal-local control group was substituted for a randomized control group (Aiken, West, Schwalm, Carroll, & Hsiung, 1998).…”

Section: Current Studiesmentioning

confidence: 99%

Examining the Efficacy of a Time Management Intervention for High School Students

Burrus

Jackson

Holtzman

et al. 2013

ETS Research Report Series

View full text Add to dashboard Cite

Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. i AbstractThe current paper reports the results of 2 quasiexperimental studies conducted to examine the efficacy of a new time management intervention designed for high school students. In both studies, there was no difference between the treatment and control groups in improvement in self-reported time management skills as a result of the intervention. However, the treatment group reported significantly greater improvement than the control group for secondary outcomes such as stress (Studies 1 and 2), anxiety (Studies 1 and 2), depression (Study 1), and knowledge of time management strategies (Study 1). Additionally, advisor ratings of student time management skills were higher for the treatment than for the control group in Study 2.Implications and suggestions for improving the intervention are discussed.

show abstract

Comparison of a Randomized and Two Quasi-Experimental Designs in a Single Outcome Evaluation

Cited by 84 publications

References 24 publications

Evaluation of the “Steps for Life” Personal and Social Skills Greek K-Curriculum

Evaluation of the “Steps for Life” Personal and Social Skills Greek K-Curriculum

A Case Study About Why It Can Be Difficult To Test Whether Propensity Score Analysis Works in Field Experiments

Examining the Efficacy of a Time Management Intervention for High School Students

Contact Info

Product

Resources

About