Why have Educational Evaluators Chosen Not to Do Randomized Experiments?

Cook, Thomas D.

doi:10.1177/0002716203254764

Cited by 64 publications

(54 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Perhaps practice may not be interested in philosophy of science; however, recommendations of a program without specified conditions may lead to disappointment when a model program is re-implemented without success. Specification of relevant conditions is essential in practice and its lack may be one reason why scholars argue against randomized experiments (Cook 2003).…”

Section: Replication In Developmental Preventionmentioning

confidence: 99%

Evidence comes by replication, but needs differentiation: the reproducibility issue in science and its relevance for criminology

Lösel

2017

J Exp Criminol

View full text Add to dashboard Cite

Recent publications in Nature, Science, and other journals raised concerns about the reproducibility of empirical findings in psychology and other scientific disciplines. This article summarizes some of these arguments and results that led to discussions about a Breplication crisis^in research. In criminology, there is not yet a similar discussion, although the need for more replications has been emphasized in the past. The present article addresses this topic with special consideration of program evaluations in early developmental crime prevention and offender treatment. In both fields, there has been substantial progress in research and practice. Most systematic reviews showed mean positive effects; however, nearly all of them demonstrated very heterogeneous findings that could not be attributed to the content of programs. This does not allow simple recommendations of Bwhat works^for policy-making and practice. In addition, there is a serious lack of long-term follow-ups and independent evaluations. The article shows remarkable similarity of the findings and problems in both fields of intervention. Problems of reproducibility prove to be highly relevant for criminology, although there is no need for using the term Bcrisis^. The article proposes various strategies that can enhance the reproducibility of findings, i.e., more systematic investigation of those differentiated conditions under which interventions are most effective. An integrative model of relevant characteristics is briefly presented. It refers to factors of the programs, contexts, participants, and evaluation methods. Confirmatory meta-analyses can play an important role on the path toward more differentiated and replicated knowledge J Exp Criminol

show abstract

Section: Replication In Developmental Preventionmentioning

confidence: 99%

Evidence comes by replication, but needs differentiation: the reproducibility issue in science and its relevance for criminology

Lösel

2017

J Exp Criminol

View full text Add to dashboard Cite

show abstract

“…24 An analysis of the reasons for not adopting the RCT design concludes that, despite serious practical objections and partial remedies, RCTs are logically and empirically superior to all currently known alternatives. 25 The view that the RCT is inappropriate to test the success of policy interventions is refuted by a bibliometric analysis, which concludes that between 6% and 15% of impact evaluations of childhood interventions in education and justice employ a randomised design. 26 Our own experience of conducting RCTs supports their use for evaluating social interventions.…”

Section: Evaluating Public Policy Interventionsmentioning

confidence: 99%

“…Similarly, divergent views about appropriate methods for evaluating interventions are found in the areas of social welfare 44 and education 25 where The Centre for Evidence-Based Social Services (www.ex.ac.uk/cebss/introduction.html) and the Evidence-based (www.cemcentre.org/ebeuk/) Education Network UK stand out from many of their British professional colleagues in social welfare and education, respectively, as advocates for randomised evaluation.…”

Section: Chaptermentioning

confidence: 99%

Randomised controlled trials for policy interventions: a review of reviews and meta-regression

Oliver

Bagnall²,

Thomas³

et al. 2010

Health Technol Assess

View full text Add to dashboard Cite

How to obtain copies of this and other HTA programme reports An electronic version of this title, in Adobe Acrobat format, is available for downloading free of charge for personal use from the HTA website (www.hta.ac.uk). A fully searchable DVD is also available (see below).Printed copies of HTA journal series issues cost £20 each (post and packing free in the UK) to both public and private sector purchasers from our despatch agents.Non-UK purchasers will have to pay a small fee for post and packing. For European countries the cost is £2 per issue and for the rest of the world £3 per issue. How to order:-fax (with credit card details) -post (with credit card details or cheque) -phone during office hours (credit card only).Additionally the HTA website allows you to either print out your order or download a blank order form. Contact details are as follows:Synergie UK (HTA Department) Digital House, The Loddon Centre Wade Road Basingstoke Hants RG24 8QW Email: orders@hta.ac.uk Tel: 0845 812 4000 -ask for 'HTA Payment Services' (out-of-hours answer-phone service) Fax: 0845 812 4001 -put 'HTA Order' on the fax header Payment methods Paying by chequeIf you pay by cheque, the cheque must be in pounds sterling, made payable to University of Southampton and drawn on a bank with a UK address. Paying by credit cardYou can order using your credit card by phone, fax or post. SubscriptionsNHS libraries can subscribe free of charge. Public libraries can subscribe at a reduced cost of £100 for each volume (normally comprising 40-50 titles). The commercial subscription rate is £400 per volume (addresses within the UK) and £600 per volume (addresses outside the UK). Please see our website for details. Subscriptions can be purchased only for the current or forthcoming volume. How do I get a copy of HTA on DVD?Please use the form on the HTA website (www.hta.ac.uk/htacd/index.shtml). HTA on DVD is currently free of charge worldwide.The website also provides information about the HTA programme and lists the membership of the various committees. HTA NIHR Health Technology Assessment programmeT he Health Technology Assessment (HTA) programme, part of the National Institute for Health Research (NIHR), was set up in 1993. It produces high-quality research information on the effectiveness, costs and broader impact of health technologies for those who use, manage and provide care in the NHS. 'Health technologies' are broadly defined as all interventions used to promote health, prevent and treat disease, and improve rehabilitation and long-term care. The research findings from the HTA programme directly influence decision-making bodies such as the National Institute for Health and Clinical Excellence (NICE) and the National Screening Committee (NSC). HTA findings also help to improve the quality of clinical practice in the NHS indirectly in that they form a key component of the 'National Knowledge Service'. The HTA programme is needs led in that it fills gaps in the evidence needed by the NHS. There are three routes to the start of project...

show abstract

“…Even if we acknowledge that randomised controlled trials (RCTs) are not always appropriate or possible, it is almost certainly the case that they could be used more often than they are (Torgerson and Torgerson, 2007;Cook, 2003;Slavin, 2008). Furthermore, there are non-randomised designs that are considerably stronger than the ones used in the examples here: for example, regression-discontinuity and time-series designs (Shadish et al, 2002;Cook et al, 2008).…”

Section: Recommendations For Researchmentioning

confidence: 99%

Unobserved but not unimportant: the effects of unmeasured variables on causal attributions

Coe

2009

Effective Education

View full text Add to dashboard Cite

This is an electronic version of an article published in Coe, R. (2009) 'Unobserved but not unimportant : the e ects of unmeasured variables on causal attributions.', E ective education., 1 (2). pp. 101-122. E ective education is available online at: http://www.informaworld.com/smpp/content db=all?content=10. 1080/19415530903522519 Additional information: Use policyThe full-text may be used and/or reproduced, and given to third parties in any format or medium, without prior permission or charge, for personal research or study, educational, or not-for-pro t purposes provided that:• a full bibliographic reference is made to the original source • a link is made to the metadata record in DRO • the full-text is not changed in any way The full-text must not be sold in any format or medium without the formal permission of the copyright holders.Please consult the full DRO policy for further details. Peer-review status:Peer-reviewed Publication status:Accepted for publication version Citation for published item:Coe, R. (2009) Use policyThe full-text may be used and/or reproduced, and given to third parties in any format or medium, without prior permission or charge, for personal research or study, educational, or not-for-profit purposes provided that :  a full bibliographic reference is made to the original source  a link is made to the metadata record in DRO  the full-text is not changed in any wayThe full-text must not be sold in any format or medium without the formal permission of the copyright holders.Please consult the full DRO policy for further details. Objective:To estimate how much difference the inclusion of plausibly important but unmeasured variables could make to estimates of the effects of educational programmes. Methods:Two examples of policy-relevant research in education were identified. A sensitivity analysis using Monte Carlo simulation was conducted to estimate the size of a possible spurious ‗effect' that could actually be entirely due to the failure to incorporate a plausible unobserved variable. Results:In all the examples the effect size reported in the original study was within the range of possible spurious effects. Conclusions: What appeared to the original researchers to be substantial and unequivocal causal effects were reduced to tiny and uncertain differences when the effects of plausible unobserved differences were taken into account. Evaluators who rely on statistical control should be more cautious in making causal claims, consider possible effects of unmeasured variables and conduct sensitivity analyses. Alternatively, stronger designs should be used.

show abstract

Why have Educational Evaluators Chosen Not to Do Randomized Experiments?

Cited by 64 publications

References 66 publications

Evidence comes by replication, but needs differentiation: the reproducibility issue in science and its relevance for criminology

Evidence comes by replication, but needs differentiation: the reproducibility issue in science and its relevance for criminology

Randomised controlled trials for policy interventions: a review of reviews and meta-regression

Unobserved but not unimportant: the effects of unmeasured variables on causal attributions

Contact Info

Product

Resources

About