Intraclass Correlation Coefficients for Designing Cluster-Randomized Trials in Sub-Saharan Africa Education

Kelcey, Ben; Shen, Zuchao; Spybrook, Jessaca

doi:10.1177/0193841x16660246

Cited by 28 publications

(20 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our study, the average value of lay around .31, whereas in the U.S. does not often exceed .25 (e.g., Bloom et al, 2007;Hedges & Hedberg, 2013;Spybrook, Westine, et al, 2016). This difference between schools in Germany and the U.S. corroborates the results of international studies pointing to a significant variation of across countries (Brunner et al, 2017;Kelcey et al, 2016;Zopluoglu, 2012). Looking at different grade levels, however, yields a more differentiated picture.…”

Section: Generalizability Of Design Parameters Across Countriessupporting

confidence: 86%

See 1 more Smart Citation

Multilevel Design Parameters to Plan Cluster-Randomized Intervention Studies on Student Achievement in Elementary and Secondary School

Stallasch¹,

Lüdtke²,

Artelt³

et al. 2020

Preprint

View full text Add to dashboard Cite

To plan cluster-randomized trials with sufficient statistical power to detect intervention effects on student achievement, researchers need multilevel design parameters, including between-classroom and between-school differences and the amounts of variance explained by covariates at the student, classroom, and school level. Previous research has mostly been conducted in the U.S., focused on two-level designs, and limited to main achievement domains (i.e., mathematics, science, reading). Using representative data of students attending grades 1 to 12 from three German longitudinal large-scale assessments (3,963 ≤ N ≤ 14,640), we used three- and two-level latent (covariate) models to provide design parameters and corresponding standard errors for a broad array of domain-specific (e.g., mathematics, science, verbal skills) and domain-general achievement outcomes (e.g., basic cognitive functions). Three covariate sets were applied comprising (a) pretest scores, (b) sociodemographic characteristics, and (c) their combination. Design parameters varied considerably as a function of the hierarchical level, achievement outcome, and grade level. Our findings demonstrate the need to strive for an optimal fit between design parameters and target research context. The application of design parameters in power analyses is illustrated.

show abstract

Section: Generalizability Of Design Parameters Across Countriessupporting

confidence: 86%

“…The corresponding average values of typically lie around .10. Importantly, this general pattern of results was also found for sub-Saharan countries in the study by Kelcey et al (2016) as well as in the analyses of Brunner et al (2017) for 81 countries participating in PISA.…”

Section: International Researchsupporting

confidence: 74%

Multilevel Design Parameters to Plan Cluster-Randomized Intervention Studies on Student Achievement in Elementary and Secondary School

Stallasch¹,

Lüdtke²,

Artelt³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Depending on the experiment, we recognize that it is not always possible to single-house mice. Our review showed that scientists often analyze clustered observations using methods that mathematically function under the assumption of data independence (student T-, Mann-Whitney, One-/Two-way ANOVAs), without implementing statistics for intra-class ('intra-cage') correlated (ICC) cage-clustered data (Multivariable linear/logistic, Marginal, Generalized Estimating Equations, or Mixed Random/Fixed Regressions) 50,84,85 . The ICC describes how units in a cluster resemble one another, and can be interpreted as the fraction of the total variance due to variation between clusters 50 .…”

Section: Action Theme 4 On 'Repeating Experiments In Different Seasons'mentioning

confidence: 99%

Artificial microbiome heterogeneity spurs six practical action themes and examples to increase study power-driven reproducibility

Basson

LaSalla

Lam

et al. 2020

Sci Rep

View full text Add to dashboard Cite

With >70,000 yearly publications using mouse data, mouse models represent the best engrained research system to address numerous biological questions across all fields of science. Concerns of poor study and microbiome reproducibility also abound in the literature. Despite the well-known, negativeeffects of data clustering on interpretation and study power, it is unclear why scientists often house >4 mice/cage during experiments, instead of ≤2. We hypothesized that this high animal-cage-density practice abounds in published literature because more mice/cage could be perceived as a strategy to reduce housing costs. Among other sources of 'artificial' confounding, including cyclical oscillations of the 'dirty-cage/excrement microbiome', we ranked by priority the heterogeneity of modern husbandry practices/perceptions across three professional organizations that we surveyed in the USA. Data integration (scoping-reviews, professional-surveys, expert-opinion, and 'implementability-scorestatistics') identified Six-Actionable Recommendation Themes (SART) as a framework to re-launch emerging protocols and intuitive statistical strategies to use/increase study power. 'Cost-vs-science' discordance was a major aspect explaining heterogeneity, and scientists' reluctance to change. With a 'housing-density cost-calculator-simulator' and fully-annotated statistical examples/code, this themedframework streamlines the rapid analysis of cage-clustered-data and promotes the use of 'study-powerstatistics' to self-monitor the success/reproducibility of basic and translational research. Examples are provided to help scientists document analysis for study power-based sample size estimations using preclinical mouse data to support translational clinical trials, as requested in NIH/similar grants or publications.According to a U.S. National Science Foundation subcommittee on science replicability, "reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results" 1 . More recently, reproducibility as a scientific concept has been proposed to be divided into three types: methods reproducibility, results reproducibility, and inferential reproducibility. While these terms are applied predominantly to the biomedical field, they are not without utility across other scientific fields, each of which are governed by their own internalized needs and criteria for "proof " 2,3 . With 73,363 PubMed publications using 'mice' in 2018, laboratory mice represent a critical

show abstract

“…Cluster trial methods are widely used among those who believe in closed retroduction, no retroduction, or pure deductivism in research (Kelcey et al 2016;Taft et al 2012 illustrate these). The cluster idea locates groups of treated and untreated people (or cases, e.g.…”

Section: Important Backgroundmentioning

confidence: 99%

Bridging to Action Requires Mixed Methods, Not Only Randomised Control Trials

Olsen

2019

Eur J Dev Res

View full text Add to dashboard Cite

Development evaluation refers to evaluating projects and programmes in development contexts. Some evaluations are too narrow. Narrow within-discipline impact evaluations are weaker than multidisciplinary, mixed-methods evaluations. A twostep process leads toward profoundly better arguments in assessing the impact of a development intervention. The first step is setting out the arena for discussion, including what the various entities are in the social, political, cultural and natural environment surrounding the chosen problem. The second step is that, once this arena has been declared, the project and triangulation of data can be brought to bear upon logical arguments with clear, transparent reasoning leading to a set of conclusions. In this second step, we do need scientific methods such as peer review, data and so on, but, crucially, the impact evaluation process must not rest upon a single data type, such as survey data. It is dangerous and undesirable to have the entire validity of the conclusions resting upon randomised control trials, or even a mixture of data types. Different contributions to knowledge exist within the evaluation process, including the interaction of people during action research, ethnography, case-study methods, process tracing and qualitative methods. The cement holding my argument together is that multiple logics are used (retroductive, deductive, and inductive, in particular). Deductive mathematics should not dominate the evaluation of an intervention, as randomised controlled trials on their own lend themselves to worrying fallacies about causality. I show this using Boolean fuzzy set logic. An indicator of high-quality development evaluation is the use of multiple logics in a transparent way. Keywords Evaluation • Randomised control trials • Comparative case-study research • Methodology • Retroduction • Mixed-methods • Impact evaluation Resume L'évaluation du développement se réfère à l'évaluation de projets et programmes dans le contexte du développement. Certaines évaluations sont trop restreintes. Les évaluations étroites, au sein de la même discipline, sont plus faibles que les évalua-Extended author information available on the last page of the article terventions en matière de développement, un procédé a deux étapes nous aide à bâtir des argumentations beaucoup plus fortes. La première étape est d'établir l'espace de la discussion, y compris les organismes sociales, politiques, culturales et naturels qui encerclent le problème choisi. La deuxième étape, une fois cet espace est défini, est d'utiliser le projet et la triangulation des données de forme logique, produisant un raisonnement transparent et clair qui nous amène à des conclusions. Cette deuxième étape nécessite de méthodologies scientifiques tels que la révision par des pairs, l'utilisation des données, etc. C'est crucial que le processus d'évaluation des impacts ne s'appuie pas que sur un seul type de données, comme par exemple les données d'enquête. Que l'ensemble de la validité des conclusions soit basé que sur des e...

show abstract

Intraclass Correlation Coefficients for Designing Cluster-Randomized Trials in Sub-Saharan Africa Education

Cited by 28 publications

References 26 publications

Multilevel Design Parameters to Plan Cluster-Randomized Intervention Studies on Student Achievement in Elementary and Secondary School

Multilevel Design Parameters to Plan Cluster-Randomized Intervention Studies on Student Achievement in Elementary and Secondary School

Artificial microbiome heterogeneity spurs six practical action themes and examples to increase study power-driven reproducibility

Bridging to Action Requires Mixed Methods, Not Only Randomised Control Trials

Contact Info

Product

Resources

About