A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

Abstract

Null hypothesis significance testing has dominated quantitative research in education and psychology. However, the statistical significance of a test as indicated by a p-value does not speak to the practical significance of the study. Thus, reporting effect size to supplement p-value is highly recommended by scholars, journal editors, and academic associations. As a measure of practical significance, effect size quantifies the size of mean differences or strength of associations and directly answers the research questions. Furthermore, a comparison of effect sizes across studies facilitates meta-analytic assessment of the effect size and accumulation of knowledge. In the current comprehensive review, we investigated the most recent effect size reporting and interpreting practices in 1,243 articles published in 14 academic journals from 2005 to 2007. Overall, 49% of the articles reported effect size-57% of which interpreted effect size. As an empirical study for the sake of good research methodology in education and psychology, in the present study we provide an illustrative example of reporting and interpreting effect size in a published study. Furthermore, a 7-step guideline for quantitative researchers is also summarized along with some recommended resources on how to understand and interpret effect size. Burnham, & Thompson, 2000;Kirk, 1996;Plucker, 1997;Robinson & Levin, 1997;Thompson, 1998bThompson, , 1999cThompson, , 1999dThompson & Snyder, 1997). The effectiveness of this recommendation is worthy of a methodological review. Thus, the purpose of the present study was to investigate the effect size reporting and interpreting practices in academic journals in education and psychology areas. To build the present study into a solid theoretical framework, we review the problems and misuses of NHST, and we emphasize the importance of effect size and its relationship with NHST, confidence intervals, and meta-analytical assessment of effect size. An illustrative example of reporting and interpreting effect size in a published study is also presented. To facilitate quantitative researchers' use of effect size, we recommend resources on how to understand and use effect size, and we provide a seven-step guideline for quantitative researchers to conclude the study. Keywords NHSTNHST is a traditional and popular approach to make statistical inference about research questions (Anderson et al., 2000). It is considered to be an objective, scientific procedure for knowledge accumulation (Kirk, 1996). It frames research questions in terms of two contrasting statistical hypotheses. For instance, when the purpose is to examine the effect of a treatment, the null hypothesis states that "the experimental group and the control group are not different with respect to [a specified property of interest] and that any difference found between their means is due to sampling fluctuation" (Carver, 1978, p. 381), whereas the alternative hypothesis states the opposite for a two-tailed test of mean difference as population parameter. When...

The impact of promoting transcription on early text production: Effects on bursts and pauses, levels of written language, and writing performance.

“…This may explain the reported discrepancies between p-values and effect sizes (Sun, Pan, & Wang, 2010). Indeed, lack of sufficient statistical power may yield less accurate estimates of effects and increase the likelihood of Type II errors (Abraham & Russell, 2008).…”

Un sistema de clasificación de los diseños de investigación en psicología

“…Aunque se han desarrollado muchos indicadores de magnitud del efecto (véanse revisiones recientes en Ellis, 2010y Grissom & Kim, 2012, se han destacado dos tipos básicos (Rosnow & Rosenthal, 2009): indicadores tipo d (que se basan en la diferencia de medias estandarizada) e indicadores tipo r (que se basan en la correlación y la proporción de varianza explicada). En ambos casos, la interpretación más usual emplea las etiquetas de Cohen (1988), "pequeño", "moderado" o "grande", pero esta práctica debería ser abandonada en favor de la utilización de intervalos de confianza y la evaluación de la discrepancia entre valores de p e indicadores de magnitud del efecto (véase Sun, Pan & Wang, 2010), además de intentar la comparación de indicadores que tratan un problema común dentro de un mismo estudio y entre diferentes estudios. En una reciente revisión, Peng, Chen, Chiang & Chiang (2013) valoran el notable incremento que se ha producido en la utilización de los indicadores de magnitud del efecto en años recientes, pero denuncian la subsistencia de algunas prácticas inadecuadas en informes de investigación tales como: (a) el uso de indicadores tipo r (R 2 y η 2 ) no ajustados, además de un casi nulo uso de otros indicadores menos sesgados como omega cuadrado (ω 2 ) y la correlación intraclase (véase Ivarsson, Andersen, Johnson & Lindwall, 2013), (b) el empleo de las "etiquetas" de Cohen (alto, moderado y bajo) sin ninguna contextualización ni interpretación, (c) la persistente falta de claridad en el uso de los indicadores eta cuadrado y eta cuadrado parcial, que coinciden en modelos ANOVA de un factor pero difieren en modelos ANOVA factoriales (véase Pierce, Block & Aguinis, 2004y Richardson, 2011, (d) el escaso uso de los intervalos de confianza tanto para los estimadores de las pruebas estadísticas como para los tamaños del efecto (véase Kelley & Preacher, 2012, Thompson, 2007, y (d) la falta de integración entre las pruebas de significación estadística y los indicadores de magnitud del efecto.…”

Deberes escolares y rendimiento académico: un estudio comparativo entre el alumnado inmigrante y nativo

“…Como medida del tamaño del efecto se ha utilizado el coeficiente eta-cuadrado parcial (η p 2 ), ya que es uno de los procedimientos más utilizados habitualmente dentro de la investigación educativa (Sun, Pan y Wang, 2010). Para la interpretación de los tamaños del efecto se utiliza el criterio establecido en el trabajo clásico de Cohen (1988) …”