Exploring perceptions of meaningfulness in visual representations of bivariate relationships

Abstract

Researchers often need to consider the practical significance of a relationship. For example, interpreting the magnitude of an effect size or establishing bounds in equivalence testing requires knowledge of the meaningfulness of a relationship. However, there has been little research exploring the degree of relationship among variables (e.g., correlation, mean difference) necessary for an association to be interpreted as meaningful or practically significant. In this study, we presented statistically trained and untrained participants with a collection of figures that displayed varying degrees of mean difference between groups or correlations among variables and participants indicated whether or not each relationship was meaningful. The results suggest that statistically trained and untrained participants differ in their qualification of a meaningful relationship, and that there is significant variability in how large a relationship must be before it is labeled meaningful. The results also shed some light on what degree of relationship is considered meaningful by individuals in a context-free setting.

An Equivalence Testing Approach for Evaluating Substantial Mediation

“…The ESM methods require the researcher to specify a priori what magnitude for the direct effect would be considered inconsequential within the nature of the study. Although this a priori determination can be challenging, researchers regularly consider what level of effect size is meaningful within the context of their research question and previous literature (Beribisky, Davidson, & Cribbie, 2019). The selection of an equivalence interval is closely related to this process -a researcher must consider, within the context of their research question and the extant literature, what effect represents an appropriate inconsequential difference.…”

“…The Quantitative Methods for Psychology pend on the nature of the study, etc. As equivalence testing becomes more mainstream it is hoped that much more research will be conducted into appropriate bounds for these types of problems (see Beribisky et al, 2019).…”

Denouncing the Use of Field-Specific Effect Size Distributions to Inform Magnitude

“…Further, working in a context-free setting, Beribisky, Davidson and Cribbie (2019) found that a correlation of r = .3 was the smallest association that was deemed meaningful when participants viewed scatterplots of associations. Hemphill (2003) and Beribisky et al (2019) are two context-free settings that come to extremely different conclusions regarding effect size magnitude; in other words, context cannot be ignored when interpreting magnitude. Thus, there is simply no valid theoretical rationale for creating benchmarks of ES values for researchers in psychology (or any other discipline) by considering the distribution of observed ES values within these disciplines (or subdisciplines).…”

“…For example, if the benchmark value for a 'large' effect was set to a relatively small value (such as r = .3 in the Hemphill study) in a specific field, there is the potential for researchers to claim meaningfulness even in situations where the specific effect of a given study has no practical significance (Pogrow, 2019). Further, working in a context-free setting, Beribisky, Davidson and Cribbie (2019) found that a correlation of r = .3 was the smallest association that was deemed meaningful when participants viewed scatterplots of associations. Hemphill (2003) and Beribisky et al (2019) are two context-free settings that come to extremely different conclusions regarding effect size magnitude; in other words, context cannot be ignored when interpreting magnitude.…”

Denouncing the use of field-specific effect size distributions to inform magnitude

“…For example, if the benchmark value for a "large" effect was set to a relatively small value (such as r = 0.3 in the Hemphill study) in a specific field, there is the potential for researchers to claim meaningfulness even in situations where the effect has no practical significance (Pogrow, 2019). Further, working in a context-free setting, Beribisky, Davidson & Cribbie (2019) found that a correlation of r = 0.3 was the smallest association that was deemed meaningful when participants viewed scatterplots of context-free associations. Thus, there is simply no valid theoretical rationale for creating benchmarks of ES values for researchers in psychology (or any other discipline/subdiscipline) by considering the distribution of observed ES values within these disciplines (or sub-disciplines).…”