An Analysis of the Effect of Graph Construction Disclaimers on Visual Analysis

Radley, Keith C.; Dart, Evan H.

doi:10.1007/s10864-023-09525-5

Cited by 2 publications

(5 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This support was limited to reducing inconsistency in ratings of effect size magnitude and not decisions regarding whether an effect existed. Although the effect size was small, the potential promise of this approach may also be supported by recent findings that providing visual analysts with disclaimers about graph manipulations did not impact their ratings of effect size magnitude (Radley & Dart, 2023). These brief prompts may be helpful for reducing rating inconsistency in circumstances when the standardization of graph construction is inhibited.…”

Section: Discussionmentioning

confidence: 84%

“…However, the utility of this approach for reducing inconsistency in applied decision making is unclear. Practitioners often make decisions that are more similar to identifying whether the effects were meaningful or not (e.g., continue, modify, or discontinue intervention) than judgements regarding the magnitude of an effect (Radley & Dart, 2023). Judgements regarding the magnitude of treatment effects may better inform decisions regarding intervention selection and not intervention response.…”

Section: Discussionmentioning

confidence: 99%

“…For example, variations in the scaling of the Y -axis (Dart & Radley, 2017), the scaling of the X -axis (Dart & Radley, 2023), and the number of data points per X - to Y -axis ratio (DPPXYR; Radley et al, 2018), all influenced visual analysts’ identification of treatment effects and ratings regarding the magnitude of treatment effects. In fact, the manner in which graphs were assembled (e.g., Y -axis range, DPPXYR) had an impact on visual analysts’ judgements of the magnitude of effects even when the graphs were accompanied by a disclaimer that identified that graph manipulation was used to magnify intervention effects (Radley & Dart, 2023). Recent reviews by Peltier et al (2021, 2022) suggested that published SCD graphs generally had appropriate ordinate scaling, but DPPXYR values that could lead to inflated Type I error rates.…”

Section: Uses Of Time-series Graphs To Support Decisions In Mtssmentioning

confidence: 99%

“…One example of a task-relevant reminder is a disclaimer that a graph has been manipulated from standardization. Radley and Dart (2023) experimentally tested such an intervention that warned participants when graph axes had been manipulated. This disclaimer failed to attenuate inflation effects of compressed graph axes.…”

Section: Uses Of Time-series Graphs To Support Decisions In Mtssmentioning

confidence: 99%

See 3 more Smart Citations

Effects of portable interventions on school psychologists’ graph-rating inconsistency.

Latham,

Klingbeil

2024

School Psychology

View full text Add to dashboard Cite

The visual analysis of data presented in time-series graphs are common in single-case design (SCD) research and applied practice in school psychology. A growing body of research suggests that visual analysts' ratings are often influenced by construct-irrelevant features including Y-axis truncation and compression of the number of data points per X-to Y-axis ratio. We developed and tested two brief interventions, based on the research in cognitive and visual science, to reduce visual analysts' inconsistency when viewing unstandardized graphs. Two hundred practicing school psychologists visually analyzed data presented on standardized graphs and the same data again on unstandardized graphs. Across all conditions, participants were more willing to identify meaningful effects on unstandardized graphs and rated the data as showing significantly larger effects than on the corresponding standardized graphs. However, participants who answered additional (task-relevant) questions about the level or trend of graphed data showed greater rating consistency across the types of graphs in comparison to participants who answered task-irrelevant but challenging questions or control participants. Our results replicated prior research demonstrating the impact of SCD graph construction on practicing school psychologists' interpretations and provide initial support for an intervention to minimize the impact of construct-irrelevant factors. Limitations and future directions for research are discussed. Impact and ImplicationsDespite calls to standardize graph construction in single-case design, variability in graph construction is likely to impact visual analysis of time-series data. Interventions that can be readily applied to reduce inconsistencies of ratings of data presented on unstandardized graphs may result in more consistent ratings of time-series data. We provide preliminary evidence that a brief, attention-based intervention reduces graph-rating inconsistency in ratings of effect size magnitude among school psychologists.

show abstract

Section: Discussionmentioning

confidence: 84%

Section: Discussionmentioning

confidence: 99%

Section: Uses Of Time-series Graphs To Support Decisions In Mtssmentioning

confidence: 99%

Section: Uses Of Time-series Graphs To Support Decisions In Mtssmentioning

confidence: 99%

See 2 more Smart Citations

Effects of portable interventions on school psychologists’ graph-rating inconsistency.

Latham,

Klingbeil

2024

School Psychology

View full text Add to dashboard Cite

show abstract

“…First, researchers should clearly describe how many measurement occasions are represented within each data point in their linear graphs of single-case data. Although recent research does not support the use of disclosures or disclaimers to mitigate the effects, these kinds of graphical features have on visual analysts’ decisions (e.g., Radley & Dart, 2023), obscuring such information may make it difficult to fully contextualize a single-case data set. Second, both researchers and practitioners are encouraged to consider using the most microscopic presentation of their data that is feasible given the number of data points.…”

Section: Discussionmentioning

confidence: 99%

The effect of the level of data presentation on visual analysts’ decisions.

Radley,

Dart

2024

School Psychology

View full text Add to dashboard Cite

Single-case design (SCD) is frequently utilized in research and applied settings to evaluate the effect of an intervention over time. Once collected, single-case data are typically graphed and analyzed visually in both research and practice contexts. Despite the ubiquity of visual analysis in SCD, this analytic framework has often been critiqued due to findings of limited reliability across visual analysts. Recent research has identified that the way a graph is constructed may contribute to the limitations of visual analysis. The present study sought to evaluate the effect of visually representing multiple measurement occasions as a single data point (e.g., combining measurements taken daily into a weekly composite data point) on visual analysts' decisions regarding the magnitude of an intervention effect. Eleven participants viewed identical data sets, plotted to show different numbers of measurement occasions combined as a single data point, and provided ratings regarding the magnitude of intervention effect depicted within the graph. Results indicated a significant main effect, with data sets with higher levels of data combination being rated as demonstrating significantly larger intervention effects. The results of the study provide additional support for standardization of data presentation and graph construction within SCD in both research and practice contexts. Impact and ImplicationsThe manner in which a single-case graph is presented has an impact on rater interpretation of intervention effectiveness. Practitioners and researchers are encouraged to avoid combining multiple measurement occasions into single data points when doing so obscures or alters data characteristics such as trend and level changes of data.

show abstract

An Analysis of the Effect of Graph Construction Disclaimers on Visual Analysis

Cited by 2 publications

References 23 publications

Effects of portable interventions on school psychologists’ graph-rating inconsistency.

Effects of portable interventions on school psychologists’ graph-rating inconsistency.

The effect of the level of data presentation on visual analysts’ decisions.

Contact Info

Product

Resources

About