An Exploratory Comparison of Three Treatment Fidelity Assessment Methods: Time Sampling, Event Recording, and Post-observation Checklist

Collier‐Meek, Melissa A.; Sanetti, Lisa M. Hagermoser; Gould, Kaitlin; Pereira, Brittany A.

doi:10.1080/10474412.2020.1777874

Cited by 7 publications

(3 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These comparisons included evaluations with known “true values” of the data and replications with data collected in more typical research contexts. To date, few similar evaluations have been published with a focus on procedural fidelity (for notable exceptions, see Collier‐Meek et al, 2020, 2021; Cook et al, 2015). Identifying the relative advantages and disadvantages of various fidelity measurement methods and corresponding methods for calculating fidelity coefficients may be an important area for further inquiry.…”

Section: Discussionmentioning

confidence: 99%

On a persisting curious double standard in behavior analysis: Behavioral scholars' perspectives on procedural fidelity

Peter

Brand

Jones

et al. 2023

J of App Behav Analysis

View full text Add to dashboard Cite

Procedural fidelity is the extent to which independent variables are implemented as designed. Despite 40 years of discussion about the importance of procedural fidelity for behavioral research, reporting of fidelity data remains an uncommon practice in behavior-analytic journals. Researchers have speculated about reasons for underreporting, but the perspectives of scholars about when reporting is warranted or necessary have not yet been explored. Thus, the purpose of this study was to evaluate possible reasons for infrequent reporting of fidelity data in behavior-analytic studies. To address this purpose, we conducted focus groups with scholars in applied behavior analysis. Five themes emerged regarding why procedural fidelity data are not typically reported. We provide a discussion about how these themes are interrelated and offer suggestions and recommendations to assist with the collection and reporting of fidelity data.

show abstract

Section: Discussionmentioning

confidence: 99%

On a persisting curious double standard in behavior analysis: Behavioral scholars' perspectives on procedural fidelity

Peter

Brand

Jones

et al. 2023

J of App Behav Analysis

View full text Add to dashboard Cite

show abstract

“…If the resources in the setting preclude collecting occurrence–nonoccurrence data or a single score for fidelity is more useful, the all-or-nothing by component method may be a good option as it is less likely to mask errors than global scores or Likert scales, does not require the observer collect occurrence–nonoccurrence data, and can be used to guide (re)training. There is still much work to do in assessing procedural fidelity of DTI, and additional research needs to be conducted on the reliability and utility of these and other procedural-fidelity assessment methods including sequential analysis (Brand et al, 2017 ), time sampling methods (Collier-Meek et al, 2021a ), and indirect assessment methods (e.g., permanent product; Collier-Meek et al, 2021b , Gresham et al, 2017 ). We hope that the topic of procedural-fidelity measurement is approached with as much rigor as studies on dependent-variable measurement (e.g., Rapp et al, 2007 , 2008 ).…”

Section: Discussionmentioning

confidence: 99%

Comparing Multiple Methods to Measure Procedural Fidelity of Discrete-trial Instruction

Bergmann

Niland

Gavidia

et al. 2023

Educ. Treat. Child.

View full text Add to dashboard Cite

Procedural fidelity is the extent to which an intervention is implemented as designed and is an important component of research and practice. There are multiple ways to measure procedural fidelity, and few studies have explored how procedural fidelity varies based on the method of measurement. The current study compared adherence to discrete-trial instruction protocols by behavior technicians with a child with autism when observers used different procedural-fidelity measures. We collected individual-component and individual-trial fidelity with an occurrence–nonoccurrence data sheet and compared these scores to global fidelity and all-or-nothing, 3-point Likert scale, and 5-point Likert scale measurement methods. The all-or-nothing method required all instances of a component or trial be implemented without error to be scored correct. The Likert scales used a rating system to score components and trials. At the component level, we found that the global, 3-point Likert, and 5-point Likert methods were likely to overestimate fidelity and mask component errors, and the all-or-nothing method was unlikely to mask errors. At the trial level, we found that the global and 5-point Likert methods approximated individual-trial fidelity, the 3-point Likert method overestimated fidelity, and the all-or-nothing method underestimated fidelity. The occurrence–nonoccurrence method required the most time to complete, and all-or-nothing by trial required the least. We discuss the implications of measuring procedural fidelity with different methods of measurement, including false positives and false negatives, and provide suggestions for practice and research. Supplementary Information The online version contains supplementary material available at 10.1007/s43494-023-00094-w.

show abstract

“…The IV in applied research and practice is conducted by humans. Therefore, measuring whether humans implement the IV accurately and consistently is both possible and crucial for research and practice (Collier‐Meek, Sanetti et al, 2021; Colllier‐Meek, Sanetti, & Fallon, 2021). Whereas procedural‐fidelity data reported in research on humans tend to be very high (McIntyre et al, 2007), descriptive assessments suggest practitioners and animal behavior professionals are likely to make errors when implementing behavior‐analytic interventions (Brand et al, 2018; Breeman et al, 2020; Carroll et al, 2013; Colón & Ahearn, 2019; Donnelly & Karsten, 2017; Foreman et al, 2021; Kodak et al, 2018; Pfaller‐Sadovsky et al, 2019).…”

Section: Measuring Procedural Fidelitymentioning

confidence: 99%

Strengthening the procedural fidelity research‐to‐practice loop in animal behavior

Kodak

Bergmann

Waite

2022

J Exper Analysis Behavior

View full text Add to dashboard Cite

Procedural fidelity is the extent to which components of an intervention are implemented as designed. Procedural fidelity is measured as a dependent variable and manipulated as an independent variable. In research and practice, procedural‐fidelity data should be collected, monitored, and reported. Procedural fidelity as an independent variable has been investigated in humans using parametric analyses, and the current article summarizes some of the research conducted on the effects of procedural‐fidelity errors in behavior‐reduction and skill‐acquisition interventions. Connections were drawn to applied animal researchers and the work of animal behavior practitioners to produce implications for practice with human and animal clients and suggestions for future research. Further, there are multiple ways to measure procedural fidelity, and different conclusions can be drawn based on the measure and computation method. The current article describes procedural‐fidelity measures that are most applicable to animal behavior researchers and professionals.

show abstract

An Exploratory Comparison of Three Treatment Fidelity Assessment Methods: Time Sampling, Event Recording, and Post-observation Checklist

Cited by 7 publications

References 47 publications

On a persisting curious double standard in behavior analysis: Behavioral scholars' perspectives on procedural fidelity

On a persisting curious double standard in behavior analysis: Behavioral scholars' perspectives on procedural fidelity

Comparing Multiple Methods to Measure Procedural Fidelity of Discrete-trial Instruction

Strengthening the procedural fidelity research‐to‐practice loop in animal behavior

Contact Info

Product

Resources

About