Applied comparative effectiveness researchers must measure learning rates: A commentary on efficiency articles

Abstract

Almost all academic skills deÞcits can be conceptualized as learning rate problems as students are not failing to learn, but not learning rapidly enough. Thus, when selecting among various possible remedial procedures, educators need an evidence base that indicates which procedure results in the greatest increases in learning rates. Previous research is reviewed that shows how failure to measure learning rates can lead to erroneous conclusions regarding the relative effectiveness of interventions. Different procedures for measuring learning rates are described and cautions regarding the interpretation of these learning rate data are provided. The article concludes with a plea for all researchers to include measures of learning rates in their applied studies. C 2009 Wiley Periodicals, Inc.If learning researchers hope to improve practitioners' attempts to prevent and remedy learning problems, they must change their behavior. As it is, researchers who are attempting to validate and compare strategies and procedures designed to promote learning may be unintentionally disseminating misleading information promoting interventions that hinder student development because they are neglecting to measure learning rates (Cates et al., 2003;Joseph & Nist, 2006;Nist & Joseph, 2008;Skinner, 1998Skinner, , 2008Skinner, BelÞore, & Watson, 1995;Skinner, Fletcher, & Henington, 1996). AN APPLIED EXAMPLEBramlett, Cates, Savina, and Lauinger (2010) found that most of the researchers investigating learning procedures have not precisely measured the amount of time spent learning. To underscore how critical it is to measure time spent learning, consider the following example. A researcher, using the theoretical and applied evidence base, develops a procedure for remedying spelling problems, the Evidence-Based Spelling Intervention (EBSI). The EBSI is a multicomponent strategy that requires students to perform the following steps:1. First, the student must look at the word, close his eyes, and picture the word in his "minds eye" (visualization). 2. Next, the student must write the word 6 times using paper and pencil (repeated practice). 3. Next, the student must write the word 6 times by tracing the word in sand (using tactile and kinesthetic senses). 4. Next, the student must spell the word 6 times by selecting tiles (using selection responses). 5. Next, the student must Þnd the word listed in a word-Þnd puzzle 6 times, left to right, right to left, top to bottom, bottom to top, and both directions diagonally (often found in spelling workbooks). 6. Next, the student must use chalk to draw the word on the playground cement and then walk along the line, tracing the word with his feet (children like to be active).Assume that an evaluation of the EBSI is conducted. As Bramlett and colleagues (2010) found, this evaluation is unlikely to include an evaluation of learning rates. Rather, researchers implement several almost perfect "true" experiments and conclude that this multicomponent intervention

An investigation of Detect, Practice, and Repair to remedy math‐fact deficits in a group of third‐grade students

“…Although many have indicated the need to remedy skill deficits with evidence-based interventions, much less attention has been paid to intervention efficiency (Bramlett, Cates, Savina, & Lauinger, 2010;Skinner, 2008Skinner, , 2010. By incorporating the metronome-paced group assessment (detect phase), DPR provides an efficient procedure for identifying specific math-fact problems in need of remediation.…”

How and to What Extent Do Two Cover, Copy, and Compare Spelling Interventions Contribute to Spelling, Word Recognition, and Vocabulary Development?

“…When conducting academic intervention comparison studies, both of these omissions can result in researchers supporting the least effective treatment (Skinner 2008(Skinner , 2010Skinner and Daly 2010). The current study may have broader heuristic value by demonstrating how researchers might conduct treatment comparison and component analysis studies that identify the most useful way to apply allotted instructional time.…”

Two experiments using learning rate to evaluate an experimenter developed tool for splay trees

“…Consequently, educators who choose a procedure or tool because researchers found it causes greater learning may actually be putting the student at a disadvantage as a different intervention may have caused the same or even more learning in the same amount of time [13]. Based on these learning rate studies [14] we recommended that learning researchers conducting comparative effectiveness studies should include a precise measure of learning time and make relative effectiveness evaluations based on learning rates, as opposed to mere learning.…”

“…School psychology researchers concerned with remedying student deficit have demonstrated the importance of measuring learning rates, or the amount of learning/amount of time spent learning [5,9,13,14,15]. To highlight the difference in learning versus learning rates, consider an instructor who has the choice of using two different learning tools.…”

“…designing educational tools, developers should keep track of the amount of time users spend with the tool and calculate an effective learning rate, as this is an important measure of the tool's usefulness [13,14]. Recent research [1] shows a steady decline in the amount of time full-time students spend studying.…”