Conducting systematic reviews of applied interventions: A comment on Cabral et al. (2022).

Abstract

In their recent systematic review and meta-analysis, Cabral et al. (2022) explored the effects of implicit motor learning under pressure conditions. As a stated focus, they aimed to address the previously inconsistent findings in the literature and provide clarity to researchers and practitioners. Although we agree that such clarity is needed, we contend that there are critical methodological and procedural concerns that prevent this systematic review from achieving its objectives. In this commentary, we lay out these specific concerns in light of recent debates in this research area and the demands of real-world sporting contexts. More generally, we also call attention to important principles to consider when planning a systematic review of interventions in order to maximize contributions to the literature and usefulness for applied psychology practice.

Attentional Focus Effects on Lower-Limb Muscular Strength in Athletes: A Systematic Review

“…As an example of such an imbalance, one study used “Attempt to jump as high as you can” for the NF, “Attempt to jump as high as you can while focusing on contracting your leg muscles as hard and as fast as you can” for the IF, and “Attempt to jump as high as you can while focusing on pushing off the ground as hard and as fast as you can” for the EF. Respectively, none of the studies explained why the task or attentional focus was of relevance to the athletes involved or their level of familiarity with the specific focus of attention instructions, nor the degree of congruence with what they would consider to be “normal practice.” Although this assessment of certainty is less objective than the “quality assessment,” these are important aspects of the review in terms of identifying gaps and informing readers of how they might critically interpret this systematic review's results for their professional practice; an important feature that is typically missing from previous systematic reviews (15).…”

“…Again, evidence has shown that the number of instructions and length of instruction creates an unfair comparison between conditions when these differ (18). Differences in the cognitive loads imparted, relevance, and familiarity of instructions, for instance, can misrepresent the effectiveness of an intervention (15), which is particularly problematic when studies do not meaningfully compare experimental conditions with what is currently considered best professional practice within the applied field. Therefore, the simple idea of comparing an IF with an EF, as researchers have often done, is far too simplistic and neither representative of the applied challenges faced by coaches and athletes nor the cognitive mechanisms that support learning and performance.…”

“…Finally, in meeting the recognized need suggested by Bobrownicki et al (15) for systematic reviews of interventions to specifically address key and known issues within the topical literature, (h) identification of applied methodological concerns highlighted by others can be considered a level of evaluating the "certainty" of findings. Given the sensitive evaluation of this latter item, methodologies were reviewed by an additional applied researcher (H.J.C.)…”

“…Respectively, none of the studies explained why the task or attentional focus was of relevance to the athletes involved or their level of familiarity with the specific focus of attention instructions, nor the degree of congruence with what they would consider to be "normal practice." Although this assessment of certainty is less objective than the "quality assessment," these are important aspects of the review in terms of identifying gaps and informing readers of how they might critically interpret this systematic review's results for their professional practice; an important feature that is typically missing from previous systematic reviews (15).…”

Motor learning in golf—a systematic review

“…We identified search terms for each PICO element except “comparison” because the diversity of different motor learning strategies makes it difficult to define a fixed comparator. However, following the recommendations of Bobrownicki et al ( 14 ), we considered the relevant comparators when evaluating abstracts and full texts according to the PRISMA guidelines. We conducted the search for “All Fields” but applied the following filters on the results: “Full text, English, peer-reviewed”.…”