Planning and Reporting Effective Web-Based RAND/UCLA Appropriateness Method Panels: Literature Review and Preliminary Recommendations

Abstract

Background The RAND/UCLA Appropriateness Method (RAM), a variant of the Delphi Method, was developed to synthesize existing evidence and elicit the clinical judgement of medical experts on the appropriate treatment of specific clinical presentations. Technological advances now allow researchers to conduct expert panels on the internet, offering a cost-effective and convenient alternative to the traditional RAM. For example, the Department of Veterans Affairs recently used a web-based RAM to validate clinical recommendations for de-intensifying routine primary care services. A substantial literature describes and tests various aspects of the traditional RAM in health research; yet we know comparatively less about how researchers implement web-based expert panels. Objective The objectives of this study are twofold: (1) to understand how the web-based RAM process is currently used and reported in health research and (2) to provide preliminary reporting guidance for researchers to improve the transparency and reproducibility of reporting practices. Methods The PubMed database was searched to identify studies published between 2009 and 2019 that used a web-based RAM to measure the appropriateness of medical care. Methodological data from each article were abstracted. The following categories were assessed: composition and characteristics of the web-based expert panels, characteristics of panel procedures, results, and panel satisfaction and engagement. Results Of the 12 studies meeting the eligibility criteria and reviewed, only 42% (5/12) implemented the full RAM process with the remaining studies opting for a partial approach. Among those studies reporting, the median number of participants at first rating was 42. While 92% (11/12) of studies involved clinicians, 50% (6/12) involved multiple stakeholder types. Our review revealed that the studies failed to report on critical aspects of the RAM process. For example, no studies reported response rates with the denominator of previous rounds, 42% (5/12) did not provide panelists with feedback between rating periods, 50% (6/12) either did not have or did not report on the panel discussion period, and 25% (3/12) did not report on quality measures to assess aspects of the panel process (eg, satisfaction with the process). Conclusions Conducting web-based RAM panels will continue to be an appealing option for researchers seeking a safe, efficient, and democratic process of expert agreement. Our literature review uncovered inconsistent reporting frameworks and insufficient detail to evaluate study outcomes. We provide preliminary recommendations for reporting that are both timely and important for producing replicable, high-quality findings. The need for reporting standards is especially critical given that more people may prefer to participate in web-based rather than in-person panels due to the ongoing COVID-19 pandemic.

A novel approach to prevent social isolation in people with severe disability

“…Finally, the use of modi ed and online Delphi Panels offers a cost-effective and convenient alternative to the traditional in-person modality. Even though it has been widely used in Health research, its diversi ed utilization calls for caution and requires the application of high-quality assurance criteria to guarantee reliable ndings 40 . Aware of this limitation, speci c recommendations from the literature 38,40 have been followed, explicitly: using validated reporting templates form the RAND/UCLA manual; establishing measures of process quality, through a quality evaluation questionnaire for the experts; and utilizing an online platform that would allow the engagement, interaction and sharing among the panelists.…”

Utility of RAND/UCLA appropriateness method in validating multiple-choice questions on ECG

“…Although diversity is expected to increase with the number of participants, it may be difficult for all to participate. In recent years, an increase in web-based panel meetings have been observed in consensus-building methods using RAM [ 19 ]. Therefore, we addressed these issues by conducting an online meeting so that remote panelists could participate and allow those who could not attend to view a video recording of the meeting.…”

A new intervention to prevent social isolation in people with complex communication needs

“…The use of modified and online Delphi Panels offers a cost-effective and convenient alternative to the traditional in-person modality. Even though it has been widely used in Health research, its diversified utilization calls for caution and requires the application of high-quality assurance criteria to guarantee reliable findings 40 . Aware of this limitation, specific recommendations from the literature 38 , 40 have been followed, explicitly: using validated reporting templates from the RAND/UCLA manual; establishing measures of process quality, through a quality evaluation questionnaire for the experts; and utilizing an online platform that would allow the engagement, interaction and sharing among the panelists.…”

“…Even though it has been widely used in Health research, its diversified utilization calls for caution and requires the application of high-quality assurance criteria to guarantee reliable findings 40 . Aware of this limitation, specific recommendations from the literature 38 , 40 have been followed, explicitly: using validated reporting templates from the RAND/UCLA manual; establishing measures of process quality, through a quality evaluation questionnaire for the experts; and utilizing an online platform that would allow the engagement, interaction and sharing among the panelists. Additional precautions were taken with the recruitment of experts to ensure cognitive diversity, by involving participants with diverse backgrounds and from different countries, as well as with the option for a clear definition and quantitative measure to outline consensus, for the sake of the transparency of the results 57 .…”

“…The authors of this approach believe that combining the best available scientific evidence (through the presentation of results of a literature review) with the collective judgement of experts (via the discussion with a panel) would be effective in providing a statement regarding the appropriateness of performing specific procedures at the level of patient-specific needs 38 . Posterior work, within the clinical setting 39 , 40 , validated the use of RAND’s Appropriateness Method, for its successful application in consulting professionals to inform the development of professional guidance and achieving evidence-based recommendations. Following the method, two rounds were necessary to achieve consensus and rate the appropriateness of the CCC model and Program 3 C .…”