Stability of response characteristics of a Delphi panel: application of bootstrap data expansion

Abstract

BackgroundDelphi surveys with panels of experts in a particular area of interest have been widely utilized in the fields of clinical medicine, nursing practice, medical education and healthcare services. Despite this wide applicability of the Delphi methodology, there is no clear identification of what constitutes a sufficient number of Delphi survey participants to ensure stability of results.MethodsThe study analyzed the response characteristics from the first round of a Delphi survey conducted with 23 experts in healthcare quality and patient safety. The panel members had similar training and subject matter understanding of the Malcolm Baldrige Criteria for Performance Excellence in Healthcare. The raw data from the first round sampling, which usually contains the largest diversity of responses, were augmented via bootstrap sampling to obtain computer-generated results for two larger samples obtained by sampling with replacement. Response characteristics (mean, trimmed mean, standard deviation and 95% confidence intervals) for 54 survey items were compared for the responses of the 23 actual study participants and two computer-generated samples of 1000 and 2000 resampling iterations.ResultsThe results from this study indicate that the response characteristics of a small expert panel in a well-defined knowledge area are stable in light of augmented sampling.ConclusionPanels of similarly trained experts (who possess a general understanding in the field of interest) provide effective and reliable utilization of a small sample from a limited number of experts in a field of study to develop reliable criteria that inform judgment and support effective decision-making.

Developing a clinical pathway for the identification and management of anxiety and depression in adult cancer patients: an online Delphi consensus process

“…Although the response rate was low, such response rates are common for webbased survey research [30], and oversampling was deliberately undertaken to ensure the sample size was adequate for a Delphi study [31]. Furthermore, participants were purposively selected to represent the multidisciplinary diversity of health professionals providing cancer care in Australia.…”

Environmental heterogeneity: Anuran diversity in homogeneous environments

“…Group stability was assessed by bootstrap analysis with 1,000 replicates (AKINS et al 2005). The dendrogram was considered representative of the original data matrix when the cophenetic correlation coefficient (r) was equal to or greater than 0.8.…”

Delphi Panels: Research Design, Procedures, Advantages, and Challenges

“…There is no standard when it comes to panel size; neither has it ever been established what constitutes a large or small panel. Akins, Tolson and Cole (2005) noted that panels have been conducted with just about any size. They also noted that panels of less than 10 are rare, as are panels over 1,000.…”

“…It's perhaps intuitive that Delphi panels that include different disciplines would look to each of those disciplines for the expert threshold. That said, no matter what the topic or problem any given panel could be asked to address, there are certain criteria that apply to membership on all Delphi Panels (Akins et al, 2005):…”