The Simulated Group Response Paradigm: A new approach to the study of opinion change in Delphi and other structured-group techniques

Bolger, Fergus; Rowe, Gene; Belton, Ian; Crawford, Megan; Hamlin, Iain; Sissons, Aileen; Lūka, Courtney Taylor Browne; Vasilichi, Alexandrina; Wright, George

doi:10.31219/osf.io/4ufzg

Cited by 3 publications

(5 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Research of this kind will often require an experimental approach—but this can be difficult with group‐based methods such as Delphi, where each group provides only a single data point. Bolger et al (2021) proposed a laboratory‐based protocol for experimentation using simulated Delphi groups—which allows individual features of a proposed Delphi survey to be evaluated and so provides an evidence‐based approach to future survey design issues and choices.…”

Section: Discussionmentioning

confidence: 99%

“…Accuracy is an important criterion when Delphi is applied within a forecasting context. Exposure to other experts’ arguments rather than just their responses may be important in this process (Bolger et al, 2021; Rowe & Wright, 1996; Rowe et al, 1991; but see Rowe et al, 2005). This is especially true in most Delphi contexts, where the reasoning behind an answer is not apparent from the answer given.…”

Section: Detailed Analysis Of Two Illustrative Examples Of Large‐scal...mentioning

confidence: 99%

“…sonality differences within (expert) participants may influence their judgments-for example, traits of morality, optimism, pessimism, risktaking propensity, and so on. Research of this kind will often require an experimental approach-but this can be difficult with group-based methods such as Delphi, where each group provides only a single data point Bolger et al (2021). proposed a laboratory-based protocol…”

mentioning

confidence: 99%

See 2 more Smart Citations

A critical evaluation of 42, large‐scale, science and technology foresight Delphi surveys

Belton

Cuhls

Wright

2022

Futures & Foresight Science

Self Cite

View full text Add to dashboard Cite

Large‐scale Delphi surveys on technology foresight started in the 1960s and involve an average of about 2000 participants answering, potentially, up to about 450 items. This contrasts sharply with the participation and content of the more common, smaller‐scale Delphi surveys. Previously, Belton et al. developed “six steps” to underpin a well‐founded and defensible Delphi process and we apply these steps in a novel evaluation of the quality of 42 large‐scale technology foresight surveys. Using a detailed analysis of two exemplar studies and a content analysis of all 42 surveys, we explore whether such surveys differ systematically from “traditional” smaller‐scale Delphi surveys and, if so, why this may be and what it may mean for the quality of data produced. We conclude that there are some problematic issues within these surveys—to do with (i) data quality in both the numerical summarizing of participant's between‐round feedback and in the reporting of final round numerical responses, (ii) the infrequent elicitation of rationales to justify participants' proffered numerical responses, and, when such rationales are elicited, (iii) the between‐round summary and presentation of the rationales. We speculate on the reasons for these design differences in the extant large‐scale surveys and conclude that extra‐survey political influences, such as differing objectives and the need to demonstrate wide‐ranging expert participation, may be the underlying cause. We conclude with considerations and recommendations for the design of future large‐scale Delphi surveys to enable the underlying process to become better‐founded and more defensible to procedural evaluation.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Detailed Analysis Of Two Illustrative Examples Of Large‐scal...mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

A critical evaluation of 42, large‐scale, science and technology foresight Delphi surveys

Belton

Cuhls

Wright

2022

Futures & Foresight Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…Simulated group response paradigm (SGRP) is a method for studying group decision making (Bolger et al, 2020). Virtual groups are created by presenting individual participants with information, advice or other responses collected from previous participants during an earlier response collection exercise.…”

Section: Methodsmentioning

confidence: 99%

Structured groups make more accurate veracity judgements than individuals

Hamlin

Bolger

Vasilichi

et al. 2021

Applied Cognitive Psychology

Self Cite

View full text Add to dashboard Cite

Groups often make better judgements than individuals, and recent research suggests that this phenomenon extends to the deception detection domain. The present research investigated whether the influence of groups enhances the accuracy of judgements, and whether group size influences deception detection accuracy. Two‐hundred fifty participants evaluated written statements with a pre‐established detection accuracy rate of 60% in terms of veracity before viewing either the judgements and rationales of several other group members or a short summary of the written statement and revising or restating their own judgements accordingly. Participants' second responses were significantly more accurate than their first, suggesting a small positive effect of structured groups on deception detection accuracy. Group size did not have a significant effect on detection accuracy. The present work extends our understanding of the utility of group deception detection, suggesting that asynchronous, structured groups outperform individuals at detecting deception.

show abstract

“…Using the Simulated Group Response Paradigm (SGRP, described in Bolger et al, 2020), we first conducted piloting in which participants who had already undergone BARD training and produced BNs using BARD were presented with new problems requiring structural solutions. We then created two alternative stimulus sets of 6 models for each problem, based on those elicited during piloting.…”

Section: Design and Materialsmentioning

confidence: 99%

Improving the production and evaluation of structural models using a Delphi process

Bolger

Nyberg²,

Belton

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Bayes Nets (BNs) are extremely useful for causal and probabilistic modelling in many real-world applications, often built with information elicited from groups of domain experts. But their potential for reasoning and decision support has been limited by two major factors: the need for significant normative knowledge, and the lack of any validated methods or software supporting collaboration. Consequently, we have developed a web-based structured technique – Bayesian Argumentation via Delphi (BARD) – to enable groups of domain experts to receive minimal normative training and then collaborate effectively to produce high-quality BNs. BARD harnesses multiple perspectives on a problem, while minimising biases manifest in freely interacting groups, via a Delphi process: solutions are first produced individually, then shared, followed by an opportunity for individuals to revise their solutions. To test the hypothesis that BNs improve due to Delphi, we conducted an experiment whereby individuals with a little BN training and practice produced structural models using BARD for two Bayesian reasoning problems. Participants then received 6 other structural models for each problem, rated their quality on a 7-point scale, and revised their own models if they wished. Both top-rated and revised models were on average significantly better quality (scored against a gold-standard) than the initial models, with large and medium effect sizes. We conclude that Delphi – and BARD – improves the quality of BNs produced by groups. Further, although rating cannot create new models, rating seems quicker and easier than revision and yielded significantly better models – so, we suggest efficient BN amalgamation should include both.

show abstract

The Simulated Group Response Paradigm: A new approach to the study of opinion change in Delphi and other structured-group techniques

Cited by 3 publications

References 47 publications

A critical evaluation of 42, large‐scale, science and technology foresight Delphi surveys

A critical evaluation of 42, large‐scale, science and technology foresight Delphi surveys

Structured groups make more accurate veracity judgements than individuals

Improving the production and evaluation of structural models using a Delphi process

Contact Info

Product

Resources

About