The Delphi process: a solution for reviewing novel grant applications

Robotin, Mónica

doi:10.2147/ijgm.s11117

Cited by 13 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…33 Asynchronous online communication has well-established benefits in promoting reflection and knowledge construction. 34 There are over 100 empirical examples of successful online Delphi studies conducted between geographically dispersed participants (for examples see Keeney et al, 32 Elwyn et al, 35 Greenhalgh and Wengraf, 36 Hart et al, 37 Holliday and Robotin, 38 and Pye and Greenhalgh 39 ). We were unable to find any online Delphi study which identified the communication medium as a significant limitation.…”

Section: The Online Delphi Methodsmentioning

confidence: 99%

“…One group commented 'our online review process was less costly, quicker and more flexible with regard to reviewer time commitment, because the process could accommodate their individual schedules'. 38 Critical commentaries on the Delphi process have identified a number of issues which may prove problematic, for example 'issues surrounding problem identification, researcher skills and data presentation' 29 or defining consensus; issues of anonymity; time requirements for data collection, analysis, feedback to participants and obtaining responses on feedback; defining and selecting experts; enhancing response rates and deciding on how many rounds to undertake. 40 These comments suggest that it is the underlying design and rigour of the research process which is key to the quality of the study and not the medium through which this process happens.…”

Section: The Online Delphi Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project

Wong

Greenhalgh

Westhorp³

et al. 2014

Health Services and Delivery Research

189

260

View full text Add to dashboard Cite

BackgroundThere is growing interest in theory-driven, qualitative and mixed-method approaches to systematic review, such as realist and meta-narrative review. These approaches offer the potential to expand the knowledge base in policy-relevant areas. However, the quality of such reviews can be difficult to assess.ObjectivesThe aim of this project was to produce methodological guidance, publication standards and training resources for those seeking to undertake realist and/or meta-narrative reviews.Methods/designWe (1) collated and summarised existing literature on the principles of good practice in realist and meta-narrative systematic reviews; (2) considered the extent to which these principles had been followed by published and in-progress reviews, thereby identifying how rigour may have been lost and how existing methods could be improved; (3) used an online Delphi method with an interdisciplinary panel of experts from academia and policy, to produce a draft set of methodological steps and publication standards; (4) produced training materials with learning objectives linked to these steps; (5) refined these standards and training materials prospectively on real reviews in progress, capturing methodological and other challenges as they arose; (6) synthesised expert input, evidence review and real-time problem analysis into more definitive guidance and standards; and (7) disseminated outputs to audiences in academia and policy.ResultsAn important element of this study was the establishment of an e-mail mailing list to bring together researches in the field (www.jiscmail.ac.uk/RAMESES). Our literature review identified 35 and nine realist and meta-narrative reviews respectively. Analysis and discussion within the project team produced a summary of the published literature, and common questions and challenges into briefing materials for the Delphi panel, comprising 37 and 33 members (for realist and meta-narrative reviews respectively). Within three rounds this panel had reached a consensus on 19 (realist) and 20 (meta-narrative) key publication standards, with an overall response rate of 90% and 91% respectively. The Realist And Meta-narrative Evidence Syntheses – Evolving Standards (RAMESES) publication standards for realist syntheses and meta-narrative reviews were published in open-access journals and quickly became highly accessed. The RAMESES quality standards and training materials drew together the following sources of data: (1) personal expertise as researchers and trainers; (2) data from the Delphi panels; (3) feedback from participants at training sessions we ran; and (4) comments made on RAMESES mailing list. The quality standards and training materials are freely available online (www.ramesesproject.org).DiscussionThe production of these standards and guidance drew on multiple sources of knowledge and expertise, and a high degree of a consensus was achieved despite ongoing debate among researchers about the overall place of these methodologies in the secondary research toolkit. As with all secondary research methods, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency of studies. We anticipate that as more reviews are undertaken, further refinement will be needed to the publication and quality standards and training materials.LimitationsThe project’s outputs are not definitive and in the future updating and further development is likely to be needed.ConclusionAn initial set of publication standards, quality standards and training materials have been produced for researchers, users and funders of realist or meta-narrative reviews. As realist and meta-narrative reviews are relatively new approaches to evidence synthesis, methodological development is needed for both review approaches.FundingThe National Institute for Health Research Health Services and Delivery Research programme.

show abstract

Section: The Online Delphi Methodsmentioning

confidence: 99%

Section: The Online Delphi Methodsmentioning

confidence: 99%

Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project

Wong

Greenhalgh

Westhorp³

et al. 2014

Health Services and Delivery Research

189

260

View full text Add to dashboard Cite

show abstract

“…Asynchronous online communication has well-established benefits in promoting reflection and knowledge construction [ 35 ]. There are over 100 empirical examples of successful online Delphi studies conducted between geographically dispersed participants (see for example [ 33 , 36 - 40 ]). We have been unable to find any online Delphi study which identified the communication medium as a significant limitation.…”

Section: Introductionmentioning

confidence: 99%

“…One group commented: "Our online review process was less costly, quicker, and more flexible with regard to reviewer time commitment, because the process could accommodate their individual schedules." [ 40 ].…”

Section: Introductionmentioning

confidence: 99%

Protocol - realist and meta-narrative evidence synthesis: Evolving Standards (RAMESES)

Greenhalgh¹,

Wong²,

Westhorp

et al. 2011

BMC Med Res Methodol

237

View full text Add to dashboard Cite

BackgroundThere is growing interest in theory-driven, qualitative and mixed-method approaches to systematic review as an alternative to (or to extend and supplement) conventional Cochrane-style reviews. These approaches offer the potential to expand the knowledge base in policy-relevant areas - for example by explaining the success, failure or mixed fortunes of complex interventions. However, the quality of such reviews can be difficult to assess. This study aims to produce methodological guidance, publication standards and training resources for those seeking to use the realist and/or meta-narrative approach to systematic review.Methods/designWe will: [a] collate and summarise existing literature on the principles of good practice in realist and meta-narrative systematic review; [b] consider the extent to which these principles have been followed by published and in-progress reviews, thereby identifying how rigour may be lost and how existing methods could be improved; [c] using an online Delphi method with an interdisciplinary panel of experts from academia and policy, produce a draft set of methodological steps and publication standards; [d] produce training materials with learning outcomes linked to these steps; [e] pilot these standards and training materials prospectively on real reviews-in-progress, capturing methodological and other challenges as they arise; [f] synthesise expert input, evidence review and real-time problem analysis into more definitive guidance and standards; [g] disseminate outputs to audiences in academia and policy. The outputs of the study will be threefold:1. Quality standards and methodological guidance for realist and meta-narrative reviews for use by researchers, research sponsors, students and supervisors2. A 'RAMESES' (Realist and Meta-review Evidence Synthesis: Evolving Standards) statement (comparable to CONSORT or PRISMA) of publication standards for such reviews, published in an open-access academic journal.3. A training module for researchers, including learning outcomes, outline course materials and assessment criteria.DiscussionRealist and meta-narrative review are relatively new approaches to systematic review whose overall place in the secondary research toolkit is not yet fully established. As with all secondary research methods, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency of studies.

show abstract

Posterolateral corner of the knee: an expert consensus statement on diagnosis, classification, treatment, and rehabilitation

Chahla

Murray

Robinson

et al. 2018

Knee Surg Sports Traumatol Arthrosc

103

102

View full text Add to dashboard Cite

Purpose To develop a statement on the diagnosis, classification, treatment, and rehabilitation concepts of posterolateral corner (PLC) injuries of the knee using a modified Delphi technique. Methods A working group of three individuals generated a list of statements relating to the diagnosis, classification, treatment, and rehabilitation of PLC injuries to form the basis of an initial survey for rating by an international group of experts. The PLC expert group (composed of 27 experts throughout the world) was surveyed on three occasions to establish consensus on the inclusion/exclusion of each item. In addition to rating agreement, experts were invited to propose further items for inclusion or to suggest modifications of existing items at each round. Pre‐defined criteria were used to refine item lists after each survey. Statements reaching consensus in round three were included within the final consensus document. Results Twenty‐seven experts (100% response rate) completed three rounds of surveys. After three rounds, 29 items achieved consensus with over 75% agreement and less than 5% disagreement. Consensus was reached in 92% of the statements relating to diagnosis of PLC injuries, 100% relating to classification, 70% relating to treatment and in 88% of items relating to rehabilitation statements, with an overall consensus of 81%. Conclusions This study has established a consensus statement relating to the diagnosis, classification, treatment, and rehabilitation of PLC injuries. Further research is needed to develop updated classification systems, and better understand the role of non‐invasive and minimally invasive approaches along with standardized rehabilitation protocols. Level of evidence Consensus of expert opinion, Level V.

show abstract

The Delphi process: a solution for reviewing novel grant applications

Cited by 13 publications

References 19 publications

Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project

Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses – Evolving Standards) project

Protocol - realist and meta-narrative evidence synthesis: Evolving Standards (RAMESES)

Posterolateral corner of the knee: an expert consensus statement on diagnosis, classification, treatment, and rehabilitation

Contact Info

Product

Resources

About