Survey research in anesthesiology: a field guide to interpretation

Abstract

BackgroundSurvey research, indispensable for assessing subjective outcomes in anesthesiology, can nonetheless be challenging to undertake and interpret.ObjectiveTo present a user-friendly guide for the appraisal of survey-derived evidence, and to apply it to published survey research in the anesthesia literature.MethodsSynthesizing published expert guidance regarding methodology and reporting, we discuss five essential criteria (with subcomponents) for evaluating survey research: (1) relevance of survey outcome to research objective, (2) trustworthiness of the instrument (testing/validation, availability), (3) collecting information well (sampling, administration), (4) representativeness (response rate), and (5) guidance towards interpretation of survey findings (generalizability, interpretation of numerical outcomes). These criteria were subsequently applied by two independent assessors to original research articles reporting survey findings, published in the five highest impact general anesthesia journals (‘Anaesthesia’, ‘Anesthesia & Analgesia’, ‘Anesthesiology’, ‘British Journal of Anaesthesia’ and ‘European Journal of Anaesthesiology’) between July 01, 2016, and December 31, 2017, which were identified using a prespecified PubMed search strategy.ResultsAmong 1107 original articles published, we identified 97 reporting survey research either employing novel survey instruments (58%), established surveys (30%), or sets of single-item scores (12%). The extent to which reader-oriented benchmarks were achieved varied by component and between survey types. Results were particularly mixed for validation (mentioned for 41% of novel and 86% of established surveys) and discussion of generalizability (59% of novel survey reports, 45% of established surveys, and 17% of sets of single-item scores).ConclusionSurvey research is not uncommon in anesthesiology, frequently employs novel survey instruments, and demonstrates mixed results in terms of transparency and interpretability. We provide readers with a practical framework for critical interpretation of survey-derived outcomes.

Ongoing Variability in Pediatric Extracorporeal Membrane Oxygenation Anticoagulation Practices—Could Consensus Change the Next Survey Results?*

“…Survey research remains a common starting point for many avenues of clinical investigation. In the absence of established survey conduct and reporting guidelines, we considered the report by Ozment et al (6) in the context of a recently proposed “field-guide” to interpretation of survey research in anesthesiology (15). The clearly stated objective by Ozment et al (6) “…to determine current practices of anticoagulation management and blood transfusion in neonatal and pediatric extracorporeal membrane oxygenation patients” established relevance of the survey, and the only nontarget question regarding willingness to participate in future clinical trials may facilitate the next phases of evolving clinical investigation within the field (6).…”

“…Development, pilot testing, and validation of the survey instrument were not described in detail, but the use of a preexisting tool (4) is referenced, and the survey is published as a supplement to the article (6). Implementation, in this case, collection of physicians’ impressions of anticoagulation and transfusion management during pediatric ECMO, can introduce bias during population selection and administration (10, 11, 15). The sampling of physicians listed as center directors of U.S. ELSO ECMO centers could variably be described as a convenience (professional organization associated) or consecutive sample because it was sent to all.…”

“…The sampling of physicians listed as center directors of U.S. ELSO ECMO centers could variably be described as a convenience (professional organization associated) or consecutive sample because it was sent to all. No universally accepted minimal response rate has been defined for survey research, but higher response rates (i.e., > 70–80%) minimize potential for bias due to absent responses, ensure generalizability of the findings to the target population and enhance validity of the questionnaire (10, 11, 15). Ozment et al (6) report their relatively high (> 70%) response rate compared with many surveys of clinician practice and reflect upon the potential impact of inclusion of only ELSO ECMO center directors as a factor which may influence generalizability of the results.…”

“…Ozment et al (6) report their relatively high (> 70%) response rate compared with many surveys of clinician practice and reflect upon the potential impact of inclusion of only ELSO ECMO center directors as a factor which may influence generalizability of the results. Their report delivers on the “field-guide’ checklist of Survey Instrument, Implementation, and Interpretation, as an established survey tool and thus without extensive development methodology included (15).…”