Evaluating Data Abstraction Assistant, a novel software application for data abstraction during systematic reviews: protocol for a randomized controlled trial

Abstract

BackgroundData abstraction, a critical systematic review step, is time-consuming and prone to errors. Current standards for approaches to data abstraction rest on a weak evidence base. We developed the Data Abstraction Assistant (DAA), a novel software application designed to facilitate the abstraction process by allowing users to (1) view study article PDFs juxtaposed to electronic data abstraction forms linked to a data abstraction system, (2) highlight (or “pin”) the location of the text in the PDF, and (3) copy relevant text from the PDF into the form. We describe the design of a randomized controlled trial (RCT) that compares the relative effectiveness of (A) DAA-facilitated single abstraction plus verification by a second person, (B) traditional (non-DAA-facilitated) single abstraction plus verification by a second person, and (C) traditional independent dual abstraction plus adjudication to ascertain the accuracy and efficiency of abstraction.MethodsThis is an online, randomized, three-arm, crossover trial. We will enroll 24 pairs of abstractors (i.e., sample size is 48 participants), each pair comprising one less and one more experienced abstractor. Pairs will be randomized to abstract data from six articles, two under each of the three approaches. Abstractors will complete pre-tested data abstraction forms using the Systematic Review Data Repository (SRDR), an online data abstraction system. The primary outcomes are (1) proportion of data items abstracted that constitute an error (compared with an answer key) and (2) total time taken to complete abstraction (by two abstractors in the pair, including verification and/or adjudication).DiscussionThe DAA trial uses a practical design to test a novel software application as a tool to help improve the accuracy and efficiency of the data abstraction process during systematic reviews. Findings from the DAA trial will provide much-needed evidence to strengthen current recommendations for data abstraction approaches.Trial registrationThe trial is registered at National Information Center on Health Services Research and Health Care Technology (NICHSR) under Registration # HSRP20152269: https://wwwcf.nlm.nih.gov/hsr_project/view_hsrproj_record.cfm?NLMUNIQUE_ID=20152269&SEARCH_FOR=Tianjing%20Li. All items from the World Health Organization Trial Registration Data Set are covered at various locations in this protocol. Protocol version and date: This is version 2.0 of the protocol, dated September 6, 2016. As needed, we will communicate any protocol amendments to the Institutional Review Boards (IRBs) of Johns Hopkins Bloomberg School of Public Health (JHBSPH) and Brown University. We also will make appropriate as-needed modifications to the NICHSR website in a timely fashion.Electronic supplementary materialThe online version of this article (doi:10.1186/s13643-016-0373-7) contains supplementary material, which is available to authorized users.

A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews

“…The DAA trial was approved by Institutional Review Boards at Johns Hopkins University (IRB number 00006521, July 13, 2015) and Brown University (August 21, 2015). The protocol for the DAA trial was published previously [13]. Appendix 1 includes a completed CON-SORT checklist for the DAA trial.…”

Features and functioning of Data Abstraction Assistant, a software application for data abstraction during systematic reviews

“…After completing data abstraction for the DAA Trial, we asked each data abstractor to a complete a brief survey designed using Qualtrics. We asked questions pertaining to the data abstractor's self-reported ease with which each of the following tasks could be completed: (1) opening source documents in Split Screen view in SRDR, (2) scrolling between pages of a source document, (3) placing flags on a source document, and (4) clicking on existing flags to automatically navigate to the relevant location on the source document. We also asked data abstractors to assess the overall ease of using DAA and to indicate the DAA feature that they liked the most.…”

“…As part of the DAA Trial, data abstractors abstracted various kinds of information pertaining to published studies, including study design, risk of bias, characteristics of study participants, treatment arms, outcomes, within‐arm results, and between‐arm results. We previously reported the detailed protocol for this trial and will report its results separately. In this section, we describe the results of surveying the trial participants, ie, the early users of DAA, with regards to the user‐friendliness of DAA.…”

“…Data abstraction is labor‐ and time‐intensive and often error‐prone . The lack of an audit trail makes the verification of the abstracted information difficult.…”

“…Data abstraction is labor-and time-intensive and often error-prone. [1][2][3][4][5][6][7] The lack of an audit trail makes the verification of the abstracted information difficult. Errors made during data abstraction, which often remain undetected by peer reviewers, editors, and readers, can impact the validity of the results of systematic reviews.…”