Combining human and machine intelligence for clinical trial eligibility querying

Abstract

Objective To combine machine efficiency and human intelligence for converting complex clinical trial eligibility criteria text into cohort queries. Materials and Methods Criteria2Query (C2Q) 2.0 was developed to enable real-time user intervention for criteria selection and simplification, parsing error correction, and concept mapping. The accuracy, precision, recall, and F1 score of enhanced modules for negation scope detection, temporal and value normalization were evaluated using a previously curated gold standard, the annotated eligibility criteria of 1010 COVID-19 clinical trials. The usability and usefulness were evaluated by 10 research coordinators in a task-oriented usability evaluation using 5 Alzheimer’s disease trials. Data were collected by user interaction logging, a demographic questionnaire, the Health Information Technology Usability Evaluation Scale (Health-ITUES), and a feature-specific questionnaire. Results The accuracies of negation scope detection, temporal and value normalization were 0.924, 0.916, and 0.966, respectively. C2Q 2.0 achieved a moderate usability score (3.84 out of 5) and a high learnability score (4.54 out of 5). On average, 9.9 modifications were made for a clinical study. Experienced researchers made more modifications than novice researchers. The most frequent modification was deletion (5.35 per study). Furthermore, the evaluators favored cohort queries resulting from modifications (score 4.1 out of 5) and the user engagement features (score 4.3 out of 5). Discussion and Conclusion Features to engage domain experts and to overcome the limitations in automated machine output are shown to be useful and user-friendly. We concluded that human–computer collaboration is key to improving the adoption and user-friendliness of natural language processing.

Seed storage allergens tackled via next-generation research assistant

“…Hence, in this global vision, PGE became an instrument, in which synthesis of DNA/RNA fragments, vector assembly, and choice of the transformation method became directly available for the researcher, depending on the final research goal. So far, ML has already proven to be a revolutionary tool in supporting clinical studies, medical and precise diagnosis (Mansour et al, 2021;Fang et al, 2022) with application in medical diagnostics and for detecting, forecasting, and predicting of immunity responses (Nawaz et al, 2021;Buendgens et al, 2022). In this context, the utilization of an NGA for the comprehensive analysis of allergens could be completely revolutionary in human health.…”

Bridging Data Models in Health Care With a Novel Intermediate Query Format for Feasibility Queries: Mixed Methods Study

“…Our ongoing communication with the HL7 (Health Level Seven International) working group, which focuses on Research Studies, gives us confidence that once a suitable FHIR Resource is established or adapted to meet the needs outlined in our publication, the established technical components could be efficiently modified to align with these changes. Parallels can be drawn to implementing structured eligibility criteria, as presented by Fang and Yuan et al [43,44]. Their publications present a half-automated approach to generate feasibility queries based on free text study protocols from ClinicalTrials.gov [23] Their system is built around the OHDSI data model and uses the concept IDs.…”

Characterisation of digital therapeutic clinical trials: a systematic review with natural language processing

“… 15 , 16 These pipelines can also map extracted concepts to existing biomedical vocabularies, such as MeSH categories, for standardisation and downstream analysis. Several NLP methods have been applied to analyse drug therapeutic clinical trials, 11 , 17 but have not yet been used to characterise the clinical development of DTx.…”