Online Information Infrastructure Increases Inter-Rater Reliability of Medical Coders: A Quasi-Experimental Study (Preprint)

Abstract

Background: Medical coding is essential for standardized communication and integration of clinical data. The Unified Medical Language System (UMLS) by the National Library of Medicine is the largest clinical terminology system for medical coders and natural language processing tools. However, abundance of ambiguous codes leads to low rates of uniform coding among different coders. Objective: To measure uniform coding among different medical experts in terms of inter-rater reliability (IR) and analyze the effect on IR by using an expert-based online code suggestion system. Methods: A quasi-experimental study was conducted. Six medical experts coded 602 medical items from structured quality assurance forms (QA) or free-text eligibility criteria (EC) of 20 different clinical trials. Medical item content was selected based on mortality-leading diseases according to WHO data. The intervention consisted of using a semi-automatic code suggestion tool that is linked to a European information infrastructure providing a large medical text corpus of more than 300,000 medical form items with expert-assigned semantic codes. Krippendorff's alpha (K alpha ) with bootstrap analysis was used for IR analysis and coding times were measured before and after intervention.