Online Information Infrastructure Increases Inter-Rater Reliability of Medical Coders: A Quasi-Experimental Study
Date Submitted: Dec 13, 2017
Open Peer Review Period: Dec 13, 2017 - Feb 7, 2018
Background: Medical coding is essential for standardized communication and integration of clinical data. The Unified Medical Language System by the National Library of Medicine is the largest clinical terminology system for medical coders and natural language processing tools. However, abundance of ambiguous codes leads to low rates of uniform coding among different coders. Objective: To measure uniform coding among different medical experts in terms of inter-rater reliability (IR) and analyze the effect on IR by using an expert-based online code suggestion system. Methods: A quasi-experimental study was conducted. Six medical experts coded 602 medical items from structured quality assurance forms (QA) or free-text eligibility criteria (EC) of 20 different clinical trials. Medical item content was selected based on mortality-leading diseases according to WHO data. The intervention consisted of using a semi-automatic code suggestion tool that is linked to a European information infrastructure providing a large medical text corpus of more than 300,000 medical form items with expert-assigned semantic codes. Krippendorff’s alpha (Kalpha) with bootstrap analysis was used for IR analysis and coding times were measured before and after intervention. Results: The intervention improved IR in structured QA form items (from Kalpha= 0.50, 95%-CI [0.43-0.57] to Kalpha = 0.62 [0.55-0.69]) and free-text eligibility criteria (from Kalpha = 0.19 [0.14-0.24] to Kalpha = 0.43 [0.37-0.50]) while preserving or slightly reducing mean coding time per item for all six coders. Regardless of intervention, pre-coordination and structured items were associated with significant higher IR, but the proportion of items that were pre-coordinated significantly increased after intervention (EC: Odds ratio: 4.92 [2.78 - 8.72]; QA: Odds ratio: 1.96 [1.19-3.25]). Conclusions: Use of an online information infrastructure improved IR towards moderate or even substantial inter coder agreement. Pre-coordination and use of structured vs. free-text data elements are key drivers for higher IR.