Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Accepted for/Published in: JMIR Medical Informatics

Date Submitted: Aug 13, 2019
Open Peer Review Period: Aug 13, 2019 - Oct 3, 2019
Date Accepted: Jan 26, 2020
Date Submitted to PubMed: Apr 29, 2020
(closed for review but you can still tweet)

The final, peer-reviewed published version of this preprint can be found here:

The Development of the Military Service Identification Tool: Identifying Military Veterans in a Clinical Research Database Using Natural Language Processing and Machine Learning

Leightley D, Pernet D, Velupillai S, Stewart RJ, Mark KM, Opie E, Murphy D, Fear NT, Stevelink SA

The Development of the Military Service Identification Tool: Identifying Military Veterans in a Clinical Research Database Using Natural Language Processing and Machine Learning

JMIR Med Inform 2020;8(5):e15852

DOI: 10.2196/15852

PMID: 32348287

The Development of the Military Service Identification Tool: Identifying Military Veterans in a Clinical Research Database using Natural Language Processing and Machine Learning

  • Daniel Leightley; 
  • David Pernet; 
  • Sumithra Velupillai; 
  • Robert J. Stewart; 
  • Katharine M. Mark; 
  • Elena Opie; 
  • Dominic Murphy; 
  • Nicola T. Fear; 
  • Sharon A.M. Stevelink; 

ABSTRACT

Background:

Electronic healthcare records (EHRs) are a rich source of health-related information, with potential for secondary research use. In the United Kingdom (UK), there is no national marker for identifying those who have previously served in the Armed Forces, making analysis of the health and well-being of veterans using EHRs difficult.

Objective:

The aim of this study was to develop a Tool to identify veterans from free-text clinical documents recorded in a psychiatric EHR database.

Methods:

Veterans were manually identified using the South London and Maudsley Biomedical Research Centre Clinical Record Interactive Search – a database holding secondary mental health care electronic records for the South London and Maudsley National Health Service Foundation Trust. An iterative approach was taken, first a Structured Query Language (SQL) method was developed which was then refined using Natural Language Processing and machine learning to create the Military Service Identification Tool (MSIT) to identify if a patient was a civilian or veteran.

Results:

A gold standard dataset of 6672 free-text clinical documents were manually annotated by human coders, 66% of were then used to train the SQL and MSIT approaches, and 34% used for testing the approaches. To develop the MSIT, an iterative two-stage approach was undertaken. In the first stage, a SQL method was developed to identify veterans using a keyword rule-based approach. This approach obtained an accuracy of 0.93 in correctly predicting civilians and veterans, a positive predictive value of 0.81, a sensitivity of 0.75 and negative predictive value of 0.95. This method informed the second stage, which was the development of the MSIT using machine learning, which, when tested, obtained an accuracy of 0.97, a positive predictive value of 0.90, a sensitivity 0.91 and a negative predictive value of 0.98.

Conclusions:

The MSIT has the potential to be used in identifying veterans in the UK from free-text clinical documents, providing new and unique insights into the health and well-being of this population and their use of mental healthcare services.


 Citation

Please cite as:

Leightley D, Pernet D, Velupillai S, Stewart RJ, Mark KM, Opie E, Murphy D, Fear NT, Stevelink SA

The Development of the Military Service Identification Tool: Identifying Military Veterans in a Clinical Research Database using Natural Language Processing and Machine Learning

JMIR Medical Informatics. 26/01/2020:15852 (forthcoming/in press)

DOI: 10.2196/15852

URL: https://preprints.jmir.org/preprint/15852

PMID: 32348287

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.