This retrospective observational study was conducted in the Sixth Medical Center of the Chinese People's Liberation Army General Hospital, which is a tertiary hospital that integrates medical care, teaching, and research. All inpatients aged ≥18 years admitted between January 1 and December 31, 2021, were included as the validation cohort. Hospitalized patients were excluded if they lacked diagnostic information or if their hospital stay was ≤24 hours. This study followed the Standards for Reporting Diagnostic accuracy studies (STARD) reporting guidelines (Multimedia Appendix 1) [25].

This cohort study was approved by the medical ethics committee of the Sixth Medical Center of Chinese People's Liberation Army General Hospital (approval HZKY-PJ-2022-21). A waiver of informed consent was obtained because this was a retrospective data-only study. All data were deidentified.

The rule-based NLP approach in DeVTEcare was embedded into the EHR to timely identify VTE based on the clinical text (admission notes, medical progress notes, surgical notes, discharge summaries, etc) and imaging reports. When a patient was admitted, DeVTEcare automatically assessed and flagged VTE risk, and then, the guideline-recommended VTE prophylaxis and treatments were further proposed at 5 time points: within 24 hours after admission, within 24 hours before and after operation, within 24 hours after transferring, and at discharge. Moreover, VTE risk assessment was automatically evaluated daily at midnight.

The NLP system detected VTE in 3 steps: data acquisition, information extraction, and information aggregation (Figure 1). This system employs a set of highly optimized and configurable NLP components that enable quick and easy clinical information extraction and aggregation (Table 1). All relevant full-text documents from patients were retrieved from the EHR. First, the data acquisition module encapsulated patient data into the JSON format. Second, in the information extraction process, the location of extraction was determined by the location detector. The information for the discharge confirmation of patients with VTE was extracted from the patient’s imaging report, discharge summaries, medical progress notes, and surgical notes, while the information for the admission confirmation of patients with VTE was extracted from the patient’s admission notes. Then, the sentence segmenter was used to break up the sentence, the entity recognizer was used to identify the target concept, and the context detector was used to find the target concept’s occurrence time and negation relationship. Next, the sentence inferencer was used to derive the sentence conclusion, and the document inferencer was then utilized to generate the document conclusion. Third, in the information aggregation process, the convergence of various documented findings yielded “discharged with VTE” and “admission with VTE,” which were then further computed by expressions to yield the results for new-onset in-hospital VTE.

To achieve these objectives, a VTE domain knowledge base was developed. We used disease definitions from the Chinese Guidelines for the Prevention and Treatment of Thrombophilia [26], the National Clinical Version of Disease Classification and Coding 2.0 [27], and the corresponding Chinese translations from the Unified Medical Language System from the National Library of Medicine. We also searched medical websites for corresponding synonyms to enrich the ontology concepts [28]. The knowledge base was further improved by a research team involving 2 NLP programmers, 3 medical specialists in thrombosis, and 2 database administrators with a weekly working meeting. During the iterative training phase, a small proportion of real clinical cases was manually verified by the designated physicians on a weekly basis, the NLP programmer refined the recognition error of the rules iteratively for 8 weeks, and eventually, 200 randomly selected patients were tested. During the testing phase, additional 260 patients were randomly selected for testing by the physicians. After the test was complete, the NLP approach achieved 85.7% sensitivity and 93.5% specificity for detecting VTE. Then, the NLP programmer repeatedly refined the rules based on the test results and research team’s comments until all correctable faults were fixed. The keywords for the NLP algorithm are listed in Multimedia Appendix 2.

The NLP approach for VTE detection was validated in different clinical settings and in departments with different VTE risk statuses, as follows.

The departments were further classified as low, intermediate, and high risks for VTE if there were <1, 1-5, >5 VTE-related deaths per year, respectively. The performance of the NLP approach was also evaluated with regard to the type of VTE, that is, PE or DVT. Manual review of medical records was taken as the reference standard for detecting PE and DVT. When there was a disagreement between NLP identification and the clinical expert’s review, an advanced artificial intelligence engineer and a cardiologist audited the event.

The primary end point was the performance of the NLP approach in DeVTEcare embedded into the EHR for VTE identification. The secondary end points were the performances to identify VTE in different clinical settings and in departments with different VTE risk statuses.

The diagnosis of VTE was confirmed on ultrasound doppler, computed tomography, or computed tomography pulmonary angiogram. The VTEs were classified as (1) on-admission VTE: new-onset VTE admitted to the hospital and (2) in-hospital VTE: new-onset VTE during hospitalization but without VTE on admission. Since this study primarily focused on DVT and PE, thrombosis in the upper extremity, internal jugular, and superior or inferior vena cava were not considered valid VTE. Patients with prior VTE were also deemed invalid.

Continuous variables were expressed as median (IQR), while categorical variables were expressed as numbers and percentages. The sensitivity, specificity, PPV, negative predictive value (NPV), area under the receiver operating characteristic curve (AUC), and F1-score were used to evaluate the performance of the NLP approach for VTE detection, in relation to the type of VTE, different clinical settings, and in the low-, intermediate-, and high-risk departments. Subgroup analyses were performed among age (>65 years or ≤65 years), sex (female or male), and the study season (first quarter, second quarter, third quarter, fourth quarter). Given that PPV might be affected by the prevalence of disease, positive and negative likelihood ratios (LR+ and LR–, respectively) were used to characterize the clinical utility of the test. The LR– CIs were obtained using a bootstrap method with 100% sensitivity [29].

The chi-square test or Fisher exact test was applied for the comparisons of the categorical variables, if appropriate. DeLong test was used for comparing AUCs, and Z test was used for comparing the F1-scores. The relative diagnostic LR method was used for comparing the LRs of 2 diagnostic tests [30]. The Bonferroni method was used to adjust for multiple comparisons. The agreement to the reference test and the interreviewer agreement were also assessed using Cohen κ test. A 2-sided P<.01 was considered to be statistically significant. All statistical analyses were carried out using the R software (version 4.2.1; R Core Team and the R Foundation for Statistical Computing).

A total of 30,152 patients were included from January 1 to December 31, 2021; the flow chart is shown in Figure 2. The median patient age was 56 (IQR 41-67) years, and 47.3% (14,247/30,152) of the patients were females. Half of the patients were from internal medicine departments with moderate risk of VTE. The enrolled patients were almost equally distributed throughout the 4 quarters of the study year. The median length of hospital stay was 8 (IQR 5-12) days. The incidence of VTE confirmed by the clinical experts was 2.1% (626/30,152), with PE at 0.6% (177/30,152) and DVT at 1.8% (532/30,152). The detailed incidence of VTE is shown in Table 2 and Multimedia Appendix 3. Overall, the prevalence of VTE increased with the increasing VTE risk in the departments (high risk: 219/6769, 3.2%; intermediate risk: 358/15,368, 2.3%; low risk: 49/8015, 0.6%; P<.001).

In DeVTEcare, of the 626 events, 563 (89.9%) VTEs were detected by the NLP approach, achieving a sensitivity of 89.9%, specificity of 99.8%, PPV of 91.1%, NPV of 99.8%, LR+ of 483, LR– of 0.1, AUC of 0.95, and F1-score of 0.9. The NLP approach detected the on-admission and in-hospital VTE, with sensitivity ranging from 84.4% to 100%, specificity from 99.8% to 100%, PPV from 81% to 92.3%, NPV from 99.8% to 100%, LR+ from 550 to 7115, LR– from 0 to 0.16, AUC from 0.92 to 1.00, and F1-score from 0.87 to 0.94 (Table 3, Table 4, and Figure 3). However, NLP performed better in detecting PE than DVT for on-admission VTE (sensitivity 100% vs 84.4%, NPV 100% vs 99.8%, LR+ 1579 vs 739, LR– 0 vs 0.16, AUC 1.00 vs 0.92, F1-score 0.94 vs 0.87, respectively; DeLong test for AUC; all P<.01), while much higher NPV (100% vs 99.9%, respectively; P<.001) and lower F1-score (0.87 vs 0.90, respectively; P<.001) were observed for in-hospital PE detection. No statistically significant differences in the other diagnostic parameters between PE and DVT for either on-admission or in-hospital detection were found (all P>.01) (Table 3, Table 4, and Figure 3). DeVTEcare and the expert’s review agreed well for any type of VTE identification, with κ values ranging from 0.87 to 0.94 (Table 4). Interreviewer agreement for the diagnosis of VTE showed κ values of 0.90-1.00.

Among the departments of surgery, internal medicine, and intensive care units, the highest specificity (100% vs 99.7% vs 98.8%, respectively), NPV (100% vs 99.7% vs 98.7%, respectively), LR+ (3202 vs 321 vs 77, respectively), and F1-score (0.95 vs 0.89 vs 0.92, respectively) of the NLP approach for VTE detection were in the surgery department (all P<.001; Bonferroni adjusted P=.003). No statistically significant differences were found in sensitivity, PPV, LR–, and AUC (all P>.01; Bonferroni adjusted P=.003) (Table 5, Table 6, and Figure 4).

The NLP algorithm consistently detected VTE in low, intermediate, and high-risk departments with good sensitivity and specificity (sensitivity 100%, 87.4%, and 91.8%, respectively; specificity 100%, 99.8%, and 99.6%, respectively), although the difference of specificity between low- and high-risk departments was statistically significant (P<.001; Bonferroni adjusted P=.003). Among low-, intermediate-, and high-risk departments, the low-risk department had the highest AUC (1.00 vs 0.94 vs 0.96, respectively), F1-score (0.97 vs 0.90 vs 0.90, respectively), NPV (100% vs 99.7% vs 99.7%, respectively), and the lowest LR– (0.00 vs 0.13 vs 0.08, respectively) (DeLong test for AUC, all P<.001; Bonferroni adjusted P=.003). The highest LR+ was associated with the low-risk department as well (2655 vs 525 vs 223, respectively), but a statistically significant difference existed only between the low- and high-risk departments (P<.001; Bonferroni adjusted P=.003) (Table 5, Table 6, and Figure 4). The error analysis of DeVTEcare in detecting VTE is shown in Table 7.

Subgroup analysis of the age, sex, and quarter of the study year demonstrated consistently good performance of the NLP tool in VTE detection with >87% sensitivity, specificity, PPV, and NPV, and >89% of AUC and F1-score. The NLP tool detected VTE in patients aged ≤65 years much better than that in patients aged >65 years (specificity 99.9% vs 99.5%; NPV 99.9% vs 99.5%; LR+ 1494 vs 178; F1-score 0.93 vs 0.89; respectively, all P<.001) (Table 5, Table 6, and Figure 4).

We developed an NLP algorithm with multisource EHR data and validated VTE detection extensively, in relation to different clinical settings with different VTE risks. The main findings of our study were as follows: (1) the NLP algorithm of DeVTEcare facilitated VTE detection with good diagnostic ability; (2) the NLP algorithm consistently detected the on-admission and in-hospital VTE well; however, the NLP algorithm showed higher sensitivity in detecting PE than DVT on admission; (3) among departments of surgery, internal medicine, and intensive care units, the highest specificity and NPV were shown in the surgery department; and (4) the NLP algorithm performed better among departments with low-risk VTE and among patients aged ≤65 years.

NLP can identify unstructured language that documents the diagnosis or findings of interest. Previous studies [10-13,16-24,31,32] on NLP embedded into EHRs to identify VTE were mainly based on imaging reports. Those studies reported about 94% sensitivity and 89% PPV for patients without surgery but only 81% sensitivity and 54% PPV for patients who had undergone a surgery (Multimedia Appendix 4) [10-13,16-24,31,32]. The NLP algorithm in our DeVTEcare not only confirmed good sensitivity, specificity, PPV, and NPV for VTE detection for patients admitted to internal medicine units but also for patients in the surgery department with higher specificity, NPV, and F1-scores compared to those for patients in the internal medicine and intensive care units.

Surgical procedures are associated with increased VTE risk, and certain surgeries, for example, orthopedic surgery, major general surgery, gynecological surgery, urological surgery, and neurosurgery are considered to pose a higher risk for VTE. Our algorithm for VTE detection was extensively validated in these clinical settings and was found to have good diagnostic ability. Our algorithm could help clinicians increase the awareness and prevention of VTE. The good performance of our NLP algorithm may be associated with the use of multisource data from EHRs besides imaging reports and the application of a set of highly optimized and configurable NLP components that enable rapid and straightforward clinical information processing. Moreover, the NLP algorithm of DeVTEcare demonstrated a better identification of on-admission PE than DVT, which was rarely reported by previous studies [19,20]. PE is a more severe clinical issue than DVT; nonetheless, it can be the preventable cause of hospital death. Thus, early identification of patients with PE on admission is critical. The adoption of adequate treatments in such patients during hospitalization would help to reduce adverse outcomes. Since the prevalence of VTE could vary across age, gender, season, and VTE risk, the performance of the NLP algorithm may be affected. However, our study showed that our NLP algorithm performed favorably in VTE identification across all subgroups. In addition, we found that the NLP algorithm detected VTE events even better for patients from departments at low VTE risk and those aged ≤65 years. This could be because VTE occurred less often in these patients, but the NLP system still recognized VTE well and thus showed better specificity while maintaining relatively high sensitivity. The NLP algorithm of DeVTEcare simplified and streamlined the screening of VTE for inpatients in different departments with different risks of VTE, and thus, it can be a user-friendly tool to facilitate the prevention of hospital-associated VTE and related morbidity.

Finally, the error analysis of our NLP tool for detecting VTE demonstrated that 81% of the false negative events were asymptomatic isolated calf muscle vein thrombosis and iliac vein thrombosis. This could partly explain the higher sensitivity of our NLP tool in PE detection than DVT detection on admission. In addition, 95% of the false positives were from patients with prior VTE or those with suspected VTE on admission but without confirmation of a diagnosis. The results from error analysis suggested that the text mining of key phrases needs to be enhanced furthermore.

There are several limitations in our study. First, although we tested the NLP algorithm extensively in different clinical settings, we cannot confirm if the findings of our study can be generalized to other hospitals, given the possible differences in the EHR structure, document formatting, and local terminologies. However, the rule-based NLP tool that employs a set of highly optimized and configurable NLP components can be easily tuned with a modest quantity of training data. Second, the VTE risk in the hospital departments was classified by the hospital quality control administration and we did not calculate the incidence rate of VTE, which is the standard procedure in epidemiology studies. Third, some physician-related factors such as the physician’s medical record writing style, awareness of VTE prevention and treatment, and imaging report quality have the potential to influence the NLP algorithm’s ability to identify VTE. Nonetheless, the hospital in our study is an academic teaching hospital in Beijing, which provides the highest quality of medical care in China. Fourth, given the retrospective design of this study, there was possible selection bias, which needs further external validation.

The NLP algorithm of DeVTEcare identified VTE with good diagnostic ability across different clinical settings, especially for the surgery units, departments with low-risk VTE, and patients aged ≤65 years. This algorithm permits us to inform the accurate in-hospital VTE rates and enhance risk-classified VTE integrated care in future research.