This study was designed as a population-based, real-world retrospective cohort analysis using data derived from the EMS system in Nanning, Guangxi Zhuang Autonomous Region, China. All emergency calls are centrally coordinated through the Nanning Emergency Medical Center, which operates the unified municipal dispatch system and provides prehospital emergency care across both urban and surrounding suburban areas.

The EMS database integrates dispatch information with standardized electronic prehospital medical records completed by on-scene physicians, allowing linkage between call-level information and clinical outcomes. This structure enabled comprehensive capture of both structured variables and free-text narratives generated during the dispatch process and subsequent patient assessment.

All EMS dispatch records between January 1, 2021, and December 31, 2025, were screened. Patients were eligible for inclusion if the emergency call was associated with symptoms suggestive of acute cardiopulmonary conditions, identified through predefined dispatch complaint categories and keyword-based filtering of call narratives (based on standard EMS dispatch guidelines). These included presentations such as chest pain, dyspnea, syncope, and altered consciousness. We excluded records related to trauma, nonmedical events, interfacility transfers, and cases with missing key variables, including dispatch narratives or outcome labels. Given the extremely low rate of missing data (<1%), a complete case analysis approach was used. After applying these criteria, a total of 38,523 cases were included in the final analysis.

To simulate real-world prospective deployment and minimize information leakage, a temporal validation strategy was adopted. Data from January 2021 to December 2024 were used as the training cohort (n=28,332) for model development and internal validation, while data from January to December 2025 were reserved as an independent temporal test cohort (n=10,191). This approach allowed the evaluation of model performance under natural variations in case mix and EMS utilization over time.

The primary outcome of interest was a composite end point uniformly defined as a “critical outcome.” This label was derived from standardized on-scene medical records completed by EMS physicians and reflected patients experiencing a critical outcome requiring urgent intervention. Operationally, these outcomes were recorded by attending EMS physicians using electronic prehospital care reports immediately following the mission. Data standardization was enforced by municipal EMS protocols, and validity was supported by routine electronic logic checks and administrative quality assurance within the EMS database. To improve objectivity and reproducibility, the composite end point was anchored to operational EMS records, including documented on-scene death, requirement for ALS interventions (eg, airway management, cardiopulmonary resuscitation), or persistent instability necessitating immediate emergency department resuscitation upon hospital arrival. These criteria were based on structured EMS documentation fields to reduce interphysician variability.

Specifically, outcome labels were generated using predefined operational criteria and determined using standardized structured fields in the prehospital electronic medical record rather than subjective free-text descriptions, thereby improving interrater consistency. Critical outcomes included (1) death at the scene, (2) persistent hemodynamic or respiratory instability (eg, systolic blood pressure persistently <90 mm Hg requiring vasopressors, or peripheral capillary oxygen saturation [SpO2] <90% despite high-flow oxygen) despite initial resuscitative efforts, and (3) refractory or rapidly worsening clinical status necessitating immediate escalation of care. This composite end point was chosen to capture clinically meaningful high-risk cases that would benefit from early identification and optimized prehospital preparation. Although not formally validated as a standardized end point, this composite outcome reflects the occurrence of operationally relevant critical outcomes and has been widely adopted in EMS-based risk stratification studies focusing on early decision-making.

To prevent information leakage, the dataset was temporally partitioned into training (2021‐2024) and test (2025) cohorts before vocabulary construction, feature selection, and model development. Free-text data, specifically the dispatch narratives documented by call-takers during the emergency call, were processed using a structured NLP pipeline. Prior to feature extraction, all text entries underwent standardized preprocessing, including removal of punctuation, special characters, and noninformative stop-words (using a customized Chinese medical stop-word dictionary), to reduce noise and improve signal consistency. Crucially, the NLP pipeline, including vocabulary construction and selection of the top 300 most frequent n-grams, was developed exclusively using the training cohort. The independent temporal test cohort remained isolated throughout model development and was subsequently transformed using only the predefined vocabulary and preprocessing rules derived from the training data. No information from the temporal test cohort was used during feature engineering, model training, or hyperparameter tuning.

The cleaned text was then transformed into character-level n-grams, specifically bigrams, to capture local semantic patterns within short phrases commonly used in emergency communication. Given that the dispatch narratives were recorded in Chinese, this character-level tokenization strategy was deliberately used. It effectively bypasses the inaccuracies commonly associated with Chinese word segmentation tools when parsing highly colloquial, fragmented, or abbreviated emergency medical texts, allowing for direct capture of native semantic structures. To ensure computational efficiency and reduce dimensionality, feature selection was performed based on term frequency across the training corpus. The top 300 most frequent n-grams were retained as the core feature set. These selected features were subsequently encoded into a structured numerical matrix. Given the ultra-short nature of emergency dispatch narratives, where term recurrence within a single record is exceedingly rare, a binary encoding scheme (presence vs absence) was used. In this specific short-text context, binary encoding effectively captures the essential semantic signal (functioning as a Boolean term frequency) without the added complexity of global weighting schemes such as term frequency-inverse document frequency (TF-IDF).

While advanced deep learning architectures such as transformer-based large language models (eg, bidirectional encoder representations from transformers, BERT) are increasingly used in medical informatics, a lightweight, explainable, n-gram-based approach was deliberately selected for this study. Dispatch narratives are inherently highly fragmented, abbreviated, and lack complex grammatical structures, rendering the contextual advantages of deep learning models marginal. Furthermore, the explicit n-gram matrix provides greater semantic transparency, which enhances the interpretability of downstream machine learning predictions—a mandatory prerequisite for building algorithmic trust in high-stakes emergency triage. Importantly, feature selection (top 300 n-grams) was performed exclusively within the training dataset to prevent information leakage into the temporal test cohort.

To leverage both structured and unstructured information, a multimodal modeling framework was constructed by combining the NLP-derived features with baseline structured variables, including age, sex, and call time. Given the class imbalance in the training cohort, with a relatively lower proportion of critical outcomes, class weighting was incorporated into the model training process to enhance sensitivity toward high-risk cases. Specifically, the scale_pos_weight hyperparameter in the Extreme Gradient Boosting (XGBoost) algorithm was defined as the ratio of negative (noncritical) to positive (critical outcome) in the training set.

The primary model was developed using XGBoost, selected for its ability to handle nonlinear relationships and interactions. Hyperparameter tuning (including parameters such as maximum depth, learning rate, and subsample ratio) was conducted via grid search using 5-fold cross-validation exclusively within the training cohort to optimize the final model parameters. For benchmarking purposes, additional models, including random forest and penalized logistic regression, were also trained under the same framework. To ensure a fair comparison and reproducibility, all model hyperparameters were optimized within the training set only, without access to the temporal test cohort.

Model performance was evaluated on the independent 2025 temporal test cohort. Discrimination was assessed using the area under the receiver operating characteristic curve (AUROC). Given the class imbalance, additional metrics, including the area under the precision-recall curve (AUPRC) and F₁-score, were calculated to provide a more comprehensive assessment of model performance. Calibration was evaluated using the Brier score.

To enhance interpretability, Shapley Additive Explanations (SHAP) were applied to quantify the contribution of individual features to model predictions, allowing the identification of key textual patterns associated with increased risk.

To assess potential clinical applicability, a risk enrichment analysis was performed by ranking patients according to predicted risk and evaluating the proportion of true critical outcomes captured within predefined high-risk strata (top 5%, 10%, and 20%). In addition, decision curve analysis (DCA) was conducted to estimate the net benefit across a range of threshold probabilities, comparing the model against default strategies of treating all or no patients as high risk.

Continuous variables were presented as medians with IQRs and compared using the Mann-Whitney U test, given their nonnormal distributions. Categorical variables were summarized as counts (percentages) and compared using the Pearson chi-square test or Fisher exact test, as appropriate.

For model performance evaluation, 95% CIs for the AUROC were computed. Statistical comparisons of discriminative performance between the baseline structured model, the NLP-only model, and the multimodal fusion model were conducted using the DeLong test. Given the inherent class imbalance in critical outcomes, the AUPRC and F₁-scores were additionally calculated as robust evaluation metrics. The clinical usefulness of the predictive models was assessed using DCA to quantify the net benefit across a range of threshold probabilities.

All data preprocessing, machine learning development, and statistical analyses were performed using R software (version 4.3.2; R Foundation for Statistical Computing). Key R packages used included xgboost for model training, pROC for ROC analysis and DeLong tests, dcurves for DCA, and relevant libraries for SHAP value computation and visualization. A 2-sided P value <.05 was considered statistically significant.

This study was approved by the Institutional Review Board of Guangxi Zhuang Autonomous Region People’s Hospital (approval number KY-KJT-2024‐239). The requirement for informed consent was waived because of the retrospective study design and the use of anonymized EMS data.

Additional methodological details and model specifications are provided in Multimedia Appendix 1. The completed TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) reporting checklist is available in Checklist 1.

A total of 38,523 EMS dispatch records with available free-text narratives were included in the final analytic cohort. Historical data from 2021 to 2024 (n=28,332) were used for model development, while an independent temporal cohort from 2025 (n=10,191) served as the temporal validation dataset (Figure 1).

The flow diagram illustrates the patient selection process and the dual-track methodological framework. From an initial pool of 38,562 dispatch records screened via cardiopulmonary symptom keywords, 38,523 eligible cases were included after excluding records with missing dispatch narratives or unknown clinical outcomes. To simulate real-world prospective deployment and prevent data leakage, the cohort was temporally split into a historical training dataset (2021‐2024, n=28,332) and an independent temporal test dataset (2025, n=10,191). The diagram outlines the NLP pipeline, machine learning development, and comprehensive model evaluation strategies.

Baseline demographic characteristics were highly comparable between the 2 cohorts (Table 1). The median patient age was 68.0 (IQR 54.0‐79.0; P=.52) years in both groups, and the proportion of male patients was similar (n=16,872, 59.6% vs n=6147, 60.3%; P=.18). The temporal distribution of EMS calls was also consistent, with a median call hour of 13:00 (IQR 8.0-18.0) in both cohorts (P=.14). Structured major complaint categories, including dyspnea (n=10,265, 36.2% vs n=3599, 35.3%; P=.11), syncope (n=6692, 23.6% vs n=2367, 23.2%; P=.44), chest tightness (n=5082, 17.9% vs n=1877, 18.4%; P=.27), and chest pain (n=2387, 8.4% vs n=894, 8.8%; P=.28), showed stable distributions across the 2 periods.

However, temporal dataset shifts were observed in the underlying clinical severity. The prevalence of critical outcomes decreased from 35.3% (n=10,012) in the historical training cohort to 24.2% (n=2464) in the 2025 temporal validation cohort (P<.001; Figure 2). Furthermore, the prevalence of certain NLP-extracted latent semantic signatures, such as second-hand reporting (n=11,070, 39.1% vs n=4199, 41.2%; P<.001) and convulsions (n=9129, 32.2% vs n=3651, 35.8%; P<.001), exhibited significant temporal fluctuations. The presence of this natural dataset shift provided a stringent real-world environment for evaluating model robustness under changing clinical distributions.

Structured triage variables alone demonstrated improved but still limited predictive ability for identifying critical outcomes in the temporal validation cohort. An expanded structured baseline model incorporating age, sex, call hour, and routinely available structured complaint categories (dyspnea, syncope, chest pain, and chest tightness) achieved an AUROC of 0.681 (95% CI 0.669‐0.693; Table 2). In contrast, models incorporating unstructured dispatch narratives substantially improved predictive discrimination beyond the expanded structured baseline model. The NLP-only model achieved an AUROC of 0.803 (95% CI 0.792‐0.814). The best performance was observed in the multimodal model integrating both structured variables and NLP-derived textual features, which achieved an AUROC of 0.808 (95% CI 0.798‐0.818; Table 2, Figure 3A).

Given the outcome prevalence of 24.2% in the temporal test cohort, precision-recall analysis was additionally performed to assess model performance under class imbalance. The multimodal model achieved an AUPRC of 0.630 (95% CI 0.611‐0.651), markedly exceeding the baseline prevalence of 0.242 (Figure 3B). Across classification metrics, the multimodal model consistently outperformed other approaches, achieving the highest F₁-score (0.637) and an improved positive predictive value (PPV 0.571) compared with the expanded structured baseline model (PPV 0.341) (Table 2, Figure 3C).

Comparative evaluation across 3 machine learning algorithms—XGBoost, random forest, and penalized logistic regression—demonstrated consistent predictive performance (Figure 4A). The XGBoost model achieved the highest discrimination (AUROC 0.808), closely followed by the random forest model (AUROC 0.807), while penalized logistic regression yielded slightly lower performance (AUROC 0.794).

These similar performances across different modeling approaches suggest that the predictive signal primarily originated from the textual features themselves rather than from a specific algorithmic architecture. Subgroup analyses further demonstrated stable model performance across demographic and spatial groups (Figure 4B). The multimodal model achieved AUROC values of 0.825 in patients younger than 65 years and 0.793 in those aged 65 years or older. Performance was also comparable between male (AUROC 0.803) and female (AUROC 0.811) patients, as well as between urban (AUROC 0.805) and suburban or rural (AUROC 0.812) dispatch locations.

Model interpretability was examined using SHAP analysis to identify the most influential textual features contributing to predictions. The SHAP summary plot revealed several clinically meaningful linguistic patterns associated with critical outcomes (Figure 5A).

Among the most influential features were terms describing impaired consciousness or respiratory distress, including “unresponsive,” “breathing,” “coma,” “no response,” “heartbeat,” and “dizziness.” Because isolated n-gram tokens can sometimes lack clinical context, reviewing the original source phrases provides better interpretability. For example, the high-risk token “heartbeat” predominantly appeared in dire phrases such as “has no heartbeat” or “heartbeat stopped.” Similarly, “breathing” was often extracted from urgent reports such as “can’t breathe” or “stopped breathing.” Conversely, features associated with lower risk, such as “conscious,” typically appeared in reassuring statements such as “the patient is still conscious.” These descriptors correspond to symptoms frequently reported by callers or bystanders when patients present with severe cardiopulmonary compromise. Notably, several influential features reflected second-hand descriptions from witnesses, suggesting that the inability of the patient to communicate directly may itself function as an important contextual signal of clinical severity. The semantic patterns identified by the model were consistent with established clinical indicators of critical illness and support the interpretability of the NLP-based approach (Figure 5B).

To evaluate the potential operational value of automated narrative-based triage, a risk enrichment analysis was conducted using the 2025 temporal validation cohort. When the model prioritized the top 10% (n=1019) of dispatch calls with the highest predicted risk scores, 32.1% (n=791) of all true critical outcomes were captured, representing a 3.2-fold enrichment compared with random allocation (Figure 6A).

Expanding the prioritization threshold to the top 20% (n=2038) of calls captured 51.9% (n=1279) of the critical outcomes, corresponding to a 2.6-fold enrichment. These findings suggest that even modest prioritization thresholds could substantially concentrate high-risk patients within a limited subset of EMS dispatch calls.

Based on the outcome prevalence of 24.2% in the temporal validation cohort, the null model yielded a Brier score of 0.183. The multimodal model achieved a Brier score of 0.188 (95% CI 0.185‐0.191, estimated via 1000 bootstrap resamples), with a calibration intercept of −1.27 and a calibration slope of 1.30. These findings indicate imperfect probability calibration (Figure 6B). The observed calibration deviation is likely attributable to the substantial prevalence shift between the development and temporal validation cohorts, together with the intentional class weighting strategy used during model development. Because the primary objective of this study was to evaluate temporal transportability and risk ranking of the original model, post hoc recalibration was not applied to the temporal validation cohort. Furthermore, DCA showed that the NLP-enabled multimodal model consistently provided greater net benefit than both treat-all and treat-none strategies across clinically relevant threshold probabilities (Figure 6C). Together, these results indicate that automated analysis of dispatch narratives may offer practical value for early risk stratification and optimized allocation of prehospital emergency resources.

In this population-based study of 38,523 EMS dispatches for suspected acute cardiopulmonary symptoms, we evaluated a narrative-based approach to early risk stratification using routinely collected dispatch texts. The findings suggest that information embedded in free-text narratives is associated with clinically relevant risk signals that are not captured by conventional structured variables. When incorporated into a multimodal framework, these semantic features were able to improve risk discrimination and remained relatively stable under temporal dataset shift. Rather than representing a stand-alone solution, these results indicate that narrative-derived features may serve as a complementary component within existing prehospital triage systems. Although the underlying NLP and machine-learning methods are well established, the primary contribution of this study lies in the rigorous temporal validation of this framework and in demonstrating its potential clinical applicability within a real-world EMS dispatch setting. These findings support the value of adapting established, interpretable machine learning methods to clinically relevant implementation challenges rather than introducing a novel prediction algorithm.

Early risk stratification is a central objective of prehospital care, particularly for acute cardiopulmonary presentations where delays in recognition may directly affect downstream management [15]. Existing dispatch systems largely rely on predefined symptom categories, structured checklists, or dispatcher experience. While such approaches provide operational consistency, they are inherently limited in capturing the complexity and evolution of acute illness prior to patient contact [16]. In parallel, there has been increasing interest in applying machine learning and NLP in clinical settings, including electronic health records, radiology reports, and clinical documentation, where unstructured text has been shown to contain meaningful prognostic information [17]. However, the use of narrative data at the dispatch level remains relatively underexplored, despite being the earliest available clinical description in the care pathway [18].

Previous work in EMS has primarily focused on structured data, such as response intervals, demographic characteristics, or predefined complaint codes, to support triage and system optimization [19]. Some studies have explored prediction models for cardiac arrest recognition or dispatch prioritization, but these approaches often rely on limited input features or are developed within static datasets [20,21]. In contrast, narrative-based approaches leverage the variability and richness of real-world communication to provide complementary prognostic perspectives on patient status.

The multimodal model remained superior even after strengthening the structured comparator by incorporating routinely available complaint categories, indicating that narrative-derived features provide clinically meaningful information beyond conventional structured dispatch variables. The observed association between narrative features and clinical outcomes likely reflects the way acute illness is described and perceived during emergency calls. Unlike structured variables, which represent relatively static patient characteristics (eg, age and sex), narrative descriptions capture dynamic information regarding symptom onset, progression, and contextual observations [22]. The modest incremental improvement achieved by multimodal integration suggests that narrative-derived features may already capture much of the clinically relevant information required for early risk stratification, thereby limiting the additional predictive contribution of basic structured variables when dispatch narratives are sufficiently informative. Nevertheless, structured variables may still provide complementary value in situations where narrative information is limited or incomplete, for example, because of communication difficulties or very brief emergency calls [23,24]. Taken together, these findings suggest that incorporating free-text dispatch narratives alongside structured information may improve early risk assessment and support more informed allocation of EMS resources. Crucially, transformer-based architectures have demonstrated excellent performance across a wide range of clinical NLP tasks by capturing complex contextual relationships. However, emergency dispatch narratives are typically short, fragmented, and highly abbreviated, and the incremental benefit of large language models in this setting remains uncertain. In addition, lightweight n-gram–based models offer greater interpretability through explicit feature attribution and generally require substantially fewer computational resources, which may facilitate future implementation in resource-constrained EMS dispatch environments. Future studies directly comparing lightweight and transformer-based approaches across diverse EMS datasets will be valuable to better define their respective strengths and optimal application scenarios [25,26].

From a clinical and operational perspective, the potential value of narrative-informed risk stratification lies in its timing [27]. Decisions regarding dispatch priority, resource allocation, and destination planning are often made before direct patient assessment. In this context, even modest improvements in early risk identification may influence preparedness at multiple levels, including equipment readiness, crew configuration, and prenotification of receiving facilities [28]. Rather than replacing existing triage protocols, a narrative-based risk signal could be integrated as an additional layer of information to support decision-making under uncertainty [29]. This may be particularly relevant in settings with constrained ALS resources, where aligning resource intensity with patient risk remains a persistent challenge [30].

Several limitations should be considered when interpreting these findings. First, the analysis depends on the accuracy and completeness of dispatch narratives, which may vary according to dispatcher training, caller communication ability, and situational stress. Misclassification or information loss at the transcription stage cannot be excluded. Second, although temporal validation was performed, the data were derived from a single EMS system in China. Differences in language, dispatch workflows, documentation practices, and health care system organization may limit the applicability of the proposed model to other regions and countries. Third, the outcome definition is based on prehospital assessments and does not capture in-hospital end points such as definitive diagnosis, survival, or neurological recovery. Furthermore, our composite “critical outcome” relies on local EMS operational criteria and intervention thresholds. Although the use of standardized structured fields likely improves internal consistency within our system, these operational practices may vary across different regional EMS networks, potentially affecting the external generalizability of our outcome definition. However, from an EMS operational perspective, the occurrence of a prehospital critical outcome represents the most direct and actionable proxy for dispatch triage because it directly informs decisions regarding on-scene ALS and transport prioritization, independent of downstream in-hospital management. Fourth, our NLP pipeline relied on character-level n-grams and tree-based ensemble models. Although these methods offer advantages in interpretability and computational efficiency, they may not fully capture complex semantic relationships compared with modern pretrained language models. Finally, although the lightweight architecture suggests favorable computational efficiency, deployment-oriented characteristics were not formally evaluated in this retrospective study. Therefore, our findings support the potential for future implementation rather than confirmed operational deployment. Future multicenter studies incorporating external validation, prospective workflow evaluation, and implementation-oriented engineering assessments will be essential to determine the feasibility of integrating this framework into routine EMS dispatch practice.

Free-text dispatch narratives contain clinically relevant information that is associated with early risk stratification in patients with suspected cardiopulmonary emergencies. Incorporating these narrative features into a structured modeling framework may provide additional insight beyond conventional variables and support decision-making at the dispatch stage. Further validation and prospective evaluation are needed to determine how such approaches can be integrated into routine prehospital care. These findings support the potential value of incorporating explainable, NLP-derived information into EMS dispatch systems, providing a foundation for future multicenter validation and prospective implementation studies.