This study was approved by the Minneapolis VA Health Care System Institutional Review Board (number 1594855). We used the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines to describe study procedures and findings [21].

We identified an initial list of 8 terms that clinicians might use to document fentanyl use through a combination of literature review; input from subject matter experts; and manual review of 100 EHR notes from patients with an ICD, 10th Revision (ICD-10), code for OUD. These terms included “fentanyl,” “fent,” “blues,” “M30s,” “tranq,” “tranq dope,” “sleep dope,” and “fetty.” This list was narrowed down to 4 terms (“fent,” “tranq,” “M30s,” and “blues”) that appeared at least once in our sample.

We identified all patients whose EHR documentation included at least one mention of any fentanyl term. For note sampling, we stratified patient encounters a priori into 5 groups based on the expected context in which fentanyl was mentioned, such as addressing OUD (eg, during mental health encounters), prescribed use of fentanyl (eg, during a procedure), or other clinical scenarios. We selected these settings because we hypothesized that terms and documentation styles may differ across these settings. Groups included (1) OUD-related encounters (patients with notes that included a fentanyl term and whose notes were associated with a clinical encounter with an ICD-10 code for OUD), (2) suicide or overdose event encounters (patients with a Suicide Behavior and Overdose Report [SBOR; a report created by VHA staff for all patients identified as having a suicide or overdose event] in which a fentanyl term was present in any note on or up to 90 days before the most recent SBOR entry), (3) non-OUD and noncrisis event clinical care setting encounters (patients with no SBOR or ICD-10 code for OUD who nonetheless had at least one fentanyl term in an inpatient or outpatient encounter in the following care settings: primary care, mental health, inpatient medicine, medical intensive care unit, inpatient surgery, social work, chronic pain and wellness, or homeless care team), (4) periprocedure encounters (patients with a Current Procedural Terminology code for 1 of the 10 most frequent procedures performed at the VHA [eg, tissue exam by a pathologist] within 24 hours of a fentanyl term mention), and (5) a bias mitigation sample (a random sample of patients’ notes from any encounter or note type with a fentanyl term to address potential sources of bias). We initially sampled approximately 200 notes per group randomly and without replacement. Because the low prevalence of the target class limited performance, we added 1000 notes per group to improve model performance. For each identified fentanyl term, we extracted a 250-character snippet of surrounding text, a text length selected based on prior work [22] demonstrating sufficient context for classification while maintaining computational efficiency. These snippets were then labeled using a Microsoft Azure machine learning text data labeling project [23].

We developed 3 classification labels for snippets containing fentanyl mentions: “nonprescribed fentanyl use,” “prescribed fentanyl use,” and “other.” “Nonprescribed fentanyl use” included mentions of fentanyl obtained and used without a prescription both currently and in the past. “Prescribed fentanyl use” included mentions of fentanyl that was prescribed for medical indications, including procedural sedation and transdermal pain treatment. The “other” category captured fentanyl mentions that did not meet the criteria for prescribed or nonprescribed use. This included concepts such as unspecified OUD; misuse; potential accidental exposure to fentanyl when consuming other nonprescribed drugs (eg, methamphetamines laced with fentanyl); fentanyl testing; fentanyl allergy documentation; uncertain or not enough context; negation of fentanyl use; fentanyl use by another person; standardized or templated clinical language about fentanyl, such as education materials about fentanyl risk that might be addended to a clinical note; and terms with alternative meanings unrelated to fentanyl use (eg, “blues” documented to describe depression). These subcategories were combined for modeling development because our primary objective was to identify individuals whose snippets contained reasonable clinical documentation of nonprescribed fentanyl use. Prior to annotation, a group of 6 physician annotators with experience treating patients with prescribed and nonprescribed fentanyl use created a conceptual classification schema and refined it over several sessions, in which annotators independently reviewed snippets and then met as a group to align on classification, developing consensus regarding the classification rules and definitions for each class. During annotation, each snippet was classified independently by at least 3 physicians prior to reaching a consensus classification or determining the need for further review. Interannotator agreement was assessed among the 6 physician annotators on approximately 30% of the snippets before consensus using mean pairwise unweighted Cohen κ values [24]. For snippets where annotators did not reach consensus, classification was determined through group discussion to prioritize precision over recall to minimize false positives, recognizing that in a clinical application, false positives could lead to unnecessary interventions. Snippets with consensus were used as the final dataset for training, validation, and testing of the NLP models.

On the basis of model performance in prior work [22], we reused the same nonneural and neural models and applied the same modeling framework: ridge-penalized logistic regression and Bio-ClinicalBERT. We also compared these 2 previously useful models to 2 other large language models (Mistral-7B [Mistral AI] and Llama 3-8B [Meta AI]) using instruction tuning, 10-fold cross-validation with 90% of data used for training and 10% held out for testing, and fixed random seeds to choose the initial value for parameters in each model to ensure reproducibility [25,26]. Cross-validation folds were split at the patient level to prevent overlap between training and test sets.

Model performance was evaluated using precision, recall, and F₁-scores for each class. To estimate the uncertainty in model performance, the test set was bootstrapped 100 times by resampling snippets with replacement. Performance metrics were calculated for each bootstrap sample, and the mean and corresponding 95% CIs were reported. To interpret model predictions, we implemented Shapley additive explanations (SHAP) [27]. The highest overall F₁-score from all folds was used to calculate SHAP values for the Bio-ClinicalBERT model. To address algorithmic bias, particularly in relation to demographic differences, a fairness analysis was performed on the Bio-ClinicalBERT model by stratifying each bootstrapped test set according to patient demographic characteristics, including sex (male or female), race (White or non-White), and ethnicity (Hispanic or non-Hispanic). Performance metrics were calculated separately for each subgroup, and the mean and 95% CIs were reported across the 100 bootstrap samples.

External validation was conducted to assess the real-world performance on identifying nonprescribed fentanyl use using data that were both temporally and patient independent from the original training and test sets. Using the same term-based snippet extraction method, we identified fentanyl mentions among VHA patients from January 2025 to June 2026, excluding all patients included in the original dataset. The fine-tuned Bio-ClinicalBERT model classified the extracted snippets, and 200 snippets classified as nonprescribed fentanyl use were randomly sampled, retaining the first snippet from the first note per patient so that each snippet referred to a unique patient to translate snippets into a patient-level identification. Each snippet was annotated using the same classification criteria. As identifying nonprescribed fentanyl use was the primary clinical objective, only model-positive cases were reviewed. Thus, only precision was calculated based on the proportion of confirmed patients upon manual review.

Despite recent declines in US overdose mortality, use of nonprescribed fentanyl remains a major driver of unintentional and intentional (eg, suicide) overdose deaths. An important challenge impeding clinical and research efforts is that health care systems lack methods to identify patients who use nonprescribed fentanyl. Using VHA clinical notes, we developed and validated NLP models to classify mentions as “nonprescribed use,” “prescribed use,” and “other.” Our fine-tuned models demonstrated strong performance, achieving F₁-scores above 0.8, indicating that all models tested in this study were largely comparable overall.

Although Llama 3-8B achieved the highest nonprescribed fentanyl use F₁-score (0.87), the substantially overlapping CIs among Llama 3-8B, Mistral-7B, and Bio-ClinicalBERT suggest comparable performance among these models, whereas the nonoverlapping CIs between Llama 3-8B and penalized logistic regression suggest a significant performance difference. Although external validation demonstrated comparable precision for identifying nonprescribed fentanyl use, this finding should be interpreted as preliminary because it did not validate full model performance. Each approach comes with trade-offs in terms of computation effort and cost, ability to identify patients across the health care system, and the certainty with which automated identification is truly indicative of nonprescribed fentanyl use. Our findings suggest that health systems with limited computational infrastructure or programming staff could reasonably implement penalized logistic regression or Bio-ClinicalBERT, which may provide improved identification compared with approaches relying solely on ICD-10 codes [29]. We used Bio-ClinicalBERT for supplementary analyses given its lower computational cost and comparable performance to that of larger large language models, making it pragmatic for real-world deployment. Conversely, if a health system aimed to conduct targeted outreach for patients with nonprescribed fentanyl use but lacked resources to contact all identified patients, choosing a model with higher precision (positive predictive value) may be preferred to maximize the probability that contacted patients truly have nonprescribed fentanyl use. In this scenario, Mistral-7B may be favored despite its lower F₁-score for nonprescribed fentanyl use (Table 2) as its reduced F₁-score performance was due to lower recall (0.72), whereas its precision was higher than that of Bio-ClinicalBERT or Llama 3-8B. However, this trade-off deserves careful consideration because potentially missing 28% of patients with non-prescribed fentanyl use may be detrimental to clinical applications aimed at identifying and treating at-risk patients. In these contexts, a missed identification represents a missed clinical opportunity to prevent the harms associated with nonprescribed fentanyl use.

Overall, model performance was comparable across subgroups, with the exception of female and Hispanic subgroups for the “nonprescribed fentanyl use” class, where performance was somewhat lower. This finding is likely due to the smaller sample sizes among these groups, as suggested by the wider CIs (n=138 female vs n=1344 male; n=231 Hispanic vs n=1251 non-Hispanic). These findings should be interpreted as preliminary given the small subgroup sample sizes and wide CIs and warrants further investigation in larger samples. Future work should explore approaches to improve performance in underrepresented groups, such as oversampling these subgroups during training or subgroup-specific model fine-tuning.

The “other” category was the most common classification, accounting for half (3699/7389, 50%) of the annotated snippets, highlighting the diverse ways in which fentanyl is referenced in clinical text. This heterogeneity included drug testing results, allergy documentation, fentanyl use by another person, and often ambiguous fentanyl mentions. A notable subset included incidental or unintentional exposure to nonprescribed fentanyl through the intentional use of other substances such as methamphetamine or cocaine that were laced with fentanyl, which highlights the clinical complexity of this domain. Although consolidating these subcategories into a single “other” label may bias the model by obscuring clinically relevant distinctions, such as unintentional exposure and diagnostic uncertainty, this decision was made to avoid additional class imbalance that would have precluded model convergence. Future work could explore the classification of these clinically relevant concepts.

The methods we describe could be applied to support clinical, operational, and research efforts to reduce the harms associated with nonprescribed fentanyl use among VHA patients. Our approach could also be adapted in other health systems to identify high-risk patients and inform the targeting of other interventions. For example, this information could be incorporated into risk prediction models for overdose and suicide risk [30-32]. Identifying patients at risk could also facilitate large-scale efforts in harm reduction, such as prescribing naloxone and xylazine test distribution. Additionally, health care systems could use this approach for epidemiologic surveillance of fentanyl use, identification of at-risk patients who might not be receiving appropriate care, or evaluations of clinical care processes and outcomes among this patient population. These applications may require the use of evaluation frameworks other than standard aggregate model performance reporting to ensure clinical reliability across patient populations [33]. The evaluation approach in this study reporting class-specific performance and fairness analysis reflects this standard. Additionally, these clinical tools may require more evaluation prior to integration into clinical decision-making systems, such as prospective evaluation, cost-benefit analysis, and further consideration of the risks associated with misclassification.

Aside from the potency of fentanyl, some of the mortality and negative outcomes associated with fentanyl at the population level are likely attributable to its dramatic increase in use in such a short time (and health care systems’ lack of methods to identify and treat vulnerable patients). Importantly, this study provides a blueprint to develop automated methods that can help in identifying use of other potentially emerging substances, such as use of nitazenes and other newly developed synthetic opioids.

Our findings also highlight opportunities for further research with important clinical and operational implications. All approaches we used demonstrated lower performance for identifying nonprescribed fentanyl use than for identifying the “prescribed use” and “other” categories. On the basis of review of thousands of snippets, and as reflected in Table 3, those differences are due to greater linguistic variability in how nonprescribed fentanyl use is documented in clinical notes. Prescribed use is usually documented with dosages and often in a list of other medications. Common examples of the “other” category include templated drug testing results or documentation in allergy lists. In contrast, documentation indicating actual or potential nonprescribed use is heterogeneous and may include contextual references that challenge model interpretation, such as drug use by family members, nonprescribed use of other substances, or historical fentanyl use mentioned alongside currently active use disorders for other substances such as alcohol. Although these scenarios were categorized as “other” during annotation to create gold-standard training data, their semantic similarity to nonprescribed use plausibly contributed to misclassification. Future research could address developing models to better recognize these complexities.

Our specific findings—such as that Mistral-7B had higher precision—may not be generalizable to non-VHA systems given differences in care delivery, EHR formats, documentation, and patient demographics between systems. However, these findings are potentially impactful given that the VHA provides care to nearly 10 million veterans. Furthermore, our general approach to identifying fentanyl terms, extracting clinical documentation among relevant patient groups, annotation, and training and validating models could be applied in other health care settings.

The SHAP-associated terms should be interpreted cautiously because several high-ranking features were subword tokens generated by the Bio-ClinicalBERT tokenizer rather than stand-alone clinical concepts. Manual review showed that some fragments were mapped to clinically meaningful parent terms when considered in context. For example, in the prescribed fentanyl use classifier, “verse” appeared as part of “versed” or midazolam, “ane” appeared as part of “anesthesia,” “bla” appeared as part of “ablation,” “losure” appeared as part of “closure,” and “gade” appeared as part of “Tegaderm,” all of which are plausibly associated with procedural settings where fentanyl may be prescribed or administered. However, these subword attributions should not be interpreted independently of their parent terms and surrounding note context. Therefore, SHAP analyses were used as supportive model inspection tools rather than definitive explanations of model behavior.

This study has other limitations. It relied on information documented in clinical notes, and the accuracy or completeness of these records is unknown. Additionally, the sampling strategy relied on predefined fentanyl terms, and while this term list was developed systematically, no new terms were identified during the annotation process. However, this approach may miss implicit mentions or emerging slang terms not captured at the time of development and could impact the sensitivity of a full identification pipeline in a real-world setting.

Identifying patients with nonprescribed fentanyl use is essential for patient care and informing public policy and system-level prevention and treatment strategies. As ICD-10 codes have low sensitivity for nonprescribed OUD and no codes are specific to patients with nonprescribed fentanyl use, extracting this information from EHR documentation is a crucial step in mitigating the harms of fentanyl use. We describe models with strong performance in identifying nonprescribed fentanyl use from clinical text, offering a potential tool for improving detection and intervention efforts.