This was a retrospective cohort study of all patients up to 21 years of age presenting to the ED of a large, free-standing, university-affiliated, pediatric hospital between March 1, 2020, and May 31, 2022.

The Boston Children’s Hospital Committee on Clinical Investigation performed ethical, privacy, and confidentiality reviews of the study and found it to be exempt from human subjects oversight. A waiver of consent was obtained to cover the targeted extraction and secure review of clinical notes by approved study personnel in protected environments within the hospital firewall.

The main dependent variables were a set of 11 COVID-19 symptoms based on Centers for Disease Control and Prevention (CDC) criteria [13]—fever or chills, cough, shortness of breath or difficulty breathing, fatigue, muscle or body aches, headache, new loss of taste or smell, sore throat, congestion or runny nose, nausea or vomiting, and diarrhea. We identified these symptoms by both NLP and ICD-10 codes. For the formative use case, the study period was divided into 3 variant eras defined using Massachusetts COVID-19 data from Covariant [14]. The pre-Delta era was from March 1, 2020, to June 20, 2021; the Delta era was from June 21, 2021, to December 19, 2021; and the Omicron era was from December 20, 2021, onward. A diagnosis of COVID-19 was defined as a positive SARS-CoV-2 polymerase chain reaction (PCR) test or the presence of ICD-10 code U07.1 for COVID-19 during the same ED encounter in which symptoms were evaluated.

A total of 3 reviewers reached a consensus on a symptom concept dictionary [15] to capture each of the 11 COVID-19 symptoms. They relied on the Unified Medical Language System [16], which has a nearly comprehensive list of symptom descriptors [17], including SNOMED (SNOMED International) coded clinical terms [18], ICD-10 codes for administrative billing, abbreviations, and common language for patients [19]. The open-source and free Apache cTAKES (Apache Software Foundation) NLP pipeline was tuned to recognize and extract coded concepts for positive symptom mentions (based on the dictionary) from physician notes [20]. Apache cTAKES uses a NegEx algorithm which can help address negation [20-23]. To further address negation, we incorporated an LLM, Bidirectional Encoder Representations from Transformers, that was fine-tuned for negation classification on clinical text [24,25].

A total of 2 reviewers established a gold standard by manually reviewing physician ED notes. After all notes were labeled by the cTAKES pipeline, a test set of 226 ED notes was loaded into Label Studio [26], an open-source application for ground truth labeling. These notes were from patients both with and without COVID-19 and were selected to ensure that each of the 11 symptoms was mentioned in at least 30 ED notes. Some notes mentioned more than 1 symptom. Using an annotation guide (Multimedia Appendix 1), 2 reviewers, who were masked from the terms identified by the NLP pipeline for note selection, each labeled 113 notes for mention of the 11 COVID-19 symptoms. As per the guide, only symptoms relevant to the present illness were considered positive mentions. Symptoms were not considered positive mentions if stated as past medical history, family history, social history, or an indication for a medication unrelated to the encounter.

The F₁-score was used to assess consistency in manual chart review. The F₁-score is the balance of sensitivity and positive predictive value (PPV) [27]. It was computed by comparing the annotations of each of the 2 initial reviewers to those of a third reviewer, who independently labeled a subset (56/226, 25%) of notes annotated by the other reviewers. The choice of F₁-score as the metric for agreement was informed by the observed high frequency of true negative annotations when they were assigned by chance [20,27,28]. Reliability analyses used Python (version 3.10; Python Software Foundation).

Accuracy measures of the true symptom percentages in the test set for each symptom included F₁-score, PPV, sensitivity, and specificity [29,30].

The impact of pandemic variant era on COVID-19 symptomatology was examined. Descriptive statistics were used to characterize patients presenting to the ED during each pandemic era. The percentage of patients in the ED with symptoms of COVID-19 was assessed in separate analyses for each symptom using chi-square analyses of 3×2 tables (pandemic era × symptom presence or absence) with α set at .05. Post hoc chi-square tests were used to compare each pandemic era with all others using a Bonferroni adjusted α of .017. To assess the effect of pandemic era, COVID-19 status, and the interaction of these variables on whether or not a patient had each symptom, logistic regression was used in separate analyses for each symptom. Bonferroni adjusted confidence limits were used for post hoc analyses. If the interaction term was not significant, the main effects of COVID-19 and variant era were reported. Data were analyzed using SAS (version 9.4; SAS Institute Inc).

There were 59,173 unique patients with 85,678 ED encounters during the study period. For each ED encounter, there was 1 final physician ED note that aggregated all ED physician documentation. Characteristics of the entire study cohort and variant-specific cohorts are summarized in Table 1. A patient could appear in the cohort more than once if they had multiple ED encounters.

High consistency was demonstrated between reviewer 3, who labeled a subset of notes, and both reviewers 1 and 2, who each labeled half of the notes chosen to establish the gold standard. The F₁-scores for the 2 reviewers were 0.988 and 0.984, respectively. The PPV was 0.976 and 0.968 and sensitivity was 1.0 for both.

As shown in Table 2, the F₁-score for NLP was higher and thus more accurate at identifying encounters in the test set with patients that had any of the COVID-19 symptoms than ICD-10. NLP also had higher F₁-score for each individual symptom. In addition, NLP sensitivity of true positive symptoms was higher than ICD-10. However, NLP accuracy of true negative symptoms (specificity) was somewhat lower compared to ICD-10.

The 2 most prevalent symptoms, cough and fever, had sensitivity scores for NLP that were among the highest of the symptoms, and much higher than those for ICD-10 codes. The greatest discrepancy between NLP and ICD-10 F₁-scores was for congestion or runny nose. The smallest difference was for diarrhea.

We find evidence that AI-based NLP of physician notes is a superior method for capturing patient symptoms for real-time biosurveillance than reliance on traditional approaches using ICD-10. NLP was more sensitive than ICD-10 codes in identifying symptoms and some symptoms could only be detected using NLP. As a form of internal validation, the symptoms identified by the CDC as associated with COVID-19 were more common in patients with than without this disease.

The study was also able to capture a nuanced picture of symptom prevalence and odds across different SARS-CoV-2 variant eras. Consistent with previous literature, symptom patterns changed over time as new variants emerged. Variants may present with differences in symptomatology as a result of a number of factors including differences in mutations in spike proteins, receptor binding, and ability to escape host antibodies [31]. As has been previously reported [11,32-35], we found that fever or chills were the most common COVID-19 symptom across the variants. In our cohort, shortness of breath was less common during the Omicron era than during the pre-Delta era. The Omicron variant has less of an ability to replicate in the lungs compared to the bronchi, which may explain why this symptom became less common [36]. Studies have reported sore throat as a common symptom in the Omicron era, but we did not observe a significant difference across eras [8,9]. It is possible that we did not see a higher percentage of sore throats in the Omicron era because it may be more challenging for pediatric patients to describe this symptom. One study found that sore throat was observed more often in those of 5-20 years of age compared to those of 0-4 years of age [8]. Similarly, a study reported that sore throat was more common in those greater than or equal to 13 years of age in the Omicron era compared to the Delta era [37]. In our study cohort, approximately half of the patients were younger than 5 years of age. As children this age may not be able to describe their symptoms well, symptoms that are also signs, such as fever or cough, might be more commonly documented in physician notes than symptoms such as sore throat. New loss of taste or smell was most common in the pre-Delta era, followed by the Delta era and then the Omicron era in this study. This symptom has been reported less commonly in the Omicron era [8,9]. Studies have postulated that patients with the Omicron variant are less likely to present with loss of taste or smell as this variant has less penetration of the mucus layer and therefore, may be less likely to infect the olfactory epithelium [38].

There were important limitations in our use of NLP. The NLP pipeline was tested with a set of notes where some symptoms were more frequent in the test set (eg, loss of taste or smell) than in the formative use case. This was done to have sufficient data to evaluate the symptom pipeline. The NLP pipeline does not account for vital signs and so fever may not have been detected with the pipeline if it was documented in a patient’s vital signs rather than the clinical text. The cTAKES tool in the pipeline lacks the temporal context to ascertain if the mention of a symptom in a note is a new symptom or a prior symptom. We modified our technique because of this but nevertheless may have overestimated the prevalence of symptoms in our study. Future work will involve filtering by note section so that certain components of a note like past medical history are not included. We used 2 techniques to recognize negation, but some negated symptoms (eg, “patient had no cough”) were still captured as positive symptom mentions leading to a possible overestimation of symptom prevalence. Finally, this NLP pipeline did involve substantial preprocessing. We plan to evaluate the implementation of Generative Pre-trained Transformer (GPT) for this task. GPT-4 was able to extract COVID-19 symptoms in a recent study [39] and it may limit the need for preprocessing.

Our formative study had some limitations. First, we examined COVID-19 symptoms in patients presenting to a single urban pediatric ED. Patients presenting to outpatient settings, who likely had milder symptoms, were not included and our results may reflect patients with more severe symptoms. And because the setting was a single site, results may not generalize to other EDs. Second, we defined COVID-19 status as positive if a patient had a PCR positive test for COVID-19 or an appropriate ICD-10 code at the ED encounter. Patients who were COVID-19 positive on a test at home or at an outside center may not have been captured by this definition even if they presented to the ED with COVID-19 [40]. Additionally, symptoms may have differed across variant eras as a result of COVID-19 vaccinations or previous infections rather than variant differences. Literature in adults shows that vaccination is associated with a decrease in systemic symptoms [41]. The United States Food and Drug Administration authorized the use of the COVID-19 vaccine in October 2021, during the Delta era and prior to the Omicron era, for children 5-11 years of age [42]. Vaccination rates for pediatric patients vary by age group in Massachusetts, as of April 3, 2023, of those 0-19 years of age, 3% to 57% have received a primary series but have not been boosted, and 3% to 18% have been boosted since September 1, 2022 [43]. As such, some patients in the Delta and Omicron eras may have been vaccinated or had previous COVID-19 infections [44].

In an era where rapid and accurate infectious disease surveillance is crucial, this study underscores the transformative potential of AI-based NLP for real-time symptom detection, significantly outperforming traditional methods such as ICD-10 coding. The dynamic adaptability of NLP technology allows for the nuanced capture of evolving symptomatology across different virus variants, offering a more responsive and precise tool kit for biosurveillance efforts. Its integration into existing health care infrastructure could be a game changer, elevating our capabilities to monitor, understand, and ultimately control the spread of emerging infectious diseases.