The rapid spread of SARS-CoV-2 and its associated disease, COVID-19, has created a pressing need for accurate and timely diagnostic tools. Traditional diagnostic methods, such as polymerase chain reaction tests, while reliable, often involve invasive procedures and can be time consuming. Consequently, there is a growing interest in developing additional diagnostic approaches that are noninvasive, affordable, scalable, and capable of delivering swift results [1].

Deep learning models have demonstrated exceptional capabilities across various domains, including medical diagnostics [2-8] and epidemiological surveillance [9]. Studies have illuminated the potential of harnessing deep learning techniques for analyzing diverse data sources, such as clinical and biological biomarkers, computed tomography scan imagery, and clinical characteristics, to predict the severity and progression of COVID-19 [10-13]. In recent studies, the analysis of cough sounds has shown potential as a noninvasive modality for COVID-19 detection [14-16]. Leveraging the power of deep learning, these models can extract crucial information from acoustic characteristics, aiding in the early identification of individuals who are infected.

However, in practice, the performance of deep learning models tends to decline during deployment and shows further deterioration over time. This phenomenon, commonly known as model degradation, can be attributed to various factors [17]. One contributing factor is the limited representation of the training data, which fails to capture the complexity of the problem space adequately. Consequently, the model exhibits unexpected behavior when confronted with input samples lying outside the distribution of training examples [18,19]. Another significant factor is the dynamic nature of the system’s environment, which undergoes continuous changes over time [18], posing challenges for a single model to maintain accurate predictions consistently. This factor is particularly critical in the context of COVID-19, given the rapid and unpredictable changes due to several reasons, including the emergence of new viral strains.

The literature refers to these 2 aforementioned factors as concept drift, which is the phenomenon where the input data and their relationship to the labels undergo changes over time. Numerous attempts have been made in the past decade to precisely define concept drift [17,20-23]. In this paper, the definition from the study by Lu et al [22] is adopted, which states that concept drift occurs when either the data distribution changes, the underlying relationship between the input and output changes, or both change.

In the context of respiratory diseases, several previous studies have acknowledged the limitations posed by concept drift in crowdsourced respiratory datasets, highlighting the variability introduced by self-reported ground truth labels, the lack of clinical validation, and the evolving symptomatology of different variants of SARS-CoV-2 [24]. These factors contribute to performance degradation and uncertainty in the extracted features. In addition, dataset biases due to demographic imbalances and variations in symptom severity further complicate model reliability over time [25].

Researchers have recognized the importance of addressing these challenges and have focused on learning in nonstationary environments [26] and mitigating the impact of concept drift [27-30]. Research studies have stressed the importance of integrating a model degradation detector within the learning framework [31] that assesses and tracks the system’s performance after deployment to effectively manage the degradation in prediction accuracy. The level of degradation in the model performance serves as an indicator for detecting concept drift within the system. By incorporating these detection components, deep learning systems develop resilience against environmental changes, thereby mitigating the performance degradation of predictive models in this ever-changing setting.

Because the presence of concept drift between training data and real postdevelopment data impedes the performance of deep learning models on out-of-distribution samples [27], applying the model on new data may necessitate adaptation. Automatic methods have emerged to tackle these challenges; however, collecting large-scale labeled datasets for different populations, emerging virus variants, or new pandemics is an arduous task. When working with limited data, it is often necessary to use more cost-efficient deep learning methodologies, such as unsupervised domain adaptation (UDA) and active learning (AL).

Domain adaptation is a technique used to address the limited generalization ability of predictive models when the training and testing data come from different distributions [32]. The goal is to adapt a model trained on a source domain to perform well on a target domain. This involves minimizing the distribution gap between the domains through learning domain-invariant features [33,34], weighing samples based on similarities [35], or using model-based techniques such as domain adversarial networks [36,37]. These approaches can improve model generalization in real-world scenarios with varying data distributions, as they enable learning from labeled data in the development set, which refers to the past, and applying this knowledge to solve tasks on postdevelopment unlabeled data.

AL, by contrast, is a machine learning approach where informative samples from a large, unlabeled dataset are selected and labeled iteratively to train a model. The objective is to minimize the amount of labeled data needed while maximizing the model’s performance [38-40]. A query strategy is selected to determine which unlabeled samples should be labeled. Various strategies exist, such as uncertainty sampling [41,42] or diversity sampling [43]. On the basis of the initially trained model, the chosen query strategy is applied to the unlabeled dataset, identifying the most informative samples based on the selected criterion. These selected samples are labeled either manually by domain experts or through an automated process. The newly labeled samples are incorporated into the labeled dataset, which is used for retraining the model.

Most recent studies focusing on COVID-19 detection based on the use of audio recordings have primarily used supervised deep learning techniques [15,16,44-50]. It is noteworthy that in most of these studies, aspects related to the existence of concept drift and the challenges of model generalization have not been addressed. In the study by Han et al [16], a rather complex model architecture has been presented with a deeper fully connected part, enhancing performance when multiple modalities (cough, voice, and breathing) are used but leading to reduced accuracy when relying solely on cough recordings. While the study by Han et al [16] acknowledges potential biases and dataset limitations, its analysis does not explicitly consider temporal aspects, such as how biases or drifts emerge over time as new data are collected.

Earlier attempts to address data drift issues in the context of respiratory diseases, including COVID-19, have primarily relied on transfer learning techniques to mitigate performance degradation [51,52]. However, these methods often assume that the source and target domains share a strong underlying similarity, which may not hold when data distributions drift significantly, potentially leading to poor performance. In general, the exploration of methods relying on concept drift detection, model degradation detection, UDA, and AL has been limited [29,53]. A few recent approaches have investigated adversarial domain adaptation to enhance model generalizability across datasets. For instance, in the study by Nguyen et al [37], a domain adaptation framework for respiratory symptom detection has been proposed, concentrating on static cross-dataset generalization. AL has also been explored as an efficient strategy for improving model performance while minimizing the labeling burden, particularly in resource-constrained scenarios such as pandemic response. In the study by Wu et al [40], a deep AL framework has been developed for COVID-19 diagnosis from computed tomography scans, using a hybrid sampling strategy to optimize labeling efforts. Thus, UDA and AL approaches appear to be highly promising and well-suited for addressing the continuously evolving nature of pandemics, as they enable the development of reliable models with the potential to address even the emergence of novel virus variants.

In this paper, a comprehensive framework is introduced for the diagnosis of infectious diseases, focusing on COVID-19 detection from cough sounds. The framework leverages deep learning models combined with UDA- and AL-based methodologies to monitor and mitigate model degradation and concept drift. The development and evaluation of the proposed framework is demonstrated on the COVID-19 data due to their continuously evolving epidemiological and virological characteristics, arising from the complex interplay among the virus, humans, vaccines, and environments. The maximum mean discrepancy (MMD) [54] distance is first used as a metric to quantify the dissimilarity between temporal data distributions. By monitoring the MMD distance between batches of postdevelopment data and data from the initial development period, the framework detects changes in both the data and the model’s performance while also providing insights into the impact of the pandemic’s evolution on the trained models’ diagnostic accuracy. If concept drift is detected, a retraining process is initiated, including two adaptation methods: (1) a UDA process, which leverages labeled development data along with unlabeled postdevelopment data to align their distributions and adapt the model to novel data instances and (2) an AL strategy, aimed at selecting informative data to include them with their labels in the retraining process. To the best of our knowledge, this is the first work leveraging UDA and AL approaches toward mitigating the impact of evolving data dynamics on model performance for COVID-19 detection, with the ultimate goal of enhancing reliability in COVID-19 detection and potentially across various diverse epidemiological contexts.

The COVID-19 Sounds dataset is a collection of respiratory sound recordings associated with COVID-19 infections, which were acquired through a crowdsourcing platform launched in April 2020 [16]. It includes demographic characteristics (ie, age and gender), along with participant-reported information about medical history and symptoms. It also comprises audio clips of voluntary cough, breathing, and voice captured from healthy individuals and individuals with COVID-19. A total of 36,364 participants contributed 75,201 samples to the project. Quality checks were performed on the audio samples to filter out incomplete or noisy recordings [16]. The data were collected in multiple languages, but for this study, the part of the dataset acquired from English-speaking participants [16] was solely considered to avoid language bias, corresponding to 1461 samples, as shown in Table 1.

The Coswara dataset is another crowdsourced database recorded between April 2020 and February 2022, which consists of 9 types of recordings, such as shallow and deep breaths, shallow and heavy coughs, sustained vowel phonation (ie, <ey> as in made, <i> as in beet, and <u:> as in cool), and number counting from 1 to 20 (normal and fast paced) [55]. Alongside this, information on the participants’ COVID-19 infection status, symptoms, comorbidities (if any), gender, age, and broad geographical location is included. In this study, shallow cough recordings were used as the models’ input space. After the exclusion of any missing, corrupted, or silent samples, a total of 72.69% (1996/2746) of samples from the initial set remained for analysis, as shown in Table 1.

Following the preprocessing approach described in the study by Han et al [16], the cough recordings were normalized, and leading and trailing silence was removed. Mel spectrograms were calculated using a 25-millisecond window size, a 10-millisecond window hop, and 64 mel bins, encompassing frequencies ranging from 125 Hz to 7500 Hz. In addition, within the framework of this study, in order to handle the varying size of the mel spectrograms’ time axis, the 0.9 quantile across all spectrograms was calculated. Subsequently, the spectrograms were either cropped or padded with repeated sections of the spectrogram accordingly. Finally, a sliding window approach was used to extract segments from the spectrogram. The width of the window used was 0.96 seconds, while the window stride length was equal to half of the window’s width (0.48 s). This setting resulted in a mel spectrogram segment with a size of 64 mel bins × 96 frames.

To facilitate model training, the COVID-19 Sounds and Coswara datasets were partitioned based on chronological order into a development set and a postdevelopment set by applying a 70:30 ratio. The development set was further divided into training, validation, and testing subsets, using a 60:20:20 split ratio, respectively. This division ensured that the model was trained on a representative portion of the data and validated and tested on separate subsets, promoting robustness and generalization.

No ethics approval was required for this study as it was not human participant research and did not include experiments on humans and the use of human tissue samples. As indicated in the data availability statement, the development of the presented methods was based on the use of a publicly available dataset and a dataset which was granted by a third party following the submission of relevant request. The Coswara data collection procedure was approved by the Institutional Human Ethics Committee, at the Indian Institute of Science, Bangalore. The informed consent was obtained from all participants who uploaded their data records. All the data collected was anonymized, and excluded any participant identity information. The COVID-19 sounds study was approved by the ethics committee of the Department of Computer Science at the University of Cambridge. Informed consent was given by the users through the mobile app.

The baseline model was assessed in terms of its ability to accurately detect COVID-19 cases in the presence of variations or shifts in the data. Particularly, the performance evaluation on the 2 datasets considered for the development and postdevelopment periods is reported in Figure 6.

On the basis of the results obtained for the COVID-19 Sounds dataset, it was observed that the baseline model achieved superior performance on the test subset of the development period in terms of the area under the receiver operating characteristic curve (AUC-ROC; 69.13%) and sensitivity (67.89%) compared to the best model performance reported in the literature [16] (AUC-ROC: 66%; sensitivity: 59%; specificity: 66%), despite considering a smaller amount of labeled data for training and validation (619 vs 1062 instances). The baseline model achieved a satisfactory F₁-score (65.2%) but demonstrated quite low specificity, correctly classifying 58.9% (63/107) of instances from the negative class. The performance of the baseline model on the postdevelopment data demonstrated a significant decline in the AUC-ROC, the F₁-score, sensitivity, and specificity, as reported in Figure 6.

In the case of the Coswara dataset, the baseline model displayed moderate discriminative ability on the development data, achieving an AUC-ROC value of 66.8%, while the accuracy, sensitivity, and specificity scores were 60.57%, 62.96%, and 60.32%, respectively. The highly imbalanced distribution of the 2 classes in the development data (165/1395, 11.82% positive vs 1230/1395, 88.18% negative) posed a significant challenge for the model, as highlighted by the notably low F₁-score, a metric that exclusively focuses on positive instances. The model’s discriminative power on the postdevelopment data presented a decline, as indicated by the AUC-ROC, specificity, and sensitivity (Figure 6). The high value obtained for the F₁-score metric was related to the presence of class imbalance with reversed minority (negative) and majority (positive) classes in the postdevelopment data with respect to the class distribution of the development data, which led to a misleading perception that the model’s performance had significantly improved. A more thorough analysis of this issue is provided in the Discussion section.

The hyperparameters of the drift detection mechanism were fine-tuned across both datasets to maximize detection accuracy. For the COVID-19 Sounds dataset, the best-performing configuration incorporated a 7-day window, 3-day overlap, a minimum batch size of 40 samples, a polynomial kernel for the MMD calculation, and CUSUM drift and threshold values of 0.2 and 0.5, respectively, and achieved an accuracy score of 91.3%, sensitivity of 88.5%, and specificity of 93.2%. For the Coswara dataset, the optimal setup included a 10-day window, no overlap, a minimum batch size of 20 samples, and CUSUM drift and threshold values of 0.2 and 0.7, respectively, leading to an accuracy score of 89.7%, sensitivity of 85.2%, and specificity of 92.1%. Further details on the tuning procedure and hyperparameter selection are provided in Multimedia Appendix 1.

A comparative assessment of the model’s performance, before and after each UDA adaptation phase, was carried out, with balanced accuracy on the test subset of the development period serving as a benchmark. For the COVID-19 Sounds dataset, considerable improvement was achieved after each adaptation phase, particularly following the fourth (up to 15%) and fifth (up to 24%) adaptations (Figure 7). The model consistently outperformed the baseline model, demonstrating the effectiveness of the proposed approach in mitigating concept drift. It is noteworthy that by correctly identifying periods of drift, the drift detection mechanism efficiently prevented the degradation of the model’s performance in a timely manner while also contributing to sustaining the model’s performance closer to the development period benchmark.

The results obtained for the Coswara dataset are depicted in Figure 8. It was observed that while some adaptations (eg, fourth) exhibited up to 15% improvement, others initially led to a significant yet short-lasting decline in the model’s performance, before ultimately demonstrating the ability to recover. Despite these fluctuations, the drift detection mechanism effectively generated timely alerts, preventing prolonged performance degradation.

A detailed study of each adaptation’s impact on various performance metrics (AUC-ROC, accuracy, balanced accuracy, sensitivity, specificity, and F₁-score) for both datasets was also carried out. The results of the COVID-19 Sounds dataset showed that the UDA approach significantly enhanced the model’s performance in terms of all evaluation metrics. However, the application of the UDA approach on the Coswara dataset produced less consistent results. Overall, the performed adaptations exhibited varying effects on the model’s performance across the considered evaluation metrics. The complete results of this analysis are presented in Multimedia Appendix 2.

The proposed AL approach was evaluated by comparing the performance of the model after each AL-based retraining phase with that of the baseline model, as well as the model following retraining, using random sampling. Considering the COVID-19 Sounds dataset, Figure 9 demonstrates the observed balanced accuracy score across the entire data stream, indicating a substantial and lasting improvement with respect to the baseline model following each adaptation. Overall, each AL-driven adaptation significantly enhanced balanced accuracy, with the third adaptation improving performance by up to 30% over a broad period of 15 batches, and the fourth adaptation surpassing a 95% balanced accuracy score while achieving an improvement of up to 25% compared to the baseline model. The superiority of the proposed AL approach over random sampling was evident across 90% (36/40) of the data batches.

In the case of the Coswara dataset, Figure 10 shows that model adaptations led to improved performance during most postalert periods, particularly after the third (up to 40%) and fifth (up to 60%) adaptations. Occasional fluctuations of limited duration were observed, mostly in the early batches of the postalert periods, where the baseline model outperformed the AL-based model. As compared to random sampling, AL remained superior in most cases, further confirming its effectiveness in selecting informative data.

Further insights were derived through per-metric performance comparisons, including AUC-ROC, accuracy, sensitivity, specificity, and F₁-score, across adaptations for the 2 studied datasets. Most AL-based adaptations for the COVID-19 Sounds dataset yielded improved performance in all the considered evaluation metrics with respect to the baseline model and the random sampling approach. In the case of the Coswara dataset, the AL approach led to an overall improvement in the model’s performance over the baseline model and the random sampling approach, except for certain metrics and adaptations. The results obtained are presented in detail in Multimedia Appendix 2.

This study proposed a drift-adaptive framework for COVID-19 detection using crowdsourced cough audio recordings and evaluated its effectiveness in addressing temporal data drifts, confirming its ability to sustain model performance over time.

The proposed framework addressed challenges imposed by dynamic, nonstationary environments caused by a pandemic by incorporating a drift detection mechanism and appropriate adaptation strategies. The proposed approach focused on the temporal evolution of data distributions in a real-world scenario, unlike previous studies that did not explicitly consider temporal aspects [15,16,46-50]. The evaluation of the introduced framework provided evidence regarding its ability to maintain model performance, thus highlighting its potential to facilitate the identification of new cases in the evolving context of a pandemic.

A baseline model that was able to detect COVID-19 positive cases using cough recordings was trained and evaluated. During the development period, the model achieved an AUC-ROC of 69.1% and 66.8% on the COVID-19 Sounds and Coswara datasets, respectively. However, in the postdevelopment period, there was a notable decline in the baseline model’s performance, reflected in an AUC-ROC of 60.7% and 59.7%, respectively, thus suggesting the potential presence of concept drift. These findings motivated the development of the proposed approach, which leveraged robust drift detection and efficient adaptation mechanisms to maintain the model’s performance in the face of evolving data distributions.

The results obtained indicated the efficacy of the proposed drift detection mechanism and provided evidence regarding its ability to enhance the robustness and adaptability of deep learning models in dynamic environments. The combination of the MMD distance monitoring and the use of the CUSUM algorithm for adaptive detection of abrupt changes, which reflect a growing divergence between the reference distribution (development data) and the postdevelopment data, enabled the timely and robust detection of performance degradation. The use of the CUSUM algorithm, tailored to the characteristics of each dataset, ensured the generation of accurate alerts for significant changes in the monitored MMD distance, thus minimizing false alerts and preventing unnecessary interventions.

Two distinct retraining strategies based on UDA and AL were used to mitigate performance degradation in this study. Notably, the use of UDA in this study focused on achieving continuous adaptation in the presence of real-world data drift, unlike previous studies [37] where UDA was used to address static cross-dataset generalization. Similarly, this study incorporated AL as an adaptation mechanism, ensuring continuous model refinement in response to data distribution shifts, in contrast to previous studies [40] where AL was used for reducing annotation costs by selecting the most informative samples for initial model training.

Regarding the use of UDA, the results obtained from the analysis of the COVID-19 Sounds dataset showed significant improvement in the model’s discriminative ability. The comprehensive examination of the adaptation phases based on multiple evaluation metrics mostly revealed improvements in the balanced accuracy with respect to the baseline model’s performance, ranging from 10% to 20%.

The aforementioned findings align with UDA’s core advantages, which include cost-effectiveness and adaptability to dynamic environments through the model’s adaptation to the target domain’s data distribution without requiring labeled target domain samples. This approach is particularly valuable when labeled data from the target domain are scarce or expensive to obtain, as is often the case in emerging pandemic scenarios.

The application of the UDA approach on the Coswara dataset yielded less consistent results. The overall comparison between the adapted models and the baseline model revealed that the adaptation had diverse effects on the model’s performance in terms of the evaluation metrics considered.

The difference in the effectiveness of the UDA approach on the COVID-19 Sounds and Coswara datasets may be attributed to differences in the datasets’ characteristics. Figures 11 and 12 illustrate selected descriptive statistics of the development and postdevelopment data of the COVID-19 Sounds and Coswara datasets. As per Han et al [16], the COVID-19 Sounds dataset used in this study had undergone meticulous curation to eliminate biases as a result of methodological decisions, thus enabling the development of unbiased models. In the case of the Coswara dataset, significant differences were observed in terms of COVID-19 prevalence and the frequency of related symptoms between the development and postdevelopment data, which may be attributed to the presence of age and gender biases [69]. In this study, handling the data in chronological order implicated different levels of data biases present across the various adaptation periods, which may arise in emerging pandemic scenarios.

The AL approach resulted in a more prominent improvement in the models’ performance compared to UDA. This underscored the power of actively selecting informative samples for labeling, which aids in refining the model’s understanding of the target domain. Thus, by optimizing both adaptation to the target data and the use of labeling resources, AL is considered promising for ensuring model performance in data-scarce scenarios, such as during a pandemic.

Given that both UDA and AL achieved varying levels of performance improvement on the COVID-19 Sounds and Coswara datasets, it is essential to consider their limitations and potential challenges. UDA relies on the assumption that the source and target domains share some underlying similarity. In the presence of significant differences, adaptation might not yield substantial improvements. By contrast, AL’s performance is determined by human labeling expertise, which is associated with the rise of the related costs and depends on the reliability of the existing diagnostic tests. If the chosen samples are mislabeled, the model’s performance could suffer. Moreover, AL’s performance is sensitive to the selection of labeled samples, which might introduce biases.

The aforementioned observations suggest that the proposed adaptation mechanisms effectively addressed the individual challenges linked to the special characteristics of each dataset and mitigated the effects of concept drift during critical periods corresponding to batches in proximity to the alert periods. Figure 13 summarizes the model’s performance obtained by applying each adaptation approach on the postdevelopment period of the 2 datasets and shows that both approaches succeeded in maintaining the models’ performance closer to the development period’s benchmark.

Further evidence regarding the proposed framework’s ability to detect concept drift and maintain model performance was provided through a detailed analysis of stratified heterogeneity (SH) using the Q statistic [70-73]. The findings obtained indicated that COVID-19 presence was stratified across time, age, and gender, with a strong correlation to performance fluctuations. Notably, detected input drift alerts aligned with statistically significant differences of SH across strata (P values for all pairwise comparisons are reported in Multimedia Appendix 3), indicating the presence of drifts in the ground truth label, which provided further evidence on the effectiveness of the proposed drift detection mechanism. Moreover, the superiority of the UDA- and AL-based adapted models’ performance over that of the baseline model and the adapted model using random sampling was observed for most batches, presenting statistically significant differences in SH. The detailed SH analysis that was conducted is provided in Multimedia Appendix 3.

These results highlight the importance of combining effective drift detection mechanisms and intelligent adaptation modules in addressing concept drift. Together, these components form a robust framework that enables the model to continuously adapt to changing data conditions, thereby maintaining its discriminative power and overall performance over time.

Certain potential limitations of the proposed study should be acknowledged. First, although the proposed framework successfully detects and responds to performance degradation based on the MMD distance, it does not explicitly interpret the underlying causes of the drift. Understanding the sources of the drift could enhance trust in the detected drifts and enable efficient targeted interventions toward sustaining model performance. Moreover, by design, this study isolates and investigates temporal drift as the primary source of distributional change. While this focused approach allows for detailed analysis, it limits exploration of other drift types, such as cross-dataset drifts or interdemographic variability, which may arise in broader deployment scenarios. Finally, the framework was applied for COVID-19 detection from cough data; the integration of multimodal inputs, including breathing and voice data, is straightforward and could further improve the model’s performance.

The significance of the proposed approach lies in its reliance on data-efficient techniques. By minimizing the dependence on labeled data, the proposed framework enables the accurate detection of COVID-19 cases even in the absence of comprehensive labeling resources. This aspect becomes particularly crucial when considering the value of a deep learning–based detection model during the early stages of a new pandemic or when dealing with emerging viral variants that may not be adequately detected by existing diagnostic tools. Thus, the proposed framework is able to contribute toward a more generalizable approach that can be applied to future pandemics or novel variants. By collecting knowledge and formulating a well-defined framework, a basis for rapid adaptation and deployment of disease detection tools is established, ensuring timely and accurate identification of infectious diseases.