Generative artificial intelligence (GenAI) is a family of artificial intelligence (AI) models that can generate synthetic content ranging across text, images, audio, and video. Well-known GenAI tools such as ChatGPT [1] and DALL-E [1] have revolutionized the way in which people view AI [2]. On the other hand, data analysis in health care has entered a new era as large amounts of electronic health records (EHRs) are generated daily. Among these data, biomedical time series have been of particular interest to researchers in tasks including tracking patient trajectories [3] and treatment estimations [4]. Compared to cross-sectional data, which represent 1 time point, time series provide more information on development over time. There are 2 main types of time-series data: real-time and longitudinal data. While real-time data usually span a shorter period (minutes or days) and have more consistent time-point entries, longitudinal data are recorded over the magnitude of months or years with various sampling frequencies and number of entries. Compared to traditional time-series data in finance and climate science, biomedical time series face unique challenges in modeling, including missingness and sampling rate [5]. Therefore, traditional methods such as foundation models (FMs) that work well on high-frequency time series are not guaranteed to work well on biomedical data. In the age of big data, GenAI models have begun to outperform traditional statistical and machine learning (ML) methods in various tasks [6]. On the other hand, GenAI for health care faces unique challenges, including data privacy, transparency of the decision process, and other ethical issues that must be addressed. Nonetheless, these tools have the potential to improve disease modeling accuracy, tailor treatment plans across a heterogeneous population, and assist clinicians in making clinical decisions.

In this paper, we aim to provide a comprehensive review of GenAI applications to biomedical time-series data and a technical overview for clinicians who wish to work with AI tools but are unsure of where to begin. We survey GenAI applications to time-series health care data and its ability to improve patient care and the diagnosis process while considering the challenges and showcasing real-world uses that highlight its importance in the field. For the selection process, we followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews; Multimedia Appendix 1) guidelines and searched in PubMed and Google Scholar for conference and workshop papers using the following keywords: “generative AI (GenAI),” “health care,” “time series,” “longitudinal,” “electronic health records (EHR),” “electronic medical records (EMR),” and “genetics.” From the initial search, we applied the following inclusion criteria—being peer reviewed, published after 2000, and focusing on time-series or longitudinal data—ultimately yielding 155 studies as of April 2024. The detailed selection process is described in the Related Works section.

We organized the review by data modality, including structured and unstructured text, imaging, waveform, and multi-omics data, and illustrate possible data sources in each modality for GenAI application in Figure 1. For each modality, we discuss preprocessing, applications, model selection, and challenges and future directions. In the Model Selection sections, we help guide how to choose the appropriate models for this type of data for clinicians who are interested in incorporating GenAI into their practice. We organize the workflow for choosing an appropriate model for the data and computational resources at hand. We have organized the paper as follows: (1) overview of the algorithms and techniques introduced in the review; (2) structured data; (3) unstructured data; (4) medical imaging; (5) physiological waveforms; (6) genetics and multi-omics data; and (7) ethical considerations, challenges, and future directions.

Our electronic search retrieved 20,978 study records: 20,003 (95.35%) from Google Scholar and 975 (4.65%) from PubMed, as shown in Figure 2. After title and abstract screening and removing duplicates, 3.24% (680/20,978) of these articles were retained for full-text review. Of these 680 articles, 533 (78.4%) were excluded, leaving 147 (21.6%) studies after the primary screening. A total of 8 more articles were added through secondary screening, resulting in a total of 155 articles included in our scoping review.

Our search for existing reviews on GenAI for disease trajectory modeling returned 39 entries broadly covering the use of AI in health care (Multimedia Appendix 2) in Google Scholar, Scopus, and PubMed. From a review of these works, although we observed an increasing number of AI-related review papers in the biomedical setting in recent years (Multimedia Appendix 3), there remained a gap in reviewing GenAI approaches working with time-series data specifically, prompting the need for our paper.

Despite a shared nomenclature, it is important to differentiate between modern AI and the field of AI. The general study of AI encompasses the class of ML methods, which includes the subset of deep learning (DL) methods (also thought of as modern AI). Previous ML approaches have been mostly discriminative such that, given a set of input predictors, they learn a strategy (decision boundary) to fit or separate data points. On the other hand, DL models can be discriminative or generative based on the choice of the objective function. Therefore, they are not inherently generative or discriminative and are the building blocks of modern AI models, although they have typically been developed and demonstrated in the discriminative setting. Finally, GenAI methods belong to the class of generative ML methods and capture the underlying data generation process. By learning the distribution domain of the data, these models can synthesize hypothetical data points that are statistically indistinguishable from the originals. For discriminative tasks, GenAI models are often combined with downstream models, which can be simply rule based or more complex. In this section, we introduce a set of traditional, ML, DL, and GenAI models that are referenced later in the paper and list existing models used in time-series forecasting in Figure 3, as well as showing a timeline for when these models were first introduced in Figure 4 and a comparison among these methods using common metrics in Table 1.

In this section, we introduce several statistical models commonly used as benchmarks in modeling time-series data. One of the most basic yet effective methods is the Naive2 model, which makes a prediction by using the value from the most recent observation or from a similar period in the past. In the class of autoregressive methods, the autoregressive moving average model [7] predicts the outcome value at a given time point using both a linear combination of multiple past observations and a linear combination of past error terms in the regression model. As an extension of the autoregressive moving average, the autoregressive integrated moving average (ARIMA) model [8] accounts for seasonal or periodical changes in the data. One of the most widely used methods is the Prophet model by Meta [9], which uses an additive model that accounts for yearly, weekly, and daily seasonality and holidays. Finally, the exponential smoothing (ETS) model [10] uses a weighted combination of multiple past observations in which the decay of the weights is exponentially distributed. While these methods have limited performance with highly complex data, they remain a popular choice due to their simplicity and transparency in the decision-making process.

DL models are the combination of an architecture (ie, the structure) and an objective (ie, the loss function). In this section, we introduce the canonical DL architectures that are the foundation of different GenAI settings but are not GenAI models themselves.

In an effort to improve patient safety and reduce medical costs, >500,000 physicians and almost 6000 hospitals in the United States are expected to have operating EHRs and health IT systems [41] and charts [42] as of 2015. As a result, big data in health care in the United States have grown rapidly in recent years, with a massive database of electronic patient data. Among these, unstructured text data, also referred to as tabular data, are the most recorded and used time series, including numeric (eg, laboratory measurements and vital signs) and categorical (demographics, medications, and diagnostic codes) features. We note here that tabular data encompass a wide range of data, and we focus only on time series in this review.

Among time-series structured data, vital signs and laboratory results are the most common sources. These are typically represented in one of two ways: (1) as one 2D matrix in which each row is an encounter or patient and each feature can be represented across time as multiple columns, such as heart_rate_time1 and heart_rate_time2; and (2) as stacked 2D matrices in which each matrix contains columns of all features at 1 time point. Depending on the recorded values, feature values can be either numerical or categorical. During preprocessing, the number of time points can be determined by the mean or median number of entries available across patients. The interval between each point can be absolute or relative time depending on whether the cohort has a large variability in terms of hospital stay time. With a large variability, a set number of time points can correspond to the relative time such that all patients have the same number of points but the intervals do not match up in absolute time. For uncommon measures such as certain laboratory tests, there may be large missingness within the cohort, which is further discussed in the next section. Finally, feature engineering can be performed through either a manual selection or feature selection algorithms [43,44].

In this section, we introduce existing work on GenAI application in 3 main categories: data augmentation and imputation, disease classification and prediction, and counterfactual estimation.

As they are most abundant and easiest to access, unstructured text data are a good place to start for clinicians who want to explore GenAI tools. In Figure 5, we present a flowchart to help guide the process of choosing the right model for the data and task at hand. The flowchart is organized in the same way as the sections in this paper, where different colors refer to different data modalities for easier visualization. In addition to the statistical methods mentioned previously, there are also a number of ML models, including logistic regression, decision tree, random forest, and Extreme Gradient Boosting (colored red in Figure 5). While we do not discuss these methods in this review, there are review papers that describe them in detail [96-98]. If the dataset is quite small, a traditional autoregressive or a simple ML method would work better as more complex models require more data. However, if the dataset is large, DL models would perform better as the relationship between features is more complex and tends to be nonlinear. Commonly, the model of choice will be RNN or transformer based, which also have the most literature available. Once one has chosen a base model, they can start with a published framework and modify certain parameters based on their goal. If the model is quite simple and only consists of linear layers, one may also consider changing part of the structure to newer architectures, such as attention layers, or chaining 2 models together to improve performance.

As ML and DL methods, including GenAI, continue to grow with an increasing amount of available data, many comparisons have been made between AI methods and traditional statistical methods. Cerqueira et al [99] compared ML methods in time-series forecasting (rule based, random forest, and Gaussian process regression) to statistical methods (ARIMA, Naive2, and ETS) using datasets with varying sample sizes up to 1000 and showed that ML methods outperformed statistical methods when the sample size was relatively large. Makridakis et al [100] jointly evaluated the accuracy of DL methods and statistical methods (Naive2, ARIMA, ETS, and ensemble methods) using the 1045 M3 dataset and showed that, among the nonensemble approaches, the statistical methods on average yielded better performance than DL methods for short-range prediction with regard to symmetric mean absolute percentage error and mean absolute scaled error, whereas the DL methods performed better in general for medium- and long-range predictions. The DL ensemble method using a combination of the 4 DL methods consistently outperformed the nonensemble methods in the study by Bonnheim et al [101]. However, in a practical setting, AI methods face the biggest challenge of low transparency due to their black-box nature. Interpretability has been an active area of research, and many techniques and tools have been proposed to understand the decision-making process of such models. Feature importance analysis tools such as Shapley Additive Explanations [102] and Local Interpretable Model-Agnostic Estimations [103] estimate feature contribution to reveal top features that are important to the decision. These tools may help shed light on how the model predicts the outcome, which is extremely important in the field of health care. A second challenge that AI methods face is the trade-off between generalization and performance as an exceptionally good performance of a model on one dataset may be caused by overfitting. As patient distributions can differ widely between hospitals and areas, the model’s robustness must be measured before deploying it in practice. Nonetheless, GenAI has shown great potential in clinical support, such as early warning systems or care assistance tools. Future GenAI tools that tackle these challenges will become paramount in health care.

Unstructured data include a wide range of clinical narratives, such as progress notes, radiology reports, patient correspondence, and discharge summaries, comprising 80% of EHRs [104]. The loosely structured nature of typed text (also known as “free text”) is effective in day-to-day clinical workflows but presents a major challenge for automating EHR-based registries. The unstructured data may contain key patient information missing in structured data, extra information complementing structured data, or even data that may contradict information presented in structured data. As such, the overall goal of modeling using unstructured data is to assist disease prediction using structured data by adding critical missing information. The complexities of unstructured data, along with the fact that existing text-mining tools and NLP applications have limited accuracy in extracting information from free text, have prompted some registries to ask for a manual chart review of individual patients before final inclusion in the registry. Unstructured data limit the application of automated computational phenotyping methods and increase the likelihood of low data quality (eg, missing data) when data are extracted from structured EHR data only [105,106].

Information retrieval from unstructured data is challenging because the information can often be of low quality and contain missing values. Commonly used data preprocessing techniques for unstructured data include data collection, data cleaning, tokenization, stemming, and the removal of stop words [107,108]. Tokenization refers to segregating, for example, words and punctuation marks in the input document as tokens and computing their frequency. Stemming is the process of identifying the roots of words, typically by removing the suffix. Stop word removal consists of removing infrequent words to reduce the dimensionality of the dataset. Once the key components of the dataset are extracted, they can be used for downstream applications such as the identification of risk factors and individuals at high risk of a particular disease.

As the application of large language models to health care time-series data is still a developing area of research, in this section, we discuss static time applications of these models. We will discuss potential research directions for time-series unstructured text based on progress in other fields such as climate and finance in the Challenges and Future Directions section. Both traditional NLP and FMs can be used to perform clinical NLP tasks such as clinical concept extraction (or named entity recognition), medical relation extraction, semantic textual similarity, natural language inference, medical question answering, and medical report summarization. The domain of medical report summarization is a popular area of research, with numerous papers published on the use of NLP and FMs to summarize medical reports, health care records, and medical dialogues [109]. A radiology report is a medical document that contains the details of an imaging examination (such as x-rays and magnetic resonance imaging [MRI]). A radiology report consists of three components: (1) the Background section, which contains the medical history of the patient; (2) the Findings section, which discusses the crucial observations and findings of the radiology examination; and (3) the Impression section, which is a short summary of the findings section. The Impression section is usually written by medical professionals, which is a time-consuming process. The only aim of radiology report summarization is to automate the generation of this Impression section. Several attention-based and FM-based models have been developed to improve the quality of summarization [110].

Medical dialogue summarization corresponds to the automatic generation of coherent summaries that capture medically relevant context from dialogues between patients and health care providers. Medical dialogue summarization can help medical providers keep a record of patient encounters and also provide the necessary context of a patient’s medical history during patient handoffs between providers. Existing studies have used techniques from computational linguistics [111], NLP (Pegasus [112]), pretrained language models, and low-shot learning to collect labeled data and perform medical dialogue summarization.

Medical question answering refers to developing a technique that automatically analyzes thousands of articles to generate a short text, ideally in less than a few seconds, to answer questions posed by physicians [113]. Such a technique provides a practical alternative that allows physicians to efficiently seek information at the point of patient care. Several question-and-answer systems have been developed using NLP [114] and FMs (generalist medical AI and MedLM).

FMs hold great potential to assist clinicians in a wide range of health care problems. However, clinicians should be aware of the risks associated with the use of these models and potential data leaks. They should also select which models to use based on the computational resources available to them, as shown in Figure 5, as the training of these models is becoming increasingly expensive (Table 1).

Compared to traditional ML models, FMs have a significantly higher computational cost, requiring massive datasets and graphics processing unit specifications for training. While they achieve high predictive accuracy [115], these models are composed of millions to billions of parameters and have poor interpretability and high latency and runtime. While these costs can be lowered over multiple downstream applications, their value may take longer to realize than that a smaller model developed for a single high-value task [30]. In addition, data privacy and security are significant concerns with FMs as they may leak protected health information through model weights or prompt injection attacks [116]. As the cost of computing decreases and the amount of data increases, FMs may become easier to train and more accessible. One active area of research is testing FMs’ potential in multimodality input, in which FMs are trained and evaluated on multimodal datasets, such as both text and images. These models can be used to generate personalized treatment plans based on the patient history as well as develop early warning systems for individuals at high risk. These advancements would transform how health care professionals interact with patient data, making health care more predictive, personalized, and precise.

Medical imaging refers to 2D or 3D images obtained from radiology procedures such as MRI and computed tomography (CT). GenAI has been applied to a wide set of problems in medical imaging to solve issues of scarcity, heterogeneity between datasets, and low resolution quality, among others. There are 3 main classes of generative vision methods: GANs, VAEs, and DDPMs. Each of these classes has been applied to medical imaging. Among all applications, medical image series input has been used mostly in data generation, anomaly detection and classification, and image segmentation and registration tasks.

There are several steps in preparing a medical imaging dataset as the quality varies largely depending on location, time, and cohort [117]. First, a denoising technique is applied to remove as much nuisance noise as possible. After denoising, data interpolation can be conducted for certain images such as CT scans to ensure the same spacing across all images. In the case of MRI images, bias field correction may be performed to adjust image intensity. Third, image registration is performed to align all images with a common template to avoid unnecessary incoherency among images, such as rotation. Finally, images can be standardized or normalized to improve algorithm efficiency. Depending on computational resources, the image may also be downsampled to a lower resolution or split into patches to reduce computational cost. Publicly available datasets such as those of the Alzheimer’s Disease Neuroimaging Initiative [118] and Open Access Series of Imaging Studies [119] do not need much preprocessing as they have been carefully curated. However, image registration is recommended to align all patients with a common template of choice if multiple datasets are combined. The medical images are usually represented as 2D matrices or 3D vectors of the pixels or voxels.

Although there are many tasks associated with imaging data input, the underlying model architecture remains largely the same for prediction, synthesis, registration, or even domain transfer tasks. For anyone using time-series or longitudinal images for any task, choosing 1 of the 3 families of vision methods following the workflow in Figure 5 is a good place to start. One may also refer to Table 1 to compare the models’ strengths and weaknesses. GANs are an earlier framework that remains powerful today, especially with synthetic generation, but are more complicated to implement as they have a generator and a discriminator. On the other hand, VAEs offer a latent space that can be used to downstream linear models such as mixed-effects models to control for variables such as age, but its generation quality may not be as ideal as that of GANs or DDPMs. Finally, diffusion models have been largely applied in recent years due to their simple architecture compared to that of the other 2 and low memory requirements but offer very little interpretability on the generation process. For clinicians with limited computing resources, 3D images may be too large to fit onto graphics processing units, and in many cases, the images can be split into smaller patches to reduce memory footprint [148]. Finally, there are publicly available general-purpose vision models that can be downloaded as a starting point, which can be further trained and fine-tuned for the specific task at hand.

With computer vision models’ success in recent years, there is a lot of potential in their applications to medical imaging. In recent years, diffusion models have shown great promise in generating images that are comparable to those generated by state-of-the-art GAN methods. Furthermore, GenAI methods that support video generation can potentially be adapted for medical imaging [149-153]. However, some issues that are present in current vision models also extend to medical imaging, such as illogical generations that do not comply with physical laws [119]. Therefore, clinicians should take the generations with a grain of salt when applying them to downstream tasks. The advent of vision models has been revolutionary in the study of medical images as previous methods must first transform the images into numeric representations, such as thickness and mean intensity. These representations are then fed into linear models for tasks such as classification and segmentation. To work with raw image inputs, linear models of estimation such as principal component analysis or single-value decomposition are often used. However, these generations often have poor resolution or fail to adequately capture variability. In contrast, modern neural network approaches (eg, CNN) jointly learn the optimal feature extraction strategy together with the downstream task. Thus, GenAI algorithms are able to consider highly nonlinear and nuanced interactions between inputs that better capture the underlying distribution of the data. While GenAI faces the challenge of transparency, interpretability methods such as Gradient-Weighted Class Activation Mapping [154] have shown convincing results. Another challenge of GenAI in medical imaging is the computational resources required to support such a model, which will become cheaper and easier to acquire as hardware technology becomes more efficient.

Waveform data capture the change of one or multiple physical quantities over time and can be viewed as either univariate or multivariate time-series data. Examples of medical waveform data include single-lead electrocardiograms (ECGs) or photoplethysmograms, which can be represented as univariate time series. Multi-channel technologies such as electroencephalograms (EEGs) are typically represented as multivariate time series. Although waveforms are considered to be closer to structured time-series tabular data, DL methods applied to structured data often face technical limitations when applied to waveforms as the sampling frequency is much higher. As such, GenAI methods applied to waveforms often resemble speech processing [155] or vision-based models adapted for 1D input signals.

Physiological waveforms are often recorded over a short period with low missingness but face the problem of noise and signal artifacts. Therefore, waveform preprocessing aims to remove as much nonphysiological noise and artifacts as possible while preserving the actual signal. This is typically accomplished by applying digital filters designed to address nonbiological signals, drifts, and spikes [155]. Depending on the signal, specific filters have also been developed and shown strong performance, such as the Wiener filter for ECGs [156]. After filtering, the waveform is usually normalized for efficiency and resampled to a common frequency.

GenAI models have gained increasing popularity among medical waveform data. Major tasks using GenAI models for this type of data include synthesis and imputation and anomaly detection.

In general, waveform data are best trained using vision models such as VAE and DDPM as the time series can be viewed as a 1D image. Therefore, the workflow to choose an appropriate model for waveform data is similar to that for imaging data. As GANs are often more complex but do not significantly outperform other vision models, they are being replaced with VAEs and DDPMs [171,172].

As GenAI models are only beginning to be applied to waveform data, there are many directions to pursue. One of these is cross-modality imputation, where, given an EEG, a GenAI model can be trained to generate the corresponding functional MRI or vice versa. Another area of study is to convert EEG signals to natural language by combining vision models with a language model [173]. As previous studies have focused on modeling a single type of waveform data in isolation, one natural direction is to use more data sources, for example, EEGs with text data or ECGs with medication data, as they are often informative about one another in tasks such as imputation [174] and forecasting [175].

Multi-omics data comprise multiple-sequencing data types and various features. To efficiently preprocess the data, different methods are used for each type of data. DNA sequences are represented as strings of adenine, thymine, guanine, and cytosine nucleotides, which can be transformed into a numerical matrix in which each nucleotide is represented as a binary vector. For transcriptomics, expression data are often represented as a matrix of continuous numeric values in which each row corresponds to a sample or a cell and each column corresponds to a gene or transcript. Proteomics data are similarly structured but represent peptide abundances or protein expression levels, and epigenomics data such as methylation status are typically captured as binary values or continuous levels indicating methylation at specific sites. Depending on the goal, different omics data can be combined to uncover time-related trends and critical factors that influence those biological shifts.

GenAI has shown great promise in analyzing multi-omics data for disease progression analysis. By using techniques such as omicsGAN, which uses GANs, and graph-linked unified embedding and MultiVI, which use VAEs, GenAI facilitates the integration and compression of multi-omics data from diverse sources into a unified representation [198-200]. This aids in a more comprehensive analysis of the data. In addition, tools such as Potential Energy Underlying Single-Cell Gradients apply generative modeling to map out potential landscapes from time-series single-cell transcriptomics data, making it possible to generate trajectories for unseen data points [201]. This opens up avenues for hypothesizing about biological perturbations and pathways of disease progression. Moreover, GenAI proves instrumental in identifying distinct disease subtypes and stratifying patients based on their multi-omics profiles. For example, gene-guided weakly supervised clustering via GANs uses a GAN to generate imaging features from brain MRIs and single nucleotide polymorphisms to explore conditions such as AD and hypertension [202], whereas the Trained GAN Discriminator model stratifies patients with breast cancer into high- versus low-risk categories based on their transcriptome profiles [203]. For clinicians interested in working with multi-omics data, we outline a procedure for finding the best tool in Figure 5 based on their specific needs, such as resource availability and the type of data analysis required, including genetics, single-cell transcriptomics, and proteomics or epigenomics.

Analyzing and modeling multi-omics data in time-series contexts presents challenges in addressing missing data across different biomolecule types, which complicates the integration of datasets obtained through various omics technologies. In addition, the complexity of biological interactions can lead to analytical difficulties when dealing with high-dimensional data. The integration of multi-omics data is becoming increasingly important in the pursuit of future progress. This approach uses various biological data layers—genomics, transcriptomics, proteomics, and metabolomics—to enhance the understanding of genetic interactions and disease mechanisms.

While there remain many challenges to deploy reliable GenAI methods in real practice, including technical, regulatory, and ethical issues, these models can have huge impacts that could benefit patients, practitioners, and administrators. As such, collaboration among policy makers, legal experts, AI developers, and health care providers is crucial in overcoming these issues. In this paper, we introduced concepts basic to DL and GenAI to those unfamiliar with the field, reviewed existing work on GenAI application to time-series health data, and offered suggestions to clinicians who are interested in applying such tools to their field. We hope that this review bridges the gap and promotes better collaboration between technical and applied disciplines to develop better methods that are suitable and robust in a practical setting.