Electronic health records are widely regarded as having significant potential to improve clinical research. However, most data contained in electronic health records are in free-text format. Unstructured text contains up to 80% of crucial clinical information, including temporal information [1-3]. Several temporal information extraction approaches have been proposed in recent years to facilitate access to this information, which can help better understand the patient’s prior medical history, disease progression, treatment efficacy, etc. Temporal information extraction includes detecting events, identifying temporal expressions, and extracting temporal relations between them.

In the context of rare diseases, extracting specific temporal information such as the date of diagnosis and the dates of key phenotypes is crucial. Such temporal anchors enable the reconstruction of patient care pathways, support prognostic assessments, and allow researchers to characterize the natural history of rare diseases that are often poorly documented. This information is also required for the curation of rare disease registries and clinical research databases, where structured entries are derived from clinical narratives for reporting and research purposes. In practice, clinicians often need to retrospectively review these narratives to identify and structure relevant temporal information. Automating this process will facilitate the construction of longitudinal patient timelines, improve registry completeness, and reduce the burden of manual reporting.

There has been a significant interest in temporal relation extraction (TRE) in the clinical domain through the 2012 i2b2 challenge [4] and the 2015‐2017 Clinical TempEval shared tasks [5-7]. Clinical TRE methods evolved from rule-based [8-10] to feature-based machine learning [11-13] and neural-based methods [14-19].

Early systems relied on manually defined rules [9,10,20], which required human expertise and limited generalizability across domains. Following rule-based systems, a range of feature-based machine learning methods has been developed, using, for instance, support vector machine classifiers and conditional random fields, leveraging a range of syntactic, lexical, and semantic features [11,12,20,21]. Tourille et al [12] evaluated random forest and support vector machine models to tackle TRE for clinical corpora in both English and French. Some research studies introduced a hybrid approach that integrates rule-based techniques with a maximum entropy classifier and evaluated its performance on the 2012 i2b2 dataset [8].

Neural approaches offered significant improvement to clinical TRE, outperforming traditional feature-based models. Early deep learning methods used convolutional neural networks (CNNs) [19,22] and long short-term memory networks [14,17,23,24]. Subsequently, the emergence of pretrained transformer-based models [25] has driven several research efforts on clinical TRE [15,26,27], applied to benchmark datasets such as THYME [28] and 2012 i2b2. Han et al [16] introduced OpenNRE, an open and extensible toolkit that implements neural relation extraction models using diverse encoding architectures, such as CNNs, piecewise convolutional neural networks (PCNNs) [29], and recurrent neural networks. Other studies have further explored the use of graph neural networks for document-level TRE, aiming to model long-range dependencies between events and temporal expressions. For instance, GRAPHTREX was introduced as a hybrid approach combining span-level entity representations with a heterogeneous graph to model both local and global dependencies [30].

Despite notable progress, TRE in clinical texts remains challenging. To obtain high-performing supervised temporal relation extractors, large amounts of annotated corpora are required. However, the annotation process is known to be time-consuming and highly expensive, particularly in the clinical domain, due to domain expertise need. Consequently, the available datasets have low interannotator agreement [31], and annotated corpora are limited for non-English languages. Only a few research works have been conducted on French corpora [12,23,32,33]. Moreover, clinical narratives often contain implicit temporal reasoning, domain-specific terminology, and long-distance relations, which continue to limit model performance in real-world applications [34].

With the emergence of generative large language models (LLMs), showing promising results in a variety of natural language processing (NLP) tasks, a few studies have evaluated their potential on clinical TRE. Generative LLMs, such as BART [35], T5 [36], GPT [37], and Llama models [38], have been used for TRE by casting the task as a text-to-text problem. Different input or output representations impact the performance of encoder-decoder models (T5 and BART) on the Clinical TempEval 2016 dataset, showing that prompting one event at a time yields the best performance [39]. Saiz and Altuna [40] fine-tuned Relation Extraction By End-to-end Language generation [41], a BART-based sequence-to-sequence model originally designed for general TRE, on the 2012 i2b2 corpus, achieving competitive results. Some research work explored the use of ChatGPT in zero-shot and few-shot settings for the TRE task [42,43], reporting that its performance remains limited for long-distance dependencies. A recent review study [44] stated that generative LLMs are not widely used for clinical TRE. Furthermore, most efficient models are offered as privately owned services, limiting their use with sensitive data. The advent of open-weight models, most notably the Llama model family [38], has facilitated the use of LLMs on real-world data. A new generation of built-in reasoning LLMs has been released, such as DeepSeek-R1 [45] and Qwen3 [46], to enhance complex task solving.

Research efforts on open-weight LLMs for clinical TRE showed that zero-shot performance is generally lower than supervised approaches and that temporal consistency remains challenging [47]. An LLM-based method combining automated prompt optimization with few-shot in-context learning to predict temporal relations [48] was proposed in the 2024 ChemoTimelines shared task [49], noting that although fine-tuned smaller LLMs perform best, few-shot prompting can produce reasonable results in low-resource scenarios. Recently, Andrew et al [33] evaluated LLMs for extracting temporal relations from French pediatric clinical reports, showing that few-shot prompting with simplified binary labels outperforms multiclass approaches. Their results highlight the potential of LLMs for clinical timeline construction while handling sensitive data securely.

Latest research has examined prompt-tuning and prompt-learning as efficient alternatives to full fine-tuning in low-resource clinical relation extraction [50-52], though only a few have focused on clinical TRE [53]. An instruction fine-tuning method obtained the highest results [54] in the 2024 ChemoTimelines shared task, using the Flan-T5 model [55]. For the 2025 edition [56], Zhao and Vydiswaran [57] combined supervised fine-tuning of Llama models for temporal relation classification with zero-shot Qwen3 prompting for normalization and rule-based postprocessing, obtaining a second-place ranking. As described by Zaghir et al [58], there is a significant lack of open-source and locally deployable LLM-based approaches for real-world clinical applications, particularly for French.

In this paper, we use open-weight on-premises generative LLMs to extract temporal relations from French clinical narratives in zero-shot and few-shot settings, relying solely on in-context learning [59], that is, the ability of the model to learn from directly embedded instructions and examples in the prompt at inference time. Our study focuses on low-resource scenarios, where only few annotated examples are available. Our main goal is to extract the temporal expressions associated with clinical events. To this end, we cast the TRE task as a question-answering problem.

The main contributions of this paper are summarized as follows:

TRE involves identifying and classifying the relation between pairs of events, pairs of temporal expressions, or between an event and a temporal expression. In this work, we are interested in extracting the temporal relations between events and temporal expressions. As illustrated in the left panel of Figure 1, traditionally, the task is addressed by first constructing all possible event-temporal expression pairs, selecting relevant candidate pairs, and then classifying the relation between each pair. As shown in the right panel of Figure 1, we reformulated this task as a question-answering problem. Given a clinical text and a clinical event, the objective is to identify and extract all temporal expressions within the text that are associated with the event. If no such temporal expressions exist, the model should return “None”. Our aim with this task definition and the focus on a single relation type is to develop a reproducible and practical method suitable for real-world clinical use. This reformulation combines the selection of relevant temporal expressions and relation identification into a unified task. Moreover, we used a prompt-chaining strategy to perform normalization on the extracted temporal expressions.

We evaluated 4 generative LLMs, namely, Llama3.3 with 70.6 billion parameters, Qwen2.5 with 72.7 billion parameters, Qwen3 with 32.8 billion parameters, and DeepSeek-R1 with 32.8 billion parameters, all hosted on an on-premises Ollama server to ensure data privacy. Models were served in an 8-bit quantized format for efficiency without compromising performance, and a default context length of 32,000 tokens is used for all experiments. To minimize the variance in outputs, we used a temperature parameter of 0. For the built-in reasoning models Qwen3 and DeepSeek-R1, the maximum number of generated tokens was limited to 2000 to prevent overly long reasoning.

We compare the performance of our LLM-based approach with 2 baseline methods: a proximity-based relation extraction method [68] and the deep learning–based OpenNRE relation extraction toolkit [16]. The following section presents some details regarding these 2 baseline approaches.

We evaluated our methods using the 3 main information extraction metrics: precision, recall, and F₁-score, focusing exclusively on the positive class (cases where a date is present, as opposed to “None”). Predicted dates were compared to gold-standard dates using exact match after lowercasing. Certain answers were allowed according to the prompt instructions: Report_date was counted as correct if it matches the document date, and Birth_date was considered correct for age- or birth-related events. 95% CIs were estimated using the empirical bootstrap method [69]. Each test corpus is sampled with replacement 1000 times, and the evaluation metrics were computed for each sample.

Evaluating LLMs remains challenging due to outputs with multiple valid interpretations, factual inconsistencies, and variability in structured tasks such as information extraction, which standard metrics often fail to capture [70,71]. Therefore, in addition to this strict evaluation, we performed an LLM-as-a-judge validation to account for minor variations in extracted temporal expressions. We prompted the gpt-oss:20b [72] model as an impartial temporal evaluator. For each clinical event, the model receives the clinical text, the list of predicted dates, and the reference dates. Each predicted date is judged as correct if it matches or is equivalent to a reference date at the same level of temporal precision (year, month, or day), and incorrect otherwise. Precision, recall, and F₁ are then derived from these judgments. This evaluation framework enables scalable and context-aware validation while leveraging the original clinical narrative.

Finally, we conducted a human-based validation on our best-performing experiments to provide a more relaxed assessment of the LLM outputs. We reviewed each predicted temporal relation and assigned a binary judgment (correct or incorrect) based on the clinical text. This precision metric measures the proportion of model predictions that are true positives among all predicted relations.

Note that the normalization step was only evaluated under strict exact-match metrics.

Carbon footprint in terms of CO₂ equivalent of our LLM-based experiments was estimated using GreenAlgorithms (version 3.0; University of Cambridge) [73], based on runtime, computing hardware, and location, covering the full extraction and normalization pipeline for our real-world corpus and the extraction stage only for the i2b2 corpus.

This work is part of the C’IL-LICO research project on ciliopathies. This project and its study protocol were approved by the French National Ethics and Scientific Committee for Research, Studies and Evaluations in the Field of Health (approval 2201437). The data processing was approved by the French Data Protection Authority (CNIL) with a waiver of informed consent under number DR-2023‐017//920398v1. The clinical data used in our study were deidentified in accordance with General Data Protection Regulation protocols and regulations, and all our experiments were conducted locally on secure machines. The 2012 i2b2 corpus is available for research purposes under a data use agreement.

Tables 2 and 3 report the exact-match performance with 95% bootstrap CIs across all models, prompting strategies, and input configurations for rare disease diagnoses and phenotype events, respectively. Few-shot prompting consistently outperforms zero-shot extraction for rare disease diagnoses, while for phenotype events, the gains are generally modest or inconsistent across models and input settings.

Configurations including rationale generation provide more balanced precision and recall, while those incorporating candidate temporal expressions achieve higher recall values, particularly when combined with rationale generation. In the zero-shot setting, the Text+rationale+givenDates configuration achieves the best results across all evaluated models for both phenotypes and rare disease diagnoses.

For rare disease diagnoses, the best results are obtained by Llama3.3 in the few-shot setting Text+rationale+givenDates configuration, achieving an F₁ of 0.73. Comparable performance is achieved by Qwen2.5, reaching an F₁ of 0.72 in the few-shot Text+rationale configuration. For phenotype events, overall scores are lower. The best F₁-scores are achieved by Qwen2.5 in the zero-shot Text+rationale+givenDates configuration, with an F₁ of 0.61, and in the few-shot setting with the same configuration, reaching an F₁ of 0.6. Llama3.3 and Qwen3 also yield competitive performance, with maximum F₁-scores of 0.59.

Across both event types, the reasoning Qwen3 and DeepSeek-R1 models obtain lower F₁-scores than Qwen2.5 and Llama3.3, with consistently higher recall and lower precision across most configurations.

Figures 4 and 5 illustrate how character-based distance between clinical events and temporal expressions influences extraction performance for rare disease and phenotype events with Qwen2.5 across all input settings. In both figures, the upper panels present F₁-scores computed within each distance bin, where distance bins are defined by percentiles of the distance distribution (Table S1 in Multimedia Appendix 1). The lower panels show the distance distribution using a Gaussian kernel density estimate.

For rare disease relations, F₁-scores are highest at short distances (up to 57 characters: 0.58‐0.73) and progressively decline across medium distance bins (57‐159 and 159‐315 characters: 0.44‐0.73 F₁) with a substantial performance drop at long distances (more than 315 characters: 0.20‐0.57 F₁ and F₁<0.30 for several zero-shot configurations). The Gaussian kernel density estimate curve indicates that most rare disease relations occur within shorter distances, whereas long-distance relations are less frequent but still more challenging.

For phenotype relations, F₁-scores remain relatively stable (0.52‐0.82) across the first 3 distance bins (13‐1097 characters) for most configurations. The configuration Text+rationale+givenDates achieved the highest performance (0.82 of F₁) in both zero-shot and few-shot settings in the shortest distance range (13‐97 characters). Performance progressively declined as distance increased, with F₁-scores dropping to 0.29‐0.36 beyond 1999 characters.

Results obtained using our LLM-as-a-judge validation framework for the TRE are presented in Tables S3 and S4 in Multimedia Appendix 1. This assessment allows for minor variations in predicted temporal expressions while maintaining strict temporal consistency. The results closely align with the exact-match evaluation but consistently yield higher F₁-scores across all models and event types.

To assess the reliability of the LLM-as-a-judge framework, we compare its judgments with human evaluations on a subset of examples. Across all configurations using Qwen2.5, the agreement between the LLM and the human annotator reaches an average F₁ of 0.96 for phenotypes and 1.0 for rare disease events, indicating a very high level of consistency.

Table 4 reports the proportion of predicted dates judged as correct by a human evaluator for rare disease and phenotype events for our best-performing Qwen2.5 experiments. For rare disease diagnoses, precision ranges from 0.65 to 0.85 across all configurations, and for phenotypes, from 0.6 to 0.71. These values exceed the exact-match precision (Tables 2 and 3), confirming a high precision of the extracted relations.

Table 5 presents the results of the proximity-based relation extraction method on rare disease and phenotype events. The method achieves higher performance on rare disease events, with an F₁ of 0.48, while performance is lower for phenotypes, with an F₁ of 0.24. The values in parentheses in Table 5 correspond to our best-performing LLM, Qwen2.5, which outperforms the proximity-based method across both event types (Tables 2 and 3).

Table 6 summarizes the performance of the OpenNRE-based models trained on clinical narratives using different encoders (CamemBERT, ModernCamemBERT, CNN, and PCNN), 4 input configurations, and 2 encoder representation strategies for transformer-based encoders, as previously described. Due to the 512-token limit of CamemBERT, which causes truncation of longer clinical texts, we restrict its experiments to passage-based configurations, rather than the full text. Similarly, for the CNN-based encoders, we report results only for the LimitedPairs+Passage configuration.

Across all encoders, CamemBERT achieves the highest F₁-scores on the passage-based configurations (up to 0.42 for phenotypes and 0.58 for rare disease events), while ModernCamemBERT yields lower overall performance, with maximum F₁-scores of 0.27 for phenotypes and 0.39 for rare disease diagnoses. For transformer-based encoders, the [CLS]-based representation generally outperforms the entity-based strategy, particularly for CamemBERT. Results also show that standard CNN-based models achieve high precision (up to 0.75) but very low recall (0.04), while PCNN-based models improve recall (up to 0.83) at the expense of precision (0.1‐0.21).

According to the exact match evaluation (Tables 2 and 3), our best LLM-based configurations outperform the OpenNRE-based models for both rare disease (0.73 vs 0.58 of F₁) and phenotype events (0.61 vs 0.42 of F₁).

Table 7 reports exact-match and LLM-as-judge evaluation results for zero-shot and few-shot for TRE on the 2012 i2b2 clinical concepts using Qwen2.5, the best-performing model in our previous experiments.

In the zero-shot setting, exact-match F₁-scores range from 0.30 to 0.40, with the best performance obtained with the Text+rationale+givenDates configuration. Similar results are observed under the LLM-as-a-judge evaluation, with a maximum F₁ of 0.40. In the few-shot setting, performance further improves, with exact-match and LLM-as-a-judge F₁-scores reaching 0.40 for configurations that include the candidate dates, with or without rationale generation. Few-shot prompting consistently increases recall, reaching up to 0.57.

Table 8 presents the exact-match evaluation of the end-to-end Qwen2.5 pipeline for TRE and normalization. The extraction component is evaluated under zero-shot and few-shot settings, whereas normalization is consistently performed in a few-shot configuration using 6 examples.

In the zero-shot settings, the highest F₁-scores are obtained when the model uses only the text as input (0.7 for rare disease and 0.62 for phenotypes). In the few-shot setting, performance increases to 0.72 for rare disease, whereas the phenotype score slightly decreases to 0.61, when rationale generation is included. These results indicate that few-shot examples enhance extraction quality for rare disease events while maintaining steady performance for phenotypes. Overall, performance remains stable across event types under the prompt-chaining configuration. The exact-match results for the remaining evaluated LLMs on the end-to-end TRE and normalization pipeline are presented in Tables S5 and S6 in Multimedia Appendix 1.

As shown in Table S7 in Multimedia Appendix 1, temporal extraction and normalization inference experiments are estimated to have produced a total of around 21 kg of CO₂ equivalent, including around 5 kg CO₂ equivalent from temporal extraction and normalization experiments on the real-world corpus and more than 15 kg CO₂ equivalent from TRE on the i2b2 corpus using only Qwen2.5.

Among the evaluated LLMs, Llama3.3 had the highest environmental impact (2.3 kg of CO₂ equivalent), while Qwen2.5 achieved comparable performance on our clinical corpus with a lower carbon footprint (1.5 kg of CO₂ equivalent). Qwen3 and DeepSeek-R1 produced less than 1 kg of CO₂ equivalent.

A complete overview of the carbon emissions is provided in Table S8 in Multimedia Appendix 1.

We compare our LLM-based approach with a proximity-based rule baseline and supervised OpenNRE-based neural models. The proximity baseline achieves an F₁ of 0.48 for rare disease diagnoses and 0.24 for phenotypes (Table 5), indicating that rare disease mentions are often temporally anchored by nearby expressions, whereas phenotype-temporal relations are more dispersed and varied, resulting in low recall. Several LLM configurations, including in zero-shot settings, substantially outperform the proximity baseline for phenotypes, with Qwen2.5 and Llama3.3 reaching F₁-scores of 0.55‐0.61 (gains of 31‐37 points). For rare disease relations, most zero-shot configurations also already outperform the proximity baseline, achieving up to 0.68 of F₁ (Qwen2.5, Text+rationale+givenDates), and only 6 few-shot examples are sufficient to exceed 0.7 of F₁.

As reported in Table 6, OpenNRE-based models, trained jointly on a limited dataset of only 117 phenotype relations and 30 rare disease relations, show highly variable performance depending on architecture, input design, and pair selection strategy. CamemBERT generally outperforms ModernCamemBERT, and entity-based embeddings yield more balanced precision-recall than CLS-based representations. Passage-level inputs with LimitedPairs achieve the best F₁ for rare disease relations (0.58), benefiting from local context and restricted candidate pairs, while FullText inputs improve recall for long-range links but reduce precision when allPairs are included. CNN-based encoders are inconsistent, often showing extreme precision-recall imbalances. Phenotype relations remain challenging, with the best neural configuration reaching an F₁ of 0.42, underscoring the difficulty of capturing dispersed, long-distance temporal relations in clinical text, especially given the low amount of annotated training data.

These findings demonstrate the effectiveness of our LLM-based approach, which with few-shot examples substantially outperforms both neural and rule-based baselines for rare disease and phenotype relations with minimal annotation effort. While proximity-based rules offer interpretability, neural models typically require a substantial amount of annotated data, and both approaches struggle with long-distance, dispersed, and implicit relations. In contrast, our LLM-based approach robustly handles such complexities, including surface variations and temporal reasoning, providing clear advantages in low-resource clinical settings.

As shown in Table 7, the evaluation of Qwen2.5 on the 2012 i2b2 clinical concepts demonstrates that our LLM-based approach generalizes effectively to a distinct clinical English dataset. While some studies [53] explored the fine-tuning and prompt-tuning of LLaMA and GatorTron variants on the original i2b2 task, reporting competitive performance, a direct comparison with our experiments is not straightforward since we adapt the corpus to our annotation scheme. These adaptations ensure consistency with our experimental framework but prevent a strict head-to-head comparison with prior LLM studies on the original corpus. Under this setting, Qwen2.5 achieves meaningful performance on 6008 annotated relations, with the best zero-shot configuration reaching an exact-match F₁ of 0.4 and recall of 0.47, while few-shot prompting with only 10 examples improves recall to 0.53 while maintaining the same F₁. These results are further validated by the LLM-as-a-judge evaluation. This performance is noteworthy, given the inherent difficulty of the i2b2 corpus, which is characterized by dense temporal annotations, implicit relations, and relatively low interannotator agreement [4]. Some errors of our LLM-based approach arise from the model’s tendency to infer temporal relations from contextual cues, occasionally producing plausible but incorrect links, while others may reflect omissions or inconsistencies in the gold annotations.

Overall, these results confirm that our LLM-based approach offers a robust and scalable solution for clinical TRE, even on challenging benchmarks, while remaining generalizable across datasets and languages in low-resource settings.

Evaluating LLMs for information extraction remains challenging because traditional metrics are limited not only in their ability to capture semantic equivalence but also in their capacity to account for contextual and annotation-related variability. LLMs frequently generate synonyms, abbreviations, paraphrases, or normalized temporal expressions that are clinically correct yet do not exactly match reference annotations [70,71,74]. We therefore conducted a human evaluation of Qwen2.5, selected as the best-performing model, to further assess the quality of its predictions beyond exact-match and LLM-as-a-judge evaluations. This human evaluation confirms that a substantial proportion of extracted temporal relations, initially judged as incorrect, is in fact correct upon contextual review. Table 4 shows that Qwen2.5 achieves 0.71 of human-validated precision for few-shot phenotype relations and 0.85 for zero-shot rare disease relations, exceeding exact-match precision scores (Tables 2 and 3). Overall, human validation recovers up to 32 points with respect to exact-match precision (0.85 vs 0.53, rare disease diagnoses), with gains ranging from 6 to 32 points across configurations.

Our qualitative analysis highlights several sources of error that account for the gap between automatic metrics and human judgments. A recurring issue is annotation incompleteness, which underrepresents the temporal information in clinical narratives. For instance, in “crise d’epilepsie en 2013” (ie, epileptic seizure in 2013), the event and its temporal relation are annotated, but subsequent mentions like “8 à 9 crises par an en 2015 à 3 crises par an en 2018” (ie, 8 to 9 seizures per year in 2015 to 3 seizures per year in 2018) include coreferring event mentions (“crises,” ie, seizures) that are not annotated, leaving the corresponding temporal relations to “2015” and “2018” absent from the gold standard. Consequently, model predictions for these relations are penalized, despite being clinically valid. Similarly, for relations in “rechute de la maladie en 2012” (ie, relapse of the disease in 2012), the model is penalized because the referenced disease entity is not explicitly annotated as an event. These cases constitute annotation omissions rather than errors in the model’s temporal reasoning.

Other mismatches arise from annotation conventions that could, for instance, prioritize certain temporal cues over others or separately annotate individual mentions of entities. In our scheme, report dates are annotated only when they correspond to clinically salient events (eg, consultation, hospitalization, or procedure dates), while document metadata are excluded. However, the model sometimes extracts nearby temporal references such as header dates (“Paris, 18/02/2021”) or validation timestamps (“letter validated on 17/09/2019”) but fall outside the annotation guidelines. Additionally, variant disease mentions such as “oligo-arthrite juvénile idiopathique” (ie, Oligoarticular juvenile idiopathic arthritis) and “AJI oligoarticulaire” (ie, Oligoarticular JIA) denote the same underlying disease but are treated as distinct in the gold standard, penalizing correct temporal associations across synonymous nomenclature. When both “AJI” (ie, JIA) and “arthrite juvénile idiopathique” (ie, juvenile idiopathic arthritis) are annotated separately, the model often assigns the same date to both mentions, but only one relation is counted as correct under exact-match scoring.

Another source of error arises from the intrinsic complexity of clinical text, including nonstandard formatting and ambiguous temporal scope. For instance, in “Le 05/05/2003: Poids=42 kg ...” (May 5, 2003: Weight =42 kg ...), the date should be associated exclusively with the weight measurement and not with subsequent events mentioned in the text. However, the model frequently incorrectly propagates such narrowly scoped dates to later events, resulting in incorrect temporal relations. Similarly, constructions such as “depuis la dernière hospitalization de jour: ...” (since the last day hospitalization: ...) can lead to temporal references being extended beyond their intended scope. The model also struggles with relative temporal expressions such as “ce jour” (on this day) when contextual anchoring is ambiguous. In sections describing disease history, it sometimes assigns the report date to longitudinal events, even when explicitly instructed to return “None” if no temporal expression is present in such sections. The model also tends to favor selecting a concrete date over “None” when candidate lists include both options, resulting in false-positive temporal links.

Finally, some errors come from prompt specification gaps. While negated event mentions were excluded from the gold standard annotations, the model was not explicitly instructed to ignore them. Thus, it occasionally extracts temporal expressions linked to negated events, which are counted as errors under exact-match evaluation. Overall, our findings suggest that adapting annotation protocols and evaluation frameworks to better reflect the capabilities of LLMs, such as considering synonyms, coreferring events, and contextually valid temporal expressions, could provide more accurate assessments of model performance in clinical information extraction.

The end-to-end prompt-chaining pipeline of Qwen2.5 integrates temporal expression extraction with subsequent normalization, illustrating the model’s ability to handle sequential clinical information tasks. In this setup, input configurations affect the extraction step, while normalization is consistently guided by a small set of 6 in-context examples applied to the previously extracted expressions. As shown in Table 8, rare disease events achieve higher combined F₁-scores than phenotype events, suggesting that extraction is the primary driver of overall pipeline performance. The stability of results across input configurations indicates that prompt chaining effectively preserves dependencies between extraction and normalization, limiting error propagation. Even with a small number of in-context examples for normalization, the model can reliably standardize temporal expressions, highlighting that improvements in overall performance are mostly attributable to extraction quality. Most normalization errors involved partial dates such as “July” being normalized as “0001-07-01” instead of “2021-07-01,” due to missing guidance for such cases and the lack of access to the surrounding clinical context during normalization. Conversely, some cases showed strong normalization despite extraction mismatches, when the model directly produced a normalized date rather than its original textual form, which is counted as incorrect for extraction but correct for normalization.

Overall, the stability of results across configurations indicates that the prompt-chaining approach effectively maintains dependencies between extraction and normalization tasks.

The estimation of carbon emissions of our inference experiments highlights a substantial variability across models, corpora, and prompting strategies. As summarized in Table S7 in Multimedia Appendix 1, the TRE experiments on the i2b2 corpus using Qwen2.5 generated the highest carbon footprint. This is mainly due to the scale of the corpus, which contains 120 documents and 10,087 relations (Table 1), compared to our clinical corpus, which contains 42 documents with 356 phenotype relations and 70 rare disease relations. Qwen3 and DeepSeek-R1 produced lower CO₂ equivalent (less than 1 kg), which could be due to the limitation of generated tokens we fixed to 2000 to prevent overly long reasoning of these models. As illustrated in Table S8 in Multimedia Appendix 1, few-shot prompting consistently increased carbon emissions compared with zero-shot prompting. The addition of rationale generation or given the list of possible dates seems to amplify costs due to longer input contexts and generated outputs.

The limited availability of annotated clinical data, which defines the low-resource settings targeted in our study, results in a relatively small evaluation dataset. Future work on larger annotated datasets would help further validate and generalize our approach.

Performance in temporal extraction is partly constrained by the dataset and task design. In our annotated corpus, nearly all temporal expressions associated with phenotypes or rare disease diagnoses are dates, with very few cases involving durations and almost none involving frequencies, limiting the assessment of model generalization to more diverse temporal phenomena. The fixed question formulation used for prompting further restricts coverage of complex or implicit temporal relations in clinical narratives, and annotation omissions or inconsistencies can penalize clinically valid predictions.

We evaluated multiple open-weight LLMs, but detailed qualitative error analysis and human validation were conducted only for the best-performing Qwen2.5. Additional experiments with smaller or more lightweight models may be needed, as they could offer lower computational cost for real-world deployment. We deliberately chose a zero- and few-shot prompting approach over fine-tuning to prioritize low-resource and easily deployable settings.

While the task of identifying and standardizing the date of an event may seem common, finding points of comparison with previous work remains difficult. We have tried to adapt the i2b2 corpus, which is widely used in the literature, to this task, but these adaptations do not allow for comparison with previous systems.

Carbon footprint measurements cover inference experiments only, excluding prompt refinement. These measurements were initially difficult to obtain reliably in the shared Ollama environment due to request queuing and dynamic model loading, though efficiency is a critical factor for clinical deployment.

In this paper, we evaluated 4 open-weight, on-premises LLMs for TRE from French clinical narratives in a low-resource setting. We framed the task as question answering, combining zero- and few-shot prompting with prompt chaining for temporal normalization. Using a constructed and annotated real-world French clinical corpus, we compared model performance against rule-based and neural baselines through exact-match metrics, LLM-as-a-judge evaluation, and human validation. Results show that strong performance is achievable with few in-context examples for both phenotypes and rare disease diagnoses. Human validation of the best-performing Qwen2.5 predictions demonstrates that clinically correct temporal relations are often underestimated by traditional exact-match evaluations. We further demonstrate the generalizability of our approach on the English 2012 i2b2 corpus. Overall, carefully prompted generative LLMs can deliver high-quality TRE and normalization. Future work should explore annotation and evaluation frameworks better aligned with the strengths of LLMs to enable more accurate and clinically meaningful assessments.