Evidence-based medicine (EBM) represents a paradigm shift in clinical practice, fundamentally anchored in “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” [1]. This approach integrates clinical research with practical expertise to optimize therapeutic decision-making. Central to EBM implementation are the population, intervention, comparison, and outcome (PICO) elements—which serve as the cornerstone for formulating precise clinical questions and evaluating medical evidence. However, the exponential growth of biomedical literature poses significant challenges in efficiently extracting these critical elements from vast textual repositories.

Recent advancements in large language models (LLMs) have revolutionized natural language processing capabilities, demonstrating exceptional performance across diverse biomedical applications, including named entity recognition (NER) [2,3], knowledge graph construction [4,5], intelligent agent development [6,7], and question-answering systems [8,9]. The emergence of open-source LLM architectures (eg, Llama, Qwen, and DeepSeek) has further catalyzed domain-specific models, exemplified by specialized models such as PMC-Llama [10] and BioMedGPT-LM [11]. These developments show that LLMs can be used for recognition and extraction tasks in medical fields. Considering graphics processing unit (GPU) resource availability and deployment efficiency without compromising the generalization of the study, we selected the Llama 3.2-3B model (Meta AI), which has a small parameter size yet high model performance, to develop a two-stage PICO element recognition and extraction method: (1) validate the reasoning capability of prompt engineering based on the base model and (2) implement parameter-efficient fine-tuning (PEFT) based on Llama 3.2-3B. Our work makes two primary contributions:

In this study, “PICO recognition task" refers to assigning text segments to predefined PICO categories, which is a classification problem, and “PICO extraction task” refers to identifying the specific spans of PICO elements in text, which is an NER problem. Current mainstream approaches primarily encompass rule-based or dictionary-based methods, machine learning techniques, deep learning architectures, and LLM-based methodologies. This section reviews relevant research by analyzing the trajectory of technological evolution.

One stream is to use predefined matching rules or domain-specific lexicons for entity recognition. Demner-Fushman and Lin [12] pioneered rule-based pattern matching in 2007 for automated PICO element extraction from structured abstracts. Cohen et al [13] enhanced literature screening efficiency by integrating rule-based systems with basic statistical methods to identify PICO-containing articles. While these methods achieve high accuracy, they suffer from poor generalization capabilities and significant maintenance challenges for rule or dictionary updates.

Conventional machine learning implementations include support vector machines, decision trees, hidden Markov models, and conditional random fields. Notable applications include the naive Bayes classifier by Huang et al [14] for PICO identification in structured abstracts, the conditional random fields–based framework with feature templates for EBM text analysis by Hassanzadeh et al [15], and the n-gram–based ExaCT system for extracting population and intervention elements from clinical trial summaries by Kiritchenko et al [16]. Boudin et al [17] conducted comprehensive comparative analyses of multiple machine learning models. Although superior to rule-based approaches, these methods remain constrained by their reliance on expert-crafted feature engineering.

Deep neural networks have substantially reduced feature engineering burdens while improving performance. Predominant architectures include bidirectional long short-term memory, convolutional neural networks, and deep neural networks. Nye et al [18] developed an attention-enhanced sequence-labeling model validated on PICO annotation datasets. Wang et al [19] implemented bidirectional encoder representations from transformers–based PICO sequence labeling with multitask learning optimization. Stylianou et al [20] proposed an end-to-end PICO recognition system using recurrent neural networks. While they outperform traditional machine learning methods, deep learning approaches require substantial annotated training data.

The emergence of LLMs has catalyzed novel recognition methodologies requiring minimal domain-specific annotations through fine-tuning. These models have demonstrated notable successes in biomedical NER tasks [2,21-23]. However, research on PICO recognition and extraction using LLMs remains relatively scarce. This study focused on investigating LLM capabilities for this specialized task.

As depicted in Figure 1, this study explored the feasibility and effectiveness of LLMs for recognizing and extracting PICO elements in EBM literature through a 4-step workflow.

First, 3 EBM corpora—the PubMed-PICO and NICTA-PIBOSO datasets for the recognition task and the EBM-NLP dataset for the extraction task—were systematically collected, preprocessed, and divided into training and testing sets.

Second, various prompt engineering strategies were designed, including in-context learning (ICL), chain of thought (COT), and multi-path reasoning (MPR), to tailor the prompts for each task, and comparative experiments were conducted to evaluate the effectiveness of these strategies.

Third, building on the results of the second stage and using the best-performing prompt, a comparative study using PEFT was carried out. In this phase, methods such as LoRA, QLoRA, and freeze were applied to refine the model further, thereby providing deeper insights into the impact of different fine-tuning algorithms and ultimately yielding a robust, fine-tuned LLM.

Finally, the model’s performance was evaluated using standard confusion matrix metrics (precision, F₁-score, and recall), thereby quantifying its effectiveness in accurately identifying PICO elements.

This study investigated the performance of LLMs in PICO element recognition and extraction tasks. To achieve this objective, we used 3 datasets with the following configurations. For the recognition task, we used 2 high-quality annotated open-source datasets: PubMed-PICO [24] and NICTA-PIBOSO [25]. The PubMed-PICO dataset, developed in 2018, originally classifies each sentence into 7 categories: aim, population, intervention, outcome, method, results, and comparisons. As comparisons are often inconsistently reported and semantically overlap with intervention descriptions (eg, placebo, standard care, or other interventions), we excluded this category to standardize task settings across datasets. In addition, following related dataset conventions and prior studies on NICTA-PIBOSO [25,26], and aligned with our research objectives, we focused on the population, intervention, and outcome triadic recognition framework. The NICTA-PIBOSO dataset originally classifies each sentence into 6 categories: population, intervention, background, outcome, study design, and other. Following the aforementioned standardization principle, we likewise restricted our analysis to population, intervention, and outcome annotations in this dataset. The population, intervention, and outcome distribution characteristics of both datasets are summarized in Table 1.

For the extraction task, we used the EBM-NLP dataset [18], which differs from the aforementioned 2 recognition datasets in that it is specifically designed for NER tasks. This corpus adopts the begin-inside-outside tagging scheme to annotate PICO elements at the token level within abstract sentences, thereby satisfying formal requirements for entity boundary identification. Notably, individual sentences in this dataset frequently contain multiple entity types—a characteristic that necessitated strategic preprocessing to optimize LLM performance. Guided by the classic computer science principle of “divide and conquer,” we designed a prompt mechanism that allows multiple inferences to be made on the same text, each targeting only one category while preserving the original context. Finally, the model aggregates these different category extraction results and outputs them in JSON format.

For all 3 datasets, we adopted the official train-test splits provided by the original benchmark settings.

In addition to conducting comparisons of various prompt engineering strategies across different tasks, our investigation extended to an empirical evaluation of fine-tuning methodologies. Given the computational intensiveness inherent in full-parameter fine-tuning, this constraint has catalyzed the emergence of numerous PEFT algorithms. These approaches maintain model performance while requiring adjustment of only minimal parameter subsets.

Through an investigation of the compatibility between various PEFT algorithms, this study selected the following PEFT methods:

All model training procedures throughout this experimental framework were conducted on an NVIDIA GeForce RTX 4090 D 24-GB GPU. For PEFT experiments, the per-device batch size was 8, with gradient accumulation steps of 1, resulting in an effective batch size of 8. All models were trained for 2 epochs using the AdamW optimizer (β1=0.9; β2=0.999; ε=1 × 10^–8), a cosine learning rate scheduler, and a warm-up ratio of 0.1. Early stopping was not used. The learning rate was 5 × 10^–5 for freeze, LoRA, and QLoRA. The validation split ratio was 0.1. For LoRA and QLoRA, the LoRA rank and α were both set to 8, with a LoRA dropout of 0.05, and the target modules were q_proj and v_proj. For QLoRA, 4-bit quantization with NF4, double quantization, and BF16 compute data type were used. For freeze, we trained the last 2 layers. The compute data type was BF16 for all PEFT settings.

For performance assessment, we adopted well-established confusion matrix–derived evaluation metrics, with particular focus on 3 critical indexes: precision, recall, and F₁-score. We used macro–F₁-score (the unweighted mean of the class-specific F₁-score for population, intervention, and outcome) as it provides a balanced overall assessment of precision and recall while reducing bias toward majority classes. Precision and recall were also reported separately as false negatives and false positives may have different practical implications in EBM applications. For the extraction task on the EBM-NLP dataset, model outputs were parsed from JSON format and evaluated using strict exact matches. A prediction was counted as correct only when the extracted mention text exactly matched the gold-standard mention text for the corresponding entity type; malformed or incomplete outputs were treated as errors. These metrics were defined as follows:

Precision = TP/(TP + FP) (1)

Recall = TP/(TP + FN) (2)

F₁-score = (2 × TP)/(2 × TP + FP +FN) (3)

In these equations, “TP,” “FP,” and “FN” denote true positives, false positives, and false negatives, respectively. To assess result stability, all experiments were repeated 3 times, and the results are reported as means and SDs. Statistical analysis for pairwise method comparisons was performed using instance-level bootstrap analysis based on test set predictions, with F₁-score differences, 95% CIs, and approximate 2-sided P values reported. Detailed statistical test results are reported in Table S1 in Multimedia Appendix 1.

This study was a secondary analysis of publicly available datasets and did not involve new human participant recruitment or direct interaction. No identifiable personal information or images were involved. Therefore, additional informed consent and participant compensation were not applicable to this study. Details of the original data collection procedures are available in the original dataset publications.

This study systematically investigated prompt engineering and PEFT techniques for PICO element recognition and extraction driven by the practical demands of intelligent EBM development. Focusing on PICO elements in medical literature, we propose a novel framework integrating prompt engineering and PEFT to automate PICO element processing. Our methodology involved two key phases: (1) designing prompt templates for specific tasks, evaluating the basic reasoning ability of models based on prompt engineering, and using the optimal prompt configuration; and (2) based on prompt engineering, implementing PEFT to further optimize the performance of the model.

The experiment revealed 3 significant findings. First, our divide-and-conquer approach using COT prompting achieved the strongest average performance among the prompt engineering strategies across recognition and extraction tasks, indicating enhanced reasoning capability for interpreting complex medical contexts. This superiority suggests that structured cognitive prompting can effectively improve semantic comprehension and precision in recognition tasks. One explanation for COT’s superior performance is that the relatively small-parameter models used in our experiments may struggle to process the intricate reasoning steps inherent in MPR or the implicit reasoning demonstrations typical of ICL. Consequently, these models likely derive greater benefit from the explicit, step-by-step reasoning provided by COT. Furthermore, considering the moderate complexity of the PICO recognition and extraction tasks, a sequential reasoning strategy such as COT aligns well with guiding model cognition in an explicit and systematic manner, thus improving task-specific performance. Second, in PEFT, for the recognition task, LoRA showed the best results on the PubMed-PICO dataset, and QLoRA showed the best results on the NICTA-PIBOSO dataset. For the extraction task, QLoRA performed best. These results validate PEFT’s effectiveness in computing power–constrained clinical scenarios, providing technical support for deploying lightweight models in real-world medical applications. To the best of our knowledge, this work presents the first systematic comparison between prompt engineering and further PEFT for PICO processing. This establishes clear methodological guidelines for PICO tasks: the optimized prompt engineering strategy is feasible, and the task effect can be further improved through supervision and fine-tuning when the training data are sufficient.

There are still some limitations to this study. First, the relatively modest performance gains in extraction tasks (eg, outcome element F₁-score increasing from an average of 65.3%, SD 1.0% to 74.0%, SD 0.5%) highlight persistent challenges in entity boundary detection and long-range dependency resolution. We also observed extraction errors in cases involving ambiguous entity boundaries and sentences containing multiple relevant entities. These difficulties may be associated with the relatively small model size and the added complexity of the prompt design. For example, the prompt instruction that “only one category can be extracted from a text” may have introduced ambiguity when sentences contained multiple distinct mentions of the same PICO element type, potentially contributing to incomplete extraction. Second, as our experiments were confined to the Llama 3.2-3B architecture, whether the observed findings generalize to larger models, other model families, or medical domain–specific models remains unclear. Third, the framework has not yet been validated in real-world clinical decision support systems, so its practical utility still requires further verification. Fourth, class imbalance in NICTA-PIBOSO was not specifically investigated in this study. This should be addressed in future research. Finally, potential data contamination cannot be fully excluded for Llama 3.2-3B given that PubMed-PICO, NICTA-PIBOSO, and EBM-NLP are publicly available benchmark datasets; therefore, part of the baseline performance may reflect prior exposure rather than true zero-shot comprehension.

Future research should explore hybrid strategies that combine syntactic parsing with contextual augmentation to better address the challenges in entity boundary detection and long-range dependency resolution. Promising directions also include integrating LLMs with knowledge graphs [39] to capture implicit medical logic and using advanced models (eg, DeepSeek 671B) [40] to distill smaller LLMs, thereby potentially enhancing efficiency while achieving superior performance. In addition, implementing our framework in clinical decision support systems could further validate its real-world utility and extend its applicability to population, intervention, comparison, outcome, and study design (PICOS) framework analysis.

This study demonstrated that the application of advanced prompt engineering and PEFT techniques can substantially enhance the ability of LLMs to recognize and extract PICO elements from biomedical literature, achieving notable performance gains across diverse datasets. These findings underscore the potential utility of LLMs in the field of EBM and provide empirical support for future innovative endeavors in this domain.