This study was designed as a pragmatic evaluation using freely accessible artificial intelligence tools (AI) for literature screening and summarization of articles with five predefined questions to assess how retracted articles are handled. The trial was registered during the data acquisition phase in the Open Science Framework (osf.io) [14].

The predefined questions were designed to outline a pathway for investigating a new topic of interest: providing an overview based on the article’s keywords, identifying the most important research articles related to the defined topic, and summarizing the retracted article. The final two questions were directly aimed at the retraction status of the respective article. Each response was scored as correct (1) or incorrect (0).

We used AI tools specifically designed for scientific work (Consensus Version July 2025, SciSpace Version 1.5.1, ScienceOS Version July 2025, OpenEvidence Version July 2025) as well as generic AI tools (ChatGPT Version 4 and 5, Microsoft Copilot Version 3.0, Gemini Version 2.5, Perplexity AI Version July 2025, Claude AI Version 3.5). AI tools with trial versions insufficient to perform all necessary research questions were excluded.

This study did not involve human participants, patient data, or identifiable personal information and was limited to the analysis of publicly available literature and AI-generated outputs. In accordance with institutional and international research ethics guidelines, ethics committee approval was not required.

The following questions were used to guide the literature assessment:

The exact titles (excluding RETRACTED or any other details on the retraction status), authors, and keywords we used can be viewed in the supplement (Table S1 in Multimedia Appendix 1, [15-29]).

A binary scoring system (one point per question) was used to emphasize functional correctness in AI-assisted literature workflows, acknowledging that this approach sacrifices linguistic nuance. Responses using equivocal or nonspecific wording (eg, “controversial” or “heavily criticized” instead of explicitly stating retraction) were rated as incorrect, as such phrasing does not clearly state retraction status. This conservative approach was chosen to avoid overestimating AI performance where precise identification of retracted literature is critical.

Data extraction was performed between June and September 2025. A total of 675 prompts were submitted, each asked twice across all models.

The primary aim of this analysis was to assess how often each GenAI tool achieved a perfect score of 5/5 correct answers per question set. None of the evaluated models consistently achieved full accuracy. Three tools - SciSpace, ScienceOS, and Consensus - failed to provide any completely correct responses. The best-performing model was ChatGPT 5, which achieved 8 out of 15 perfect results (53.3%). Model-specific performance data can be seen in Figure 1.

To assess the risk of incorporating retracted studies into literature syntheses, topic overviews and combined topic overviews with article summaries were analyzed separately.

When examining only topic overviews (ie, the first two questions), Microsoft Copilot gave an incorrect answer in 2 (13.3%) out of 15 cases. SciSpace, on the other hand, had the most incorrect cases with 8 out of 15 cases. Exact model performances are presented in the supplementary file figure S1 in Multimedia Appendix 1 and Table 1.

To assess the risk of incorporating retracted studies into literature syntheses, topic overviews, and combined topic overviews with article summaries were analyzed separately.

Each GenAI tool was asked every question twice to assess consistency. Differences between repeated answers resulting in a rating change occurred in each GenAI tool. ChatGPT 4 had the lowest variability with only two inconsistent answers (2.67%). Consensus had the highest incidence with 19 differences, approximately 1 in every 4 questions. Detailed results for each AI are shown in Table 1. We additionally calculated Cohen kappa, which ranged between 0.43 in Perplexity to 0.92 in ChatGPT, with an overall Cohen kappa of 0.73 (table S3 in Multimedia Appendix 1)

General-purpose AI tools achieved a median of 6 fully correct responses out of 15 (range 4‐8), corresponding to an aggregated accuracy of 40%. In contrast, research-focused tools did not achieve any fully correct responses (median 0, range 0‐0). The absolute difference in aggregated proportions between groups was 40 percentage points. Given the limited number of tools per group, this comparison is descriptive and exploratory.

Keyword overlap was assessed using a moderately generous matching strategy designed to account for realistic linguistic variation while maintaining methodological rigor. Specifically, orthographic variants (eg, “COVID-19” vs “covid19”), singular and plural forms (eg, “risk factor” vs “risk factors”), hyphenation differences (eg, “meta-analysis” vs “meta analysis”), and established abbreviations versus full terms (eg, “RCT” vs “randomized controlled trial,” “AI” vs “artificial intelligence”) were treated as equivalent. In addition, clearly synonymous or lexically overlapping technical terms referring to the same scientific entity or method (eg, “Alzheimer’s disease” vs “Alzheimer pathology,” or “clinical trials” vs “clinical trial”) were considered matches. In contrast, broader conceptual or hierarchical relationships were explicitly excluded from matching; for example, general versus specific terms (eg, “mental health” vs “depression”), different abstraction levels (eg, “hypertension” vs “blood pressure control”), or loosely related thematic concepts (eg, “public health” vs “disease prevention”) were not treated as equivalent. Pairwise comparisons showed substantial keyword overlap across all model pairs, ranging from 68% to 78%. The greatest overlap was observed between Claude and Perplexity (78%), followed by ChatGPT-4 and Perplexity (74%). ChatGPT-4 and Claude, and Gemini and Claude, each demonstrated 72% overlap, while Gemini and Perplexity showed 70% overlap and ChatGPT-4 and Gemini showed the lowest overlap at 68%. Comparable overlap magnitudes are observed both in comparisons involving the reference model and in comparisons among nonreference models, indicating a high degree of thematic convergence across LLMs. This matching strategy was selected to balance linguistic flexibility with analytical conservatism, avoiding both overly strict string-based matching and overly permissive conceptual aggregation. Nonetheless, we want to raise awareness that ChatGPT may have had a benefit as the keywords used for our search were generated with ChatGPT 4 and no perfect overlap was shown between AI tools.

In this cross-sectional evaluation of 9 freely available GenAI tools, none were able to consistently recognize or correctly handle retracted scientific publications. Across all 5 predefined research tasks, no model achieved perfect accuracy, and even the best-performing system, ChatGPT 5, correctly processed less than two-thirds of retracted articles (8 of 15; Figure 1). By contrast, AI tools marketed specifically for scientific use—Consensus, SciSpace, and ScienceOS—performed particularly poorly, each failing to produce a single fully correct set of answers. As shown in Table 1, intra-tool variability was observed across all GenAI systems, with discordance rates between repeated queries ranging from 2.67% in ChatGPT 4% to 25.33% in Consensus, indicating that even the most consistent model failed to produce fully stable responses. These findings collectively indicate that current GenAI systems cannot reliably detect or flag retracted literature, even when explicitly queried about retraction status.

The key result of this study is the persistent inability of LLMs to identify retracted papers or to transparently communicate their status to users. This deficiency was observed both in exploratory topic overviews and in direct article queries, suggesting that the problem is systemic rather than situational. As illustrated in Figure 2, the models most frequently cited retracted studies without warning, and even when asked whether a given article had been retracted, many systems returned incorrect statements. These outcomes highlight a fundamental gap in the factual grounding of current AI tools—a limitation that directly undermines their use in scientific research and evidence synthesis. Our analysis cannot determine the exact reason why retracted articles are not marked correctly. Possibilities include the insufficient data retrieval from the original databases (including missing source details or incomplete retrieval) or errors in the reasoning of the respective GenAI. Nevertheless, the scientific end user, who ultimately carries the responsibility for the correctness of his or her article, should be aware of and cautious of this risk.

The contrast between general-purpose and domain-specific AI systems is notable. While one might expect specialized research tools to perform better, they were instead among the weakest performers. Several of these systems rely on static, proprietary databases that are not regularly synchronized with dynamic resources such as PubMed or CrossMark. This architectural rigidity may explain their inability to capture recent retraction events, especially among newly withdrawn papers included in our sample. Conversely, models such as ChatGPT 5 and Claude, which use broader retrieval mechanisms and more frequent model updates, achieved somewhat higher accuracy but still lacked robustness and transparency. Importantly, the results presented represent a performance snapshot of freely accessible AI tools at the time of data collection and may not generalize to subscription-based versions or to future model iterations, which may incorporate improved retrieval mechanisms or retraction-aware features.

Our findings extend previous work that has documented the use of retracted literature by GenAI. Gu et al [11] reported that ChatGPT produced answers referencing retracted oncology studies, without acknowledging their retraction status, in more than 70% of tested prompts. Similarly, Jan et al [10] and Ge et al [6] have shown that GenAI tools frequently fabricate or misattribute citations during literature summarization tasks. Importantly, these earlier studies typically evaluated individual models or single fields within the scientific literature. The present analysis broadens the evidence base by systematically comparing multiple AI systems across heterogeneous scientific domains, including both general and medical literature. The consistency of failure across all tools underscores that this is not an isolated design flaw but a pervasive, cross-platform limitation in current GenAI architectures.

The continued citation and reuse of retracted work represent a well-recognized threat to research integrity and, in clinical disciplines, may even interfere with patient safety as illustrated by the infamous article of Wakefield et al [13,29,32]. Our results demonstrate that GenAI tools risk amplifying this long-standing problem by automating the propagation of unreliable information. Particularly concerning is the tendency of GenAI tools to reproduce incorrect content with high linguistic confidence, which may give users a false sense of authority and discourage verification. Without built-in mechanisms to cross-check bibliographic metadata or retraction notices, AI-assisted literature synthesis may inadvertently erode rather than enhance scientific reliability.

Given the accelerating adoption of AI systems in academic and clinical workflows, these findings have immediate practical implications. Researchers increasingly use GenAI tools to generate literature summaries, draft background sections, and identify relevant studies. In medicine and health sciences, where evidence accuracy is critical, the unrecognized inclusion of retracted data could distort systematic reviews, meta-analyses, and ultimately clinical recommendations. As such, reliance on unverified AI output in research and clinical contexts may indirectly contribute to patient harm by propagating retracted or unreliable evidence.

Importantly, expectations regarding the handling of retracted literature may reasonably differ depending on the intended purpose of an AI system. For AI tools explicitly designed to support scientific research or evidence synthesis, excluding retracted publications by default or clearly flagging their status aligns with established principles of research integrity and evidence-based practice. In contrast, for general-purpose AI systems, outright exclusion may not always be appropriate, as retracted articles can still be relevant for historical, methodological, or meta-scientific purposes. In such cases, clearly communicating the retraction status to users may represent a more practical and responsible design approach. Given that retracted information cannot be removed from a trained LLM, handling is more realistically achieved through post-generation checks rather than through changes to the underlying model itself.

To mitigate this risk, several structural measures should be prioritized. AI developers should integrate real-time bibliographic verification pipelines using authoritative sources such as Crossref or PubMed Retraction Notices. AI interfaces intended for academic use should provide transparent confidence indicators or citation provenance, enabling users to identify potentially unreliable results. Journals and peer reviewers should require explicit disclosure of any AI assistance in literature searches and verify cited sources independently. These steps are consistent with recent policy recommendations emphasizing “responsible AI use” and “model interpretability” in scientific contexts [33]. These recommendations align with JMIR editorial policies and ICMJE guidance, which emphasize transparent disclosure of AI use, author responsibility for content accuracy, and independent verification of cited sources.

This study offers a pragmatic, reproducible evaluation of AI performance under conditions that mirror real-world research behavior. By using predefined and repeated queries on a diverse set of retracted papers, the design captured both intermodel variability and intramodel inconsistency (Table 1). The inclusion of the 10 most-cited and 5 most-recently retracted papers provided a balanced sample, revealing that both historically influential and newly withdrawn studies were equally mishandled.

Nonetheless, several limitations merit consideration. First, the study was confined to freely accessible versions of each AI tool, excluding subscription models that may have more advanced retrieval capabilities and was taken at a specific point in time. Second, while the test set was broad, it remains a small subset of the retraction landscape and may not generalize to all fields, especially since the majority of articles (13/15, 86.7%) were medical articles. While this reflects the intended application of many AI-assisted literature workflows, the predominance of biomedical publications in our sample may limit the generalizability of these findings to other scientific fields, where differing publication cultures, citation structures, and retraction practices could influence AI performance. Third, AI models evolve rapidly; the specific results reported here represent a snapshot of performance as of mid-2025 and may not reflect future iterations. Fourth, given the limited number of retracted articles (n=15), the study was not powered to detect small-to-moderate differences between tools, and nonsignificant results should therefore be interpreted cautiously due to the risk of type II error. Accordingly, between-tool comparisons were considered exploratory and intended to describe observed performance patterns rather than to support confirmatory claims. Finally, the binary scoring approach (correct vs incorrect) captures factual accuracy but does not account for nuances such as ambiguous phrasing (heavily criticized and scrutinized). While distinguishing whether AI systems correctly identify the specific reasons for retraction could provide additional nuance, our analysis intentionally focused on the more fundamental step of detecting and flagging retracted literature, as failure at this stage already represents a critical risk for research integrity.

These factors should be addressed in future validation studies.

This study has several methodological flaws. One being that our way of questioning needed a uniform, comparable way. This was achieved by combining the keywords in a single prompt string, which could differ from a prompt by a common user searching for a specific topic. Furthermore, there are multiple ways why a retracted article was not cited by an AI. This could be due to the fact that the AI decided against citing it, due to an incomplete or outdated database, due to the fact that this article does not report data of interest, or that there are better articles describing similar aspects. In conclusion, we cannot specifically differentiate between all of those possibilities and therefore decided to create real-world judgment in which we just decide whether this article was included without warning or not. We additionally created an exploratory control group in which we used publication-order matched articles to estimate the ability of each AI to detect a specific article. While the results showed that AI tools found the respective article in up to 90% of cases, we want to highlight the limitation that this extraction was performed in newer iterations of each AI tool.

As GenAI becomes increasingly embedded in research ecosystems, systematic benchmarking and transparent reporting of factual reliability are essential. Future studies should expand sample sizes, include subscription-based models, and explore automated cross-validation with bibliometric databases. In parallel, developers should implement retraction-aware pipelines capable of dynamic metadata synchronization. Collaborative efforts between publishers, AI developers, and database curators could enable “AI integrity layers” that automatically flag or exclude retracted content before output generation. Ultimately, ensuring that AI systems respect the integrity of the scientific record is both a technical and ethical imperative.

No currently available free-access GenAI system can be considered reliable for detecting or handling retracted literature. Even the best-performing models failed to consistently recognize the retraction status of scientific articles, while research-focused tools performed particularly poorly. As AI tools become ubiquitous in the preparation and evaluation of scientific work, independent verification of sources and explicit retraction checks remain indispensable to maintain research integrity.