Colorectal polyps are a significant risk factor for colorectal cancer (CRC). When polyps are identified during an index colonoscopy, patients may require further colonoscopic surveillance depending on the endoscopic and pathological characteristics of the lesions [1]. Colorectal polyp surveillance improves the detection of CRCs, particularly in high-risk groups defined by specific endoscopic and pathological findings [2-5]. Endoscopy represents a substantial workload in the United Kingdom, with around 2 million procedures performed each year. Polyp surveillance accounts for approximately 15% of the half a million colonoscopies performed annually [1,6].

Postpolypectomy surveillance decisions are particularly vulnerable to inaccuracies. Reported compliance with the updated 2020 postpolypectomy surveillance guidelines varies from 66.2% to 86.5% [7,8]. Poor compliance can often be attributed to surveillance intervals not being updated once pathology results become available [9]. Nonadherence to guidelines can lead to both under- and oversurveillance, with one study reporting 73% of nonadherent patients having earlier-than-recommended colonoscopies [9]. These findings highlight the need for continued efforts to improve guideline adherence in colorectal surveillance and the need for accurate calculation of surveillance intervals based on evolving guidelines.

Several studies have proposed to automate portions of the decision-making workflow, but they frequently rely on applying inflexible rules or pattern recognition to endoscopy and pathology report text (regular expressions) that struggle to accommodate variations in clinical report language. Existing studies are often limited to single-center data or focus on partial tasks such as adenoma detection without extending to full guideline-based follow-up recommendations [10-13]. Wu et al [14] developed an automatic surveillance system for postpolypectomy follow-up that encodes multiple international guidelines and combines free-text endoscopy and pathology reports to assign surveillance intervals. Their approach relies heavily on regular expressions, which are used to identify patients and map extracted polyp attributes. In practice, this design requires a long list of exclusion categories (no pathology result; poor bowel preparation; previous CRC, colorectal surgery, or endoscopic submucosal dissection; nonpolyp lesions; hamartomas; inconsistent endoscopic vs pathological reports; and missing or malformed size information) so that many real-world reports fall back to manual review rather than fully automated surveillance. The system was developed and evaluated in three hospitals in China, which may limit direct transferability to English-language histopathology workflows in UK practice.

Recent studies have evaluated general-purpose large language models (LLMs) for postpolypectomy surveillance interval prediction [15-17]. The larger study by Acharya et al [17] leveraged GPT-4o to generate recommendations from colonoscopy and pathology report impressions using prompted US Multi-Society Task Force guidance. These approaches treat surveillance interval generation as an end-to-end language model task, which leaves clinical decision-making implicit without auditable intermediate representation for clinicians to review.

Our study used a different task breakdown: explicit extraction of specimen-level variables followed by separately implemented British Society of Gastroenterology (BSG) guideline rules. This approach allows for evaluation of the extracted variables and the final interval suggestion. This also allows decision rules to be modified alongside guideline changes independently of the language model. Additionally, we compared automated recommendations with observed care under real-world deployment conditions, looking at patients’ full electronic health record (EHR) history beyond the index colonoscopy.

The primary objective of the current study was to develop and retrospectively evaluate an implementable and interpretable hybrid decision-making tool that accurately suggests follow-up timings for patients undergoing colorectal surveillance based on colonoscopy and associated histopathology reports grounded in BSG guidelines. We validated our tool on reports from two geographically disparate National Health Service (NHS) trusts in the United Kingdom (Guy’s and St Thomas’ NHS Foundation Trust [GSTT] and Sandwell and West Birmingham NHS Trust [SWB]).

A secondary objective was to assess the tool’s potential clinical utility in routine NHS care. We compared automated recommendations with observed surveillance timing provided by gastroenterologists’ review of full EHRs. This approach enabled us to understand whether automated decision support could reduce under- or oversurveillance and identify limitations of decisions based on the index reports.

Figure 1 summarizes the entire pipeline from data access to the evaluation of the AI component. We adhered to the TRIPOD-LLM (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis–Large Language Models) guidelines [18] and completed the checklist (Checklist 1) ().

Data access and processing at GSTT was authorized by the Guy’s and St Thomas’ Electronic Records Research Interface (Integrated Research Application System [IRAS] ID 257283; Research Ethics Committee ID 20/EM/0112). This approval was effective from June 3, 2023. Data access for SWB was authorized through a data sharing agreement between the institutions in compliance with the General Data Protection Regulation. This work used deidentified, routinely collected clinical data with no direct participant contact. Individual informed consent was not required under the governance and ethics approvals for the data access route. Patients in England can register in the National Data Opt-Out service to prevent use of their confidential patient information for purposes beyond individual care (research and planning).

We extracted colonoscopy and histopathology text reports from GSTT’s EHR system covering procedures performed between January 2017 and September 2023.

The pipeline aims to extract guideline-relevant polyp features from paired colonoscopy and histopathology reports and then apply the BSG guidelines to generate a follow-up recommendation. A key challenge is that the evidence needed for surveillance is fragmented across sections: endoscopic findings describe the lesions sampled, whereas histopathology reports describe multiple specimens (“pots”) with macroscopic and microscopic text that must be linked to determine which pots correspond to polyps, which represent nonpolyp tissue, and which findings are unresected or described only endoscopically. The GSTT test set was annotated by a committee of 3 clinical experts to establish the gold standard. For each procedure, annotators reviewed the endoscopic findings and the histopathology macroscopic and microscopic descriptions to derive the number of histology pots and extract pot-level variables listed in Table 2. “Premalignant” and “advanced” polyps were defined according to the BSG criteria [1]. Disagreements between annotators were resolved through consensus. When consensus could not be reached, a fourth clinical expert (SZ) resolved the discrepancy. The SWB test set was annotated by a single gastroenterologist (SZ).

The structured outputs produced by the final LLM prompt were parsed into histopathology pot-level variables (Table 2) and passed to a deterministic rule-based algorithm that encoded the guidelines. The algorithm first raised an alert if cancer was detected, then filtered out nonpolyp specimens; aggregated polyp counts and sizes across all pots; identified premalignant and advanced polyps using guideline-aligned criteria; and, finally, output a surveillance recommendation according to the BSG, Association of Coloproctology of Great Britain and Ireland, or Public Health England risk categories (no surveillance, 3-year surveillance, 5-year surveillance, or site check for large nonpedunculated polyps with uncertain excision). If age was missing, the algorithm ignored age-based conditions and defaulted to the non–age-specific guideline branches.

The pseudocode functions “IsPremalignantPolyp,” “IsAdvancedPolyp,” and “IsHighRisk” implemented guideline definitions of premalignant polyps, advanced polyps, and high-risk patients. In summary, premalignant polyps included adenomatous polyps and clinically relevant serrated lesions (excluding diminutive rectal hyperplastic polyps), and advanced polyps were defined by size and/or dysplasia according to the UK guideline thresholds. High risk was assigned when either 2 or more premalignant polyps were present, including 1 or more advanced polyps, or 5 or more premalignant polyps were present.

The algorithm compared the endoscopic polyp count with the number of histopathology specimens to detect specimen fragmentation and reduce pot-level size and count estimation bias. The algorithm preferentially used the endoscopic size as pathology-reported measurements may underestimate true size due to specimen shrinkage after formalin fixation [23]. The pseudocode for the rule-based algorithm can be found in Multimedia Appendix 3.

In addition to per-report extraction metrics, we performed a retrospective, real-world eligibility study where we compared the automated system recommendations and observed care to a clinician-derived reference based on full EHR review. The automated system used the index colonoscopy and associated histopathology reports, with age where available. This analysis aimed to evaluate how the automated surveillance recommendations would align with guideline-based care when full longitudinal patient information was available.

A single gastroenterologist (SDM) reviewed the complete EHR of all the patients in the retrospective test set (n=127). The clinician determined the gold-standard guideline-concordant surveillance interval for each patient accounting for but not limited to previous CRC or colorectal surgery, prior large or advanced polyps, hereditary or other high-risk conditions, and comorbidities or contraindications that would alter surveillance timing. We then compared 3 intervals: the clinician gold-standard interval from full EHR review, the interval recommended by the automated system using only the index colonoscopy and associated histopathology reports, and the observed care interval based on real-world scheduling (a completed follow-up colonoscopy or a future booking recorded in the EHR). Observed care was assessed using all information available in Epic. A completed procedure, future booking, or absence of surveillance were classified looking at all post–index colonoscopy data available in Epic. Reference assessment was blinded to model output. This analysis assessed the extent to which this system could improve alignment with guideline-concordant surveillance and, therefore, its potential impact on clinical practice. Full EHR review also identified patients whose management required a clinical pathway beyond the postpolypectomy recommendations generated by the system.

Table 3 shows that the extraction of key fields for the final surveillance decision was highly reliable (mean comprehensiveness 0.968, SD 0.112), with close to perfect accuracy for location, type of specimen, and histopathological size (0.981, 0.995, and 0.995, respectively). The model also showed high accuracy for diagnosis (0.981) and excision completeness (0.949). The endoscopic size, number of specimens, and extraction of unresected polyps showed the lowest accuracy (0.935, 0.935, and 0.903, respectively).

Table 4 summarizes the performance of the decision support system for surveillance recommendations at GSTT and SWB. The system achieved an overall accuracy of 0.96 at GSTT and 0.93 at SWB, with strong performance for site check (F₁-score of 0.93 at GSTT and 0.91 at SWB) and no follow-up (F₁-score of 0.98 at GSTT and 0.97 at SWB). Cancer alerting exhibited perfect performance (F₁-score=1.00) for both sites.

Detailed robustness results across perturbation levels are provided in Multimedia Appendix 5. Across progressively perturbed test sets (1 to 5 perturbations per report), F₁-scores decreased only modestly compared with the clean test set, indicating that the pipeline is robust to realistic typographical noise and minor formatting artifacts. In particular, performance on the number of pots, type of specimen, and diagnosis remained stable, whereas size extraction showed the greatest sensitivity to perturbations.

Using gold-standard guideline-concordant intervals derived from full chart review, we audited current practice within the department for 127 patients in the test cohort and compared it to the system’s suggestions. Overall, only 59.1% (75/127) of patients were booked in line with the recommended surveillance interval. A further 15.7% (20/127) deviated from guideline-concordant care: 5.5% (7/127) were potentially oversurveilled (either booked earlier than recommended or scheduled for an unnecessary colonoscopy), and 10.2% (13/127) were potentially undersurveilled (6/127, 4.7% were booked later than recommended, and 7/127, 5.5% had no surveillance booked despite needing one). A further 25.2% (32/127) were managed via alternative pathways (CRC or other colonoscopy pathways), where surveillance scheduling was driven by additional clinical findings (Table 5). Even excluding these alternative-pathway cases, 21.1% (20/95) of patients in routine care had potentially missed or unnecessary surveillance relative to guideline-concordant recommendations. In contrast, our system provided a consistent, automated, and guideline-based recommendation for every patient based on structured information from index colonoscopy and histopathology reports. Overall, the decision support tool improves alignment with guideline-concordant care (Table 5), particularly by reducing undersurveillance. These findings highlight a real opportunity to identify patients at risk of undersurveillance and potentially reduce unnecessary procedures, supporting efficient use of endoscopic resources.

This study introduced a decision support tool that combines open-weight, medium-sized language models for extraction of guideline-relevant information with a deterministic rule-based algorithm for postpolypectomy recommendations. Focusing on implementation encouraged the use of a computationally lightweight LLM to only perform report standardization and information extraction. We demonstrated that such a decision aid system presents an opportunity to improve guideline adherence by conducting a retrospective chart review using patients’ full medical histories, showing that routine surveillance sometimes occurs later than guideline-recommended intervals. We identified cases with no indicated surveillance booking, highlighting a concrete opportunity to strengthen safety netting and reduce missed surveillance in routine care.

Acharya et al [16,17] evaluated direct LLM-based surveillance interval assignment in an initial 100-case feasibility study and subsequently in a 2-institution cohort of 1000 cases. The latter study reported a mean GPT-4o accuracy of 94.6% across 10 runs. Our recommendation accuracy was 96% at GSTT (95% CI 90%-98%) and 93% at SWB (95% CI 86%-97%). These independently obtained results cannot be compared directly as the studies differ in guideline systems, populations, output categories, and evaluation scale. The later study by Acharya et al [17] also provides evidence from a substantially larger validation cohort.

In the larger study by Acharya et al [17], accuracy was lower for cases containing 4 or more polyps than for those with 1 to 3 polyps. This finding highlights the importance of evaluating complex cases. Our framework separates specimen-level extraction and aggregation from deterministic guideline application, which facilitates granular evaluation at the individual polyp level.

There are several key strengths to this study. While general-purpose LLMs are increasingly used for a wide range of tasks, our work focuses on a specialist clinical setting in which smaller open-weight models are applied to narrow, auditable subtasks. This design choice supports transparency and increases clinician confidence by making the extracted information and decision logic available for inspection. We divided the overall task into components that closely mirrored clinical practice: collating endoscopy and histopathology reports, extracting information pot by pot, aggregating findings across pots, and deriving a surveillance recommendation. It is important to highlight that the language model was only tasked with extracting and normalizing information already present in the reports and did not generate the surveillance recommendation. The LLM did not generate any clinical decision-making, and the surveillance decision itself was made by a rule-based algorithm anchored in the guidelines, which can be audited and updated independently as guidelines change. By separating extraction from decision-making, the framework could also be adapted to other gastrointestinal surveillance pathways by modifying the extraction layer and introducing condition-specific rules.

The evaluation of the language model information extraction was performed using a range of metrics (accuracy, comprehensiveness, F₁-score, robustness, and deployment considerations) that provide a more complete picture of performance than accuracy or F₁-score alone. The SWB evaluation provided evidence of performance in a second environment with differing documentation styles and reporting practices subject to its small sample size and the restriction concerning age-dependent rules.

This study deliberately focused on small- to medium-sized language models to reflect typical on-premises hardware constraints in NHS settings. While some hospitals can process clinical text via external LLM APIs, this capability varies widely across institutions due to governance and privacy requirements, and on-premises inference is generally preferred.

Future work should include larger and more geographically diverse sites to better assess robustness and generalizability.

The retrospective analysis highlighted that a subset of patients (n=32) required referral to alternative pathways (eg, inflammatory bowel disease or ulcerative colitis), which is currently out of scope for this study. This reinforces why this study introduced a decision support tool rather than a fully automated system: clinicians sometimes need to combine multiple guideline pathways and clinical context to make the correct decision. The SWB and the retrospective audit annotations were provided by a single gastroenterologist, which may introduce reference standard bias.

The two 100-procedure test sets provided limited precision, particularly for minority classes. Larger cohorts with diverse representations of rare outcomes are needed to draw broader generalizability conclusions. The small number of ALERT cases resulted in wide CIs.

The GSTT test set was older (mean 69.4, SD 9.35 vs 57.3, SD 14.50 years) and had a higher recorded proportion of procedures involving White patients (93/100, 93% vs 7625/14,735, 51.7%). Interpretation was also affected by the larger combined “other or not specified” category in the source cohort (4610/14,735, 31.3% vs 3/100, 3%). These differences limit representativeness and may reflect differences in the sampled clinical population. The small numbers in some demographic groups precluded formal subgroup performance analyses across sex or ethnicity. Instead, we report overall performance and highlight fairness as an important direction for future work, particularly as larger and more diverse validation cohorts become available.

Finally, although we compared models of different sizes, the final configuration used a 14 billion–parameter model with 4-bit quantization on the reported hospital server. While our institution has the necessary infrastructure, we acknowledge that this may not be the case in other settings. However, the modular design and code available upon request allow the language model component to be swapped for smaller or fine-tuned alternatives, enabling adaptation to local computation constraints.

This study demonstrates that a hybrid automated system can generate accurate and auditable postpolypectomy surveillance recommendations from paired colonoscopy and histopathology free-text reports. The framework enables inspection of intermediate information and rule application. External validation across two NHS trusts suggests that the approach could generalize across different reporting environments. The retrospective comparison with routine care suggests potential improvements over current surveillance practice in specific patient groups. It also identified 25.2% (32/127) of patients requiring alternative clinical pathways, highlighting the importance of broader patient history. Future research could therefore investigate whether integrating longitudinal clinical information helps distinguish patients eligible for routine postpolypectomy surveillance from those requiring alternative pathways. Future research could also explore whether smaller open-weight language models can be used to extend this framework and more broadly support gastroenterology surveillance pathways under realistic deployment constraints.