To evaluate the DUMS system’s real-world application performance, the prescription details of older patients with diabetes were extracted. After inputting the prescription details, the DUMS system extracts the established drug characteristics and constraints from the cloud-based drug database and does not adjust the output results (referred to as the “DUMS group”). Pharmacists then refined these preliminary results to create active output schemes that are better suited to clinical practice (hereinafter referred to as the “intervention group”). For comparison, the same prescription details were entered into ChatGPT-4, prompting it to generate medication schedules and self-management programs based on the drug label and relevant literature guidelines, where appropriate (hereafter referred to as the “GPT-4 group”; refer to Table S1 in Multimedia Appendix 1 for details).

Health care experts with professional titles at the associate senior level or above were invited to participate in the evaluation. They rated the results from the DUMS, intervention, and GPT-4 groups using a 5-point Likert scale following a horizontal comparison approach. Table 1 outlines the assessment indicators and their corresponding scoring criteria.

SPSS (version 27.0; IBM Corp) was used to statistically analyze the data. Measurement information was expressed as mean (SD). Independent-sample ANOVA was used, followed by the least significant difference test for multiple comparison. The expert opinion harmonization coefficient (w) was calculated using Kendall W, with differences assessed through the chi-square (χ²) test. A P<.05 difference was considered statistically significant. Furthermore, the Bland-Altman method was used to generate scatter plots for pairwise comparisons of the 3 methods and to assess the degree of consistency.

This study was reviewed and approved by the Hospital Ethics Committee of Xiangya Hospital, Central South University (approval 2023030556). This study exclusively used prescribing information for test evaluation. There was no direct interaction with patients, in line with the principles of ethical conduct in the study. All data were deidentified to protect privacy and ensure confidentiality, with access restricted to authorized members of the research team.

A total of 12 specialists, comprising 8 general clinicians and 4 pharmacists, participated in this study, with a mean clinical experience of 3.08 (SD 0.99) years. The evaluation involved prescriptions for 20 older patients with diabetes, through 60 medication plans presented in 3 different output modes. Among the patients, more than half (n=14, 70%) were female, with an average medication frequency of 12.65 (SD 4.10) doses per day. The basic information about the patient cases is shown in Table 2.

The Kendall W coordination coefficient was used to calculate the agreement between raters (Kendall W for overall mark=0.21; P<.001), suggesting that the different scoring systems are reliable in producing reproducible results. Then, we supplemented the comparison between the two groups using the Bland-Altman method (Figure 4), revealing a mean bias between the DUMS group and the GPT-4 group as 1.57. The 95% CI for the lower limit of agreement (LoA) was –0.66 to 0.65, and the 95% CI of the upper LoA was 2.50-3.81. The mean bias between the intervention group and the DUMS group was 1.60, with 95% CI –0.65 to 0.67 for the lower LoA and 95% CI 2.54-3.86 for the upper LoA. No points of difference exceeded 95% LoAs. Comparing the intervention group to the GPT-4 group, the mean bias was 3.18. Most of the data points fell within the LoA, indicating no significant systematic bias or trend. Although some bias existed, the overall consistency was good.

The evaluation results revealed that the DUMS group achieved significantly higher accuracy and safety scores than the GPT-4 group (P<.001), with full consideration of multiple aspects of DDI, pharmacokinetics, and dietary restrictions. However, in order to reduce the occurrence of adverse events, the medication frequency was increased, which may reduce adherence to the medication plan. Regarding medication schedule usefulness, the DUMS group scored slightly higher than the GPT-4 group, and both DUMS and ChatGPT-4 were able to provide valuable information for pharmacists to assist patients in taking effective medication, but the difference was not statistically significant (P=.23). The self-management programs provided by DUMS not only deliver easy-to-understand instructions on medications but also offer multifaceted guidance on disease education, lifestyle, and dietary management, with positive feedback on accuracy, comprehensiveness, and richness (P<.001).

The total evaluation scores showed that the mean values of the DUMS group, intervention group, and GPT-4 group were 37.96 (SD 0.56), 39.52 (SD 0.80), and 36.35 (SD 0.89), respectively. The “adjusted” DUMS results (ie, the intervention group) were better than those of the other groups, with statistically significant differences (P<.001), as shown in Table 3.

Faced with the challenge of ensuring that patients with diabetes take the correct medication at the right time, the emergence of telemedicine and AI has provided researchers with new directions. In this study, we established constraint management of essential pharmaceutical products by summarizing the relationship between drugs and evidence-based arguments and created an individualized medication reminder platform.

Our system extends beyond traditional auxiliary models by offering a visually interactive personalized medication plan and a user-friendly interface. Through secondary data analysis, we establish a robust medicine constraint management and DDI risk assessment framework to guide health care providers in precision medication timing. A variety of complications in patients with diabetes often leads to complex medication management and puts patients at an increased risk of moderate to severe DDIs, which may threaten their lives or may deteriorate their quality of life [26,27]. To mitigate the above risks, a triple rationale for DDI is provided in our system, allowing health care providers to balance multiple drug demands with the possibility of harmful drug poisonings. Additionally, the system promotes DSMES by strengthening blood glucose management, ensuring patient take drugs correctly and on time, and developing exclusive planning [28].

ChatGPT is an evolving model in natural language processing, as Microsoft and OpenAI continue to develop it, and its potential applications in medical education have garnered a great deal of attention [29]. Previous studies indicate that ChatGPT-4 performs well in providing descriptive and objective content, often generating largely accurate information for a variety of medical queries [30-32]. In this study, to ensure consistency, the commands entered into ChatGPT-4 were manipulated by a single researcher, who entered the following instruction:

However, the accuracy, comprehensiveness, and richness of the health education component of self-management provided by ChatGPT-4 could not replace that in the DUMS group and the “experienced” intervention group. A conceivable explanation is that, within the medical field, examinations often prioritize testing theoretical knowledge, frequently deviating from genuine patient-focused clinical environments. Such deviations might be more pronounced for LLMs, which are well versed in rapidly integrating available information. Another notable limitation is ChatGPT-4’s reliance on English-language training data, which affects its performance in other languages. The lack of culturally specific knowledge and the latest medical literature and data in other languages is a significant limitation of ChatGPT, which may lead to irrelevant or incorrect answers and conclusions [33,34]. Moreover, many scientific experts have pointed out that ChatGPT presents information redundantly and irrationally in most cases, offering repeated content and references, which may not be of obvious practical utility [35,36]. Although the development of machine learning and AI methods represents a significant technological advancement, and the output of ChatGPT-4 has a certain level of practicality, achieving reliable performance in various data scenarios still requires proper training of the model algorithms to maximize efficiency.

However, there are some limitations with the DUMS system. Its accuracy and usability rely heavily on the integrity of the drug information management database. Failure to update or expand the database promptly can result in suboptimal administration suggestions. Therefore, to continue expanding the number of drugs in the database and promptly improve the information resources, it is essential to develop a more efficient input mechanism for maintaining and managing the database, thus minimizing the risk of human errors or omissions. Additionally, comparing the performance of DUMS solely against that of ChatGPT-4 limits the generalizability of the findings to other intelligent systems or devices. Further validation is required to determine whether DUMS can perform consistently within its intended domain. Future iterations of DUMS will aim to enhance output quality by leveraging the imitative learning capabilities of LLMs, thereby improving the scientific and systematic nature of the output.

The DUMS system represents an innovative and cost-effective mobile health solution. Without making any changes to the existing health care system, health care institutions can deploy our proposed system to provide safer and more rational medication services for patients with diabetes. The drug information and constraint management database, combined with the user-friendly interface developed in this study, supports health care professionals in creating accurate, personalized medication schedules and self-management programs. In addition to serving as a reminder, the inquiry component allows older individuals living independently to receive timely solutions when treatment issues arise. However, it remains unclear whether mobile health can help health care workers improve patient health literacy. Significant further research will be required from managers and end users to ensure the safety, reliability, and effectiveness of this technology.