To assess the nutritional management status of patients with T2DM, we conducted an electronic questionnaire survey involving patients and doctors. The questionnaire was designed to address patients’ lifestyle habits and dietary cognition, including basic information (eg, height, age, gender, weight, and level of medical-related education), lifestyle habits (eg, frequency and duration of weekly exercise), dietary therapy cognition (eg, selection of common foods), and evaluation of medical quality in terms of dietary therapy (eg, doctor-patient communication, available medical resources, and expectations). The survey was conducted electronically and disseminated via QR codes by the researchers from May 1, 2023, to May 16, 2023. The location was the Endocrinology Department of Peking University Shenzhen Hospital. The specific inclusion criteria were as follows:

The research procedure complied with the national legislation and local guidelines in China and was approved by the Ethics Committee of Peking University Shenzhen Hospital (PUSH2023069).

To systematically assess the expertise of ChatGPT in dietary therapy, we conducted a systematic evaluation from the perspective of nutrition and the most challenging ketogenic diet. We asked all questions in Chinese and translated the results into English for presentation.

Using the multilabel image classification model and ChatGPT application programming interface, we developed a WeChat miniprogram, FoodMed Companion (FMC), for patients with T2DM to complement medical visits. The decision to create a WeChat miniprogram instead of a standalone app was because WeChat has over 1 billion monthly active users in China. Android and iOS users can access the WeChat miniprogram without downloading a separate app. The main functions of FMC are diet recording and nutrition consulting. Users enter their disease-related information and the diet their doctor prescribed the first time they use the program. Using the Multilable Image Recognition Model model, they can obtain quick ingredient recognition and instruction by uploading their meal pictures and recording their blood sugar levels before and after meals for diet tracking. The program displays recognized ingredients in three colors, as follows: (1) green for foods recommended from the selected diet for best practice, (2) yellow for foods that require limited consumption under the best practice of the selected diet, and (3) red for foods of the selected diet that are not recommended under the best practice. Patients can communicate directly with the AI nutritionist or send their list of ingredients to the AI nutritionist for pre- and postmeal consultation, achieved by prompt engineering.

Based on the conditions described in the Methods section, we collected results from 206 patient questionnaires and 26 endocrinology clinical doctor questionnaires. The questionnaire results and option distribution are presented in Tables S1-S2 (Multimedia Appendix 1). Figures 2 and 3 provide visualizations of some representative results. According to the survey results, patients diagnosed with prediabetes comprised the largest proportion, accounting for 55.34% (114/206) of the surveyed population. Most of them (161/206, 78.16%) were between 20 and 60 years of age, mainly prediabetic, or with a disease duration of less than 5 years (Figure 2). Regarding lifestyle habits, only 14.56% (30/206) of patients would persist in exercising. In terms of MNT knowledge, 85.44% (176/206) of patients believed that unhealthy eating habits were the leading cause of diabetes, and 50% (103/206) of patients believed that dietary management itself is the most effective measure to lower blood sugar (Figure 3A). However, in terms of food cognition, many people believe that pork belly can raise blood sugar more than carrots and yams (Figure 3B), and many still need knowledge of the glycemic index (Figure 3C). The results of the medical quality assessment questions showed that less than half of the patients (79/206, 38.35%) felt that the doctor gave practical and detailed dietary advice. Most patients (182/206, 88.35%) hoped to obtain immediate and comprehensive dietitian services to guide them in controlling blood sugar through an appropriate diet, but they wanted it to be affordable (Figure 3D). The results of the clinical doctor survey showed that 53.85% (14/26) of the doctors believed that dietary control was an effective way to control diabetes. Almost all doctors (25/26, 96.15%) believed that unhealthy eating habits are one of the leading causes of diabetes. However, only 34.62% (9/26) of them could comprehensively assess the patient’s condition and give appropriate and detailed dietary advice.

The experimental result of the Dino V2–based model and the inverse cooking model on the validation set of 11 ingredients is shown in Figure 5. The Dino V2 model achieved an average F₁ score of 0.825, whereas the inverse cooking model achieved an average F₁ score of 0.477.

This study presents the development and validation of an AI nutritionist program designed to address China’s growing epidemic of T2DM. Recognizing the limitations of traditional preclinical evaluation methods for AI technologies, we have used alternative approaches to ensure the safety and effectiveness of our program. Our results show strong evidence that the LLM-based AI nutritionist program can perform tasks comparable to a registered dietitian in both examination and clinical settings.

Our study reveals a significant demand for more immediate and comprehensive nutritional management and education among patients between 20 and 60 years of age predominantly diagnosed with prediabetes or diabetes within the last 5 years. Time constraints prevent these patients from regular exercise, a crucial lifestyle modification. Although exercise is more effective than diet control in managing diabetes, dietary control alone still proves highly effective [23,24]. However, nearly half of the patients surveyed incorrectly identified pork belly as the food with the highest glycemic index among the other options, including carrot, yam, and onion. This suggests a limited understanding of nutrition. Alarmingly, 23% of endocrine doctors also made the same error, casting doubt on their reliability as nutritional knowledge sources in China. These findings highlight the urgent need for accurate nutritional education and resources for both patients and medical professionals. Moreover, research from Europe has also shown that the duration of patient interaction with nutritional educators is crucial for managing and reducing hemoglobin A_1c levels [25]. However, the limited supply of qualified dietitians in China restricts accessibility, depriving most patients of the opportunity for sufficient nutritional education from a professional. To compound the situation, most patients reported having only 10-minute interactions with their doctors during each medical visit [26]. Considering these findings, LLM-based software could offer accessible care to Chinese patients with diabetes, providing essential support for nutritional management and education.

Previous research in medical LLMs has demonstrated that ChatGPT excels in responding to open-ended questions, but its performance tends to falter during examinations [27,28]. Our study revealed that the latest ChatGPT and GPT 4.0 models successfully passed the Chinese Registered Dietitian Examination, a challenging test with an annual pass rate of only 30%-40%. Most of the errors they made were information errors, which could be improved with more targeted training data. Nonetheless, the examination questions do not accurately reflect real-world clinical scenarios. Therefore, we evaluated ChatGPT’s ability to answer common patient inquiries and compared its food recommendations to established best practices. For safety considerations, we focused on the ketogenic diet, a challenging yet effective treatment for diabetes, as a primary example in our experiments. As anticipated, ChatGPT provided reliable answers to open-ended questions about the ketogenic diet and frequently reminded users to adhere to their doctor’s advice. Among the 6 responses that received a score of 1 from the nutritionist judges, 3 responses (questions 10, 20, and 27) were found to be less informative than the expert responses, 2 responses (questions 8 and 17) addressed the question from a different perspective, and 1 response (question 12) contained inaccurate information, incorrectly suggesting red meat should be avoided in a ketogenic diet. After the evaluation, we queried question 12 from the model 10 more times, and it did not repeat the error about red meat. This result serves as a reminder that the LLM operates as a statistical model, generating responses from its training data in a nontransparent or “black box” manner. In the future, an LLM specializing in medical consulting should find a balance between randomness and accuracy for more reliable use. We also discovered that ChatGPT’s food recommendations generally aligned with best practices. However, it needed more certainty when asked about Chinese-specific food ingredients. This issue likely arises from a need for more training data, as we found that open-source recipe data often excludes these ingredients [16].

Class imbalance happens in multilabel classification problems where the classes exist in long-tailed distribution, resulting in reduced recognition accuracy [29]. This issue frequently happens in ingredient recognition problems because a high proportion of food images consist of very few popular ingredients, thus making it challenging to construct an unbiased food image data set [30]. These problems had often impeded previous efforts in developing an effective ingredient recognition model [22]. For example, this severely affects the recognition of peanuts in the inverse cooking model (Figure 5). In this context, a universal feature extractor, such as Dino V2, simplifies and optimizes the development of a robust and efficient multilabel classification model. We effectively circumvented the challenges above by using single-label images to train each ingredient in its recognition model separately. This approach was made possible due to the capabilities of Dino V2, which can extract features pertinent to identifying all food ingredients, reducing each training image into 1024 numbers. In our research, we demonstrated that our model, based on Dino V2, significantly surpassed inverse cooking, a multilabel classification model trained on 1 million recipe images, which was previously hindered by the bottleneck mentioned earlier. Our layered model of 11 ingredients can identify most of the carbohydrate content in an image in less than 2 seconds, and the number of identifiable ingredients can be scaled up without additional time constraints. Furthermore, we can specifically enhance the recognition of a particular ingredient by adjusting the training data associated with that specific ingredient.

The successful management of diabetes hinges mainly on the patients’ adherence to the dietary recommendations provided by their health care provider [31]. Our innovative miniprogram, FMC, is designed to enhance this adherence by performing a series of tasks in 2 seconds. These tasks include recognizing and recording dietary contents at the ingredient level from images of the patient’s meals and providing immediate feedback on the appropriateness of each recognized ingredient in the patient’s diet. This rapid and efficient process represents a significant improvement over traditional management applications that require manual input, thereby increasing the patient’s commitment to maintaining a consistent dietary log and overall diet [6,32]. Moreover, FMC has the potential to revolutionize the methodology used in assessing patient adherence to dietary recommendations. Current research methods, which predominantly rely on patient questionnaires, often fail to represent accurately the patient’s dietary habits. New digital tools like FMC allow for more precise and rapid patient diet recording. Collecting these data, in turn, enables us to gain a more comprehensive understanding of patient adherence. Furthermore, it provides a more robust platform for studying the impact of various diets, thereby contributing to developing more effective dietary strategies for diabetes management.

Although we have shown that advanced LLM like ChatGPT demonstrated promising results in the registered dietitian exam, responding to frequently asked questions and providing food recommendations, it is essential to note that the model was exposed to only some possible questions a patient might ask. In addition, it can generate different answers to the same question. These limitations are inherent in any AI model training process, as predicting and including every potential query is impossible. An effective strategy to mitigate the concern about the LLM’s ability to respond to all possible patient questions at this stage could be to define a specific scope of questions that the model is trained and validated to handle. By focusing on common and critical nutritional queries, we can ensure that the LLM provides accurate and reliable responses within this defined scope. As the model continues to learn and evolve, the range of questions it can handle can be gradually expanded. This approach allows us to maintain control over the quality of the LLM’s responses while still benefiting from its ability to process and analyze language at scale.

Our positive results advocate for a clinical pilot study to advance our AI nutritionist program. This study should evaluate its effectiveness in a real-world clinical setting, focusing on patient adherence to dietary recommendations and the subsequent impact on their health outcomes. Furthermore, the pilot study should explore the potential of our AI nutritionist program to serve as a tool for continuous patient education, providing patients with timely and accurate nutritional information to help them make informed dietary decisions. Regarding safety measures, the AI nutritionist program must be limited to providing nutritional information and dietary recommendations. Additionally, all generated responses should be subject to human expert review within 48 hours. Upon discovery, the patient should receive immediate notification if an incorrect response is generated. This approach ensures the safety and accuracy of the information provided while maximizing the potential benefits of the AI nutritionist program.