Our research comprised a qualitative study that aimed to directly compare automated analysis using ChatGPT and manual analysis by human researchers to evaluate the utility and challenges of AI in qualitative data analysis. Verbatim transcripts from semistructured interviews conducted in Japanese were used as a dataset to assess the performance of AI analysis. In particular, we compared the AI and human analysis results in terms of theme consistency and depth of interpretation, focusing on the differences in cultural elements and emotional nuances. The agreement rates presented in this study are derived from relative frequency comparisons of themes identified by ChatGPT and human analysis. As the General Thematic Approach does not involve systematic frequency coding per interview, quantitative agreement metrics such as Cohen kappa were not applicable. Instead, we report descriptive agreement rates to provide a structured comparison.

The data were derived from semistructured interviews centered on experiences of “sacred moments.” The participants were 30 individuals, including health care professionals and patients, from an urban community hospital in Tokyo (Hospital A) and a rural university hospital (Hospital B). The interview guide was developed based on previous studies [13] (Multimedia Appendix 1).

The interviews were conducted by experienced qualitative researchers, RS (General Medicine) and KS (Family Medicine). RS, a Japanese female with expertise in psychology and health-related research, has experience working at a Japanese university hospital. KS, a Japanese male, has a background in general medicine and public health, and has experience working at rural hospitals in Japan.

The interviews were conducted in a one-on-one format, with an average duration of 28.43 (range 16.75-48.00) minutes at Hospital A and 24.33 (range 15.42-40.20) minutes at Hospital B. All interviews were conducted in Japanese and focused on topics related to “sacred moments,” including participants’ experiences, the significance of such moments, and specific examples from medical practice. Transcripts were recorded verbatim (Multimedia Appendix 1). Participant selection criteria included intrinsic interest and experience related to the interview topics, with an emphasis on recruiting participants from diverse backgrounds to obtain a broad range of perspectives. The verbatim transcripts were transcribed using the automated NVivo (version 14; Lumivero) tool. Transcription accuracy was reviewed and the data were prepared for qualitative analysis.

To compare the results of ChatGPT’s analysis, the co-author RS (General Medicine) independently analyzed the interview transcripts without accessing ChatGPT’s output or prompts. The human researcher was provided with the same initial instructions provided to ChatGPT (“Please thematically analyze this transcript”) to allow for a flexible yet independent approach to analysis. Thematic analysis was conducted following the 6-phase approach proposed by Braun and Clarke [15]. First, the researchers familiarized themselves with the data by reading and re-reading the interview transcripts to gain an in-depth understanding. Second, initial codes were generated through the systematic coding of relevant features across the dataset. Third, similar codes were organized into potential themes. Fourth, themes were reviewed for coherence and refined to ensure consistency. Fifth, the final themes were clearly defined and named, refining their scope and specificity. Finally, the analysis was synthesized into a structured report, summarizing the key thematic findings. The human reviewer employed a General Thematic approach, which did not involve systematic frequency coding. This approach allows for greater flexibility in interpreting the meaning behind qualitative data without reducing the insights to numerical frequency counts. Cohen kappa could not be calculated due to the absence of data with structured presence or absence for each theme.

Data Saturation and Reliability

Data saturation was assessed throughout the analysis in accordance with the work by Hennink et al [18]. No new themes emerged after the 27th interview, suggesting that data saturation had been reached. To confirm the stability of thematic patterns, 3 additional interviews were analyzed, and the results remained consistent. To enhance the rigor and trustworthiness of the thematic analysis, we incorporated reflexivity, confirmability, dependability, credibility, and transferability as key methodological considerations. Reflexivity was ensured by maintaining a research journal throughout the analysis, allowing the researcher to critically reflect on potential biases. Confirmability was enhanced through systematic documentation of coding decisions and maintaining an audit trail. In addition, 20% of the interview data was independently recoded by a second researcher to assess reproducibility. Dependability was established through multiple coding consistency checks at different stages, with two independent researchers comparing their coding and resolving discrepancies through discussion. Credibility was reinforced by implementing a member-checking process, where selected participants reviewed preliminary findings to ensure the accuracy of identified themes. Transferability was considered by providing detailed descriptions of the data collection process and thematic categories, allowing researchers in other settings, particularly within East Asian medical contexts, to assess the applicability of the findings.

This study was approved by the institutional review board of Shimane University Hospital (approval number: KS20230706-1). As this study involved the secondary analysis of previously collected interview data, it was exempt from further ethical review under institutional guidelines. All participants were provided with a detailed explanation of the study objectives, procedures, and potential risks before their participation. Written informed consent was obtained from all participants, and no individuals who were recruited declined to participate. The informed consent explicitly allowed the reuse of collected data for further research purposes without requiring additional consent. To ensure privacy and confidentiality, all collected data were fully anonymized before analysis, and no personally identifiable information was included in the transcripts. The anonymized data were securely stored in a password-protected institutional repository, accessible only to the research team. Participants received a gift card worth 2000 JPY (approximately US $13) as compensation for their time and contribution to the study. No additional financial incentives or reimbursements were provided.

No images or figures included in this manuscript contain identifiable information about the participants. If any identifiable images were included, explicit consent from the individuals would have been obtained, and the relevant consent forms would have been submitted as supplementary materials.

Among the extracted themes, high consistency was observed for “personal experience of a sacred moment” (28/30, 93%), “strong connections and emotionally stirring experiences between healthcare workers and patients” (28/30, 93%), and “building relationships” (28/30, 93%). These themes exhibited agreement rates exceeding 80% between ChatGPT and human researchers, suggesting that ChatGPT demonstrates strong capabilities in identifying recurrent themes in qualitative data, even in Japanese-language contexts. Furthermore, the reproducibility of these findings emphasizes the potential utility of ChatGPT as a supportive tool for qualitative research.

Conversely, themes with strong cultural and emotional connotations, such as “difficult to answer, no experience of sacred moments” and “fate,” were identified in fewer than 3 interviews (<10%) and exhibited agreement rates below 30%. These results highlight ChatGPT’s limitations in interpreting Japanese-specific sociocultural expressions and implicit emotional nuances. To enhance interpretability, we have supplemented these themes with explicit definitions and representative example sentences. For instance, the theme of “fate” was characterized by statements where participants attributed life events or medical outcomes to forces beyond human control. An example statement includes: “No matter how much we try, some things are determined by fate.” This refinement enhances the transparency and objectivity of our results.

Using the grounded theory approach, certain themes demonstrated high occurrence and agreement rates, including “personal experience of a sacred moment” (22/30, 73%), “strong connections and emotionally stirring experiences between healthcare workers and patients” (21/30, 70%), and “building relationships” (26/30, 86%). These results suggest that ChatGPT can identify fundamental thematic elements that emerge consistently across different analytical frameworks. However, other themes, such as “dialogue” (4/30, 13%) and “fate” (0/30, 0%), showed lower frequency and agreement, particularly for culturally and interpretatively rich topics.

Similarly, under the five-step framework approach, high frequency and agreement rates were observed for “personal experience of a sacred moment” (25/30, 83%), “strong connections and emotionally stirring experiences between healthcare workers and patients” (22/30, 73%), and “building relationships” (25/30, 83%). These findings reinforce the trend observed in the grounded theory approach, indicating that ChatGPT consistently recognized themes that were explicitly described within the interview data. In contrast, themes requiring deeper interpretation, such as “benefits for patients and healthcare providers” (3/30, 10%), “difficult to answer, no experience of sacred moments” (2/30, 6%), and “fate” (0/30, 0%), exhibited significantly lower frequency and agreement rates. These results emphasize ChatGPT’s limitations in analyzing nuanced and culturally embedded themes, which require deeper contextual understanding beyond surface-level textual analysis.

In this study, we analyzed and compared 30 Japanese interviews focusing on “sacred moments” in clinical settings using ChatGPT and human researchers. Themes such as “personal experience of a sacred moment,” “strong connections and emotionally stirring experiences between healthcare workers and patients”, and “building relationships” were identified in over 83% of interviews and exhibited high agreement rates (≥83%) between ChatGPT and human researchers. These findings indicate that ChatGPT possesses strong capabilities in extracting frequently occurring themes within Japanese-language qualitative data, supporting its potential as a supplementary tool for qualitative research [19].

Conversely, themes deeply embedded in sociocultural contexts, such as “difficult to answer, no experience of sacred moments” and “fate,” demonstrated agreement rates below 30%, highlighting the limitations of AI in interpreting culturally and emotionally nuanced expressions in Japan. To improve the transparency of these findings, we supplemented key themes with explicit definitions and example sentences. For instance, the theme of “fate” was characterized by statements in which participants attributed life events or medical outcomes to forces beyond human control. An example statement includes: “No matter how much we try, some things are determined by fate.” This refinement enhances the interpretability of the results [20]. While ChatGPT’s ability to quantify themes provides a structured numerical comparison, human thematic analysis inherently emphasizes interpretative depth. This methodological difference should be considered when evaluating the comparative outputs of AI-assisted and human-led qualitative research.

These results align with previous studies that identified the strengths of ChatGPT in extracting descriptive themes [19,21]. Notably, its ability to achieve high agreement rates for descriptive themes in non-English languages underscores the potential of LLMs to support qualitative analyses across multiple languages [22]. Recent studies, such as those conducted by Pattyn [23], have explored the application of generative AI models in qualitative research. While these studies demonstrate AI’s potential in automating thematic identification and data synthesis, key challenges remain, including the risk of overgeneralization, context misinterpretation, and ethical concerns regarding AI-generated analyses. Our study extends this line of research by systematically comparing ChatGPT’s performance against human thematic analysis in a linguistically and culturally complex context, highlighting both its capabilities and its inherent limitations [24].

Future research should continue to explore best practices for integrating AI into qualitative methodologies, ensuring rigor, interpretability, and contextual accuracy. Furthermore, previous research has suggested that AI models trained primarily on Indo-European languages may struggle with Asian language structures [25]. Jalali and Akhavan [22] explored AI’s role in multi-language qualitative analysis and demonstrated its potential in thematic identification across diverse linguistic contexts. However, these studies primarily focused on European and Indo-European languages, which differ structurally from Japanese. Our study contributes to this body of research by assessing AI’s performance in analyzing qualitative data presented in Japanese, a language characterized by subject omission, honorifics, and indirect expressions [26]. This distinction highlights the need for further investigation into AI’s adaptability across linguistically complex settings, particularly in East Asian contexts [22].

While this study demonstrates valuable insights, several limitations must be acknowledged.

First, the study sample consisted of 30 interviews, which provides exploratory insights but limits broad generalizability. However, data saturation was reached after analyzing the 27th interview, and subsequent interviews confirmed the consistency of thematic patterns, thereby suggesting that key themes were well captured. Given that qualitative research prioritizes depth and rich contextual understanding over large-scale representativeness, our findings serve as a foundational step for future studies that may apply larger sample sizes across diverse health care settings to validate and extend these insights [20].

Second, this study focused exclusively on ChatGPT (GPT-4). Advancements in AI language models, such as GPT-4o and GPT-o1, may offer improved performance in analyzing culturally nuanced qualitative data. These newer models incorporate enhanced multilingual capabilities, improved contextual understanding, and greater adaptability to complex linguistic structures such as honorifics, subject omission, and indirect expressions in Japanese [27]. Future studies should explore comparative evaluations of these models to assess their potential in refining AI-assisted thematic analysis [28].

Third, limitations exist in ChatGPT’s ability to analyze deeply cultural and emotional themes. While most of the themes identified by ChatGPT aligned with those recognized by human researchers (themes 1-4), some themes (themes 5 and 6) required minor renaming to ensure consistency with human-coded themes. These adjustments were made to improve comparability while preserving the core meaning of the themes [29].

Fourth, methodological constraints should be considered. We recognize that future studies could explore structured coding schemes, such as Framework Analysis, to enable the calculation of interrater reliability metrics like Cohen κ or Krippendorff α. However, in this study, the focus was on interpretative thematic analysis rather than numerical coding [30].

Fifth, ethical and privacy concerns remain critical considerations in AI-assisted qualitative research. Although all data were anonymized and securely stored, further discussions are needed regarding data protection frameworks, algorithmic biases, and the responsible implementation of AI in qualitative analysis [31].

To further refine AI-driven qualitative analysis in Japanese, future research should focus on developing structured AI models adapted to the linguistic and cultural characteristics of Japanese. Collaborations between computational linguists, AI researchers, and domain experts could facilitate the construction of a Japanese-specific corpus tailored for thematic analysis. Adjusting the model architecture to incorporate context-aware processing mechanisms would enhance AI’s ability to interpret nuanced conversations [32]. Expanding training datasets to include diverse Japanese-speaking communities can further improve the robustness of AI-assisted qualitative analysis [33]. While statistical agreement measures could strengthen future comparisons between AI and human qualitative analysis, our study primarily focuses on conceptual consistency rather than numerical agreement. Future studies employing structured coding frameworks may facilitate statistical assessments such as the McNemar test or chi-square analysis.

In addition, as AI capabilities continue to evolve, comparative studies of newer models should be conducted to assess whether they offer improved interpretability and accuracy in non-English qualitative research [28]. Finally, a more extensive exploration of hybrid AI-human analytical frameworks may be beneficial in leveraging the strengths of both AI-driven automation and human interpretative depth [34]. To further validate the cultural specificity of extracted themes, future studies could involve independent expert reviewers who are blinded to the source (AI vs human). These reviewers would assess whether the themes represent culturally unique or globally universal concepts. Such external evaluations may enable quantifiable comparisons between AI- and human-generated analyses regarding cultural sensitivity.

This study demonstrated that ChatGPT possesses a high capability for extracting descriptive themes from Japanese qualitative data while revealing its limitations in addressing themes requiring cultural context and deep interpretation. These findings suggest a supplementary role for AI in qualitative research, emphasizing the continued importance of human researchers’ insights. Future research should focus on evaluating the applicability of AI across different cultural and linguistic contexts and address the ethical considerations associated with its use. Note: Language editing was supported by ChatGPT-4.0 (OpenAI) to enhance clarity and readability. All AI-assisted outputs were reviewed and validated for accuracy by the authors prior to submission.