This exploratory evaluation used quantitative and qualitative methods and was conducted with community members and health care providers recruited from Indlela’s Behavioral Hub (B-Hub) in Gauteng province, South Africa. The B-Hub is a research initiative focused on testing behavioral interventions for HIV prevention and care [18]. The hub is located at the Health Economics & Epidemiology Research office (HE²RO) in Johannesburg, a large urban metropolitan area. Participants were recruited from urban and semiurban communities across the province. Despite being the smallest province in South Africa, Gauteng is the most densely populated province with an estimated population of 16.1 million people. The province is highly urbanized and serves as the country’s economic center, attracting people from different social and economic backgrounds. While Gauteng includes some peri-urban and informal settlement areas, it is predominantly urban compared to other provinces in South Africa.

Participants were recruited from the B-Hub database, which served as the sampling frame. The B-Hub maintains a consented research participants panel of community members and health care providers from Gauteng province who agreed to be contacted for future research studies. We used purposive sampling to ensure inclusion of eligible community members who met the study eligibility criteria and health care providers with experience in HIV service delivery. Community members and health care providers were contacted telephonically by trained research staff to assess interest and confirm eligibility. Eligible community members were adults aged ≥18 years who were willing to provide written informed consent in English, self-reported an HIV-negative or unknown HIV status, and were able to travel to the study site. Individuals who self-reported as HIV-positive were excluded. Eligible health care providers included professional nurses, HIV counseling and testing (HCT) counselors, and health coaches, as these cadres are directly involved in HIV testing, prevention counseling, linkage to care, and ongoing patient support within routine HIV service delivery. Health care providers were eligible if they were able to provide informed consent in English and had at least one year of experience providing HIV services. Recruitment and data collection took place between November 2024 and January 2025. Data collection with community members was conducted at the Indlela@HE²RO research office, while interviews with health care providers were conducted at their respective clinics. Only the researcher and the research assistants were present during data collection. As this was a rapid prototype pilot evaluation, sample sizes were determined pragmatically to assess usability, acceptability, and perceived appropriateness of the tool among intended users rather than to test effectiveness outcomes.

Eligible community members were randomly assigned to a mock HIV-negative or positive test result scenario prior to the simulated HIV self-test and interaction with the Your Path tool. Participants were asked to envision themselves in their assigned scenario when completing study activities and interacting with the tool. This approach was chosen because the study’s primary aim was not to replicate real-world HIVST procedures, but to evaluate the usability and acceptability of the AI tool. Providing the mock result beforehand allowed participants to engage fully with the Your Path tool, which offers step-by-step guidance on performing the test and tailored recommendations through a chat with an AI companion for next steps in accessing HIV services based on their assigned result.

Community members then completed a pretest baseline survey (Table S1 in Multimedia Appendix 1). They were then provided with study phones and instructed to open the Your Path WhatsApp thread by typing “hi,” which triggered an automated welcome message and menu options. From the main menu, participants selected the “Self-test support” option and completed a simulated HIVST using the Mylan HIVST kit (Viatris) option, guided step-by-step by Your Path. Your Path AI-Companion is a rule-guided conversational chatbot designed to simulate post–HIVST counseling and linkage support. The tool provides structured information and conversational guidance across several topics, including interpreting HIV self-test results, emotional support following testing, HIV treatment initiation and linkage to care (for positive results), HIV prevention options including PrEP (for negative results), and planning next steps for accessing services. Upon completion of the simulated HIVST, participants were prompted to discuss their assigned mock test results with the Your Path AI-companion. The posttest chat provided an opportunity for the participant to ask free-text questions, with the tool responding in a conversational style to discuss the participant’s mock result and next steps. During the chat, the AI-companion posed context-specific questions including (1) support interpreting HIV self-test results; (2) recommended next steps and linkage guidance (eg, confirmatory testing and treatment linkage after a positive result; prevention and repeat testing guidance after a negative result); (3) emotional support and normalization of common concerns; and (4) responses to common questions about HIVST and services. We structured the sessions to provide about 10‐15 minutes for participants to engage in the posttest chat. Additionally, chat transcripts were logged for qualitative analysis, and participants consistently completed the posttest chat component as part of the simulated workflow. Chats occurred in a private space to ensure participant comfort and confidentiality. Participants were asked not to enter any personal identifying information. All interactions were conducted using study access accounts, and chatbot interaction logs were securely recorded for analysis, including conversation length and number of message exchanges.

After engaging with Your Path, community members completed the posttest survey (Table S2 in Multimedia Appendix 1), which also included validated System Usability Scale (SUS) questions [19], to measure the usability of Your Path. A subset of community members was then purposively selected for semistructured interviews to explore the acceptability of Your Path features, as well as barriers and motivators to engagement. To allow for variation, community members were purposively selected based on age, gender, and mock test results (negative or positive). The interviews were audio recorded and included prompts (Table S3 in Multimedia Appendix 1) regarding the community member’s experience using Your Path and suggestions for improvement. Each interview lasted about 20 minutes on average and was conducted in a private room where only the participant and the interviewer were present. The interviews were conducted by 2 female study team members (NM and OM) with master’s degrees who are experienced researchers. The sample size was adequate to see a recurrence in themes, and data saturation was assessed during the data collection process through frequent revision of the probes.

Health care providers engaged in 2 steps. First, they reviewed 2 transcripts of interactions between community members and Your Path together with the AI-generated summaries corresponding to those same transcripts. Secondly, they participated in semistructured interviews, each lasting approximately 30 minutes. During interviews, health care providers examined whether the tool provided clear and accurate guidance on next steps after an HIV self-test, offered appropriate emotional support, and facilitated posttest counseling conversations. Health care providers also evaluated the completeness and relevance of the AI-generated summaries, including whether the summaries captured key user concerns and counseling needs reflected in the transcripts.

Development of the Your Path conversational content and decision pathways was informed and grounded with a knowledge base of World Health Organization guidelines, South African national guidelines, and other relevant local-specific HIV testing and posttest counseling guidelines to structure the conversational goals and recommended next steps. The AI-companion is powered by LLMs, using prompt engineering techniques and referencing the knowledge base to improve the consistency and appropriateness of counseling-style responses, support capture of key data elements (eg, HIV status) from each conversation to maintain continuity and ground context across subsequent chat interactions. Prior to the study evaluation, iterative prestudy testing was conducted that included: (1) scenario testing using simulated user conversations representing expected HIVST outcomes and common user concerns; (2) pilot testing of the WhatsApp-based interaction flow to confirm usability and technical performance; and (3) review of simulated transcripts to assess alignment with guideline-recommended counseling, clarity of next steps, and appropriateness of tone and supportive language. Findings from these activities informed the refinement of the conversation flow and response behavior before deployment.

Statistical analyses were conducted using Stata 18 (StataCorp LLC) [20]. Descriptive statistics, including percentages, means, medians, and SDs, were used to summarize sociodemographic variables and survey responses. To assess changes in participants’ intentions to engage in HIV-related services, we compared paired pre- and postintervention responses across a series of Likert-scale outcomes. These outcomes captured intentions, confidence, and perceived importance of engaging in HIV prevention or treatment services, depending on mock HIV test results. Results were not compared across HIV test result scenarios (HIV-negative vs. HIV-positive), as the underlying questions were not directly comparable: HIV-negative scenarios referred to engagement in prevention services, whereas HIV-positive scenarios referred to engagement in treatment services. While the results of the intention to engage in HIV services before and after using the intervention are reported, it should be noted that the study was not powered to measure efficacy, and results should therefore be interpreted with caution.

Given the ordinal nature of the responses, we used two complementary nonparametric approaches to test for statistical significance pre- and post-interacting with the Your Path intervention. Firstly, we used the Wilcoxon signed-rank test to evaluate within-individual changes over time, testing whether there was a systematic directional shift in responses. Secondly, we applied the Stuart-Maxwell test to assess whether the overall distribution of responses across categories differed between pre- and postintervention. This provides a complementary assessment of whether responses were redistributed across categories, irrespective of direction. Statistical significance was assessed at the 5% level (P<.05). Community members’ perception of the usability of Your Path was assessed using the SUS, a validated 10-item questionnaire in which each item is rated on a 5-point Likert scale [21]. A mean SUS score of 68 or higher is considered above average, while a score of 80 or above indicates excellent usability. However, interpretation of SUS scores can vary based on the system’s complexity, target audience, and context of use.

Qualitative data were analyzed using NVivo (version 14, Lumivero). Audio recordings were transcribed using a professional transcription service. Thematic analysis using a deductive approach was used to derive and analyze themes from the data. Codebooks were developed and refined through an iterative approach by three researchers (NM, CG, and OM). During discussions, interrater reliability across themes was assessed where similar themes were retained while others were dropped when no consensus was reached. Qualitative findings are reported in accordance with the Consolidated Criteria for Reporting Qualitative Research (COREQ) guidelines (Table S4 in Multimedia Appendix 1). The study results were aligned with the Theoretical Framework of Acceptability (TFA) [22]. We did not stratify themes by HIV result because the study was designed to explore overall acceptability, usability, and clinical appropriateness, rather than to compare experiences by HIV status. Additionally, the small number of participants interviewed limits meaningful stratified analysis.

The Wits Human Research Ethics Committee (medical) approved the study (reference number 211122). Written informed consent was obtained from all participants prior to their participation in the study. All identifying information has been removed or anonymized to protect participants' privacy and maintain confidentiality.

Participants included community members and health care providers. A total of 154 community members were invited to participate, of whom 100 were enrolled. Screening failures (n=54) were primarily due to inability to travel to the site, work schedule conflicts, and a lack of interest in participating. Similarly, 37 health care providers were invited, with 25 enrolled; screen failures (n=12) were largely due to scheduling conflicts or not meeting inclusion criteria. Table 1 presents the sociodemographic characteristics of the enrolled community members and health care providers. The median age was 28.8 (IQR 18‐61) years for community members and 37.5 (IQR 24‐61 years for health care providers. 59 out of 100 (59%) community members were female, 92 out of 100 (92%) self-reported HIV-negative, and 55 out of 100 (55%) were unemployed. Among health care providers, 21 out of 25 (83%) were female, 15 out of 25 (60%) were experienced HCT counselors, and 18 out of 25 (81%) had been employed for more than three years.

Chatbot interaction logs showed active engagement with the Your Path AI-Companion. On average, participants interacted with the AI-Companion for 15 minutes (range: 10‐15). Most participants asked at least 5 free-text questions during the interaction, in addition to responding to structured prompts. These interactions reflected conversational engagement similar to a posttest counseling session. For participants with mock HIV negative test results, there was limited evidence of change in intentions to engage in HIV prevention services following the intervention (Table 2). Perceived importance of engaging in prevention services remained high at both timepoints, with no statistically significant change (Wilcoxon P=.79; Stuart-Maxwell P=.38). Similarly, intentions to seek services if testing positive and confidence in accessing resources showed no significant shifts, although confidence in accessing prevention services increased significantly (Wilcoxon P=.0002; Stuart-Maxwell P=.0039), suggesting both a directional improvement and a redistribution toward higher confidence categories. A marginal improvement was observed in confidence to find resources (Wilcoxon P=.012), although this was not accompanied by a significant change in the overall distribution (Stuart-Maxwell P=.16). No significant changes were observed in the reported likelihood of engaging in prevention or treatment services.

For individuals with a positive HIV mock test result, the perceived importance of seeking treatment services remained consistently high, with no significant change between pre- and postintervention responses (Wilcoxon P=.68; Stuart-Maxwell P=.67; Table 3). Confidence in the ability to find resources to seek treatment services showed a statistically significant directional increase (Wilcoxon P=.04), although the overall distributional change was not statistically significant (Stuart-Maxwell P=.10), indicating a modest upward shift within categories rather than a broad redistribution. No significant changes were observed in the stated likelihood of engaging in treatment services. Overall, the intervention appears to have had its strongest effect on participants’ confidence in accessing HIV-related services, particularly in the HIV-negative (prevention) context, while intentions and perceived importance were already high at baseline and remained stable over time.

Following interaction with Your Path, its usability was rated highly, achieving a mean SUS score of 81.6 (SD 17.5; Table 4). Most community members (84/100, 83.7%) found the tool easy to use, 94.9% (95/100) would use it again, and 90.8% (91/100) felt confident using it. However, 33,6% (34/100) preferred assistance before using the tool, and 15.3% (15/100) noted that it did not always provide consistent responses to questions where they felt that responses to some queries varied or were unclear.

The deductive themes developed in accordance with the TFA and derived from the in-depth interviews are presented below and summarized in Table 5.

Community members described the interactions with Your Path as warm, welcoming, and offering personalized support that was comforting and reassuring.

However, when health care providers were reviewing the transcripts and summaries, 5 out of 25 (20%) felt that Your Path did not provide adequate precounseling and the interaction seemed robotic, lacking the warmth and responsiveness of human interaction.

Six out of 25 (24%) community members and 10 out of 25 (40%) health care providers raised concerns about privacy, language limitations, and overall accessibility of Your Path.

Community members perceived interactions with Your Path as friendly and nonjudgmental when compared to engaging with health care providers during counseling sessions.

Community members described Your Path as user-friendly, simple, and convenient to use, enabling straightforward access to information and support.

Community members and health care providers felt that Your Path provided useful and relevant step-by-step guidance for HIV self-testing and appropriate information on HIV prevention and treatment, supporting linkage to appropriate HIV services. Health care providers also viewed it as a helpful tool for individuals hesitant to visit clinics.

In this controlled exploratory study, Your Path, an AI-powered tool designed to facilitate access to HIV prevention and treatment services following HIVST, showed limited change in intentions to engage with HIV services, with intentions remaining high pre and post intervention. Most community members found the tool easy to use and appreciated its private and nonjudgmental tone, which helped reduce perceived stigma. Health care providers considered the tool to be clinically appropriate, informative, and relevant for supporting access to HIV care, although some limitations were noted regarding emotional responsiveness and language accessibility. To our knowledge, this study is among the first conducted in a resource-limited setting to evaluate a digital AI-powered technology specifically focused on facilitating post-HIVST linkage to care, highlighting the potential of AI-powered interventions to address gaps in HIV service delivery. These findings underscore the promise of user-centered digital tools to support timely connection to care, particularly in contexts where access to health care resources is constrained.

Most studies that evaluated the use of digital tools for public health services have been conducted in high-income countries with well-established digital infrastructure [23-26]. Although recent research in Africa has begun to explore the role of mobile health (mHealth) interventions, such as SMS reminders and chat-based platforms, in improving HIV testing uptake, medication adherence, and clinic appointment reminders [27-33]. Despite progress in research evaluating the integration of digital tools into public health, few studies have specifically examined how AI-powered conversational agents can support and strengthen linkage to HIV prevention and treatment services in South Africa. Our study contributes to closing this gap by evaluating the usability, acceptability, and clinical appropriateness of the AI-powered Your Path tool, designed to facilitate linkage to HIV prevention and treatment services following HIVST in South Africa. Most studies exploring the use of digital tools in public health focused primarily on the perspective of clients [25,26,28,29], while our study included both the perspectives of clients and health care providers to capture a comprehensive understanding.

Within HIVST programs in sub-Saharan Africa, linkage to care following an HIVST is poor, and this is attributed to behavioral barriers such as stigma and lack of guidance on the next steps. For example, the Tanzania HIV self-testing education and promotion project targeted men to improve HIV testing uptake [34], and a program in central Uganda offered HIV oral self-testing to male partners of women attending antenatal care [35]; both studies reported challenges in ensuring linkage to care. AI tools present an opportunity to address barriers, particularly through the application of digital conversational agents. Studies have shown that AI-based conversational agents are acceptable within an HIV context and can engage users on the next steps post HIVST in a manner that is private, confidential, and nonjudgmental [28]. For instance, studies have shown that these features make conversational agents especially valuable for high-risk populations, offering stigma-free entry points into HIV services [36,37]. Our study findings align with this, where community members expressed a preference for engaging with the Your Path AI companion over a real counselor due to fear of stigma, judgment,w and perceived lack of confidentiality with human interaction. These findings are also consistent with results from another study [38], where users often preferred digital health care agents over human providers when discussing sensitive issues.

Our findings also highlight important divergent views between community members’ and health care providers’ perspectives of the AI companion. While community members valued the AI companion’s nonjudgmental tone, privacy, and accessibility, a few health care providers raised concerns about the quality of interactions. They described these interactions as impersonal and felt that the AI companion was less able to provide emotional support compared to human counseling. These differing perspectives may reflect differences in expectations, with providers emphasizing relational and empathetic aspects of counseling as core components of care, while community members value nonhuman interactions, as they allow for more open discussion of sensitive issues without fear of judgment. Despite the health care providers’ concerns about the quality of interactions, other studies have shown that patients preferred to use AI over talking to a human provider, as they deemed it more empathetic as compared to a human health care provider [39,40]. To address the health care providers’ concerns, tools such as Your Path could be provided as options alongside in-person human interaction so community members can choose whether they want to interact with chatbots or health care providers. For future implementation of AI tools such as Your Path in health care, it’s important to position AI-powered tools as complementary tools to provider-delivered counseling.

Our findings indicate that Your Path is acceptable, which aligns with studies conducted in Malaysia [35] and South Africa [28,41,42] as well as a broader scoping review [43], which show that digital tools integrated with AI are particularly well-received in the context of HIV care. Acceptability of digital interventions is influenced by multiple factors, including usability, confidentiality, and emotional support [44]. In our study, there was a high willingness to use Your Path, reflecting openness to incorporating digital technologies into HIV self-testing and posttest support. Qualitative interviews highlighted emotional safety and confidentiality as key contributors to user satisfaction. Many people living with HIV still face significant psychological challenges as they are often discriminated against, socially isolated, and stigmatized for their condition [45-47]. Our findings, along with those of other studies [39,45], suggest that a non-judgmental, anonymous conversational agent like Your Path can provide a reassuring and safe space for users to discuss sensitive topics, such as HIV self-testing results, without fear of being criticized. This may encourage users to seek information freely and might enhance their willingness to use Your Path.

Our findings also highlight the strong usability of Your Path, with a mean usability score of 81.6/68 (SD 17.5), exceeding the preset threshold for success. Participants reported high ease of use and described the tool as accessible and simple to navigate, consistent with other studies showing the value of digital interventions when designed for familiarity and convenience [48,49]. The predominance of young participants in our sample may have contributed to these positive perceptions, given the widespread use of WhatsApp and uptake of mobile technology among youth, consistent with previous studies [28,50,51]. Together, these findings suggest that Your Path is well-positioned to engage users by combining high acceptability with strong usability, both of which are critical for sustaining adoption of digital health tools in HIV care.

Several concerns and limitations, including data privacy, security of personal information, and data access, were highlighted despite positive perceptions of Your Path, a finding consistent with other studies on digital health interventions in sub-Saharan Africa [52-54]. To align with the South African Protection of Personal Information Act (2013) [55] and strengthen user trust, future versions of Your Path could implement explicit user consent processes, minimize the collection of personal identifiers, and provide users with the option to delete their data. The reliance of the current version of Your Path on mobile data and the exclusive use of English are limitations that could hinder accessibility, particularly for users in low-connectivity areas or those not proficient in English. Addressing these challenges may require offering data-free access, a strategy already used in South African mHealth programs such as Mom Connect, B-Wise, and the Moya app, which expanded reach by eliminating data costs for end-users [56-58]. To address language limitations, future versions of Your Path could include other local and regional languages to increase reach and access.

This study has contributed important information to the acceptability, usability, and clinical appropriateness of AI-powered tools for HIV care in South Africa, but it has limitations. Recruitment was limited to a single urban site, with predominantly younger participants who are more likely to have access to and be comfortable with digital technologies. As a result, our findings may not be fully generalizable to rural communities and older populations where access to and familiarity with such technologies may differ. The limitations in our study underscore the importance of future field-based evaluation with a more diverse population. Such studies will be essential to assess the acceptability, usability, and clinical appropriateness of Your Path in a real-life context where logistical, behavioral, and system-level factors play a larger role. We conducted our study in a controlled simulated environment. However, the simulated environment was necessary to allow for safe, ethical testing of feasibility before moving to real-world implementation. This evaluation did not develop or validate a clinical prediction model; therefore, TRIPOD or TRIPOD-AI performance reporting (eg, discrimination, calibration, and predictor-weight specification) was not applicable to the aims of this acceptability, usability, and clinical appropriateness assessment. A further limitation is that the qualitative results reflect the overall usability experiences of the total sample and are not separated by HIV assigned status. This limits the ability to identify potential differences in experiences or needs between participants who received a negative or a positive HIV self-test result. Future research could be conducted to understand the effectiveness of this app for achieving health outcomes for individuals with reactive and nonreactive self-test. Strengths of this study include its mixed methods design, which allowed for a comprehensive, user-centered evaluation of acceptability, usability, and clinical appropriateness. By incorporating direct input from both community members and health care providers, the study ensured that Your Path was evaluated in alignment with the real-world needs and preferences of its end users.

In conclusion, this study found Your Path to be usable, acceptable, and clinically appropriate as a digital health tool to support linkage to services after an HIVST. Participants valued its ability to provide convenient access to accurate information, step-by-step guidance, and personalized support for connecting with HIV services. Insights from this study will inform future builds and adaptations of Your Path to enhance user experience, improve integration within health systems, and ensure responsiveness to emerging needs in HIV prevention and treatment. Future studies should evaluate Your Path in clinical settings to determine its real-world effectiveness, and conduct cost-effectiveness analyses to assess the value, scalability, and sustainability of AI-powered digital health tools for supporting HIV care and broader health service delivery.