Development and Evaluation of a Mobile App Designed to Increase HIV Testing and Pre-exposure Prophylaxis Use Among Young Men Who Have Sex With Men in the United States: Open Pilot Trial

Background: HIV disproportionately affects young men who have sex with men (YMSM) in the United States. Uptake of evidence-based prevention strategies, including routine HIV testing and use of pre-exposure prophylaxis (PrEP), is suboptimal in this population. Novel methods for reaching YMSM are required. Objective: The aim of this study is to describe the development and evaluate the feasibility and acceptability of the MyChoices app, a mobile app designed to increase HIV testing and PrEP use among YMSM in the United States. Methods: Informed by the social cognitive theory, the MyChoices app was developed using an iterative process to increase HIV testing and PrEP uptake among YMSM. In 2017, beta theater testing was conducted in two US cities to garner feedback (n=4 groups; n=28 YMSM). These findings were used to refine MyChoices, which was then tested for initial acceptability and usability in a technical pilot (N=11 YMSM). Baseline and 2-month postbaseline assessments and exit interviews were completed. Transcripts were coded using a deductive approach, and thematic analysis was used to synthesize data; app acceptability and use data were also reported. Results: The MyChoices app includes personalized recommendations for HIV testing frequency and PrEP use; information on types of HIV tests and PrEP; ability to search for nearby HIV testing and PrEP care sites; and ability to order free home HIV and sexually transmitted infection test kits, condoms, and lube. In theater testing, YMSM described that MyChoices appears useful and that they would recommend it to peers. Participants liked the look and feel of the app and believed that the ability to search J Med Internet Res 2021 | vol. 23 | iss. 3 | e25107 | p. 1 https://www.jmir.org/2021/3/e25107 (page number not for citation purposes) Biello et al JOURNAL OF MEDICAL INTERNET RESEARCH


Introduction
Background HIV incidence remains high in the United States among young men who have sex with men (YMSM). In 2018, more than 20% of new HIV infections in the United States were among young people aged between 13 years and 24 years, with YMSM accounting for 83% of newly diagnosed HIV infections in this age group [1]. New HIV infections also disproportionately impact men who have sex with men (MSM) of color. In 2018, more than half (52%) of the new diagnoses of HIV among MSM aged between 13 years and 24 years were seen among Black individuals and 27% were identified among Latinx individuals [1]. In addition to experiencing a high HIV incidence, a higher proportion of YMSM living with HIV do not know that they are infected in comparison with their adult peers [2,3]. Moreover, individuals aged between 13 years and 24 years are less likely to be linked to HIV care upon diagnosis and present lower levels of viral suppression in comparison with older individuals [2,4,5]. Consequently, YMSM will have delays seeking effective treatment and are more likely to transmit HIV to others [6].
Overwhelming evidence shows that routine HIV testing and expanded use of pre-exposure prophylaxis (PrEP) would drastically reduce the population burden of HIV [7][8][9][10][11][12][13]; however, uptake of both interventions is suboptimal among young adults. For example, although the Centers for Disease Control and Prevention recommends that sexually active MSM be tested for HIV at least annually [14], data suggest that nearly half of YMSM reported not being tested for HIV in the past year and one-third reported they had never been tested [15]. In addition, PrEP awareness and uptake are low among younger people [16,17]. For example, only 5% of MSM aged between 18 years and 24 years with PrEP indications reported ever using PrEP, compared with 14% of those aged 25 years and more [18]. Furthermore, only 0.1% of PrEP prescriptions in the United States were given to individuals aged under 18 years [19]. Moreover, young individuals have lower levels of adherence to and retention in PrEP care after initial prescription [20][21][22][23], decreasing the impact of PrEP on HIV prevention among this group.
Risk taking, behavioral experimentation, and confronting a host of difficult choices regarding identity formation are all part of the normal developmental trajectory of adolescence and young adulthood [24]. In addition, beliefs about invincibility, sensation seeking, and the still-developing cognitive processes of adolescents may have a role in increased HIV risk-taking behaviors and a lower prioritization of prevention strategies for this age group [25][26][27]. Developing innovative ways to intervene to increase engagement in HIV prevention behaviors among youth is crucial, particularly interventions that are accessible and responsive to the diverse needs of youth.
Smartphones are used by nearly all youth in the United States, across race and social class, and as such are a direct way to meet youth where they are [28]. In addition, the use of mobile phone apps by YMSM is nearly ubiquitous; apps may offer unique opportunities for public health interventions, and previous studies have demonstrated the feasibility and potential efficacy of this approach [29,30]. In a systematic review of various mobile health (mHealth) interventions, Muessig et al [29] noted that internet-and mobile-based interventions can increase dissemination of HIV prevention interventions to wider populations while also providing consistency and lower cost in intervention delivery once fully developed. In addition, mHealth tools could promote behavior change and improve aspects of the HIV care continuum, including linkage to care, retention in care, and adherence to both PrEP and antiretrovirals [30,31]. As such, a mobile phone app that aims to increase HIV testing and PrEP uptake among YMSM has the potential to provide greater access to and uptake of these prevention services for this population.

MyChoices App
The development of the MyChoices app has been described previously [32]. In brief, MyChoices is a social cognitive theory-driven mobile app adapted from HealthMindr, an HIV prevention app developed through an iterative process for adult populations of MSM [33][34][35]. MyChoices built upon the initial framework and was subsequently adapted for youth by an interdisciplinary team of researchers with input from a diverse sample of YMSM at every stage of adaptation and refinement. The goal of the app is to increase HIV testing and PrEP uptake among YMSM in the United States by supporting goal setting, increasing self-efficacy, and enhancing self-regulation [36,37]. Key features of the MyChoices app include facilitation of the development of HIV testing plans with personalized recommendations; inclusion of reminder systems for HIV testing and GPS-enabled maps with local HIV and sexually transmitted infection (STI) testing locations and PrEP providers; ability to order free condoms, condom-compatible lubricants, and at-home HIV and STI test kits; and sexual health information using a variety of media (eg, videos, Graphics Interchange Format [GIF], infographics, frequently asked questions [FAQs], and quizzes) [32]. Figure 1 shows the MyChoices app home screen and the testing plan feature.

Study Population
Eligible participants were cisgender men who were aged 15-24 years; did not have an HIV test in the past 3 months; self-reported being HIV uninfected or HIV status unknown at screening; owned an iOS or Android mobile phone and were willing and able to download the MyChoices app; were able to understand, read, and speak English; were not taking PrEP; and had self-reported evidence of being at risk for HIV acquisition (details on risk criteria are given in the study by Biello et al [32]).
Through the University of North Carolina/Emory Center for Innovative Technology (iTech) [38], a part of the National Institutes of Health's Adolescent Medicine Trials Network for HIV/AIDS Interventions [39], participants were recruited across 2 sites: Boston, Massachusetts (study site: Fenway Health), and the Bronx, New York City (study site: the Adolescent AIDS Program at Montefiore). Recruitment methods included posting on social media (eg, Craigslist, social networking ads, and gay networking mobile apps); distributing posters, flyers, and palm cards about the study; direct outreach at local venues frequented by YMSM (eg, community-based organizations, schools, bars, health fairs, and balls); and clinic-based recruitment.

Theater Testing for App Refinement
After an initial prototype of the MyChoices app was developed through multiple rounds of formative research with YMSM [33,40], we conducted theater testing with 28 YMSM in 4 groups across the 2 iTech sites (5-8 participants per group). Theater testing allows for groups of participants to interact with the product being tested and provide feedback in situations that approximate real-life experiences and has been used commonly in mHealth app development [33,35,41,42]. Testing was conducted in a private room at each site by a research staff member who had training in qualitative methods and group facilitation. All participants completed a written informed consent or assent process before data collection commenced. Participants completed a brief demographic and behavioral questionnaire before theater testing to contextualize the group data collected. During theater testing, participants interacted with the MyChoices app prototype and provided feedback on the functionality, appearance, and usability of the platform. We also asked participants to comment on ways to maximize acceptability (eg, update language and improve flow) and to identify the components of the app that were liked or disliked and aspects that could be improved to increase HIV testing and PrEP uptake among YMSM. Groups lasted 60 minutes to 90 minutes and were audio recorded and professionally transcribed verbatim. Participants received US $50 as reimbursement for their time.
Members of the iTech Analytic Core [43] reviewed transcripts for quality and identified emergent themes. We then used Dedoose Version 8.0.35 (SocioCultural Research Consultants, LLC) software to apply the final codes to all transcripts. Thematic analysis involved using a primarily deductive approach to synthesize data coded for app acceptability, particularly around functionality, appearance, usability, and potential for improving HIV testing and PrEP uptake [44,45]. Findings are illustrated in the following sections using representative quotes. These data were used to refine the app before the initiation of the open technical pilot.

Technical Pilot to Assess Feasibility and Acceptability of the MyChoices App
Once the MyChoices app had been refined, a technical pilot with 11 YMSM across the 2 iTech sites was conducted to assess the initial feasibility and acceptability and to identify any final areas for improvement. Eligible participants (mentioned earlier) attended a visit at the study site, at which they completed a web-based behavioral and psychosocial assessment, which included measures to assess sociodemographics (ie, enrollment city, age, race or ethnicity, educational status, and insurance status), sexual behaviors (ie, frequency of condomless anal sex), HIV testing history, and PrEP awareness. Study staff then assisted youth with app download and provided them with brief instructions on the purpose of the MyChoices app and an overview of how to use it; participants were encouraged to use the app over the course of 2 months.
At month 2, the participants completed a web-based assessment. In addition to the measures collected at baseline, we assessed the acceptability of the MyChoices app using the System Usability Scale (SUS) [46]. SUS is a validated 10-item measure that assesses the subjective usability of a system or, in this case, an app [46]. SUS has been extensively used in mHealth research and provides reliable results even with small sample sizes [47]. It is scored from 0 to 100, and a score of ≥50 indicates that the app is acceptable [48]. Feasibility was assessed using app analytics to determine whether the app was used, how often it was used, and what components were used most and least frequently.
Finally, we conducted exit interviews with participants to obtain feedback on app functionality, technical performance, errors and software bugs encountered, overall experiences using the app, feedback for further refinement, and subjective impact of the app on HIV testing and PrEP uptake. Exit interviews were conducted by study staff on the web using videoconferencing technology that was compliant with the Health Insurance Portability and Accountability Act. Interviews were transcribed and analyzed using the same approach outlined above.
Participants received US $50 for completing the baseline visit, US $25 for the 2-month assessment, and US $50 for the exit interview.
The study procedures were reviewed and approved by the University of North Carolina Institutional Review Board (IRB) as a single IRB-of-Record. IRB authorization agreements with all participating research entities were enacted. The MyChoices protocol is registered at ClinicalTrials.gov (NCT03179319). Table 1 provides the characteristics of the 28 YMSM who participated in theater testing. Participants' ages ranged from 16 years to 24 years, with a mean age of 20 years. YMSM of color made up 57% (16/28) of the sample, and 57% (16/28) of the participants were still in school.

Findings From Theater Testing
A total of four key themes emerged from theater testing, and thus, we structure the presentation of our results to highlight these findings: (1) general utility and acceptability of the app, (2) feedback and suggestions for user interface, (3) opinions on language, and (4) suggestions for additional content and features.

General Utility and Acceptability of MyChoices
Participants described that an HIV prevention app needs to be perceived as useful to its target audience and that they saw value in MyChoices. The participants described: Although participants acknowledged that they could find similar information from other sources, they appreciated that the information on MyChoices came from a reliable source because of its association with the study and clinic and that using the app was a better avenue for ascertaining health information than searching the internet for resources they could not be sure were accurate:

Suggestions for User Interface
Many participants liked the look and feel of the app, noting that it was nondescript enough to sufficiently maintain privacy: Participants appreciated the places in the app that allowed them to personalize the language so that they could decide what type of language-more or less direct, more or less casual-suited them personally. They also felt that personalization of the language also protects against privacy and confidentiality concerns because you "can put whatever you want":

Suggestions for Content and Features
In addition to feedback on the current MyChoices prototype, participants provided suggestions on additional content and functions that might enhance the acceptability and utility of the app. One participant suggested including a section on how to talk to your partner about PrEP: In addition to new content, participants included suggestions for new features. For example, we described a potential feature that would use geofencing to notify a user when they are near an HIV testing location and they are due for an HIV test according to their created test plan. Participants were excited about this option, saying "that's pretty cool" and "that's a cool feature." Participants also suggested additional tools for interacting with health professionals, including allowing HIV testing sites to provide results directly through the app:

Part of me wished that if -so let's say, you got tested from [Health Center] and there was a way [Health Center] could coordinate so that your results just like pop-up in the app, like you don't have to put it manually and that is like your way of receiving them, too. So it is all in one place and you don't do it manually...And they just like get a push notification like, "Your results are in!" And that's like -you go in and you just like find out that way. [White, gay, age 21 years, Boston]
Participants also appreciated information about postexposure prophylaxis (PEP) but felt that the need for immediate action warrants easier access to this information:

Summary of Changes Made to MyChoices After Theater Testing
As noted earlier, the participants saw the value in the content and functionalities of the MyChoices app. They believed that the suggestions that came from brief quizzes were useful, that the testing plan would be helpful to encourage regular testing, that the PrEP information was instructive, and that being able to order HIV or STI test kits and safer sex supplies would be beneficial. However, participants also provided suggestions on how to build on and improve some of these components. As a result of these suggestions, before initiating the technical pilot, we refined some of the language in the app, updated GIFs and icons, and streamlined the flow through the app. Moreover, we expanded app functionalities to include (1) a geolocator function that pings individuals when they are near a testing site and due for HIV testing based on their personalized testing plan, (2) the Need PEP?! button that is available at the bottom of every screen on the app to directly connect participants with information about PEP and locations where it is available, (3) additional videos to demonstrate how to use the home testing kits, and (4) emails that are sent to users after downloading the app to introduce them to key features that they may have otherwise missed. Some suggestions made by participants were unable to be implemented into the app, although they were noted as potential ways to enhance future iterations, including receiving test results from clinics through the app, adding a real-time chat feature, syncing reminders with phone calendars, and being able to schedule HIV testing or PrEP appointments through the app.

Findings From the Technical Pilot
The open pilot enrolled 11 participants (Boston, n=6; the Bronx, n=5), and retention at the 2-month follow-up was 100%. Participants' ages ranged from 15 years to 23 years, with a median age of 19 years. YMSM of color made up 91% (10/11) of the sample (Black, non-Hispanic, n=5; Hispanic or Latinx, n=5; Table 2).

App Feasibility
Participants in the technical pilot accessed the app, on average, 8 times (SD 5.0; range 2-18) over 2 months, with an average duration of 28 minutes per session (SD 38.9). Across all participants, the cumulative time spent in the app ranged from 1.8 minutes to 20.5 hours, with an average of 4 hours and 39 minutes (SD 7 hours). All participants used the test plan feature (11/11, 100%; average number of accesses to this feature 20.2, SD 18.2; range 1-63), and nearly all participants (10/11, 91%) used MyChoices to order HIV or STI self-testing kits and safer sex supplies (average number of accesses 22.6, SD 29.8; range 0-85). Most participants (7/11, 64%) used the app to locate HIV or STI testing centers or PrEP providers (average number of accesses 5.3, SD 5.8; range 0-17), and 4 participants (4/11, 36%) accessed the FAQ feature of the app to access information about HIV prevention and PrEP (average number of accesses 1.5, SD 3.0; range 0-9).

Exit Interviews
All 11 participants completed follow-up exit interviews. Overall, there was a high level of acceptability of the content, interface, and features. Participants commented on the relevant information presented in the app, one participant noting that "[they] learned a lot about PEP and PrEP" and noted that the app facilitated their scheduling a PrEP counseling appointment with their primary care provider. Another participant noted that they "didn't know anything about [PrEP] until [they] used the app." Furthermore, the feature to schedule PrEP appointments "was the best part for [them]" because "it shows you, like, you are and, like, the different centers, testing centers around you, the different communities around you." The participant endorsed how the app helped him feel more connected to the community where they lived. Another participant highlighted how the app provided him with helpful information about health clinics in his area, of which he was unaware despite living in the neighborhood "a good portion of [his] life." Participants also provided suggestions to improve the app and provided specific feedback on additional resources and features that they found interesting and potentially helpful. One participant suggested including a myth busting section in the app to offset false information spread on the web about sexual health. A few participants also noted that some of the information in the app remains repetitive. One participant suggested defining PrEP differently in different areas of the app to reduce repetition and to mitigate a potential lack of understanding of health information.

Principal Findings
This study describes the findings from two phases of formative data collection (theater testing of the MyChoices mobile app prototype and the technical pilot of MyChoices fully functional app) to create a mobile app to increase HIV testing and PrEP uptake among young MSM in the United States. HIV prevention apps are proliferating; however, using evidence-based methods for intervention development, including community-centered approaches with iterative feedback from the community, is essential to maximize their impact and reach [35,49], and few HIV prevention apps for youth have been developed and tested using the approaches described in this paper.
In the context of previously reported findings about preferences for mobile HIV prevention apps in young people, our findings support and extend previous reports by providing a sample including a higher proportion of teenage participants, confirming some aspects of previous reports, and documenting some novel findings. Like others [50][51][52], we found that youth valued having an app that presents credible information relevant to their health and interests in direct and understandable language and the value of having educational information available. Notably, other qualitative studies of preferences for HIV prevention or HIV management eHealth tools have identified some themes that did not emerge from the youth who participated in our study. For example, other researchers have found youth preferences for features that allow them to interact with other youth [53], and young MSM in other studies reported concerns about privacy and confidentiality of data [54]. Young MSM have also raised questions about what the scope of sexual health apps should be, with some suggesting that related health issues, such as substance use, could be included in a mobile app resource [52]. Our participants also provided suggestions for developing geospatial tools to guide users to prevention services. These issues, which have been rarely reported in other studies, may have emerged from our participants because geospatial tools are becoming more refined and younger people have grown up relying on smartphones as a primary source of navigation. Due to the younger age of our participants compared with many previous studies and because our data were collected more recently, youth expectations around enhanced navigation and geolocation services may be seen as emerging expectations for mobile prevention apps.
Theater testing revealed high levels of interest in the content provided through MyChoices, with youth indicating that the availability of this type of information is both lacking and necessary. Although youth have access to huge amounts of health information through the internet and report using the internet frequently to access this type of information [55,56], they do not always know what to trust [57,58]. In theater testing, youth indicated that one of the major strengths of the app was to have access to a wide range of sexual health information in one place, in multiple formats, that they knew they could trust. In addition, the app's usability score is comparable with the HealthMindr app on which MyChoices was based, although it had a higher proportion of people who would recommend it to friends [33]. This may suggest a successful adaptation for YMSM.
In addition to the content, the features of the MyChoices app were also broadly seen as favorable and useful. In both theater testing and the technical pilot, the most popular features included ordering free HIV and STI self-testing kits, condoms, and lube; ability to search for nearby HIV testing and PrEP care sites; and PrEP information provided in multiple formats (ie, text, videos, GIFs, and infographics). This suggests that YMSM are open to accessing multiple means of HIV prevention support and that providing a large toolbox of HIV prevention options using diverse modalities is essential for reaching this group at the highest risk for HIV acquisition [59].
Although there was consensus on the importance of the content and utility of the features of MyChoices, there was a wide range of views on preferences for the user interface. Some participants appreciated that the language and presentation of information was more formal, as it was viewed as more trustworthy and legitimate. However, others felt that the interface was bland and potentially even anxiety producing and that more casual language would render the app more relatable. This dichotomy highlights the ongoing difficulty in developing an app that aims to reach large populations; although YMSM in the United States are a subpopulation of a larger group, they are not a monolith and will have diverse needs and preferences [60][61][62]. In addition, with technology constantly changing and advancing, mobile apps must be flexible and responsive to these changes in technology and end user preferences [63,64].

Limitations
These results should be interpreted in light of the following limitations. First, the technical pilot was small and used a nonrandomized design, and as a result, it was not powered or designed to evaluate efficacy. Sample sizes for technical pilots are often determined based on practical considerations rather than inferential statistical power calculations [65]. Still, the evaluation of feasibility and acceptability of behavioral interventions in open pilot studies is an important part of the intervention development process and helps inform subsequent randomized controlled trials (RCTs) to evaluate the efficacy of the intervention [66]. Second, social desirability bias may have led participants in the theater testing and technical pilot to speak more positively about their experience of the app during focus groups and exit interviews. To minimize the potential for these biases, participants were continuously reminded that there were no right or wrong answers and that it was important to provide honest responses. Third, for both phases of the study, individuals were enrolled in the Bronx, New York City, and Boston, Massachusetts, only, potentially limiting the generalizability of the findings. Future studies should expand to other regions of the country and outside of large metropolitan areas, particularly in the South, where the HIV epidemic is spreading most rapidly among young Black MSM. Finally, in both phases of the study, participants had to report sexual risk for HIV, no current PrEP use, and no recent HIV testing. This allowed us to ensure that we received input from those at the highest risk and from those who may benefit the most from the app; however, it also limits the generalizability of our findings to less risky populations.

Conclusions
HIV incidence in the United States remains disproportionately high among YMSM, compared with other risk and demographic groups. YMSM are also less likely than their adult peers to know that they are infected with HIV, highlighting an imminent need to increase routine HIV testing and expand access to HIV prevention interventions, including PrEP. Smartphone use is ubiquitous in the United States, and mobile apps offer an opportunity to reach YMSM "where they're at." mHealth apps have proliferated in recent years; however, only a limited number have been theory-driven and developed using evidence-based methods for intervention development. These data from theater testing and a technical pilot show the initial promise, feasibility, and acceptability of the MyChoices app to improve HIV testing and PrEP uptake among YMSM in the United States. The next step involves further pilot testing using an RCT design to determine more accurate effect size estimates; a full-scale RCT efficacy trial; and ultimately, if efficacious, an implementation study to ensure it is disseminated in such a way that maximizes its reach and utility. At each step, iterative evaluation and refinement based on the reflections and experiences of YMSM will be prioritized.