HIV incidence remains high in the United States among young men who have sex with men (YMSM). In 2018, more than 20% of new HIV infections in the United States were among young people aged between 13 years and 24 years, with YMSM accounting for 83% of newly diagnosed HIV infections in this age group [1]. New HIV infections also disproportionately impact men who have sex with men (MSM) of color. In 2018, more than half (52%) of the new diagnoses of HIV among MSM aged between 13 years and 24 years were seen among Black individuals and 27% were identified among Latinx individuals [1]. In addition to experiencing a high HIV incidence, a higher proportion of YMSM living with HIV do not know that they are infected in comparison with their adult peers [2,3]. Moreover, individuals aged between 13 years and 24 years are less likely to be linked to HIV care upon diagnosis and present lower levels of viral suppression in comparison with older individuals [2,4,5]. Consequently, YMSM will have delays seeking effective treatment and are more likely to transmit HIV to others [6].

Overwhelming evidence shows that routine HIV testing and expanded use of pre-exposure prophylaxis (PrEP) would drastically reduce the population burden of HIV [7-13]; however, uptake of both interventions is suboptimal among young adults. For example, although the Centers for Disease Control and Prevention recommends that sexually active MSM be tested for HIV at least annually [14], data suggest that nearly half of YMSM reported not being tested for HIV in the past year and one-third reported they had never been tested [15]. In addition, PrEP awareness and uptake are low among younger people [16,17]. For example, only 5% of MSM aged between 18 years and 24 years with PrEP indications reported ever using PrEP, compared with 14% of those aged 25 years and more [18]. Furthermore, only 0.1% of PrEP prescriptions in the United States were given to individuals aged under 18 years [19]. Moreover, young individuals have lower levels of adherence to and retention in PrEP care after initial prescription [20-23], decreasing the impact of PrEP on HIV prevention among this group.

Risk taking, behavioral experimentation, and confronting a host of difficult choices regarding identity formation are all part of the normal developmental trajectory of adolescence and young adulthood [24]. In addition, beliefs about invincibility, sensation seeking, and the still-developing cognitive processes of adolescents may have a role in increased HIV risk-taking behaviors and a lower prioritization of prevention strategies for this age group [25-27]. Developing innovative ways to intervene to increase engagement in HIV prevention behaviors among youth is crucial, particularly interventions that are accessible and responsive to the diverse needs of youth.

Smartphones are used by nearly all youth in the United States, across race and social class, and as such are a direct way to meet youth where they are [28]. In addition, the use of mobile phone apps by YMSM is nearly ubiquitous; apps may offer unique opportunities for public health interventions, and previous studies have demonstrated the feasibility and potential efficacy of this approach [29,30]. In a systematic review of various mobile health (mHealth) interventions, Muessig et al [29] noted that internet- and mobile–based interventions can increase dissemination of HIV prevention interventions to wider populations while also providing consistency and lower cost in intervention delivery once fully developed. In addition, mHealth tools could promote behavior change and improve aspects of the HIV care continuum, including linkage to care, retention in care, and adherence to both PrEP and antiretrovirals [30,31]. As such, a mobile phone app that aims to increase HIV testing and PrEP uptake among YMSM has the potential to provide greater access to and uptake of these prevention services for this population.

The development of the MyChoices app has been described previously [32]. In brief, MyChoices is a social cognitive theory–driven mobile app adapted from HealthMindr, an HIV prevention app developed through an iterative process for adult populations of MSM [33-35]. MyChoices built upon the initial framework and was subsequently adapted for youth by an interdisciplinary team of researchers with input from a diverse sample of YMSM at every stage of adaptation and refinement. The goal of the app is to increase HIV testing and PrEP uptake among YMSM in the United States by supporting goal setting, increasing self-efficacy, and enhancing self-regulation [36,37]. Key features of the MyChoices app include facilitation of the development of HIV testing plans with personalized recommendations; inclusion of reminder systems for HIV testing and GPS-enabled maps with local HIV and sexually transmitted infection (STI) testing locations and PrEP providers; ability to order free condoms, condom-compatible lubricants, and at-home HIV and STI test kits; and sexual health information using a variety of media (eg, videos, Graphics Interchange Format [GIF], infographics, frequently asked questions [FAQs], and quizzes) [32]. Figure 1 shows the MyChoices app home screen and the testing plan feature.

Eligible participants were cisgender men who were aged 15-24 years; did not have an HIV test in the past 3 months; self-reported being HIV uninfected or HIV status unknown at screening; owned an iOS or Android mobile phone and were willing and able to download the MyChoices app; were able to understand, read, and speak English; were not taking PrEP; and had self-reported evidence of being at risk for HIV acquisition (details on risk criteria are given in the study by Biello et al [32]).

Through the University of North Carolina/Emory Center for Innovative Technology (iTech) [38], a part of the National Institutes of Health’s Adolescent Medicine Trials Network for HIV/AIDS Interventions [39], participants were recruited across 2 sites: Boston, Massachusetts (study site: Fenway Health), and the Bronx, New York City (study site: the Adolescent AIDS Program at Montefiore). Recruitment methods included posting on social media (eg, Craigslist, social networking ads, and gay networking mobile apps); distributing posters, flyers, and palm cards about the study; direct outreach at local venues frequented by YMSM (eg, community-based organizations, schools, bars, health fairs, and balls); and clinic-based recruitment.

After an initial prototype of the MyChoices app was developed through multiple rounds of formative research with YMSM [33,40], we conducted theater testing with 28 YMSM in 4 groups across the 2 iTech sites (5-8 participants per group). Theater testing allows for groups of participants to interact with the product being tested and provide feedback in situations that approximate real-life experiences and has been used commonly in mHealth app development [33,35,41,42]. Testing was conducted in a private room at each site by a research staff member who had training in qualitative methods and group facilitation. All participants completed a written informed consent or assent process before data collection commenced. Participants completed a brief demographic and behavioral questionnaire before theater testing to contextualize the group data collected. During theater testing, participants interacted with the MyChoices app prototype and provided feedback on the functionality, appearance, and usability of the platform. We also asked participants to comment on ways to maximize acceptability (eg, update language and improve flow) and to identify the components of the app that were liked or disliked and aspects that could be improved to increase HIV testing and PrEP uptake among YMSM. Groups lasted 60 minutes to 90 minutes and were audio recorded and professionally transcribed verbatim. Participants received US $50 as reimbursement for their time.

Members of the iTech Analytic Core [43] reviewed transcripts for quality and identified emergent themes. We then used Dedoose Version 8.0.35 (SocioCultural Research Consultants, LLC) software to apply the final codes to all transcripts. Thematic analysis involved using a primarily deductive approach to synthesize data coded for app acceptability, particularly around functionality, appearance, usability, and potential for improving HIV testing and PrEP uptake [44,45]. Findings are illustrated in the following sections using representative quotes. These data were used to refine the app before the initiation of the open technical pilot.

Once the MyChoices app had been refined, a technical pilot with 11 YMSM across the 2 iTech sites was conducted to assess the initial feasibility and acceptability and to identify any final areas for improvement. Eligible participants (mentioned earlier) attended a visit at the study site, at which they completed a web-based behavioral and psychosocial assessment, which included measures to assess sociodemographics (ie, enrollment city, age, race or ethnicity, educational status, and insurance status), sexual behaviors (ie, frequency of condomless anal sex), HIV testing history, and PrEP awareness. Study staff then assisted youth with app download and provided them with brief instructions on the purpose of the MyChoices app and an overview of how to use it; participants were encouraged to use the app over the course of 2 months.

At month 2, the participants completed a web-based assessment. In addition to the measures collected at baseline, we assessed the acceptability of the MyChoices app using the System Usability Scale (SUS) [46]. SUS is a validated 10-item measure that assesses the subjective usability of a system or, in this case, an app [46]. SUS has been extensively used in mHealth research and provides reliable results even with small sample sizes [47]. It is scored from 0 to 100, and a score of ≥50 indicates that the app is acceptable [48]. Feasibility was assessed using app analytics to determine whether the app was used, how often it was used, and what components were used most and least frequently.

Finally, we conducted exit interviews with participants to obtain feedback on app functionality, technical performance, errors and software bugs encountered, overall experiences using the app, feedback for further refinement, and subjective impact of the app on HIV testing and PrEP uptake. Exit interviews were conducted by study staff on the web using videoconferencing technology that was compliant with the Health Insurance Portability and Accountability Act. Interviews were transcribed and analyzed using the same approach outlined above.

Participants received US $50 for completing the baseline visit, US $25 for the 2-month assessment, and US $50 for the exit interview.

The study procedures were reviewed and approved by the University of North Carolina Institutional Review Board (IRB) as a single IRB-of-Record. IRB authorization agreements with all participating research entities were enacted. The MyChoices protocol is registered at ClinicalTrials.gov (NCT03179319).

Table 1 provides the characteristics of the 28 YMSM who participated in theater testing. Participants’ ages ranged from 16 years to 24 years, with a mean age of 20 years. YMSM of color made up 57% (16/28) of the sample, and 57% (16/28) of the participants were still in school.

A total of four key themes emerged from theater testing, and thus, we structure the presentation of our results to highlight these findings: (1) general utility and acceptability of the app, (2) feedback and suggestions for user interface, (3) opinions on language, and (4) suggestions for additional content and features.

The open pilot enrolled 11 participants (Boston, n=6; the Bronx, n=5), and retention at the 2-month follow-up was 100%. Participants’ ages ranged from 15 years to 23 years, with a median age of 19 years. YMSM of color made up 91% (10/11) of the sample (Black, non-Hispanic, n=5; Hispanic or Latinx, n=5; Table 2).

This study describes the findings from two phases of formative data collection (theater testing of the MyChoices mobile app prototype and the technical pilot of MyChoices fully functional app) to create a mobile app to increase HIV testing and PrEP uptake among young MSM in the United States. HIV prevention apps are proliferating; however, using evidence-based methods for intervention development, including community-centered approaches with iterative feedback from the community, is essential to maximize their impact and reach [35,49], and few HIV prevention apps for youth have been developed and tested using the approaches described in this paper.

In the context of previously reported findings about preferences for mobile HIV prevention apps in young people, our findings support and extend previous reports by providing a sample including a higher proportion of teenage participants, confirming some aspects of previous reports, and documenting some novel findings. Like others [50-52], we found that youth valued having an app that presents credible information relevant to their health and interests in direct and understandable language and the value of having educational information available. Notably, other qualitative studies of preferences for HIV prevention or HIV management eHealth tools have identified some themes that did not emerge from the youth who participated in our study. For example, other researchers have found youth preferences for features that allow them to interact with other youth [53], and young MSM in other studies reported concerns about privacy and confidentiality of data [54]. Young MSM have also raised questions about what the scope of sexual health apps should be, with some suggesting that related health issues, such as substance use, could be included in a mobile app resource [52]. Our participants also provided suggestions for developing geospatial tools to guide users to prevention services. These issues, which have been rarely reported in other studies, may have emerged from our participants because geospatial tools are becoming more refined and younger people have grown up relying on smartphones as a primary source of navigation. Due to the younger age of our participants compared with many previous studies and because our data were collected more recently, youth expectations around enhanced navigation and geolocation services may be seen as emerging expectations for mobile prevention apps.

Theater testing revealed high levels of interest in the content provided through MyChoices, with youth indicating that the availability of this type of information is both lacking and necessary. Although youth have access to huge amounts of health information through the internet and report using the internet frequently to access this type of information [55,56], they do not always know what to trust [57,58]. In theater testing, youth indicated that one of the major strengths of the app was to have access to a wide range of sexual health information in one place, in multiple formats, that they knew they could trust. In addition, the app’s usability score is comparable with the HealthMindr app on which MyChoices was based, although it had a higher proportion of people who would recommend it to friends [33]. This may suggest a successful adaptation for YMSM.

In addition to the content, the features of the MyChoices app were also broadly seen as favorable and useful. In both theater testing and the technical pilot, the most popular features included ordering free HIV and STI self-testing kits, condoms, and lube; ability to search for nearby HIV testing and PrEP care sites; and PrEP information provided in multiple formats (ie, text, videos, GIFs, and infographics). This suggests that YMSM are open to accessing multiple means of HIV prevention support and that providing a large toolbox of HIV prevention options using diverse modalities is essential for reaching this group at the highest risk for HIV acquisition [59].

Although there was consensus on the importance of the content and utility of the features of MyChoices, there was a wide range of views on preferences for the user interface. Some participants appreciated that the language and presentation of information was more formal, as it was viewed as more trustworthy and legitimate. However, others felt that the interface was bland and potentially even anxiety producing and that more casual language would render the app more relatable. This dichotomy highlights the ongoing difficulty in developing an app that aims to reach large populations; although YMSM in the United States are a subpopulation of a larger group, they are not a monolith and will have diverse needs and preferences [60-62]. In addition, with technology constantly changing and advancing, mobile apps must be flexible and responsive to these changes in technology and end user preferences [63,64].

These results should be interpreted in light of the following limitations. First, the technical pilot was small and used a nonrandomized design, and as a result, it was not powered or designed to evaluate efficacy. Sample sizes for technical pilots are often determined based on practical considerations rather than inferential statistical power calculations [65]. Still, the evaluation of feasibility and acceptability of behavioral interventions in open pilot studies is an important part of the intervention development process and helps inform subsequent randomized controlled trials (RCTs) to evaluate the efficacy of the intervention [66]. Second, social desirability bias may have led participants in the theater testing and technical pilot to speak more positively about their experience of the app during focus groups and exit interviews. To minimize the potential for these biases, participants were continuously reminded that there were no right or wrong answers and that it was important to provide honest responses. Third, for both phases of the study, individuals were enrolled in the Bronx, New York City, and Boston, Massachusetts, only, potentially limiting the generalizability of the findings. Future studies should expand to other regions of the country and outside of large metropolitan areas, particularly in the South, where the HIV epidemic is spreading most rapidly among young Black MSM. Finally, in both phases of the study, participants had to report sexual risk for HIV, no current PrEP use, and no recent HIV testing. This allowed us to ensure that we received input from those at the highest risk and from those who may benefit the most from the app; however, it also limits the generalizability of our findings to less risky populations.

HIV incidence in the United States remains disproportionately high among YMSM, compared with other risk and demographic groups. YMSM are also less likely than their adult peers to know that they are infected with HIV, highlighting an imminent need to increase routine HIV testing and expand access to HIV prevention interventions, including PrEP. Smartphone use is ubiquitous in the United States, and mobile apps offer an opportunity to reach YMSM “where they’re at.” mHealth apps have proliferated in recent years; however, only a limited number have been theory-driven and developed using evidence-based methods for intervention development. These data from theater testing and a technical pilot show the initial promise, feasibility, and acceptability of the MyChoices app to improve HIV testing and PrEP uptake among YMSM in the United States. The next step involves further pilot testing using an RCT design to determine more accurate effect size estimates; a full-scale RCT efficacy trial; and ultimately, if efficacious, an implementation study to ensure it is disseminated in such a way that maximizes its reach and utility. At each step, iterative evaluation and refinement based on the reflections and experiences of YMSM will be prioritized.