Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 03.12.20 in Vol 22, No 12 (2020): December

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/20158, first published May 12, 2020.

This paper is in the following e-collection/theme issue:

    Review

    Effectiveness, Acceptability, and Feasibility of Digital Health Interventions for LGBTIQ+ Young People: Systematic Review

    1Telethon Kids Institute, Perth, Australia

    2School of Psychological Science, The University of Western Australia, Perth, Australia

    3Discipline of Psychology, College of Science, Health, Engineering and Education, Murdoch University, Perth, Australia

    4Centre for Child Health Research, The University of Western Australia, Perth, Australia

    Corresponding Author:

    Dylan Gilbey, BSc (Hons)

    Telethon Kids Institute

    15 Hospital Avenue

    Perth

    Australia

    Phone: 61 63191195

    Email: Dylan.Gilbey@telethonkids.org.au


    ABSTRACT

    Background: Young people (aged 12-25 years) with diverse sexuality, gender, or bodily characteristics, such as those who identify as lesbian, gay, bisexual, transgender, intersex, or queer (LGBTIQ+), are at substantially greater risk of a range of mental, physical, and sexual health difficulties compared with their peers. Digital health interventions have been identified as a potential way to reduce these health disparities.

    Objective: This review aims to summarize the characteristics of existing evidence-based digital health interventions for LGBTIQ+ young people and to describe the evidence for their effectiveness, acceptability, and feasibility.

    Methods: A systematic literature search was conducted using internet databases and gray literature sources, and the results were screened for inclusion. The included studies were synthesized qualitatively.

    Results: The search identified 38 studies of 24 unique interventions seeking to address mental, physical, or sexual health–related concerns in LGBTIQ+ young people. Substantially more evidence-based interventions existed for gay and bisexual men than for any other population group, and there were more interventions related to risk reduction of sexually transmitted infections than to any other health concern. There was some evidence for the effectiveness, feasibility, and acceptability of these interventions overall; however, the quality of evidence is often lacking.

    Conclusions: There is sufficient evidence to suggest that targeted digital health interventions are an important focus for future research aimed at addressing health difficulties in LGBTIQ+ young people. Additional digital health interventions are needed for a wider range of health difficulties, particularly in terms of mental and physical health concerns, as well as more targeted interventions for same gender–attracted women, trans and gender-diverse people, and people with intersex variations.

    Trial Registration: PROSPERO International Prospective Register of Systematic Reviews CRD42020128164; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=128164

    J Med Internet Res 2020;22(12):e20158

    doi:10.2196/20158

    KEYWORDS



    Introduction

    Young people who are lesbian, gay, bisexual, transgender, intersex, queer and other people of diverse sexuality, gender, or bodily characteristics (LGBTIQ+) are known to experience a range of disparities in health outcomes compared with their peers [1]. These include higher rates of mental health difficulties, such as depression and suicidality [2-4]; physical and sexual health problems, such as the incidence of HIV [5]; cigarette and alcohol use [6-8], obesity [9], and teen pregnancy [10]. Later in adulthood, the confluence of these health issues conveys further risk for cancer [11] and cardiovascular disease diagnoses [12-14]. The burden of disease that these disparities carry is a public health issue that urgently requires safe, effective, and early intervention.

    These health disparities are compounded by barriers that negatively impact the ability of LGBTIQ+ young people to access health services that are safe and adequately meet their needs. Young people in the general population face many barriers to help seeking, including inadequate resources and lack of accessibility, desire for self-reliance, and anticipated stigma for reporting certain health difficulties such as mental illnesses or HIV [15,16]. LGBTIQ+ young people face a range of additional difficulties, such as low parental support, which can lead to homelessness [17-19], and unemployment due to discrimination [20,21], which may intensify these help-seeking barriers [22]. This group also faces unique help-seeking difficulties, such as anticipated and experienced stigma surrounding their identities [22-24], concerns about disclosure or their compromised confidentiality [24], and low perceived confidence in the ability of service providers to deliver LGBTIQ+ supportive care [23-26]. These problems may be particularly pronounced among queer and trans and gender-diverse young people [25,27]. The factors impeding health care access and use may significantly contribute to the maintenance of health disparities in LGBTIQ+ young people [28,29]. Accordingly, removing these obstacles is an important step toward improving health in this population.

    Digital health interventions, such as those delivered via computers, websites, smartphones, or tablets, have been identified as an important potential avenue to improve health care access and use among young people in this group [30-33]. Accessing support digitally allows young people to bypass many of the aforementioned barriers. Anonymity facilitates private access to support and minimizes stigma [33]. In addition, digital health interventions confer further benefits beyond traditional clinical environments, being available without travel, accessible at all hours, and having no waitlists [34]. Self-guided digital health interventions are also cost- and resource-effective to access and disseminate, requiring less direct input from clinicians [34], giving consumers greater control and empowerment regarding their own health needs [35]. These considerations are especially pertinent for widening the support available to populations that are restricted from accessing traditional health care services [36]. However, digital health interventions are also commonly reported to have very high rates of attrition and disengagement [37,38], with up to 60% to 80% of users discontinuing their use [39-41], and the quality of evidence supporting the effectiveness of digital health interventions is also frequently lacking [42-45]. Along with the limited availability of many evidence-based digital health interventions beyond the research context, these issues call into question the real-world utility of these interventions despite their proposed theoretical benefits.

    LGBTIQ+ young people are adept and frequent users of digital technologies [33,46,47]; however, research indicates that technology use can present several challenges. Evidence suggests that the internet (including social media and online communities in particular) can be harmful in this population (as well as young people more broadly) [31] due to the potential for toxic interactions and exposure to distressing content [48], such as the normalization of self-harm and suicidal behaviors [49]. However, an array of benefits associated with technology use has also been documented in LGBTIQ+ young people. The internet allows LGBTIQ+ young people to explore their identities in an anonymous and safe way, seek out peers who share their identities, and come out to others in a low-risk environment [47]. The internet also facilitates an important component of the sexual development of LGBTIQ+ young people, enabling the exploration of same-gender attraction for some and seeking out romantic or sexual partners [50]. LGBTIQ+ young people may also already use the internet to access resources that are relevant and safe for them [47], suggesting that digital health interventions targeting this group may be useful. The concept of digital delivery of interventions is generally acceptable to this group, and this is particularly true when they are specifically targeted with LGBTIQ+ themes [32]. Many existing digital health interventions are not specifically applicable to LGBTIQ+ people [51]; however, untailored interventions may exacerbate feelings of alienation [31,52,53]. Given these factors, the development and evaluation of targeted digital health interventions for LGBTIQ+ young people may represent an opportunity to improve the delivery of health care to this group, should the benefits outweigh the known challenges associated with digital health care delivery discussed above.

    In response, there has been a rapid increase in the number of such interventions over the past decade. To date, however, there has not been a comprehensive review summarizing the scope and use of digital health interventions that currently exist for this population. Knight et al [54] recently reviewed web-based interventions for HIV or sexually transmitted infection (STI) risk reduction in young men who have sex with men (MSM); however, this review did not capture digital health interventions that are delivered through other digital modalities (eg, mobile apps), those that address other health issues, or those that target women, gender-diverse individuals, or individuals with intersex variations. There have also been several recent reviews focused on the mental health of LGBTIQ+ young people [55,56] and adults [57], which have referenced a combined total of 4 digital interventions for this population. However, these reviews were not explicitly focused on the use of digital technology nor did they consider interventions designed to improve a broad range of health outcomes in this population.

    A more extensive summary of this rapidly growing field of research will assist in identifying gaps in the development of interventions and determining the overall evidence for their use across the full diversity of the young LGBTIQ+ community. Therefore, this review aims to answer the following questions: (1) What are the characteristics of evidence-based digital health interventions for improving mental, physical, and sexual health outcomes in LGBTIQ+ young people? (2) Are targeted digital health interventions effective at improving health outcomes in LGBTIQ+ young people? (3) Are targeted digital health interventions acceptable and feasible for LGBTIQ+ young people?


    Methods

    Protocol and Registration

    The protocol for this review was registered using PROSPERO (Prospective Register of Systematic Reviews; ID CRD42020128164) in accordance with PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) recommendations [58].

    Eligibility Criteria

    Types of Participants

    The population of interest was LGBTIQ+ young people. The LGBTIQ+ term was used in its broadest sense to capture young people of diverse sexuality (including but not limited to those that identify as gay, lesbian, bisexual, or pansexual), diverse gender (including but not limited to those who identify as trans or nonbinary), with diverse sex characteristics (including but not limited to people with intersex variations), or people falling across any combination of these categories. The search strategy (below) was designed to be as inclusive as possible of the wide variety of identities that LGBTIQ+ people may hold including, for example, people who fall within these aspects of diversity without explicitly identifying as such (eg, MSM). Young people was defined as being primarily people between the ages of 12 and 25 years; the mean age of study participants was required to fall within this range for a study to be eligible to be included in the review. In addition, studies were required to have intentionally and specifically recruited young people.

    Types of Intervention

    The review focused on interventions designed to effect change through predominantly digital means (eg, using a computer, website, tablet, or smartphone). To be included, interventions were required to be targeted or intended to specifically effect change in health outcomes in LGBTIQ+ people. Interventions delivered via telephone with no technological function or an implantable device that is remotely monitored were excluded. Interventions were also required to have minimal human guidance in the intervention itself if present at all. Specifically, the action, process of intervening, or behavior change techniques must have been delivered by the digital technology itself not a health professional working over a digital medium. This criterion was implemented because, in interventions that blend digital and human support, the impact of the intervention cannot be meaningfully attributed to the digital component alone [59].

    The judgment of the level of human guidance was made by considering the ratio of clinicians or staff to users, and the centrality of the human guidance to effecting change in the health outcome, which itself relied on factors such as the ratio of guided versus unguided time during the intervention. No hard limits on these factors were set prior to conducting the review, as making this judgment required the full context of the intervention to be considered holistically. A judgment about the duration of the human guidance, for example, could not meaningfully be made without consideration of the purpose of that period of guidance and how it fits into the goals and process of the intervention as a whole. What was counted as minimal human guidance was therefore determined on a case-by-case basis requiring consensus from the reviewers.

    Types of Studies

    To be included, studies should have conducted an evaluation of a specific intervention as described above. Evaluation in some form was required in keeping with standards of evidence-based practice. The term evaluation was inclusive of examination of efficacy, effectiveness, acceptability, or feasibility, with a minimum of any one form of evaluation required for inclusion. All quantitative, qualitative, or mixed methods studies were eligible for inclusion. Comparators or control groups were not necessary for inclusion in the review. Protocols describing an intervention without any evaluation were not included in the review; however, when protocols were identified, steps were taken to determine if the corresponding data were publicly available. Studies evaluating the concept of digital health interventions generally or studies describing the initial development of an intervention were also ineligible. Studies evaluating digital health interventions not specifically designed for LGBTIQ+ young people were also excluded, even if conducted with an LGBTIQ+ sample.

    No specification was made for the location of the study; however, studies were required to be published in the English language. With the aim of reducing the risk of publication bias [60], gray literature was considered eligible and studies were not required to be peer reviewed to be included in the review. The search was restricted to studies published after January 1, 2000. This cut-off was selected because of the types of interventions eligible for this review, only web-based interventions may have existed at this time, and the likelihood of such an intervention existing specifically tailored for a select, marginalized group was deemed to be extremely low. Scoping searches conducted before deciding on this cut-off did not determine any evidence of existing interventions contrary to this conclusion.

    Types of Outcomes

    The review was designed to capture interventions seeking to improve health outcomes or to prevent negative health outcomes. This was inclusive of mental health outcomes (eg, symptoms or diagnoses of mental disorders, well-being, distress), physical health outcomes (eg, smoking, weight loss), or sexual health outcomes (eg, pre-exposure prophylaxis [PrEP] adherence, condom use). Any outcome reasonably perceived to represent some aspect of health and well-being was considered relevant to the review. For studies evaluating efficacy, changes must have been reported in measures of at least one of these outcomes. For studies evaluating acceptability or feasibility, at least one index of these factors (eg, surveys of participant experiences, adherence, or attrition rates) must have been reported.

    Search Strategy

    Internet databases such as PsycINFO (Ovid) and MEDLINE (Ovid) were systematically searched on August 13, 2019, and potentially relevant peer-reviewed publications were extracted. These searches were conducted using a combination of subject headings and keywords corresponding to the following themes: LGBTIQ+, Youth/Young People (aged 12-25 years), Mental Health, Physical Health, Sexual Health, Digital and Intervention. The search terms for LGBTIQ+, Youth/Young People, and Mental Health themes were adapted from those previously reported by Gilbey et al [61] and Lee et al [62]. The search terms for the other themes were devised from a broad initial scoping search of relevant articles to identify key terms and phrases. The search strategy for PsycINFO (Ovid) is presented in Multimedia Appendix 1.

    Additional searches were made using Scopus, ProQuest Dissertations, Google, Google Scholar, OpenGrey, WorldCat, ClinicalTrials.gov, and JMIR Publications, during July and August 2019. Each of these searches were conducted with several simple keyword searches (eg, LGBTQ, transgender) as the relatively low number of relevant articles available in each source made a more comprehensive, and therefore restrictive, search process unnecessary. The reference lists of several other related reviews and key articles on the subject were also hand-searched for potentially relevant articles during July 2019. Google Scholar alerts were monitored for any additional articles published until April 2020. The searches of PsycINFO (Ovid) and MEDLINE (Ovid) were repeated on March 25, 2020, and articles published since August 2019 were manually searched for any newly published studies.

    Screening

    The titles and abstracts of the articles identified by the search were screened for relevance by the lead author (DG), removing articles with no clear relevance to the topic of the review. Two authors (DG and HM) then screened the full-text of the remaining articles independently, with differences in opinion resolved in discussion with a third author (YP) in which full agreement was sought.

    Data Extraction

    The following data items were extracted from eligible studies: author(s), year, participant age (mean and range), description of sample (eg, LGBTIQ+ status), sample size, study design, study setting, intervention type, content and delivery, digital platform, control group type (if relevant), degree of human guidance in the intervention, health outcome(s), acceptability outcome(s), and feasibility outcome(s). A second reviewer (HM) cross-checked these data.

    Critical Appraisal

    Following data extraction, studies were evaluated using the Mixed Methods Appraisal Tool (MMAT) [63] by 2 reviewers (DG and HM). The MMAT is designed for the assessment of methodological quality of studies with a range of designs (qualitative, quantitative, and mixed methods), such as those reported herein. On the basis of the recommendations of the authors of the tool, quantitative quality scores were not derived; instead, the results of the appraisal are discussed qualitatively.

    Synthesis of Results

    Owing to the wide array of interventions, targets of intervention, intervention modality, and health outcomes measured, it was anticipated that a quantitative synthesis (of those studies reporting quantitative data) would be neither feasible nor informative. Therefore, the results of the studies were synthesized qualitatively.


    Results

    Study Selection

    The search and screening process is displayed in a PRISMA flow chart in Figure 1. A total of 2192 studies were identified in the search. Following title and abstract screening, 165 studies were retained for full-text screening. The final number of studies retained for the review following full-text screening was 38.

    Figure 1. Study flowchart. LGBTIQ+: lesbian, gay, bisexual, transgender, intersex, queer.
    View this figure

    Study Characteristics

    The 38 studies captured the results of studies examining 24 unique interventions conducted across 3 countries (the United States, the United Kingdom, and New Zealand). Of these, 5 targeted primarily mental health–related issues, one targeted primarily physical health–related concerns, one targeted primarily sexual health and well-being, and 17 targeted risk reduction or management of STIs. A total of 22 interventions focused specifically on young people who are attracted to the same gender (referred to with a variety of terms, eg, sexual minority, lesbian/gay/bisexual/queer people, MSM), of which 19 interventions were described as being focused on young men (eg, gay/bisexual men, MSM). Several studies that described their target audience as MSM also included trans women under this descriptor. One study targeted people who identify as lesbian, gay, bisexual, transgender and queer (LGBTQ) generally [64] and one intervention targeted transgender men and nonbinary people specifically [65]. A total of 3 interventions specifically targeted young LGBTQ+ people of color, although several other interventions were also conducted with predominantly non-White participants. No studies were identified that sought to improve health in young people with intersex variations, and no studies were identified that sought to improve health in young women alone. The characteristics of the included studies are reported in Table 1. Because of the length of the table, the characteristics of the included studies addressing sexually transmitted infection risk reduction and management are reported separately in Multimedia Appendix 2 [66-91]. Brief summaries of each intervention as well as their core findings are also provided in Multimedia Appendix 3 [64-100].

    Table 1. Summary of digital mental, physical and sexual health interventions for lesbian, gay, bisexual, transgender, intersex or queer young people.
    View this table

    Risk of Bias in Individual Studies

    Mixed Methods Studies

    Mixed methods designs were used well among the included studies overall, with most meeting all of the criteria in the MMAT for such designs. This is not to say that these studies did not have methodological flaws, as the MMAT mixed methods subsection does not generally consider the quality of the individual qualitative and quantitative components but rather their intersection and integration. These components of the studies were also, therefore, considered individually and are included among those described below.

    Quantitative Studies

    The majority of included quantitative trial studies were described as pilot or feasibility studies (17 out of 27), and their methodological quality was lacking in many cases. In total, 15 of the 27 trials were randomized trials, of which 4 did not report appropriate randomization procedures, 12 did not report blinding procedures, and participant adherence to the intervention was only reported in 2 studies. Of the 12 nonrandomized trials, 10 did not report accounting for confounding variables in their design and analysis; however, representativeness in the study samples was generally adequate. The majority of the 12 quantitative descriptive studies’ methodologies were vulnerable to nonresponse bias.

    Qualitative Studies

    Studies’ methodologies were generally sufficient to meet the MMAT criteria for qualitative studies. It is worth noting, however, that most of the studies did not specifically outline the methodological framework underpinning the study, and it is unclear whether this was due to inadequate reporting or the absence of such structures entirely. Some studies did not appear to adequately substantiate their conclusions with the data, but again it was difficult to interpret whether this was due to flaws in methodology or omission of reporting.

    Synthesis of Results

    Content of Digital Health Interventions for LGBTIQ+ Young People

    Of the 5 interventions targeting mental health difficulties, 2 focused on internalizing symptom reduction [93-95], 1 targeted drug abuse prevention [92], and 2 focused on nonspecific aspects of psychological well-being [65,97]. Although few in number, existing digital mental health interventions targeted several LGBTIQ+ subgroups, and there was relatively little conceptual overlap between them. Four of these interventions were theory-driven, and only one mental health intervention showed noteworthy community involvement in the development of the intervention. With one exception, mental health interventions tended to rely on some form of skill-building or otherwise didactic content delivery.

    There were few interventions targeting physical health problems or sexual health and well-being in LGBTIQ+ young people. Only one digital intervention focused on smoking cessation and targeted physical health in LGBTIQ+ young people [98,99]. This intervention showed aspects of both community-driven and theoretical designs. One digital intervention targeting sexual and reproductive well-being overall in LGBTIQ+ young people was developed with a theoretical framework but no community input [64]. Some aspects of this intervention overlapped with interventions targeting the risk of STIs, such as increasing condom use; however, other aspects diverged, such as including content on healthy relationships more broadly.

    The majority of the interventions identified in this review were targeted toward risk reduction or the management of STIs. Of the 17 interventions identified as having focused on risk reduction and management of STIs, 1 focused on pre-exposure prophylaxis adherence [66], 2 focused on reducing unprotected sex [67-71], 7 targeted multiple aspects of HIV prevention [72-83], 2 focused on HPV prevention [84,85,101], 3 focused on increasing STI testing [86-89], and 2 focused on antiretroviral medication adherence [90,91]. All 17 interventions were targeted toward young men who are attracted to men, 2 of which were specifically for adolescents. The majority of these interventions included some aspect of community involvement in their design, although the extent varied from iterations based on user feedback to more central participatory design, which few interventions involved. All but one intervention was theory-driven in nature, with most interventions drawing from either the Information-Motivation-Behavioral Skills Model or Social Cognitive Theory in their design. Therefore, there was notable conceptual overlap among these interventions, which largely varied only in their delivery format and the breadth of their focus (ie, targeting specific aspects of risk reduction, such as condom use, versus a variety of such behaviors).

    Delivery of Digital Health Interventions for LGBTIQ+ Young People

    The most common platforms for digital health interventions were websites and mobile apps that, combined, represented over half of the interventions identified. A smaller number of interventions were delivered via computer software. Many interventions used gamification, or elements of game playing such as point scoring, in their delivery; however, only 2 interventions were fully gamified in nature [90,95]. Few interventions incorporated social interactivity and where present, they were typically minimal [72,75,78,90,98]. Most interventions were multimedia, incorporating a number of different delivery formats and types of content.

    The vast majority of interventions delivered information to effect change either in terms of building awareness about the health issue in question or teaching skills to enable behavior change. This information is typically delivered via text or videos. For some interventions, this was the entirety of their scope; however, others included quizzes, games, or practice scenarios to consolidate the knowledge being presented. Interventions varied significantly in the extent of their personalization; some interventions delivered the same content to all participants, while others provided opportunities for personalized input and then delivered information specific to the individual’s situation or needs at the time.

    Interventions varied in the duration and intensity of their delivery. In total, 16 of the 24 interventions appeared to be intended to be a perpetually available resource that could be accessed at any time and largely completed in a single instance if desired. Intervention duration ranged from very brief completion times, as low as 10 min [84], to up to 3.5 hours [95] to complete. In total, 8 of the 24 interventions staggered their delivery in some respect, such as presenting new content over a certain period [66,90], or otherwise incentivizing users to return to the intervention over a period of up to several months [96]. Few interventions described periods of use over 2 months; however, this may reflect the limitations of trial periods rather than their ideal dissemination. Overall, a small minority of interventions appear to be available for public use at present.

    Effectiveness of Digital Health Interventions for LGBTIQ+ Young People

    With regard to the effectiveness of these interventions, there was consistent evidence from a number of interventions that digital health interventions could improve HIV testing rates in young MSM [79,80,88,89]. Aside from this, STI-focused interventions appeared to effect change more consistently in cognitive or attitudinal outcomes, such as HIV awareness and preparedness to use condoms, than in behavioral outcomes in practice, such as condom use and unprotected anal sex. Some studies observed changes in these outcomes [69,73,74,83], while others did not [67,79,80]. There was insufficient evidence to suggest that interventions targeting a specific risk-related outcome were more effective at improving that outcome than interventions that sought change in a variety of risk-related outcomes.

    Comparatively, few interventions were targeted toward improvement of mental or physical health issues; however, the interventions that did exist in this sphere were more targeted and overlapped less in scope. With the exception of an expressive writing intervention [97], digital health interventions demonstrated preliminary effectiveness in reducing internalizing symptoms such as depression and psychological distress [65,95]. Digital health interventions have also been reported to be effective at reducing substance use, including cigarette use [99] and peer drug use [92]. Owing to the limited number of interventions targeting these difficulties, it is difficult to determine any patterns regarding the factors predicting greater effectiveness; however, the only intervention that did not show notable effectiveness was also the only intervention to not present any didactic or skill-building content to the user.

    Given that the majority of the interventions included in the review were multimedia in some form, it was not possible to draw conclusions about the delivery aspects that would most reliably effect change in outcomes. Furthermore, and importantly, given that the overall quality of the interventions included in this review was suboptimal, their effectiveness must be viewed in light of limitations associated with methodology and reporting.

    Acceptability and Feasibility of Digital Health Interventions for LGBTIQ+ Young People

    Overall, digital health interventions were generally acceptable to LGBTIQ+ young people, and there were some clear themes in aspects of these interventions that determined users’ level of interest. Gamification stood out as a component of interventions that tended to be highly regarded by participants. Information presented with brevity and in a relatable way tended to receive greater ratings of acceptability from users and, although infrequent, social aspects of interventions, such as the ability to share experiences with others, were generally highly rated as well. Common concerns raised about the interventions included information being too text heavy, patronizing or contrived, and tasks feeling too laborious or homework-like. Importantly, users voiced concerns regarding the targeting of the interventions to LGBTIQ+ people, for example, being targeted at a superficial level, or coming across as stereotypical and alienating in its presentation of LGBTIQ+ people.

    Regarding the feasibility of the interventions, measures of engagement and adherence were often not reported by the included studies; however, those that did report levels of use (eg, screen time, clicks, communication with other users) appeared adequate, given the intended scope of the intervention. One study that included an in-person component (collecting rewards earned in the web-based component) reported low user engagement with this feature (5%-27%). Overall, rates of attrition among the included studies were low, with several interventions reporting retention rates of 90% to 100% in their trials [66,69,77]. Notably, however, a trial of one of these interventions in a community setting reported a much lower retention rate of 45.4% [67], indicating that engagement may be lower in reality than controlled trials would suggest. Other studies reported retention rates of 70% to 90%. Two other studies reported notably higher rates of attrition than the others [65,89]. These high rates of attrition did not appear linked to acceptability, with both interventions reporting largely positive responses from participants. There were no notable differences in acceptability and feasibility based on the health outcome interventions.


    Discussion

    LGBTIQ+ young people have a substantially higher risk of a range of health difficulties than the general population [1], and targeted digital health interventions have the potential to play a crucial role in reducing these disparities [30,33]. The aims of this review were to (1) synthesize the scope of evidence-based, targeted digital health interventions that exist for this population, (2) to identify the overall effectiveness, acceptability, and feasibility of these interventions in this population, and (3) to provide recommendations for their development. The review identified many interventions designed to improve health in LGBTIQ+ young people, and these interventions have shown preliminary effectiveness in producing changes in some health outcomes in this group. Particularly promising evidence was found for the effectiveness of digital health interventions in certain aspects of managing the risk of STIs, notably increasing HIV testing rates, and emerging evidence was also found for reductions in internalizing symptoms and substance use. The review observed a trend that digital health interventions for LGBTIQ+ young people may more consistently effect change in cognitive and affective outcomes than behavioral outcomes, though this was not prescriptive. All of these findings must be considered in light of the preliminary nature of the majority of the studies included in this review and their resultant methodological limitations.

    In addition to showing potential for effectiveness, the interventions were found to be generally acceptable and feasible overall. Acceptability appeared closely linked to collaborative intervention design development with LGBTIQ+ young people and the digital modality of delivery. Notably, one study, which had regular check-ins with a clinician, found that participants were in fact deterred by this contact, citing difficulty scheduling and desire to remain discreet [96]. There is currently insufficient evidence to conclude that digital health interventions would be more effective than untargeted or face-to-face interventions in controlled conditions, as most of the studies outlined in this review did not provide such comparisons. However, results such as these suggest that digital health interventions may not need to be more effective than other forms of intervention to be valuable; these interventions would likely engage sections of the LGBTIQ+ youth population who would otherwise be deterred from seeking any support at all. In addition, rates of attrition in the included interventions were lower than those reported in similar interventions for other groups [37,39-41]. When combined with the innately greater potential for dissemination and cost effectiveness that comes with the digital medium, these findings support the premise that digital health interventions may be an important avenue for reducing health disparities in LGBTIQ+ young people in the future.

    Future Directions

    Overall, the results of this review are therefore promising for the continued development of digital health interventions for LGBTIQ+ young people, and there are some clear paths forward for how this field of research could be developed further. Most of the interventions included in this review have thus far only been evaluated in terms of usability, acceptability, and preliminary efficacy, and due to the preliminary nature of most of these studies, when efficacy was evaluated, aspects of methodology such as randomization and blinding were often lacking in rigor. The findings of this review are therefore consistent with others in the literature that frequently report low quality of evidence associated with digital health interventions [43,45,102,103]. In addition, only 1 study trialed a digital health intervention in a community setting [61], and its rate of user retention was approximately half that of its clinical trial, consistent with previous studies that have found a similar pattern [104]. The generalizability of the findings of these studies to the wider population of LGBTIQ+ young people, therefore, cannot be determined. There is a clear need to build on the emerging evidence base through more rigorous randomized controlled trials and trials in community settings, as this evidence base is crucial for further funding and dissemination. While the evidence thus far is promising, it needs significant development.

    Outside of these methodological concerns, there is also notable progress in terms of expanding the scope of the digital health interventions that exist for this population. Most of the interventions identified in this review were directed toward improving the health of young men who identify as gay, bisexual, and queer and largely within the scope of improving sexual health–related concerns alone. Given the high rates of mental and physical health difficulties in LGBTIQ+ people, resources should be directed toward the development of digital health interventions targeting these issues, commensurate with the attention being given to STI and other sexual health–related concerns. The health concerns faced by LBTIQ+ women, trans and gender-diverse people, and people with intersex variations demand attention as well. Trans and gender-diverse people in particular face increased and wide-ranging health difficulties and barriers to health care access, even when compared with other members of the LGBTIQ+ community [25,27,105], and their marked underrepresentation in the interventions included in this review reveals a significant missed opportunity to begin to address these inequalities. While minority subpopulations are typically numerically small, and some argue that developing tailored interventions for such groups may be unnecessary [106], there is also evidence to suggest that minority groups appreciate tailoring of interventions. For example, in a recent study examining the attitudes of transgender and gender-diverse young people toward mental health gaming interventions, participants noted that TGD representation and inclusion of meaningful, specific tailored content was favorable [31].

    Limitations

    This review is the first to provide a wide overview of digital health interventions for LGBTIQ+ young people, enabling gaps such as these to be highlighted. However, the review is limited in several ways. Owing to the restrictions placed on the degree of human guidance permissible for inclusion in the review, telehealth interventions are notably absent from those discussed here. The review also did not cover interventions for which their development has been documented but has not yet been evaluated in some respects, and the review also did not include studies documenting the effectiveness of nontargeted interventions for LGBTIQ+ young people. Furthermore, the requirement for included studies to be published in the English language may have resulted in a biased sample, potentially excluding reports on interventions published in other languages. Finally, it is possible that limiting the search to studies published after January 1, 2000, may have excluded relevant studies; however, given that the earliest published study identified in the review was not until 10 years later, this is unlikely to have been the case. The review therefore largely covers the scope of interventions that accumulate an evidence base but should not be taken to cover any and all digital health interventions that may benefit the health of LGBTIQ+ youth.

    Conclusions

    Although not sought out specifically in the process of conducting this review, many protocols have been identified for continued research into the development of digital health interventions [107-112]. The development of digital health interventions for LGBTIQ+ young people is a burgeoning field of study, and we expect the evidence base to advance quickly. Going forward, this advancement should ideally occur across the breadth of health difficulties and inequalities that the entire LGBTIQ+ community faces and with appropriate methodological rigor. Given the number of interventions already targeting risk reduction or management of STIs, rather than developing more, future studies should ideally seek to refine and adapt those that exist for public use, and explore implementation barriers and facilitators to enhance translation. Given the lower scope and evidence base for interventions targeting mental and physical health difficulties, future studies should focus predominantly on expanding the available interventions and evidence base in these domains, particularly in terms of addressing difficulties such as alcohol use and suicide for which no digital health interventions were detected in this review at all. Digital health interventions for LGBTIQ+ young people show the potential to improve health disparities in this population, and the expansion of research along these lines is crucial to realize this potential.

    Acknowledgments

    This work was funded by the Giorgetta Charity Fund. DG and HM are supported by Research Training Program Stipends funded by the Australian Government. Associate Professor AL is funded by a National Health and Medical Research Council Career Development Fellowship (#1148793). Dr YP is supported by a fellowship from the Giorgetta Charity Fund.

    Authors' Contributions

    DG developed and executed the search strategy, co-screened the papers, co-assessed their quality, extracted and interpreted the data, and contributed to writing and editing the manuscript. HM co-conducted the screening process, assessed the quality of the papers, and contributed to writing and editing the manuscript. Associate Professor AL contributed to the interpretation of the data and to writing and editing the manuscript; Dr YP conceived and coordinated the study, advised on the development of the search and screening processes, and contributed to writing and editing the manuscript. All authors have read and approved the final manuscript.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Search strategy for PsycINFO (Ovid).

    DOCX File , 14 KB

    Multimedia Appendix 2

    Summary of digital sexually transmitted infection risk reduction and management interventions for LGBTIQ+ young people.

    DOCX File , 75 KB

    Multimedia Appendix 3

    Summary of interventions.

    DOCX File , 66 KB

    References

    1. Zeeman L, Sherriff N, Browne K, McGlynn N, Mirandola M, Gios L, Health4LGBTI Network. A review of lesbian, gay, bisexual, trans and intersex (LGBTI) health and healthcare inequalities. Eur J Public Health 2019 Oct 1;29(5):974-980 [FREE Full text] [CrossRef] [Medline]
    2. Eskin M, Kaynak-Demir H, Demir S. Same-sex sexual orientation, childhood sexual abuse, and suicidal behavior in university students in Turkey. Arch Sex Behav 2005 Apr;34(2):185-195. [CrossRef] [Medline]
    3. Marshal MP, Dietz LJ, Friedman MS, Stall R, Smith HA, McGinley J, et al. Suicidality and depression disparities between sexual minority and heterosexual youth: a meta-analytic review. J Adolesc Health 2011 Aug;49(2):115-123 [FREE Full text] [CrossRef] [Medline]
    4. Jones T. The needs of students with intersex variations. Sex Educ 2016 Mar 11;16(6):602-618. [CrossRef]
    5. HIV and Youth. Centers for Disease Control and Prevention. 2020.   URL: https://www.cdc.gov/hiv/pdf/group/age/youth/cdc-hiv-youth.pdf [accessed 2020-11-16]
    6. Marshal MP, Friedman MS, Stall R, King KM, Miles J, Gold MA, et al. Sexual orientation and adolescent substance use: a meta-analysis and methodological review. Addiction 2008 Apr;103(4):546-556 [FREE Full text] [CrossRef] [Medline]
    7. Corliss HL, Rosario M, Wypij D, Wylie SA, Frazier AL, Austin SB. Sexual orientation and drug use in a longitudinal cohort study of US adolescents. Addict Behav 2010 May;35(5):517-521 [FREE Full text] [CrossRef] [Medline]
    8. Day JK, Fish JN, Perez-Brumer A, Hatzenbuehler ML, Russell ST. Transgender youth substance use disparities: results from a population-based sample. J Adolesc Health 2017 Dec;61(6):729-735 [FREE Full text] [CrossRef] [Medline]
    9. Bishop A, Overcash F, McGuire J, Reicks M. Diet and physical activity behaviors among adolescent transgender students: school survey results. J Adolesc Health 2020 Apr;66(4):484-490. [CrossRef] [Medline]
    10. Saewyc EM, Poon CS, Homma Y, Skay CL. Stigma management? The links between enacted stigma and teen pregnancy trends among gay, lesbian, and bisexual students in British Columbia. Can J Hum Sex 2008;17(3):123-139 [FREE Full text] [Medline]
    11. Quinn GP, Sanchez JA, Sutton SK, Vadaparampil ST, Nguyen GT, Green BL, et al. Cancer and lesbian, gay, bisexual, transgender/transsexual, and queer/questioning (LGBTQ) populations. CA Cancer J Clin 2015;65(5):384-400 [FREE Full text] [CrossRef] [Medline]
    12. Case P, Austin SB, Hunter DJ, Manson JE, Malspeis S, Willett WC, et al. Sexual orientation, health risk factors, and physical functioning in the Nurses' Health Study II. J Womens Health (Larchmt) 2004 Nov;13(9):1033-1047. [CrossRef] [Medline]
    13. Caceres BA, Brody AA, Halkitis PN, Dorsen C, Yu G, Chyun DA. Sexual orientation, health risk factors, and physical functioning in the nurses' health study II. LGBT Health 2018 Jul;5(5):284-294 [FREE Full text] [CrossRef] [Medline]
    14. Caceres BA, Brody AA, Halkitis PN, Dorsen C, Yu G, Chyun DA. Cardiovascular disease risk in sexual minority women (18-59 years old): findings from the national health and nutrition examination survey (2001-2012). Womens Health Issues 2018;28(4):333-341 [FREE Full text] [CrossRef] [Medline]
    15. Kang M, Bernard D, Booth M, Quine S, Alperstein G, Usherwood T, et al. Access to primary health care for Australian young people: service provider perspectives. Br J Gen Pract 2003 Dec;53(497):947-952 [FREE Full text] [Medline]
    16. Gulliver A, Griffiths KM, Christensen H. Perceived barriers and facilitators to mental health help-seeking in young people: a systematic review. BMC Psych 2010 Dec 30;10(1):-. [CrossRef]
    17. Ray N. Lesbian, Gay, Bisexual and Transgender Youth: an Epidemic of Homelessness. National Gay and Lesbian Task Force Policy Institute and the National Coalition for the Homeless. 2006.   URL: https:/​/myhealth.​alberta.ca/​myhealthrecords#:~:text=You%20need%20to%20sign%20up,License%20or%20Alberta%20Identification%20card [accessed 2020-11-16]
    18. Durso L, Gates G. Serving Our Youth: Findings from a National Survey of Service Providers Working with Lesbian, Gay, Bisexual, and Transgender Youth who are Homeless or At Risk of Becoming Homeless. The Williams Institute with True Colors Fund and The Palette Fund. 2012.   URL: https://williamsinstitute.law.ucla.edu/publications/serving-our-youth-lgbtq/ [accessed 2020-11-16]
    19. Morton MH, Dworsky A, Matjasko JL, Curry SR, Schlueter D, Chávez R, et al. Prevalence and correlates of youth homelessness in the United States. J Adolesc Health 2018 Jan;62(1):14-21 [FREE Full text] [CrossRef] [Medline]
    20. Sears B, Mallory C. The Williams Institute. 2011.   URL: https://williamsinstitute.law.ucla.edu/publications/employ-discrim-effect-lgbt-people/ [accessed 2020-11-16]
    21. Takács J. Social Exclusion of Young Lesbian, Gay, Bisexual and Transgender (LGBT) People in Europe. ILGA Europe. 2006.   URL: https://www.salto-youth.net/downloads/4-17-948/ReportSocialExclusionIGLYOilga.pdf [accessed 2020-11-16]
    22. Brown A, Rice SM, Rickwood DJ, Parker AG. Systematic review of barriers and facilitators to accessing and engaging with mental health care among at-risk young people. Asia Pac Psychiatry 2016 Mar;8(1):3-22. [CrossRef] [Medline]
    23. Sanchez NF, Sanchez JP, Danoff A. Health care utilization, barriers to care, and hormone usage among male-to-female transgender persons in New York City. Am J Public Health 2009 Apr;99(4):713-719. [CrossRef] [Medline]
    24. Mayer KH, Bradford JB, Makadon HJ, Stall R, Goldhammer H, Landers S. Sexual and gender minority health: what we know and what needs to be done. Am J Public Health 2008 Jun;98(6):989-995. [CrossRef] [Medline]
    25. Safer JD, Coleman E, Feldman J, Garofalo R, Hembree W, Radix A, et al. Barriers to healthcare for transgender individuals. Curr Opin Endocrinol Diabetes Obes 2016;23(2):168-171. [CrossRef]
    26. McIntyre J, Daley A, Rutherford K, Ross LE. Systems-level barriers in accessing supportive mental health services for sexual and gender minorities: insights from the provider's perspective. Can J Commun Ment Health 2011 Sep;30(2):173-186. [CrossRef]
    27. Macapagal K, Bhatia R, Greene GJ. Differences in healthcare access, use, and experiences within a community sample of racially diverse lesbian, gay, bisexual, transgender, and questioning emerging adults. LGBT Health 2016 Dec;3(6):434-442 [FREE Full text] [CrossRef] [Medline]
    28. Kates J, Ranji U, Beamesderfer A, Salganicoff A, Dawson L. Health and Access to Care and Coverage for Lesbian, Gay, Bisexual,and Transgender (LGBT) Individuals in the U.S. Kaiser Family Foundation. 2018.   URL: https:/​/www.​kff.org/​racial-equity-and-health-policy/​issue-brief/​health-and-access-to-care-and-coverage-for-lesbian-gay-bisexual-and-transgender-individuals-in-the-u-s/​ [accessed 2020-11-16]
    29. Ward B, Dahlhammer J, Galinsky A, Joestl S. Sexual Orientation and Health Among US Adults: National Health Interview Survey. National Health Statistics Reports. 2014.   URL: https://www.cdc.gov/nchs/nhis/sexual_orientation/statistics.htm [accessed 2020-11-16]
    30. Perry Y, Strauss P, Lin A. Online interventions for the mental health needs of trans and gender diverse young people. Lancet Psychiatry 2018 Feb;5(2):e6. [CrossRef] [Medline]
    31. Strauss P, Morgan H, Wright Toussaint D, Lin A, Winter S, Perry Y. Trans and gender diverse young people's attitudes towards game-based digital mental health interventions: A qualitative investigation. Internet Interv 2019 Dec;18:100280 [FREE Full text] [CrossRef] [Medline]
    32. Lucassen MF, Hatcher S, Stasiak K, Fleming T, Shepherd M, Merry SN. The views of lesbian, gay and bisexual youth regarding computerised self-help for depression: An exploratory study. Adv Mental Health 2014 Dec 17;12(1):22-33. [CrossRef]
    33. Bowen D, Jabson J, Kamen C. mHealth: an avenue for promoting health among sexual and gender minority populations? Mhealth 2016;2:36 [FREE Full text] [CrossRef] [Medline]
    34. Aguilera A. Digital technology and mental health interventions: opportunities and challenges. Arbor 2015 May 12;191(771):a210. [CrossRef]
    35. Berry N, Lobban F, Bucci S. A qualitative exploration of service user views about using digital health interventions for self-management in severe mental health problems. BMC Psychiatry 2019 Jan 21;19(1):35 [FREE Full text] [CrossRef] [Medline]
    36. Hollis C, Morriss R, Martin J, Amani S, Cotton R, Denis M, et al. Technological innovations in mental healthcare: harnessing the digital revolution. Br J Psychiatry 2015 Apr;206(4):263-265. [CrossRef] [Medline]
    37. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res 2009 Apr 24;11(2):e13 [FREE Full text] [CrossRef] [Medline]
    38. Eysenbach G. The law of attrition. J Med Internet Res 2005 Mar 31;7(1):e11 [FREE Full text] [CrossRef] [Medline]
    39. Lie SS, Karlsen B, Oord ER, Graue M, Oftedal B. Dropout from an ehealth intervention for adults with type 2 diabetes: a qualitative study. J Med Internet Res 2017 May 30;19(5):e187 [FREE Full text] [CrossRef] [Medline]
    40. Schubart JR, Stuckey HL, Ganesh A, Sciamanna CN. Chronic health conditions and internet behavioral interventions. Comput Info Nurs 2011;29(2):81-92. [CrossRef]
    41. Yeager CM, Benight CC. If we build it, will they come? Issues of engagement with digital health interventions for trauma recovery. Mhealth 2018;4:37 [FREE Full text] [CrossRef] [Medline]
    42. Kiluk BD, Sugarman DE, Nich C, Gibbons CJ, Martino S, Rounsaville BJ, et al. A methodological analysis of randomized clinical trials of computer-assisted therapies for psychiatric disorders: toward improved standards for an emerging field. Am J Psychiatry 2011 Aug;168(8):790-799 [FREE Full text] [CrossRef] [Medline]
    43. Payne HE, Lister C, West JH, Bernhardt JM. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. JMIR Mhealth Uhealth 2015 Feb 26;3(1):e20 [FREE Full text] [CrossRef] [Medline]
    44. Zhao J, Freeman B, Li M. Can mobile phone apps influence people's health behavior change? An evidence review. J Med Internet Res 2016 Oct 31;18(11):e287 [FREE Full text] [CrossRef] [Medline]
    45. Marcolino MS, Oliveira JA, D'Agostino M, Ribeiro AL, Alkmim MB, Novillo-Ortiz D. The impact of mhealth interventions: systematic review of systematic reviews. JMIR Mhealth Uhealth 2018 Jan 17;6(1):e23 [FREE Full text] [CrossRef] [Medline]
    46. Strauss P, Cook A, Winter S, Watson V, Wright-Toussaint D, Lin A. Trans-pathways: the Mental Health Experiences and Care Pathways of Trans Young People-summary of Results. Telethon Kids Institute. 2017.   URL: https://www.telethonkids.org.au/projects/trans-pathways/ [accessed 2020-11-16]
    47. Craig SL, McInroy L. You can form a part of yourself online: the influence of new media on identity development and coming out for LGBTQ youth. J Gay Lesbian Mental Health 2014 Jan 30;18(1):95-109. [CrossRef]
    48. McDermott E, Hughes E, Rawlings V. Queer Futures. Understanding Lesbian, Gay, Bisexual and Trans (LGBT) Adolescents' Suicide, Self-harm and Help-seeking Behaviour. Department of Health Policy Research Programme Project. 2016.   URL: http://www.queerfutures.co.uk/wp-content/uploads/2016/06/Queer-Futures-Final-Report.pdf [accessed 2020-11-16]
    49. Marchant A, Hawton K, Stewart A, Montgomery P, Singaravelu V, Lloyd K, et al. A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: The good, the bad and the unknown. PLoS ONE 2017 Aug 16;12(8):e0181722. [CrossRef]
    50. Mustanski B, Lyons T, Garcia SC. Internet use and sexual health of young men who have sex with men: a mixed-methods study. Arch Sex Behav 2011 Apr;40(2):289-300 [FREE Full text] [CrossRef] [Medline]
    51. Rozbroj T, Lyons A, Pitts M, Mitchell A, Christensen H. Assessing the applicability of e-therapies for depression, anxiety, and other mood disorders among lesbians and gay men: analysis of 24 web- and mobile phone-based self-help interventions. J Med Internet Res 2014 Jul 3;16(7):e166 [FREE Full text] [CrossRef] [Medline]
    52. Steinke J, Root-Bowman M, Estabrook S, Levine DS, Kantor LM. Meeting the needs of sexual and gender minority youth: formative research on potential digital health interventions. J Adolesc Health 2017 May;60(5):541-548 [FREE Full text] [CrossRef] [Medline]
    53. Lucassen M, Samra R, Iacovides I, Fleming T, Shepherd M, Stasiak K, et al. How LGBT+ young people use the internet in relation to their mental health and envisage the use of e-therapy: exploratory study. JMIR Serious Games 2018 Dec 21;6(4):e11249 [FREE Full text] [CrossRef] [Medline]
    54. Knight R, Karamouzian M, Salway T, Gilbert M, Shoveller J. Online interventions to address HIV and other sexually transmitted and blood-borne infections among young gay, bisexual and other men who have sex with men: a systematic review. J Int AIDS Soc 2017 Nov;20(3):- [FREE Full text] [CrossRef] [Medline]
    55. Hobaica S, Alman A, Jackowich S, Kwon P. Empirically based psychological interventions with sexual minority youth: a systematic review. Psychol Sex Orientat Gend Divers 2018 Sep;5(3):313-323. [CrossRef]
    56. van der Pol-Harney E, McAloon J. Psychosocial interventions for mental illness among LGBTQIA youth: a prisma-based systematic review. Adolescent Res Rev 2018 Aug 2;4(2):149-168. [CrossRef]
    57. Coulter RW, Egan JE, Kinsky S, Friedman MR, Eckstrand KL, Frankeberger J, et al. Mental health, drug, and violence interventions for sexual/gender minorities: a systematic review. Pediatrics 2019 Sep;144(3):- [FREE Full text] [CrossRef] [Medline]
    58. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009 Aug 18;151(4):264-9, W64 [FREE Full text] [CrossRef] [Medline]
    59. Howarth A, Quesada J, Silva J, Judycki S, Mills PR. The impact of digital health interventions on health-related outcomes in the workplace: a systematic review. Digit Health 2018;4:2055207618770861 [FREE Full text] [CrossRef] [Medline]
    60. Mahood Q, van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods 2014 Sep;5(3):221-234. [CrossRef] [Medline]
    61. Gilbey D, Mahfouda S, Ohan J, Lin A, Perry Y. Trajectories of mental health difficulties in young people who are attracted to the same gender: a systematic review. Adolescent Res Rev 2019 Aug 9;5(3):281-293. [CrossRef]
    62. Lee JG, Ylioja T, Lackey M. Identifying lesbian, gay, bisexual, and transgender search terminology: a systematic review of health systematic reviews. PLoS One 2016;11(5):e0156210 [FREE Full text] [CrossRef] [Medline]
    63. Hong Q, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, et al. Mixed Methods Appraisal Tool (MMAT), Version 2018. IC Canadian Intellectual Property Office, Industry Canada. 2018.   URL: http:/​/mixedmethodsappraisaltoolpublic.​pbworks.com/​w/​file/​fetch/​127916259/​MMAT_2018_criteria-manual_2018-08-01_ENG.​pdf [accessed 2020-11-16]
    64. Mustanski B, Greene GJ, Ryan D, Whitton SW. Feasibility, acceptability, and initial efficacy of an online sexual health promotion program for LGBT youth: the Queer Sex Ed intervention. J Sex Res 2015;52(2):220-230. [CrossRef] [Medline]
    65. Martin S. Developing and Evaluating Queervibe: an Online Intervention to Empower Trans and Non-binary Youth. Doctoral Dissertation, Anglia Ruskin University. 2019.   URL: https://arro.anglia.ac.uk/704647/ [accessed 2020-11-16]
    66. Mitchell JT, LeGrand S, Hightow-Weidman LB, McKellar MS, Kashuba AD, Cottrell M, et al. Smartphone-based contingency management intervention to improve pre-exposure prophylaxis adherence: pilot trial. JMIR Mhealth Uhealth 2018 Sep 10;6(9):e10456 [FREE Full text] [CrossRef] [Medline]
    67. Greene GJ, Madkins K, Andrews K, Dispenza J, Mustanski B. Implementation and evaluation of the keep it up! Online HIV prevention intervention in a community-based setting. AIDS Educ Prev 2016 Jun;28(3):231-245. [CrossRef] [Medline]
    68. Madkins K, Moskowitz DA, Moran K, Dellucci TV, Mustanski B. Measuring acceptability and engagement of the keep it up! Internet-based HIV prevention randomized controlled trial for young men who have sex with men. AIDS Educ Prev 2019 Aug;31(4):287-305 [FREE Full text] [CrossRef] [Medline]
    69. Mustanski B, Garofalo R, Monahan C, Gratzer B, Andrews R. Feasibility, acceptability, and preliminary efficacy of an online HIV prevention program for diverse young men who have sex with men: the keep it up! intervention. AIDS Behav 2013 Nov;17(9):2999-3012 [FREE Full text] [CrossRef] [Medline]
    70. Mustanski B, Parsons JT, Sullivan PS, Madkins K, Rosenberg E, Swann G. Biomedical and behavioral outcomes of keep it up!: an ehealth HIV prevention program RCT. Am J Prev Med 2018 Aug;55(2):151-158 [FREE Full text] [CrossRef] [Medline]
    71. Christensen JL, Miller LC, Appleby PR, Corsbie-Massay C, Godoy CG, Marsella SC, et al. Reducing shame in a game that predicts HIV risk reduction for young adult men who have sex with men: a randomized trial delivered nationally over the web. J Int AIDS Soc 2013 Nov 13;16:18716. [CrossRef]
    72. Hightow-Weidman LB, Fowler B, Kibe J, McCoy R, Pike E, Calabria M, et al. HealthMpowerment.org: development of a theory-based HIV/STI website for young black MSM. AIDS Educ Prev 2011 Feb;23(1):1-12 [FREE Full text] [CrossRef] [Medline]
    73. Hightow-Weidman LB, LeGrand S, Muessig KE, Simmons RA, Soni K, Choi SK, et al. A randomized trial of an online risk reduction intervention for young black MSM. AIDS Behav 2019 May;23(5):1166-1177 [FREE Full text] [CrossRef] [Medline]
    74. Hightow-Weidman LB, Pike E, Fowler B, Matthews DM, Kibe J, McCoy R, et al. HealthMpowerment.org: feasibility and acceptability of delivering an internet intervention to young Black men who have sex with men. AIDS Care 2012;24(7):910-920 [FREE Full text] [CrossRef] [Medline]
    75. Ybarra ML, DuBois LZ, Parsons JT, Prescott TL, Mustanski B. Online focus groups as an HIV prevention program for gay, bisexual, and queer adolescent males. AIDS Educ Prev 2014 Dec;26(6):554-564 [FREE Full text] [CrossRef] [Medline]
    76. Ybarra ML, Liu W, Prescott TL, Phillips G, Mustanski B. The effect of a text messaging based HIV prevention program on sexual minority male youths: a national evaluation of information, motivation and behavioral skills in a randomized controlled trial of guy2guy. AIDS Behav 2018 Oct;22(10):3335-3344 [FREE Full text] [CrossRef] [Medline]
    77. Ybarra ML, Prescott T, Mustanski B, Parsons J, Bull SS. Feasibility, acceptability, and process indicators for guy2guy, an mhealth HIV prevention program for sexual minority adolescent boys. J Adolesc Health 2019 Sep;65(3):417-422 [FREE Full text] [CrossRef] [Medline]
    78. Ybarra ML, Prescott TL, Philips GL, Bull SS, Parsons JT, Mustanski B. Iteratively developing an mhealth HIV prevention program for sexual minority adolescent men. AIDS Behav 2016 Jun;20(6):1157-1172 [FREE Full text] [CrossRef] [Medline]
    79. Ybarra ML, Prescott TL, Phillips GL, Bull SS, Parsons JT, Mustanski B. Pilot RCT results of an mhealth hiv prevention program for sexual minority male adolescents. Pediatrics 2017 Jul;140(1):- [FREE Full text] [CrossRef] [Medline]
    80. Solorio R, Norton-Shelpuk P, Forehand M, Montaño D, Stern J, Aguirre J, et al. Tu amigo pepe: evaluation of a multi-media marketing campaign that targets young Latino immigrant MSM with HIV testing messages. AIDS Behav 2016 Sep;20(9):1973-1988. [CrossRef] [Medline]
    81. Cho H, Powell D, Pichon A, Thai J, Bruce J, Kuhns LM, et al. A mobile health intervention for HIV prevention among racially and ethnically diverse young men: usability evaluation. JMIR Mhealth Uhealth 2018 Sep 7;6(9):e11450 [FREE Full text] [CrossRef] [Medline]
    82. Muessig K, Knudtson KA, Soni K, Larsen MA, Traum D, Dong W, et al. 'I didn't tell you sooner because I didn't know how to handle it myself.' Developing a virtual reality program to support HIV-status disclosure decisions. Digit Cult Educ 2018;10:22-48 [FREE Full text] [Medline]
    83. Bauermeister JA, Tingler RC, Demers M, Connochie D, Gillard G, Shaver J, et al. Acceptability and preliminary efficacy of an online HIV prevention intervention for single young men who have sex with men seeking partners online: the mydex project. AIDS Behav 2019 Nov;23(11):3064-3077. [CrossRef] [Medline]
    84. Goldstone S. LBO-1.6 efficacy of the quadrivalent HPV vaccine to prevent anal intraepithelial neoplasia among young men who have sex with men. Sex Transmit Infect 2011 Jul 10;87(Suppl 1):A352. [CrossRef]
    85. McRee A, Shoben A, Bauermeister JA, Katz ML, Paskett ED, Reiter PL. Outsmart HPV: acceptability and short-term effects of a web-based HPV vaccination intervention for young adult gay and bisexual men. Vaccine 2018 Dec 18;36(52):8158-8164 [FREE Full text] [CrossRef] [Medline]
    86. Bauermeister JA, Pingel ES, Jadwin-Cakmak L, Harper GW, Horvath K, Weiss G, et al. Acceptability and preliminary efficacy of a tailored online HIV/STI testing intervention for young men who have sex with men: the Get Connected! program. AIDS Behav 2015 Oct;19(10):1860-1874 [FREE Full text] [CrossRef] [Medline]
    87. Horvath KJ, Bauermeister JA. Ehealth literacy and intervention tailoring impacts the acceptability of a HIV/STI testing intervention and sexual decision making among young gay and bisexual men. AIDS Educ Prev 2017 Feb;29(1):14-23 [FREE Full text] [CrossRef] [Medline]
    88. Washington TA, Applewhite S, Glenn W. Using facebook as a platform to direct young black men who have sex with men to a video-based HIV testing intervention: a feasibility study. Urban Soc Work 2017;1(1):36-52 [FREE Full text] [CrossRef] [Medline]
    89. McCoy SI, Buzdugan R, Grimball R, Natoli L, Mejia CM, Klausner JD, et al. pilot study results of an intervention using gamification to increase HIV screening among young men who have sex with men in California. Mhealth 2018;4:40 [FREE Full text] [CrossRef] [Medline]
    90. LeGrand S, Muessig KE, McNulty T, Soni K, Knudtson K, Lemann A, et al. Epic allies: development of a gaming app to improve antiretroviral therapy adherence among young HIV-positive men who have sex with men. JMIR Serious Games 2016 May 13;4(1):e6 [FREE Full text] [CrossRef] [Medline]
    91. Hightow-Weidman L, Muessig K, Knudtson K, Srivatsa M, Lawrence E, LeGrand S, et al. A gamified smartphone app to support engagement in care and medication adherence for HIV-positive young men who have sex with men (allyquest): development and pilot study. JMIR Public Health Surveill 2018 Apr 30;4(2):e34 [FREE Full text] [CrossRef] [Medline]
    92. Schwinn TM, Thom B, Schinke SP, Hopkins J. Preventing drug use among sexual-minority youths: findings from a tailored, web-based intervention. J Adolesc Health 2015 May;56(5):571-573. [CrossRef] [Medline]
    93. Lucassen M, Merry S, Hatcher S. That's [not] so gay: computerised cognitive behavioural therapy for depressive symptoms in sexual minority youth. Neuropsychiatrie de l'Enfance et de l'Adolescence 2012 Jul;60(5):S141. [CrossRef]
    94. Lucassen MF, Hatcher S, Fleming TM, Stasiak K, Shepherd MJ, Merry SN. A qualitative study of sexual minority young people's experiences of computerised therapy for depression. Australas Psychiatry 2015 Jun;23(3):268-273. [CrossRef] [Medline]
    95. Lucassen MF, Merry SN, Hatcher S, Frampton CM. Rainbow sparx: a novel approach to addressing depression in sexual minority youth. Cogn Behav Pract 2015 May;22(2):203-216. [CrossRef]
    96. Fleming JB, Hill YN, Burns MN. Usability of a culturally informed mhealth intervention for symptoms of anxiety and depression: feedback from young sexual minority men. JMIR Hum Factors 2017 Aug 25;4(3):e22 [FREE Full text] [CrossRef] [Medline]
    97. Pachankis JE, Goldfried MR. Expressive writing for gay-related stress: psychosocial benefits and mechanisms underlying improvement. J Consult Clin Psychol 2010 Feb;78(1):98-110. [CrossRef] [Medline]
    98. Vogel EA, Belohlavek A, Prochaska JJ, Ramo DE. Development and acceptability testing of a Facebook smoking cessation intervention for sexual and gender minority young adults. Internet Interv 2019 Mar;15:87-92 [FREE Full text] [CrossRef] [Medline]
    99. Vogel EA, Thrul J, Humfleet GL, Delucchi KL, Ramo DE. Smoking cessation intervention trial outcomes for sexual and gender minority young adults. Health Psychol 2019 Jan;38(1):12-20 [FREE Full text] [CrossRef] [Medline]
    100. Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Lucassen MF. The effectiveness of SPARX, a computerised self help intervention for adolescents seeking help for depression: randomised controlled non-inferiority trial. Br Med J 2012 Apr 18;344:e2598 [FREE Full text] [CrossRef] [Medline]
    101. Reiter PL, Katz ML, Bauermeister JA, Shoben AB, Paskett ED, McRee A. Increasing human papillomavirus vaccination among young gay and bisexual men: a randomized pilot trial of the outsmart HPV intervention. LGBT Health 2018 Jul;5(5):325-329 [FREE Full text] [CrossRef] [Medline]
    102. Newman MG, Szkodny LE, Llera SJ, Przeworski A. A review of technology-assisted self-help and minimal contact therapies for anxiety and depression: is human contact necessary for therapeutic efficacy? Clin Psychol Rev 2011 Feb;31(1):89-103. [CrossRef] [Medline]
    103. Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med 2013 Aug;45(2):228-236 [FREE Full text] [CrossRef] [Medline]
    104. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012 Nov 14;14(6):e152 [FREE Full text] [CrossRef] [Medline]
    105. Su D, Irwin JA, Fisher C, Ramos A, Kelley M, Mendoza DA, et al. Mental health disparities within the LGBT population: a comparison between transgender and nontransgender individuals. Transgend Health 2016;1(1):12-20 [FREE Full text] [CrossRef] [Medline]
    106. Pachankis JE. The scientific pursuit of sexual and gender minority mental health treatments: toward evidence-based affirmative practice. Am Psychol 2018 Dec;73(9):1207-1219 [FREE Full text] [CrossRef] [Medline]
    107. Koblin B, Hirshfield S, Chiasson MA, Wilton L, Usher D, Nandi V, et al. Intervention to match young black men and transwomen who have sex with men or transwomen to HIV testing options (all about me): protocol for a randomized controlled trial. JMIR Res Protoc 2017 Dec 19;6(12):e254 [FREE Full text] [CrossRef] [Medline]
    108. LeGrand S, Muessig KE, Platt A, Soni K, Egger JR, Nwoko N, et al. Epic allies, a gamified mobile phone app to improve engagement in care, antiretroviral uptake, and adherence among young men who have sex with men and young transgender women who have sex with men: protocol for a randomized controlled trial. JMIR Res Protoc 2018 Apr 5;7(4):e94 [FREE Full text] [CrossRef] [Medline]
    109. Bauermeister JA, Golinkoff JM, Horvath KJ, Hightow-Weidman LB, Sullivan PS, Stephenson R. A multilevel tailored web app-based intervention for linking young men who have sex with men to quality care (get connected): protocol for a randomized controlled trial. JMIR Res Protoc 2018 Aug 2;7(8):e10444 [FREE Full text] [CrossRef] [Medline]
    110. Coulter RW, Sang JM, Louth-Marquez W, Henderson ER, Espelage D, Hunter SC, et al. Pilot testing the feasibility of a game intervention aimed at improving help seeking and coping among sexual and gender minority youth: protocol for a randomized controlled trial. JMIR Res Protoc 2019 Feb 15;8(2):e12164 [FREE Full text] [CrossRef] [Medline]
    111. Kuhns LM, Garofalo R, Hidalgo M, Hirshfield S, Pearson C, Bruce J, et al. A randomized controlled efficacy trial of an mHealth HIV prevention intervention for sexual minority young men: MyPEEPS mobile study protocol. BMC Public Health 2020 Jan 15;20(1):65 [FREE Full text] [CrossRef] [Medline]
    112. Reiter PL, Gower AL, Kiss DE, Malone MA, Katz ML, Bauermeister JA, et al. A web-based human papillomavirus vaccination intervention for young gay, bisexual, and other men who have sex with men: protocol for a randomized controlled trial. JMIR Res Protoc 2020 Feb 24;9(2):e16294 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    HPV: human papillomavirus
    LGBTIQ+: lesbian, gay, bisexual, transgender, intersex or queer
    MMAT: mixed methods appraisal tool
    MSM: men who have sex with men
    PrEP: pre-exposure prophylaxis
    PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
    PROSPERO: Prospective Register of Systematic Reviews
    STI: sexually transmitted infection


    Edited by G Eysenbach; submitted 12.05.20; peer-reviewed by M Lucassen, M Embrett; comments to author 16.06.20; revised version received 30.07.20; accepted 26.10.20; published 03.12.20

    ©Dylan Gilbey, Helen Morgan, Ashleigh Lin, Yael Perry. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 03.12.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.