This review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [39]. We identified studies for inclusion through searching the electronic databases PubMed, Embase, PsycINFO, CINAHL, and Web of Science. The first set of search terms were related to SMI: “psychosis” OR “psychotic” OR “psychoses” OR “schizophr*” OR “schizoaffective” OR “bipolar disorder” OR “mood disorder” OR “personality disorder” OR “severe mental illness” OR “serious mental illness” OR “severe mental health” OR “serious mental health” OR “SMI”. The second set of search terms were related to online and mobile technologies: “computer” OR “technolog*”, OR “digital” OR “internet” OR “online” OR “website” OR “web-based” OR “mobile” OR “phone” OR “smartphone” OR “text message” OR “SMS” OR “mHealth” OR “eHealth”. These sets of search terms were linked with the Boolean operator AND.

To increase the likelihood of obtaining all of the relevant studies in the area, we took the following steps: 1) we included unpublished materials such as conference abstracts in the search; where abstracts were relevant, we contacted the lead authors for full results, 2) we used the “cited-by” function in Google Scholar to identify any eligible papers that had cited the included studies, 3) we screened references lists of included studies to gather any papers that the search terms had not identified, and 4) we contacted key authors in the area who were identified as potentially having further unpublished results.

We produced a flowchart of each stage of the database search, with the search yielding a total of 8017 papers (Figure 1).

We conducted full database searches in April 2015, with the inclusion and exclusion criteria identified prior to the collection period. Due to the relatively new nature of the field of the Internet and mobile phones and mental health, we considered only studies from 2005–2015. We included studies if they recruited participants with a diagnosis of psychosis, bipolar disorder, or personality disorder. Studies where these diagnoses were self-reported (not confirmed by a clinician or through an initial assessment) were still included due to Web-based recruitment strategies often used in the field. We defined interventions as a method used online or via a mobile phone with the aim of modifying a participant’s behavior or psychological well-being. Therefore, we aimed to provide a broad range of interventions, from more simplistic interventions, such as medication and appointment reminders, to more complex interventions, such as cognitive behavioral therapy and interactive psychoeducation. We included studies if they involved the delivery of an intervention online or via a mobile phone or if they investigated participants’ interest in and willingness to receive interventions delivered in these formats. Supported interventions (where participants were supported by a trained facilitator or where online and mobile phone-delivered interventions were implemented in conjunction with face-to-face therapy) were also included because some of the studies offered optional support and often continued face-to-face treatment as usual. Finally, we included studies if they measured acceptability in the form of module completion rates, intervention use, or participant views. Therefore, we included qualitative, quantitative, or mixed-methods study designs.

We excluded studies if the intervention was solely focused on diagnostic assessments, mood assessments, or symptom monitoring where feedback was not provided to participants. In addition, we excluded studies involving telepsychiatry, for example, video and telephone calls. We included studies that involved a combination of caregivers and service users but we report characteristics and outcomes only for the participants from the service user sample. Studies that involved mixed samples were excluded if ≤50% of the participants sampled had a diagnosis of psychosis, bipolar disorder, or personality disorder. For those where >50% of participants had 1 of these diagnoses, we contacted the authors of these papers for separate results for participants with a diagnosis of psychosis, bipolar disorder, and personality disorders.

A key aim of this systematic review was to determine factors (eg, demographic and clinical characteristics) that could influence the acceptability of online and mobile phone-delivered interventions for SMI. Therefore, we screened included studies for information about the analysis and findings relating to potential predicators of acceptability. We considered relationships between factors and acceptability as significant if they were below the .05 level of significance. However, due to the early nature of this research, we also report associations on the 10% level.

We excluded studies on the title level if there was no mention of online or mobile phone-delivered interventions or mental health. The first author and an independent researcher screened the 230 paper abstracts for eligibility, with a moderate level of agreement obtained (κ=.66). This moderate level of agreement was primarily due to the first author being overly inclusive while screening abstracts due to the secondary outcome nature of acceptability. The research team resolved any disagreements until a consensus was reached about study inclusion. The first author screened full texts before the research team discussed and agreed on final papers for inclusion. We identified 3 additional studies through contacting authors and reference screening. We contacted 4 authors for clarification because their studies included participants with a diagnosis of major depressive disorder, and 3 of these authors provided additional unpublished material, which we have included in this review. The final number of included studies after screening on title, abstract, and full paper level was 49.

Multimedia Appendix 3 presents results for the hypothetical acceptability of online and mobile phone-delivered interventions for SMI. The results reported focused on 1) the proportion of participants who agreed that they would be interested in receiving online and mobile phone-delivered interventions, 2) the impact of demographic characteristics on hypothetical acceptability, 3) whether participant levels of interest in online and mobile phone-delivered interventions differed between email and text message delivery, and 4) geographical differences in hypothetical acceptability.

The hypothetical acceptability of obtaining support via social media websites was relatively high (75.5%), and two-thirds of participants (64%) were willing to have clinicians contact them via social media during symptom emergence [41] (A Rizvi, MA, written communication, May 2015). However, levels of interest in receiving information and support varied between social media platforms (85% YouTube; 58% Facebook; 39% Twitter) [44]. Interest in mobile phone check-ins to inform health care providers about symptoms was relatively low (14.4% to 41%) [34,40,43] (D Ben-Zeev, PhD, written communication, May 2015), while interest in receiving text messages from health care providers was moderate to high (45% to 76%) [45,46]. Additionally, the hypothetical acceptability of online and mobile phone-delivered appointment and medication reminders varied extensively between studies (26% to 92.5%) [34,40,42-45] (personal communication with D Ben-Zeev, May 2015).

Participant interest in mobile phone-delivered information about treatment, services, and psychoeducation was relatively low (31% to 48.1%) [34,40,43] (personal communication with D Ben-Zeev, May 2015). However, a 2015 study specifically recruiting young people with first episode psychosis reported that 90% of participants liked the idea of receiving information about physical and mental health online or via mobile phones [44].

Multimedia Appendix 4 presents results for the actual acceptability of online and mobile phone-delivered interventions for SMI. The results reported focused on 1) intervention satisfaction ratings reported by participants, 2) intervention use and module completion rates, 3) the impact of demographic and clinical characteristics on actual acceptability, and 4) a comparison of the actual acceptability between supported and unsupported online and mobile phone-delivered interventions.

Some studies measured the actual acceptability of online and mobile phone-delivered interventions for SMI through satisfaction ratings. The proportion of participants who were satisfied with the ease of use, perceived helpfulness, and perceived usefulness of the interventions, and were willing to recommend the intervention to others, was moderate to high (41% to 90.6%), although the majority of studies tended to report values of around 75% [40,47,52,60,66-68,77-79,85]. Other studies measured actual acceptability through satisfaction ratings on Likert scales. Ratings for overall satisfaction and perceived helpfulness and usefulness were on the upper ends of the Likert scales, while moderate to high ratings were reported for ease of use [32,50,62-65,80,84,86] (personal communication with L Warner, June 2015).

Intervention use was also included as a measure of actual acceptability. In the HORYZONS study, 60% of participants used the intervention for the full 4 weeks and 70% used it for at least 3 weeks, as measured by the number of intervention log-ins [47]. Additionally, the FOCUS system [40] was used by participants on 86.5% of the study days [60], while 71% of participants in the WEGWEIS project used every feature on the website [85]. In the SOAR study, every person who participated was reported to have engaged with the website (used the education material on at least 4 visits and contributed to the forum on at least 13 visits) [78,79]. However, rates of engagement with the MyRecoveryPlan website were relatively low after 3 weeks (program only: 9%; program plus coaching: 38%) [81]. Participant response rates to assessments and prompts ranged from 65% to 93.33% [32,62,63,72,73,86].

Actual acceptability was also measured through module completion rates and session attendance. Between 45% and 81% of participants completed at least half the modules of their assigned intervention [47,57,59,66,71,76], while 81% of participants in the Mental.Net project attended at least 3 of the 5 sessions [53]. Additionally, the average program completion rate of participants who remained in the LWB trial was 60% [83].

Some projects included qualitative studies to investigate participant views about the acceptability of online and mobile phone-delivered interventions for SMI and ideas for future developments to increase acceptability [49,51,53-56,58,61,62,69,70,74,75,82,84,85] (Multimedia Appendix 5). To aid conceptual understanding, we collected common participant quotes relating to acceptability by screening the papers and creating a set of key themes that emerged. The key common themes identified were 1) concerns about the safety, privacy, and security of online and mobile phone-delivered interventions, 2) the importance of an engaging and appealing intervention delivery format, 3) participants’ desire for the inclusion of remote peer support, 4) individual differences in computer and mobile phone literacy and technical issues as potential barriers to acceptability, and 5) the potential impact of psychological state on motivation to engage.

This review had several notable strengths. First, the range of databases we searched and the list of search terms we created were comprehensive, which ensured that we obtained eligible studies in the field. Second, the review included studies with quantitative, qualitative, and mixed-methods designs, thus enabling a broad and in-depth analysis of the current work in the field. Third, studies were coextracted on the abstract level by a researcher independent of the research team to ensure eligibility criteria were accurate. In addition, our extraction of the data was systematic and we contacted authors if we required any further information.

Findings from the studies in this review should be considered in the context of some limitations. Many of the studies reviewed measured actual acceptability through module completion rates and intervention use. The sole use of these measures of acceptability is problematic due to the potential influence of other factors, for example, the number of modules available to complete, the intervention duration, financial incentives for high completion rates, technical issues with the interventions delivered, participants’ engaging in other activities while logged in, and time pressures preventing engagement. Therefore, module completion rates and intervention use are unlikely to be robust direct measures of acceptability. Across studies, module completion rates and intervention use were also reported in different ways, for example, some reported the average number of modules completed, while others reported the average duration participants spent accessing an intervention. It is impossible to determine what value constitutes an “acceptable” intervention without a universal measurement applied. Future research is needed to develop more accurate ways to assess acceptability.

The majority of the studies we reviewed that measured hypothetical acceptability asked participants about their general interest and willingness to use interventions delivered online and via mobile phones. However, participants were not asked about their interest in receiving these interventions online or via mobile phones in comparison with, or in addition to, face-to-face delivery. Had these questions been phrased differently, overall hypothetical acceptability may have been very different. The review findings also highlight the very limited amount of relevant information regarding predictors of acceptability being reported in studies. We could not draw conclusions about the influence of demographic and clinical characteristics on both hypothetical and actual acceptability.

Issues relating to the heterogeneity of approaches for participant recruitment are an important consideration for the recruitment strategies in the field. Specifically, it could be argued that online recruitment methods may bias the sample toward favorable attitudes toward online and mobile phone-delivered interventions, thus increasing levels of acceptability. However, potential bias toward the acceptability of interventions delivered via these modalities may also be prevalent in more traditional routes of recruitment (ie, through service providers and clinicians). While different recruitment strategies may attract different samples within the population, broadening intervention choice and examining acceptability remain important considerations within the samples identified, regardless of the recruitment method that is used.

There were also some limitations to the method of analysis we used. First, the review excluded papers published in a language other than English. Second, some studies included both participants with SMI and participants with other mental health problems such as depression. To combat this issue, we excluded studies if ≤50% of the total study sample had SMI, so we may have missed relevant findings for those who did experience SMI.

Although this review highlights the relatively high acceptability of online and mobile phone-delivered interventions for SMI, it also demonstrates the complex nature of acceptability and the need for continued focus in this area. A recent systematic review concluded that, rather than concentrating on acceptability, researchers should instead investigate whether online and mobile phone-delivered interventions are effective [37]. While efficacy is undoubtedly important, we argue that acceptability remains equally important because, ultimately, if an intervention is not acceptable to service users and relevant stakeholders, people are unlikely to engage with the approach, thereby directly affecting efficacy. Based on the findings in this review, we recommend that research groups measure hypothetical and actual acceptability of online and mobile phone-delivered interventions, with an increased focus on the factors that could influence acceptability.

There were different reporting styles evident in studies detailing intervention usage, module completion rates, and session attendance. For example, some studies reported the overall proportion of participants who used the intervention over the whole study period, while others reported proportions over specific weeks during the study period. Therefore, it was not possible to investigate how the acceptability of online and mobile phone-delivered interventions changed over time. Future research should explore and report how intervention usage, module completion rates, and session attendance change throughout intervention delivery to determine whether acceptability of interventions delivered via these modalities changes across time points.

Module completion rates and intervention use may be indirectly related to actual acceptability; however, participant satisfaction ratings and qualitative views provide rich data about acceptability. These rich data can be used to develop and refine online and mobile phone-delivered interventions in order to improve the overall acceptability for people with SMI. The need to use qualitative analysis to inform the design and development of interventions delivered online and via mobile phones has been recognized in protocols for future studies in the field [90-92]. It is recommended that, if researchers choose to measure acceptability through module completion rates and intervention use, satisfaction ratings and qualitative interviews be conducted to obtain the rich information needed to identify intervention acceptability. In addition, the use of qualitative interviews examining what participants feel they have actually gained from an intervention may help to highlight the specific areas of their lives that they feel may have been improved by participating in the intervention.

This review also found that hypothetical acceptability tended to be low or varied, while actual acceptability tended to be high, indicating that people with SMI may be initially reluctant to engage with online and mobile phone-delivered interventions. However, hypothetical acceptability results are largely limited and not necessarily directly comparable with the interventions being delivered. Issues with comparing hypothetical versus actual acceptability are primarily due to discrepancies in the types of interventions being explored. Specifically, studies investigating hypothetical acceptability investigated interest in using the Internet or mobile phones to facilitate health care provider contact and medication and appointment reminders, whereas studies investigating actual acceptability tended to implement more complex interventions such as cognitive behavioral therapy and psychoeducation. It is also likely that the sample of people who were asked about actual acceptability were already hypothetically open to the idea of receiving online and mobile phone-delivered interventions. We therefore suggest that researchers measure acceptability both before (hypothetical acceptability) and after (actual acceptability) an intervention is delivered. These measurements will aid the comparison of hypothetical versus actual acceptability without the limitations associated with separate samples.

Crucially, this review showed that the majority of the studies reviewed recruited participants who were already in contact with mental health services. One of the potential advantages of the Internet and mobile phones is that they could improve access to evidence-based interventions for people who are not receiving support but who need or want it. Therefore, it is important to investigate whether people who could potentially benefit the most from online and mobile phone-delivered interventions actually find these delivery formats acceptable.

Clearly, interventions delivered online and via mobile phones do have their place in the provision of self-care for people with SMI. However, research has yet to identify predictors of acceptability and whether people who are not engaged with services also find online and mobile phone-delivered approaches acceptable. The measurement of both hypothetical and actual acceptability in future studies would enable the investigation of the impact of prior expectations on acceptability and potential changes in acceptability after access. In order to obtain rich data about acceptability, we recommend the measurement of satisfaction ratings and participant views and the continued involvement of service users throughout all aspects of intervention development and delivery.