Relapse is common for many people diagnosed with schizophrenia [1]. Relapses are linked to increased disability from loss of important relationships and reduced education and employment opportunities [2]. One estimate suggests that psychotic relapse costs £10,950 (at 6 months) compared with £2532 for no relapse, with 75% of the difference in these costs coming from inpatient treatment [3]. In the United States, excess costs from relapse range from US $6033-$32,753 [4]. Commonly, relapses are preceded by so-called early warning signs (EWS) that reflect a combination of symptoms such as anxiety, depression, suspiciousness, and uniquely personal experiences. EWS-based prevention strategies assume that identifying relapse early enough enables preventative action and averts full relapse [5]. Guidelines for psychosis in both Scotland, the United Kingdom, [2] and Australia [6] recommend early signs-based strategies as crucial for relapse prevention in routine psychosis care.

Research into reliable and valid signs of relapse is essential for early intervention aimed at minimizing the harms associated with relapse [7]. A review [8] to determine the validity of early signs as predictors of relapse in people with nonaffective psychosis found that the sensitivity (correct relapse prediction by staff) ranged from 10% to 80% (median 61%), and specificity (nonrelapses correctly identified) ranged from 38% to 100% (median 81%). Therefore, existing systems used to identify EWS have an uncertain prognostic utility and may result in an unnecessary intervention that engenders fear of relapse in service users and carers [9]. Delayed help-seeking narrows the window for timely intervention [10] and can result in the use of coercive treatment measures that confirm negative expectations [11] and make disclosure of EWS more threatening for service users. Therefore, new interventions that address problems associated with help-seeking and disclosing EWS appear warranted [12].

One emerging application of technology in mental health care is remote self-monitoring [13]. Remote self-monitoring may improve upon traditional face-to-face monitoring by allowing more regular sampling of symptoms and, potentially, earlier detection of relapse signs. EMPOWER (Early signs Monitoring to Prevent relapse in psychosis and prOmote Well-being, Engagement, and Recovery; ISRCTN99559262) aims to develop and evaluate a mobile app for use with adults who experience psychosis. The app enables routine self-monitoring for a variety of different experiences, including psychosis (eg, hearing voices and suspicious thoughts), anxiety, mood, self-esteem, and interpersonal support. Furthermore, each time people complete an app questionnaire, they receive an EMPOWER message, which (depending on user input) provides links to further relevant information, practical advice, or helpful quotes. The EMPOWER algorithm aims to tailor these messages to individual changes in user well-being to promote a greater sense of control over mental health and to support self-management. EMPOWER participants will use the app for an initial 28-day baseline period to identify their typical variation in personal well-being. Significant changes from baseline will be then be triaged by a clinician. Peer support workers will be involved in setting up and personalizing the daily questionnaire, alongside regular fortnightly follow-up meetings where they will support service users in using the app.

Digital interventions can help address clinical priorities in psychosis, such as increasing access to psychological interventions for symptoms such as paranoia [14]. However, many effective digital interventions have failed to generalize from clinical trials into clinical practice [15,16]. Owing to concerns about generalization beyond trial contexts, the UK Department of Health [17] encourages systematic implementation research to increase an understanding of how interventions are adopted or rejected. The effectiveness of interventions (including their success in reaching the target population) can be influenced by how an intervention interacts with the context in which it was implemented [18,19]. When appraising the results of a clinical trial, it can be challenging to know whether the intervention will generalize into real-world contexts of clinical practice. Process evaluations assess the implementation of interventions and help predict generalizability in different contexts. The Medical Research Council (MRC) framework for process evaluation [18] recommends clear descriptions of assumptions about how the intervention is expected to be implemented within a specific context. In addition, consulting multiple stakeholder groups is recommended because this can reveal across-group variance in understandings of what the intervention is and differences in assumptions about why and for whom the intervention is expected to work. Collecting data at different time points is also recommended to characterize changes in implementation factors such as participants’ attitudes toward an intervention.

Typically, the majority of implementation research on engagement with interventions has been retrospective [20]. The MRC framework for process evaluations recommends that implementation research should proactively include key stakeholders because those expected to engage with an intervention are likely to have relevant experiential knowledge, which is useful in understanding the implementation process during a trial [18]. Qualitative research carried out during a trial (eg, asking service users about their experiences) can aid in understanding why an intervention might work and how context affects implementation [21]. However, befor e interacting with an intervention, stakeholders may have pre-existing expectations regarding implementation that will shape how they interact with a planned intervention (hypothetical acceptability). Hypothetical acceptability is measured by key stakeholders’ willingness to engage with a proposed intervention and in previous trials of digital interventions for severe mental health problems actual acceptability (assessed postintervention) is typically higher than hypothetical acceptability [22].

Theory in implementation science implies some predictive capacity [23]. Typically, implementation theory aims to create conceptual tools that enable researchers to describe, identify, and explain crucial elements of the implementation process and its outcomes [24]. Developing implementation theories in advance of empirical testing provides a framework for developing predictions about how interventions will interact with the context in which they are tested. Furthermore, completing this work allows researchers to make informed predictions about what implementation barriers that might be reasonably expected [25]. One such implementation theory, normalization process theory (NPT) [25] focuses on the work that groups and individuals do when interacting with an intervention and how they make sense of it, many intervention studies have successfully utilized NPT as a framework to guide research to more fully understand the implementation process [26]. Despite the recommended involvement of patients and members of the public within implementation research [27] and widespread assumptions that consultation work can help researchers anticipate stakeholders’ needs, capacities, and priorities [28], the MRC guidelines on process evaluation [18] report substantial empirical uncertainty regarding the value of Patient and Public Involvement (PPI) work. However, stakeholders are likely to offer insights beyond the acceptability of digital interventions (eg, predicting intervention implementation barriers during testing) and arguably have a right to be involved in research, which impacts them. Adding the insight of carers, service users, and mental health staff should lead to a clearer understanding of barriers and facilitators to implementation.

To the best of our knowledge, only one other study has [29] explored staff, carers, and service users’ perspectives of acceptability and implementation of a digital intervention for psychosis before engagement. Inclusion of these stakeholders enabled potentially diverse perspectives to be integrated into system design requirements for a mobile intervention for people who were considered to have treatment-resistant schizophrenia. Although this study is in a different population, the inclusion of multiple perspectives is a strength that could be applied to the prospective investigation of stakeholder engagement with digital interventions. In addition, there is little longitudinal research comparing stakeholder predictions pre-intervention with what happens when people interact with a digital intervention. Developing implementation theories for the EMPOWER intervention based on the expectations of staff, service users, and carers within a longitudinal process evaluation will allow for the assessment of the accuracy and the changing nature of these predictions over time, potentially highlighting the value of contextual knowledge that comes from consulting with stakeholders. We anticipate that developing an a priori implementation theory derived from stakeholder consultation will enhance implementation of the intervention in the context of a clinical trial and provide meaningful data to enable later generalization into clinical practice, a clear priority for services [15,17].

This study aimed to summarize the implementation expectations expressed within focus groups by mental health staff, carers, and service users in consultation work before a clinical trial to be able to compare these with the actual experiences of implementation observed within a feasibility study.

This study forms part of the qualitative phase conducted before a cluster randomized controlled trial for the EMPOWER intervention (ISRCTN: 99559262). The methods are reported in line with the consolidated criteria for reporting qualitative research reporting recommendations for qualitative work [30]; a full checklist can be seen in Multimedia Appendix 1. Before the start of the study, ethical approvals were provided by West of Scotland REC (16/WS/0042) and Melbourne Health (REC/15/MH/344). Managerial approval was given by National Health Service Greater Glasgow and Clyde (NHSGG&C; GN14CP229) and North Western Mental Health Services (Project Number: 2015.286). The protocol is available in the National Institute of Health Research website [31]

All participants came from 1 health board area in the United Kingdom and 1 in Australia, where the intervention will be tested in a multisite clinical trial. Staff who support people with psychosis within Community Mental Health Services (CMHS) were invited to take part through initial researcher contact with clinical team leaders. Service user participants were invited to take part in focus groups through mental health staff and organizations providing support or representation to people with mental health difficulties. Service user participants were eligible if they were in contact with CMHS, had experienced a relapse within the previous 2 years, had received a diagnosis of Diagnostic and Statistical Manual of Mental Disorders-5 psychosis-related condition, and were able to provide informed consent. People who identified as carers for someone with psychosis were recruited from both mental health services and support organizations.

Using focus groups rather than individual interviews enabled respondents to interact with and respond to the ideas and comments of other participants with whom they shared a role [32]. Focus groups were held in private rooms (of either CMHS or support organizations) and conducted by members of the research team using a topic guide. We did not collect demographic data beyond whether the participant was a carer, service user, or mental health staff. Following best practice guidelines [18], we used an explicit theoretical framework to guide our focus group schedule. An interview schedule informed by NPT [33] was developed to explore stakeholders’ expectations. A copy of the topic guide for each of the stakeholder focus groups is provided in Multimedia Appendices 2-4.

A total of 25 focus groups were held across Melbourne and Glasgow from July 20, 2016, to September 9, 2017. Participants were 88 mental health staff, either working in the NHS in the United Kingdom (n=54, 9 focus groups) or NorthWestern Mental Health (public run) services in Australia (n=34, 4 focus groups). Focus group length ranged from 57 min to 2 hours and 9 min. A total of 21 service users were recruited from the United Kingdom (n=5, 3 focus groups) and Australia (n=16, 4 focus groups) and 40 carers from the United Kingdom (n=20, 2 focus groups) and Australia (n=20, 3 focus groups). Carers and service users received UK £20 or Aus $40 for participation. Staff received no cash reimbursement and participated during their usual working day. All participants gave written consent before taking part. All focus group facilitators (AG, SB, AC, ML, JG, JH, JF, and SA—a mix of genders) identified themselves as researchers to conduct the research and were transparent if they also held a clinical role. All participants received a presentation about the EMPOWER intervention. The focus groups were audio recorded and then transcribed verbatim. NVivo software (QSR International) was utilized to perform analysis.

SA is a Doctor of Philosophy student investigating the implementation of digital interventions for psychosis. Facilitating focus groups was a task shared by all coauthors. Data analysis was primarily completed by SA, who has previously utilized qualitative methods. Supervision and code checking for all analysis (including discussions about saturation) were provided by AG and HM, both of whom are clinical psychologists. AG is chief investigator for the EMPOWER study and was responsible for the overall design and conduct of the research.

The analysis comprised 2 stages. Thematic analysis is a qualitative method used to construct, analyze, and report on patterns within text data [34]. This is commonly utilized within qualitative aspects of process evaluations to identify key barriers and facilitators for implementation of a diverse range of digital interventions [35-37]. In stage 1, we performed an inductive thematic analysis [34] for each unique stakeholder group in turn. This was justified because in a pilot clinical trials such as EMPOWER, study evaluators are encouraged to use exploratory research to identify facilitators and barriers to interventions so that strategies can be put in place in time for an evaluation of effectiveness [18].

For stage 2, the MRC process evaluation framework [18] was identified as a suitable deductive coding framework [38] for placing the themes in an implementation theory context more relevant to the needs for a feasibility study where it may be too early to decide if normalization should be the goal. This was the rationale for moving away from our original plan (EMPOWER ISRCTN: 99559262) to use the NPT [39] framework for qualitative work. The MRC framework goes beyond barriers and facilitators to implementation and provides a taxonomy of implementation constructs. Expected barriers and facilitators (on their own) can be seen as singular aspects of a predicted overall process. However, during the analysis of focus group conversations, it was clear that barriers and facilitators were expected to interact together into an overall expected implementation process for EMPOWER. Therefore, we selected implementation constructs from the MRC process evaluation to structure our barriers and facilitators findings in a theoretically driven hypothetical implementation theory (presented as a deductive framework) for the EMPOWER trial:

Coding and analyzing the data within this framework resulted in the implementation issues highlighted during inductive analysis being more meaningfully constructed as implementation barriers and facilitators. Through our initial thematic analysis, we developed 16 themes (Table 1). The implementation diagram (Figure 1) represents implementation expectations for the EMPOWER intervention across staff, service users, and carers with facilitators (green) and barriers (red) within the implementation framework. The framework analysis was completed across all stakeholder groups simultaneously.

Both stages of qualitative analysis were completed by SA and triangulated through discussion with AG and HM. Resource limitations meant that strategies such as member checking (where participants check over themes proposed by the researcher as an interpretation validity check [40]) were not utilized. However, it has been highlighted that employing this technique may increase the validity of findings in qualitative research exploring user views of digital interventions in psychosis [41] and better ensure participant views have not been misrepresented.

Barriers impacting upon reach (who consents to participate in the trial) are expected early in the implementation process. For example, carers expect that service users with previous negative experiences such as coercive treatment will be less likely to consent to the study (a reach barrier). Mental health staff expected that service users who have low general levels of functioning and/or high levels of paranoia would not consent or struggle to use the app if they do. However, mental health staff expected that younger service users would be more likely to be willing to participate in a digital intervention study because their generation are digital natives. Implementation issues that impact upon fidelity (such as service users inputting inaccurate data) are expected slightly later in the implementation process. However, even if the implementation is successful (with service users completing daily self-monitoring) and the data are perceived to be an accurate reflection of their mental state—problems in using EMPOWER data for relapse prevention are still expected. For example, staff predicted that EMPOWER data will not be applicable within the context of early intervention services because EWS of relapse will still be unclear for people experiencing first episode psychosis (an implementation barrier). Barriers such as a lack of staff time were constructed as a predicted barrier across all levels (ie, expected to impact upon everything from service user consents into a feasibility study all the way up to generalizing into clinical practice if clinical outcomes in a definitive randomized controlled trial were favorable). The results of this deductive analysis can be seen in Figure 1.

Table 1 presents the themes as barriers and facilitators constructed during the inductive analysis.

Figure 1 presents the hypothetical implementation framework that scaffolds both barriers and facilitators themes that came up during focus group discussions. The diagram shows that throughout all stages, barriers and facilitators reach, fidelity, and implementation were constructed as coming from context.

This study is among the first to assess and record implementation expectations across mental health staff, carers, and service users for a newly developed digital intervention for psychosis in advance of testing in a clinical trial, building on previous multistakeholder work [29]. We have identified and theoretically framed the most common implementation expectations expressed by mental health staff, service users, and carers in advance of the EMPOWER clinical trial. Understanding the context behind empirical outcomes from novel digital mental health interventions is key in deciding if an intervention can be easily implemented within current practice [16] or will require significant resources and effort to do so [42]. Within a standard implementation science approach, context is defined as a shared environment, which can provide either barriers or facilitators for implementation [18]. However, within a complexity science–informed understanding, context is defined by an intervention interacting with multiple enacted environments of different social actors [43]. Although the MRC process evaluation framework provides a theoretical framework, creating the framework shown in Figure 1 means that it is more tailored to the clinical context of relapse management as reported by carers, mental health staff, and service users. Our findings provide a complexity science–informed account of how different stakeholders expect EMPOWER to interact within the multistakeholder actions that already occur during routine relapse prevention.

Key to the proposed framework (Figure 1) is a similarity between groups regarding expectations of what would constitute successful implementation. For successful implementation, it was agreed that EMPOWER must enable service user participants to self-monitor to a level of granularity that results in data allowing for visualization of potential personal indicators of relapse while also giving a comprehensive insight into overall service user mental health. Despite this implementation expectation appearing similar across groups, there were some role differences between staff, service users, and carers. The context of health care settings is constructed as being institutionalized [44] because behaviors by social actors are described in terms of the roles people are expected to act out. Our findings suggest that implementing the use of EMPOWER data in relapse prevention is only expected to be successful if the data are symbolically meaningful [15] to each stakeholder’s role. For example, in the case of staff, this means having data that enables them to understand better how a participant feels and can help them differentiate EWS of relapse from a false alarm. For carers, useful data were constructed as staff becoming more attuned and being able to differentiate relapse signals from false alarms. Although both staff and carers emphasized data access as being an implementation facilitator that could improve service responses, service users were more curious about the impact of having access to a record of their self-reported day-to-day well-being. Previous qualitative research conducted with service users exploring potential [29,41,45] and actual [46] acceptability of digital self-management interventions for psychosis has reported that having access to personal data may have positive impacts such as enhancing self-management. However, this previous study also highlights more negative impacts reported by service users such as creating concerns about data privacy [41] and paranoia [46] and that using digital interventions may eventually lead to a reduction in mental health services [41]. Therefore, the mixed findings from our study appear mainly in line with previous research.

Similar to previous work exploring hypothetical implementation expectations held by staff, service users, and carers for a digital intervention for an online portal for schizophrenia [29], we found key differences in implementation expectations across staff, service users, and carers. Service user implementation expectations for both barriers and facilitators most frequently focused on individual experience. For example, the importance of EMPOWER providing a good user experience was highlighted as a key implementation facilitator throughout all stages of the implementation process and will be very important for sustained intervention use. User experience has been described as a neglected area within digital intervention research [47] and psychosis more specifically [48]. A recent study examining a mobile health platform for clinical monitoring in psychosis indicates that implementation was low because of the app frequently crashing [49], perhaps highlighting the importance of exploring user experience in implementation research. Carers (similar to findings from previous qualitative work [50]) and staff generally reflected how they foresee EMPOWER influencing service user interactions with staff. Furthermore, staff foreseeing digital interventions having an impact on staff roles and responsibilities is similar to previous qualitative research work conducted with mental health staff [29,51]. Carers expected that previous negative experiences of mental health care could act as a barrier toward initial engagement with the app. For carers, this expectation appeared to be related to a fear that EMPOWER would come to emulate existing dynamics within relapse prevention that can block timely communication of EWS. These findings are in line with previous research demonstrating that different stakeholders can hold different perspectives on digital mental health interventions [29,52,53] and suggest value in seeking out all relevant stakeholder perspectives.

This consultation work was helpful to the EMPOWER study because it highlighted key concerns of key stakeholders. For example, staff reporting a concern that app-generated data would be decontextualized data that may not be useful for clinical decision making. Going forward into the feasibility study, the role of a clinician in triaging data from the intervention to place app data within a meaningful context was emphasized to staff during recruitment.

This study has several limitations. First, focus groups may result in some participants feeling reluctant to share their views fully. Second, the implementation barriers and facilitators highlighted in this paper were those that were most commonly discussed throughout the focus groups. However, the quantity of discussion of barriers and facilitators may not equal their importance or relevance [54]. Third, participants were given a presentation that covered the EMPOWER rationale and how the intervention works. Participants might have formed different expectations if they were presented with an actual prototype. A recent recommendation for undertaking complexity science–informed implementation research within health care services is to abandon attempts to simplify implementation research but rather explore implementation more inductively from multiple perspectives [55]. Therefore, there is a concern that adopting existing implementation taxonomy from the MRC process evaluation framework [18] within our analytic approach may have overly simplified construction of the hypothetical implementation framework. Moreover, following the NPT framework in designing research questions may have minimized the range of potential responses from participants. Finally, PPI can range from consultation to stakeholders having decision making over the aims and conduct of a study [56]. Therefore, these findings should be considered in light of them coming from consultation and not direct stakeholder involvement.

The field of digital self-monitoring interventions in psychosis is rapidly expanding [46,48], and there is a need to optimize interventions for implementation. One critical implementation-focused strategy is intervention co-design with stakeholders to develop digital psychosis interventions more suitable to the needs of end users [57,58]. After completion of the EMPOWER feasibility trial, we will utilize observations amassed during the trial to base comments on how stakeholder expectations identified from this analysis compare with actual trial implementation. This qualitative work done in advance of the EMPOWER trial provides insight into very early implementation expectations that form when people are first told about a digital intervention. These implementation expectations seem associated with the role that a person plays in managing a health problem (such as being a patient or a carer) as well as their previous experiences. Furthermore, these expectations extend across different levels of implementation [59], from early engagement to posttrial implementation—indicating that expectations are complex and wide ranging. Our results suggest that potential participants may quickly form implementation-related expectations about interventions and make predictions about how they (and others) will interact with the intervention. These findings indicate that potential participants do not arrive at interventions in a naïve state and may develop expectations and assumptions about new technology before they even use it for themselves.