This study was a multicenter, parallel-arm, pragmatic randomized controlled trial. Participants seeking services at specialized mental health and addiction programs at the participating sites were randomized 2:1 to receive immediate access to the BWW for 3 months (immediate treatment group, ITG) or delayed access after a 3-month control period (delayed treatment group, DTG). The trial protocol has been previously published and is available open access [16]. The trial was sponsored by Ontario Telemedicine Network and Canada Health Infoway, both government-funded organizations. Sponsors specified the recruitment target and study settings, but had no involvement in the study design, procedures, data collection, or analysis.

This study received ethical approval from the research ethics boards at all participating sites. All participants gave informed consent before taking part in the study.

In response to automated follow-up surveys going to email junk boxes, personalized emails were sent with the survey link embedded. The frequency of contact to remind participants to complete follow-up surveys was increased, and surveys were completed over the phone when possible. To meet sponsor recruitment targets, recruitment was extended early in the study to individuals attending clinics rather than the initial approach targeting those on waitlists or being discharged. Given a large amount of referral between programs, early on in recruitment, it became apparent that participants were commonly on waitlists for specialized clinics while receiving active treatment in other recruitment settings, so this change had little impact on the overall composition of the study sample.

Participants were recruited from outpatient mental health programs affiliated with 3 hospitals in Ontario: (1) a public psychiatric hospital in a medium-sized city with satellite treatment sites in smaller urban centers; (2) a large community hospital located in another medium-sized city with satellite addictions programs; and (3) a large ambulatory academic hospital located centrally in a large metropolitan area. The aim of the study was to determine the utility of the BWW as a broad reaching multicomponent Web-based intervention that crosses all mental health and addiction-related needs. The BWW contains guided educational and course content applicable to a wide range of mental health needs, and the peer support component offers various opportunities to engage in discussion relevant to one’s personal needs. Thus, we recruited from a range of outpatient programs which included: adult mood and anxiety psychiatry programs, a substance use program, an emergency department, an urgent care clinic, youth mood and anxiety programs, mood and anxiety psychotherapy programs, trauma therapy programs, and a borderline personality disorder program. Program coordinators, clinicians, and administrators at each site reviewed referral waitlists and clinic rosters to identify potential participants and refer them to the study coordinator, who reached out by telephone or met the individual at the clinic. Eligible participants were aged 16 years or older, had access to the internet and an email address, were able to read English, and willing and able to access and use a Web-based mental health intervention. There were no exclusion criteria but referring clinicians were asked to consider if person would be able to participate appropriately in Web-based peer interactions. There were no imposed restrictions on the use of concomitant care, including accessing other Web-based interventions that may be like the one under investigation.

All participants provided informed consent and baseline variables either in person or by phone with a study team member. Baseline assessment included sociodemographic data and all outcome measures. Participants were also asked about baseline belief in the treatment credibility and outcome expectancy. Participants were asked to rate their agreement with the author-generated question, “Self-help tools including web-based services and books are helpful for people with mental health problems.” Item 4 from the Credibility and Expectancy Questionnaire [17] was adapted for this study, “By the end of the [BWW access period], how much improvement in your symptoms do you think will occur?” Participants were asked to rate their response from 0% to 100% with options available in 10% increments. This single item has been shown to correlate strongly with mental health treatment outcomes such as psychotherapy [18]. Participants were given the option to complete additional symptom, function, and service utilization measures over the phone or by computerized survey. Secure computerized assessments were sent electronically to each participant’s email address. All data were entered into a REDCap database [19]. Trial recruitment took place between July 2016 and January 2017.

The team at each study site monitored adverse events. A data safety and monitoring board consisting of 3 experts external to the research team reviewed the trial data at 3 and 6 months after recruitment started, specifically examining change in item 9 (suicidal ideation) scores on the Patient Health Questionnaire-9 item (PHQ-9), as well as all serious adverse events reported by the participating sites throughout the trial.

The BWW is a multicomponent, self-directed, and moderated Web-based intervention founded in 2007 and hosted in the United Kingdom. Access to the BWW was free for study participants for 3 months (ITG during the 3-month trial period and DTG after the initial 3-month trial period). The developers advocate that the BWW can improve mental health symptoms through increasing social engagement, normalizing experiences, educating, and equipping with skills to manage difficulties. All participants maintain anonymity on the site through a unique nonidentifiable user alias and use is participant dependent [15]. The BWW is monitored 24/7 by Wall Guides who are trained mental health professionals employed by the BWW and based in the United Kingdom and New Zealand. These individuals constantly review user activity and posts to ensure the content is appropriate and sensitive to all users. They will engage with users through instant communication and in the case of identified risk, the Wall Guides can identify the location of the user and direct them to use local crisis services. Contact information for local crisis services was provided on the unique landing page created for the Ontario users. The BWW components include (1) educational material, (2) guided support courses based on principles of cognitive behavioral therapy and behavior change, and (3) text communication posts as either 1:1 with another member or a Wall Guide or open to discussion groups composed of any members of the peer community. The educational materials and guided support courses cover a wide range of mental health related topics including grief, depression, anxiety, smoking cessation, substance use, trauma, among others. Users can also post and comment on bricks—creative self-expressions whereby the user designs a brick which they can place in the digital wall. See Figures 1-4 for screenshots of the BWW components. The BWW reaches out to users through their registration email if there has been a prolonged period of inactivity, encouraging users to log on. In this study, all participants received an email alias that linked to their personal email and a unique prescription for a BWW account. A follow-up call was made by a study team member 3 days after sending the prescription and again after 2 weeks, if the account had not been activated. Technical support was available from the research team, the BWW, and the Ontario Telemedicine Network.

Follow-up data were collected between October 2016 and April 2017. All follow-up data were collected by self-report via electronic surveys through REDCap or collected by phone or in person by a study team member and subsequently input into the REDCap database.

Participants received an automated survey link by email 1 week before 3 months postrandomization with a follow-up personalized email containing the survey link. In these emails, DTG participants were reminded that they would receive access to the BWW once the survey was completed, although all participants were ultimately given access regardless of survey completion. At 3 months and 3 months plus 1 week, reminder phone calls were made. Surveys were closed 2 weeks after the 3-month time point. Study team members collecting the outcome assessments were blinded to group allocation. Both baseline and 3-month postrandomization surveys were completed via the Web-based survey in 95% of cases, with the remainder by phone or in person.

The primary outcome was mental health recovery at 3 months assessed with the Recovery Assessment Scale-revised (RAS-r). This outcome assesses an individual’s orientation toward recovery and self-management across 5 domains: (1) personal confidence and hope, (2) willingness to ask for help, (3) goal and success orientation, (4) reliance on others, and (5) not dominated by symptoms [20]. The study intervention claims to promote self-management specifically, and the RAS-r was chosen as an outcome relevant to participants across all diagnoses. The use of the RAS-r also reflects the current recovery era for mental health policy and services with the focus of treatment shifting to consumers finding satisfying and fulfilling lives, rather than being symptom free [20]. The RAS-r is the most widely used and validated recovery assessment tool and has been studied in several patient populations and shown to correlate with symptoms and function [20]. It is a 24-item scale, with all items scored on a 5-point scale from strongly disagree to strongly agree, with total scores ranging from 24 to 120.

Secondary outcomes were the PHQ-9 to assess symptoms of depression, Generalized Anxiety Disorder Questionnaire-7 item (GAD-7) to assess symptoms of anxiety, EuroQOL 5-dimension quality of life questionnaire (EQ-5D-5L) to assess quality of life, and the Community Integration Questionnaire (CIQ) [21]. The PHQ-9 scale contains 9 items rated on a 4-point Likert scale from never to almost every day, with a higher score representing more symptoms [22]. The GAD-7 scale has 7 items rated on a 4-point Likert scale from never to almost every day, with a higher score representing a higher likelihood of an anxiety disorder. The EQ-5D-5L questionnaire from the EuroQOL group [23] comprises 5 dimensions: mobility, self-care, usual activities, pain or discomfort, and anxiety/depression, which are rated on 5 levels and can be interpreted as an index score by comparing with available normative values for adult populations. The EQ-5D-5L is paired with a visual analog scale (VAS) rated from 0 to 100 to assess perceived overall health at the time of survey completion. The CIQ consists of 15 items and is intended as a brief, reliable measure of a person’s level of integration into the home and community. The overall score can range from 0 to 29 with a higher score indicating better integration [24].

Actual BWW utilization data were obtained for all ITG participants including activation of account and number of logins.

The target sample size for our trial was set by the sponsors at 1000 participants. We aimed to recruit this number of participants, with an expectation that loss to follow-up would be approximately 30% based on data from other similar trials [25]. We calculated the minimal detectable difference between the 2 treatment groups for a linear regression analysis controlling for baseline score assuming a 0·8 correlation between baseline and 3-month follow-up RAS-r measurements. Assuming 30% loss to follow-up, for a sample size of 700 after attrition, allocated in a 2:1 ratio, using an alpha of .05 and power of 0.9, the minimal detectable difference was 1.35 on the RAS-r.

Participants were randomized 2:1 to the ITG and DTG groups. A 2:1 randomization ratio was used to offer the intervention immediately to a higher number of participants and increase recruitment. Randomization sequences, using block sizes of 3 or 6 were computer generated by an organization external to the research team, with stratification by site and recruitment setting. Group allocation sequence was concealed but once allocated, participants were not blinded. Participants allocated to the DTG group received a telephone or email notification that they would gain access to the website after a 3-month delay. Most follow-up data were collected by Web-based survey, but in the case where follow-up data were collected by an assessor, the assessor was not the same as the person who did the initial data collection, as this person also disclosed randomization to the participant and would not be blinded at follow-up. Data analysts remained blinded throughout the study.

The primary outcome, RAS-r, was analyzed at 3 months for all participants regardless of whether they activated their BWW account, but individuals with missing data at 3 months were excluded. We planned to repeat the analysis using a marginal structural model to account for attrition; however, owing to higher than expected attrition that was nonrandom, this analysis was not completed. RAS-r scores at 3 months were modelled with a linear regression controlling for baseline RAS-r score and treatment group. A second analysis controlled for prespecified covariates including baseline PHQ-9, baseline GAD-7, age, gender, recruitment setting, and age of first onset of mental health problems. The same analysis was repeated for all secondary outcomes at 3 months. Analyses were completed after data collection was complete. BWW utilization data were analyzed descriptively to illustrate uptake of the intervention in the ITG participants only.

Among ITG participants who completed the 3-month outcome measures, we examined whether there was an association between actual utilization of the application and the primary and secondary outcomes. Utilization of the application was defined by number of BWW logins, with users categorized into 4 groups based on distribution quartiles: 0 to 1 login, 2 to 3 logins, 4 to 9 logins, and 10 or more logins. Owing to degree of missingness, primary and secondary outcomes were separately modeled with repeated measures ANOVA using time as a repeated measure. We tested for an interaction between BWW use and time, after adjusting for same covariates as in the main analysis. Significant main effects were explored with post-hoc Bonferroni tests adjusted for multiple comparisons.

The primary analysis showed a statistically significant difference between the ITG and DTG groups for the RAS-r at 3 months, with the ITG participants having an RAS-r score on average 5.34 points higher than the DTG participants at 3 months (Table 2). In the adjusted model, the effect remained significant but slightly lower in magnitude (Table 2).

PHQ-9 and GAD-7 scores were significantly lower in the ITG group compared with the DTG group in the main and adjusted analyses (Table 2). No statistically significant differences were found for EQ-5D-5L index score, EQ-VAS, or CIQ (Table 2).

The data safety monitoring board reviewed data at 3 and 6 months after the start of recruitment and did not find any substantial increase in suicidal ideation between groups to warrant investigation or early termination. Over the course of the study, one death was reported to the study team and deemed unrelated to the intervention.

Immediate access to the BWW resulted in small, significant improvements in mental health recovery, as well as depressive and anxiety symptoms at 3 months compared with those randomized to delayed access. These statically significant findings are limited by high, differential drop out between treatment groups and overall, and were below minimal clinically important differences for these outcome measures (eg, a difference of 1.8 on the PHQ-9 where a clinically important change is 5 points) [22]. Engagement with the intervention varied highly and interestingly, we observed the commonly found Pareto principle for population effects—also known as the 80/20 rule [26]. That is, 20% of users accounted for approximately 80% of the activity. We found some evidence of a user effect whereby those who completed follow-up and engaged with the application were more likely to experience improvements in symptoms of depression and anxiety. This was not linear, however, with the user group having 2 to 3 logins experiencing the most consistent improvements relative to the lowest user group. To our knowledge, this is the first randomized evaluation of the BWW, and one of a few large multicenter pragmatic trials of any multicomponent internet-based mental health intervention. The study population represents more treatment-refractory and severely symptomatic individuals than are usually included in studies of Web-based mental health applications.

Although the BWW is unique in comparison with other studied applications, the observed low engagement is similar [25]. A comparatively large pragmatic trial that examined 2 computerized modular CBT programs with added telephone support reported that fewer than 20% of participants had completed all modules at follow-up [25]. Modular-style courses are only one component of the BWW, the other components being peer support, artistic self-expression, and more general psychoeducation. Compared with a modular program, it is more difficult to define an adequate dose for multicomponent interventions such as the BWW [13]. We examined different self-directed doses of the BWW based on logins and did not find a linear relationship between higher use and better outcomes, although it appeared that some engagement may have been better than none. Some users may benefit from a few targeted uses to get direction or motivation. A systematic review examining the relationship between e-therapy adherence and outcomes reported that studies targeting depression did not find a significant impact of total number of logins on outcomes in contrast to studies examining physical health outcomes like weight management or smoking [9]. Studies using modular interventions, however, did report that module completion led to better outcomes. Conversely, one of the main features of the BWW used more often by the higher engagers is its moderated virtual community of peers [15], an intervention for which the evidence has yielded some very mixed results [27,28]. Some authors have described a potential harm through aggravated symptoms and what Takahashi et al termed the downward depressive spiral which was linked to higher depressive tendencies at baseline [29]. This may have tempered the improvements in the higher engager groups or conversely may have led to early disengagement from the application as a result of negative reactions to the peer community.

This study experienced loss to follow-up that was proportionally higher in the intervention group. High drop out of up to 50% in trials of Web-based mental health interventions has been reported [10] and retention in digitized trials has specifically been discussed as a challenge [30]. In our study, the issues encountered with our automated survey were addressed quickly but may have impacted survey completion. More specifically, other studies of Web-based mental health programs have also reported disproportionate dropout in the intervention group [10]. This trend likely represents some study disengagement for intervention-specific reasons such as dislike of the intervention format or lack of perceived utility. In this study, some very early technology issues with BWW activation likely affected early user engagement and the Web-based platform did not include a mobile version of the site or application which could have deterred use for some. To partially address this limitation, we conducted the exploratory analysis on study completers.

We chose the pan-diagnostic RAS-r as the primary outcome, given that our study sample was recruited from a range of mental health settings and participants had a range of mental health needs. The promise of internet-based mental health interventions specifically to support recovery has been discussed [31]; however, this outcome lacks established clinically important cutoffs. Secondary outcomes may have been affected by the lack of diagnostic specificity relative to other studies that have focused on specific diagnoses such as depression and anxiety [6]. In this study, we evaluated the BWW as a solution for any mental health or addiction-related need; although we adjusted for recruitment program, it is possible that the BWW may work better for certain diagnostic or need subgroups, which represents an area for further evaluation.

From this trial, we cannot definitively conclude the effectiveness of this or similar solutions at a population level for individuals accessing specialized mental health care. However, internet-based interventions built on evidence-based principles of mental health care are likely to be beneficial where users are motivated to engage and where the format of the application is a good fit. Determining this subset and/or how to effectively motivate more people to engage with the interventions is a critical next step. Pragmatic studies that build in process evaluations and subset analyses that examine moderators of use and related outcomes are required. This study was undertaken as part of a large provincial implementation program with multiple research, clinical, and policy stakeholders involved throughout, a type of integrated, real-world approach to evaluation that is essential to establish effective health systems solutions [32].