The EFI comprised brief, tailored, written, and audio-visual content (approximately 5 minutes) presented to participants in the EFI condition (EFI + myCompass 2) on a series of click-through linear webpages. The participants in this condition viewed the EFI after they were randomized and before commencing the E–MH program for depression and anxiety (myCompass 2). The EFI was delivered on an internet-based platform that also housed the surveys and control group content. Figure 1 presents some examples of the EFI. The look and feel of the EFI was based on the design of the myCompass 2 program to create a seamless flow of the EFI intervention into myCompass 2.

We used principles of participatory design to create the EFI, as these can improve the perceived relevance and uptake of interventions for end users [35]. We developed the EFI through a focus group study [36] of community members who had personal lived experience of depression or anxiety, or both (n=24, four groups; male=3, female=21; see the study by Gulliver et al [36] for further details on the EFI and its development). As noted in our study [36], very few males participated in these groups. Community members in the groups suggested that the EFI content should target barriers to using these programs through the provision of personalized symptom level feedback to demonstrate program needs, information about the program content, data security, program efficacy, and finally content challenging potential social norms around using E–MH programs (eg, the belief that others do not use E–MH programs). In particular, we used written and video content that was informed by theory that examines how social norms influence the acceptability of E–MH programs (eg, “online programs and apps are being increasingly used by people in the community to look after their mental health in their own time”) [15,25,27].

The EFI comprised the following components:

The E–MH program used was an updated version of the myCompass program that we named for this study as myCompass 2. myCompass has previously been demonstrated to reduce symptoms of depression and anxiety in two community samples (n=89; n=720) [37,38]. The myCompass 2 program is a fully automated and interactive self-help program that is free to the end user and was designed for people with mild-to-moderate symptoms of stress, anxiety, and depression. It is delivered without therapist assistance and can be accessed privately, at any time, on a variety of platforms, including mobile phones, tablets, and computers.

The myCompass 2 program is similar to its predecessor and contains 14 modules derived primarily from cognitive behavioral therapy, problem-solving therapy, interpersonal psychotherapy, and positive psychology. The program was delivered over 7 weeks to enable sufficient time to complete the 14 modules. The user is expected to complete two modules per week. However, symptom reduction may occur with the completion of fewer modules [38,39]. Each module takes 30-45 minutes to complete; thus, the expected commitment of the program is approximately 60-90 minutes per week. Half (7/14, 50%) of the modules provided core transdiagnostic cognitive behavioral therapy, whereas the other half provided content targeting specific concerns related to mental health (eg, sleep). Additional interactive features are also included such as quizzes providing real-time self-monitoring of thoughts, feelings, and behaviors; self-monitoring reminders, feedback, facts, and mental health care tips or motivational statements provided via SMS text messaging or email. Similar to the original myCompass [38], myCompass 2 was designed to be tailored to the user’s needs. The program screens the user and then directs them to modules that are likely to be suitable for them. The clinical approaches of the original myCompass and myCompass 2 programs were identical. However, the manner in which the content is delivered, the design of the program, and the user experience were all upgraded for myCompass 2, focusing on personalization and ease of use. This process was influenced by input from clinicians, consumers, and information technology specialists, resulting in the following key differences: (1) faster and easier sign up for users, (2) a more comprehensive user dashboard with more options for personalization, (3) modules that are more clearly displayed and easier to navigate, and finally (4) more comprehensive symptom trackers that are easier to personalize for the user.

The attention control condition (HealthWatch) has previously been shown to have high credibility [40,41]. It was approximately matched to the myCompass 2 program for the time taken to complete and comprised 14 brief modules of written information offered over 7 weeks. The information was taken from public domain health and lifestyle information and was deliberately unrelated to mental health. Module topics included Keeping bones strong and healthy, Your microbes and you, and The power of your pancreas. At the end of the 6-month trial period, after the follow-up data were collected, the control group was sent information about how to access the myCompass 2 program if they wished.

Participants were recruited from the general community via a social network, Facebook, from January to March 2019. As outlined in the study protocol [42], the recruitment target of at least 231 participants per condition (n=693) to meet the power requirements for our primary hypotheses (see Outcomes and Data Analysis section for further details), was met within 2 months. All follow-up data were collected in November 2019. Facebook was used to ensure that a broad cross-section of the community could be reached and to maintain ecological validity, as internet-based interventions are often marketed on the web directly to consumers. We set up a Facebook page to describe the study and used paid Facebook advertisements using nature imagery (eg, trees, waterfalls) that asked, “Want to learn more about your mental health? Complete a survey and 7-week online program now” along with the ethics approval information listed earlier. We also ran a concurrent advertisement targeting male Facebook users only and using typical masculine-targeted imagery (eg, images of road journeys). This was to increase male participants and address the commonly higher ratio of female to male participants in internet-based mental health trial research [43]. To increase the representation of males in the study, we continued this advertisement targeting male participants after meeting the original target sample of 693 participants after the first month of recruitment. This advertising strategy was slower and took another month; however, at that time, we increased the percentage of males in the study from 12% to 21.9%. Participants who then clicked sign up were directed to the information and consent page, where they were provided the key details of the study and asked to provide their consent to participate on the web.

Participants were invited to read the information sheet and consent to participate before completing the screening measures. After consenting, participants were screened using a two-stage screening process. First, they read a list of eligibility criteria that they had to endorse to be eligible for the trial. These were as follows: (1) had not previously used the myCompass web-based program, (2) were not currently receiving psychological therapy, (3) had not made a suicide plan in the past month, (4) had not been diagnosed with psychosis or bipolar disorder, (5) were aged 18 years or older, and (6) were currently living in Australia. The second part of the screening process involved completing the Generalized Anxiety Disorder 7-item (GAD-7) [44] and Patient Health Questionnaire-9 (PHQ-9) [45] instruments. Consistent with the approach that internet-based programs are highly suitable for those with mild-to-moderate symptoms in the community [10], potential participants were eligible if they reported current symptoms of depression, anxiety, or both in the mild-to-moderate range (score 5-14) using the screening instruments (PHQ-9 and GAD-7) described as follows. If they scored too low (0-4) on both instruments or too high (15+) on either instrument, they were not eligible. Those who were screened as not eligible at any point in the two-stage screening process were excluded from the study and provided with relevant mental health resources. All participants endorsing the suicide screening item of the PHQ-9 were provided with a prompt that asked them to telephone Lifeline, the Suicide Call Back Service, or 000 in the case of emergency.

We delivered the trial using the digital infrastructure portal of the Black Dog Institute, Sydney, Australia. The portal allowed for computer-generated random allocation, automatic assessments, intervention materials, and reminders to be delivered seamlessly to the participants. The portal collected web use data automatically for all conditions, allowing us to assess participants’ uptake and usage (adherence) of their assigned program. After completing the two-stage screening process, participants provided an email address and selected a password they could use to log in to access their assigned treatment. Participants were then randomized to one of the three conditions, using computer-based randomization stratified by general psychological distress symptom severity (as measured by the Distress Questionnaire-5 [DQ5] [46], at pretest score 5-13 vs 14-25), age (18-45 vs ≥46 years), and gender (female or male; permuted block randomization, block size of 6 within each stratum) to ensure balance across conditions. Prefer not to answer and other were categorized as the group expected to be smaller (ie, male gender and age ≥46 years). This was completed using a computerized randomization algorithm embedded in the trial portal. Those who were assigned to the myCompass 2 conditions (conditions 1 and 2) were required to log in again when automatically redirected to the myCompass 2 program.

The intervention period ran for 7 weeks. By using their email and password, participants were able to access their assigned programs as much or as little as they preferred. Through the automated system, weekly reminder emails were sent to participants to encourage them to engage with the myCompass 2 (conditions 1 and 2) or the attention control websites (condition 3). Participants were also sent emails to complete the 7-week posttest and 6-month follow-up surveys. Participants were sent reminder emails to complete if they had not completed the questionnaire after 1 and 2 weeks. Individual participant access to surveys was closed 4 weeks after the initial email was sent to each participant.

We emailed participants with small incentives in the form of e-gift cards for the completion of each internet-based assessment across all conditions (posttest incentive: Aus $15 [US $11.3]; follow-up incentive: Aus $25 [US $18.8]), regardless of their level of engagement with the intervention. These incentives were considered a token of appreciation for the participants’ time and effort for each survey.

The trial was double-blinded. The participants were blinded to whether they received active or attention control interventions. They were informed that they would be randomized to receive one of three programs: (1) strategies for challenging unhelpful thoughts and behaviors (myCompass 2), (2) education about internet-based interventions plus program (EFI + myCompass 2), or (3) general health and lifestyle information (attention control). They were not provided with information about which of these interventions was expected to be the most effective. Assessments were also blinded, as they were self-report. The statistician performing the analyses was also blinded to the conditions.

We hypothesized that uptake (initiation of at least one module) would be higher in the EFI + myCompass 2 condition than in the myCompass 2 alone condition (H1). We also expected greater adherence to be observed (ie, higher number of modules completed) in the EFI + myCompass 2 condition relative to the myCompass 2 alone condition (H2).

We expected that efficacy (reduction in symptoms of depression and anxiety) would be higher in the two active myCompass 2 intervention conditions than the HealthWatch attention control condition at posttest and 6-month follow-up (H3). Finally, we also expected that efficacy would be higher in the EFI + myCompass 2 condition than in the myCompass 2 alone condition at posttest and follow-up, and this difference would be mediated by adherence to the program (H4).

We expected that uptake (H1), adherence (H2), and efficacy (H3) would be moderated by a range of sociodemographic and psychological characteristics, including gender, age, cultural and linguistic background, education, social support, symptoms of depression and anxiety, acceptability of psychosocial internet-based programs, attitudes toward professional psychological treatment, familiarity and use of technology, personality, stigma, and mental health literacy (H5). We also predicted that secondary indices of efficacy (reductions in suicidality, distress, and disability; increases in acceptability of internet-based psychosocial interventions and quality of life) would be greatest in the EFI + myCompass 2, followed by myCompass 2 alone, which would also outperform the attention control condition (H6).

The following demographic characteristics were assessed: gender, age category, language spoken at home, level of education (primary school, some secondary school or year 10 equivalent, year 12, certificate level I-IV, diploma or associate degree, bachelor degree, graduate diploma or graduate certificate, master’s degree, or doctoral degree), employment status (full-time, part-time or casual, unemployed, not working due to study or maternity leave, retirement, etc), and region or area of residence (metropolitan area, regional area, or rural or remote area). Other assessments including previous psychological treatment, mental health literacy, and stigma were also measured as per the protocol [42] but were not included here because we were not able to investigate H5 due to null findings on H1, H2, and H3.

The PHQ-9 [45] was used to assess the symptoms of depression. This scale consists of nine items that assess the frequency of Diagnostic and Statistical Manual of Mental Disorders (DSM)-IV symptoms of major depression during the past two weeks. Participants rated items on a 4-point scale ranging from 0 (not at all) to 3 (nearly every day). Scores from each item were summed to produce an overall severity score, which ranges from 0 to 27. Higher scores indicate a higher severity of depression symptoms. The PHQ-9 has previously shown good sensitivity and specificity for detecting major depression in clinical and general population samples and has also demonstrated sensitivity to change over time [48]. Internal reliability using Cronbach α was .45 (N=849) at pretest. However, Cronbach α is not recommended when participants’ scores fall within a restricted range [49], such as in this study where we recruited participants with symptom scores of 5-14 only for both PHQ-9 and GAD-7. In the posttest, when the range was broader (0-26), Cronbach α for the PHQ-9 was .85.

We used the GAD-7 to assess anxiety symptoms. The GAD-7 scale comprises seven items that correspond to the DSM-IV and DSM-V criteria for generalized anxiety disorder [44]. Participants rated items on the same 4-point scale as the PHQ-9. Scores for each item were summed and ranges from 0 to 21, with higher scores indicating greater symptom severity. Previous research has demonstrated that the GAD-7 has good psychometric properties in general population samples, has similar properties in detecting changes to the PHQ-9, and provides accuracy compared with clinical diagnosis [48,50]. This study sample Cronbach α was .75 (N=849) at pretest, but as above, this may not be a reliable estimate. In the posttest, Cronbach α was .88 at posttest.

We used the DQ5 [46] to measure general psychological distress. Given that DQ5 provides coverage of both depression and anxiety symptoms, for the purpose of this study, this measure was also used to stratify participants at randomization. We selected the case finding cut-off point of ≥14 (lower distress=5-13; higher distress=14-25) for stratification [46]. The DQ5 comprises five items that ask respondents to indicate the frequency of a range of distressing situations, thoughts, and feelings over the previous 30 days using a 5-point Likert-type scale (1=never, 2=rarely, 3=sometimes, 4=often, and 5=always). Scores are summed, and total scores range from 5 to 25, with higher scores indicating more severe levels of general psychological distress. Previous studies have demonstrated that DQ5 displays high internal consistency and external validity [46,51]. The current sample DQ5 Cronbach α was .72 at pretest in the restricted sample of individuals randomized into the study.

The acceptability of internet-based programs was assessed using items developed and compiled by Ebert et al [15], based on the Unified Theory of Acceptance and Use of Technology. This measure assesses acceptability, with each item rated on a 5-point scale ranging from totally disagree to totally agree. For example, one item is as follows:

Scores are summed (range 4-20); higher scores indicate higher acceptability of internet-based programs. This scale has acceptable internal consistency [15]. Out of the eight total scales created by Ebert et al [15], we selected the following three scales for this study: performance expectancy (4 items from Wilson and Lankton [52] and Schomerus et al [53], example item: “Using an internet-based training would reduce my mental health problems”), effort expectancy (3 items, from the studies by Wilson et al [52] and Schomerus et al [53], example item: “Using an internet-based depression intervention would be an easy task for me”), and finally concerns regarding data security (2 items, developed by Ebert et al [15], example item: “When participating in an online-training I would trust, that all information I disclose would be treated in strict confidence”). All items were rated on the same 5-point scale as previously described.

We used five suicide-specific items from the Psychiatric Symptom Frequency scale [54] to measure suicidal ideation. Items measure suicidal ideation and suicidal behavior, using items that cover suicidal thoughts, plans, and attempts. Respondents chose yes or no to indicate whether any of these aspects of suicidal ideation or behavior were present in the previous six months; higher scores indicated higher severity of suicidal ideation and actions. Psychiatric Symptom Frequency scale suicide items display high internal reliability and validity [54]. These items were assessed at the pretest and 6-month follow-up only.

Disability and the days out of role were measured by two items. The first assessed the number of days out of role: “In the last 30 days, how many days were you totally unable to work, study, or manage your day-to-day activities because of emotional problems (such as feeling depressed or anxious)?” The second assessed days of disability: “Aside from those days, in the last 30 days, how many days were you able to work, study, or manage your day-to-day activities but had to cut back on what you did or did not get as much done as usual because of emotional problems?”

We used the European Health Interview Survey Quality of Life 8-item index (EUROHIS-QOL) [55] to measure quality of life. This scale comprises eight items measuring the psychological, physical, social, and environmental components of quality of life. Two examples include, “How would you rate your quality of life?” and “How satisfied are you with your ability to perform your daily activities?” Respondents rated the items on a 5-point scale, with response categories ranging from very dissatisfied to very satisfied, very poor to very good, and not at all to completely. A total score ranging from 0 to 32 is produced by summing the scores from each item; higher scores indicate a higher perceived quality of life. Previous research has shown adequate internal consistency for EUROHIS-QOL in multiple samples [55,56].

We also asked participants at the end of the posttest survey to complete a measure of self-reported adherence by asking, “Did you complete all of the program?” For those that responded “No,” we asked, “What were some of the reasons you didn’t complete the program?”

The primary outcomes were uptake and adherence to the myCompass 2 program. Uptake (H1) was assessed as the number of individuals who accessed at least one therapeutic module of the program. We compared the rate of uptake in EFI + myCompass 2 to that in the myCompass 2 alone condition using a chi-square test. Adherence (H2) was assessed using a Mann-Whitney U test to compare the number of modules that started and completed myCompass 2 during the intervention period of 7 weeks. We selected modules completed as they captured the dosage of the therapeutic content received. We also examined qualitative data on self-reported reasons for nonadherence using thematic analysis.

Efficacy (H3 and H4) was assessed on the basis of reduced symptoms of depression (PHQ-9 [45]) and GAD-7 [44] at the posttest and 6-month follow-up. Comparisons of efficacy were made between participants in each of the active intervention conditions and those in the attention control condition. We analyzed the secondary efficacy outcomes using an intention-to-treat framework, using data collected from the three measurement occasions (pretest, posttest, and 6-month follow-up). We calculated the effects with IBM SPSS 26.0 for Windows (IBM Corporation) using mixed model repeated measures (MMRM) analyses of variance, conservatively estimated using unstructured covariance matrices. Mixed models techniques incorporate all available data, including data from participants who did not complete assessments at posttest or follow-up, under the assumption that data were missing at random [57].

The MMRM processes described above were also used to examine the effects of the interventions on secondary efficacy outcomes (H6). The exploratory logistic regression and negative binomial regression models, and growth mixture model analyses (planned for H5) [42] did not proceed due to null findings on H1, H2, and H3.

First, as noted above, the EFI may have been too brief or insufficiently tailored to the needs of users. Although our development approach involved considerable collaboration with people who had lived experience of depression or anxiety, it is possible that the intensity of the EFI was insufficient or that it did not meet the diverse needs of users in the trial. We also did not assess participants’ engagement with the EFI (eg, how much time they spent on it or if they read or watched the content, or just clicked through the pages). Attrition was the greatest in the EFI condition. It is possible that there was greater disengagement in the study in this condition, which may have occurred if the information from the EFI had a demotivating effect (ie, provided information that the psychological intervention was not of interest to the participant). When educating potential users about psychosocial interventions, there remains a risk of either overwhelming users with information or inadvertently reducing their motivation to engage.

Overall, the attrition from the study was considerable. Although the study remained well-powered to detect hypothesized effects on uptake and adherence, it was slightly underpowered to examine efficacy outcomes, as the final samples in the active conditions were less than the targets (n=111). Attrition may have also led to biases in the analyses, although rigorous MMRM models were used to account for all available data. Incentives were used to minimize attrition but clearly provided insufficient motivation for most participants. Attrition from fully internet-based trials remains a challenge, which indicates that some form of human contact in a research trial is likely to be necessary to maintain samples over extended periods. Moreover, there were some technical challenges in the delivery of the trial. Participants could not be automatically logged into the myCompass 2 program and were required to sign up for the intervention as a separate process, which may have led to reductions in uptake for both active conditions and raised the slight possibility of dual accounts that we may not have been able trace (ie, greater adherence than observed). The lack of efficacy of the myCompass 2 program in this study suggests that the intervention may be better suited to participants with different symptom profiles or that further work is needed to refine the intervention based on the low adherence rates. Nevertheless, the lack of evidence for efficacy did not limit our ability to compare the levels of uptake or adherence within the RCT design of the study.

The use of Facebook or certain imagery in advertisements may have attracted a certain type of user; however, we believe this is not a significant limitation as it reflects a similar process for real-world marketing of internet-based interventions and typical users of internet-based programs, and the promotion of the trial was identical across the three arms of the trial. Participants in the trial may have been aware of their allocation based on the content, despite interventions not being explicitly labeled as active or control. Finally, the composition of the sample was biased toward female participants. Although this imbalance reflects the usage of E–MH programs in the community, it may have limited the generalizability of the results for males.