Published on in Vol 25 (2023)

Preprints (earlier versions) of this paper are available at, first published .
Factors Influencing Community Participation in Internet Interventions Compared With Research Trials: Observational Study in a Nationally Representative Adult Cohort

Factors Influencing Community Participation in Internet Interventions Compared With Research Trials: Observational Study in a Nationally Representative Adult Cohort

Factors Influencing Community Participation in Internet Interventions Compared With Research Trials: Observational Study in a Nationally Representative Adult Cohort

Original Paper

1Centre for Mental Health Research, College of Health and Medicine, The Australian National University, Acton ACT, Australia

2The Matilda Centre for Research in Mental Health and Substance Use, University of Sydney, Sydney, Australia

3Hunter Medical Research Institute, Newcastle, Australia

Corresponding Author:

Philip Batterham, BSc, MPH, PhD

Centre for Mental Health Research

College of Health and Medicine

The Australian National University

63 Eggleston Road

Acton ACT, 2601


Phone: 61 2 61251031


Background: Digital mental health (DMH) programs can be effective in treating and preventing mental health problems. However, community engagement with these programs can be poor. Understanding the barriers and enablers of DMH program use may assist in identifying ways to increase the uptake of these programs, which have the potential to provide broad-scale prevention and treatment in the community.

Objective: In this study, we aimed to identify and compare factors that may influence participation in DMH programs in practice and research trials, identify any respondent characteristics that are associated with these factors, and assess the relationship between intentions to use DMH programs and actual uptake.

Methods: Australian adults aged ≥18 years were recruited from market research panels to participate in the study. The sample was representative of the Australian adult population based on age, gender, and location. Participants completed a cross-sectional web-based survey assessing demographic characteristics, mental health symptom measures, attitudes and use of DMH programs in practice and in research studies, and the factors influencing their use in both settings.

Results: Across both research and practice, trust in the organization delivering the service or trial was the top-ranked factor influencing participation, followed by anonymity or privacy and adequate information. There was little variation in rankings across demographic groups, including intentions to use DMH programs or mental health status. Intentions to use DMH programs were a strong predictor of both current (odds ratio 2.50, 99% CI 1.41-4.43; P<.001) and past (odds ratio 2.98, 99% CI 1.71-5.19; P<.001) use behaviors.

Conclusions: Efforts to increase the uptake of DMH programs or participation in research trials should focus on clearly communicating the following to users: the legitimacy of the organization delivering the program, security and use of participant data, and effectiveness of DMH programs.

J Med Internet Res 2023;25:e41663




Digital mental health (DMH) programs are web-based interventions and apps that are designed to deliver evidence-based therapeutic content to individuals, with limited or no in-person clinical support. Low-intensity DMH programs, including self-help DMH programs, can be effective in treating and preventing high prevalence mental health problems such as depression or anxiety [1,2]. However, community uptake and participation in these programs remain poor [3,4], with studies in primary care reporting uptake rates as low as 3%-25% [5]. A range of factors have been proposed to explain this lack of participation, including low levels of awareness of DMH programs and the perception that web-based therapy is less effective than face-to-face care [6]. Thus, the capacity of web-based programs to provide broad-scale prevention and treatment in the community is currently limited, even though most people report that they would be willing to try a DMH program [7,8]. Identifying barriers and enablers to the use of DMH programs and developing strategies to address them are critical for optimizing their potential. An additional limitation of implementing evidence-based interventions, including both digital and in-person therapies, is that outcomes in research trials may not reflect real-world outcomes [9]. Delivery of DMH interventions in practice may not realize the success seen in controlled clinical trials [10], which typically have strict eligibility criteria and support from research staff. By examining the factors associated with engagement in research trials and comparing them with the factors associated with engagement in evidence-based interventions, we might be able to (1) better characterize implementation gaps and (2) understand differences in efficacy and use when interventions move from research to real-world delivery settings, reflecting the limited ecological validity of research trials.

Barriers and Enablers of Engaging With Internet Interventions

Previous research has documented factors influencing the uptake (enablers) or lack of uptake (barriers) of DMH programs. Much of the previous research has described individual factors influencing adherence to or ongoing engagement with the program; these factors include female gender [11], sufficient time [11], less severe symptoms at baseline [12,13], younger age [12], or realization of sufficient benefit without completing the full intervention [11,12,14-16]. Research has also examined factors impeding the uptake or initiation of DMH programs, including a lack of familiarity or acceptability of these types of programs; data security concerns; poor attitudes toward help-seeking; assumptions that e–mental health programs are not as effective as face-to-face therapy; a lack of internet access or anxiety around its use; and other personal factors such as poor education, low knowledge of technology, personality (eg, conscientiousness), and a lack of time [2,4,5,17-22]. Much of this research, with a few recent exceptions [23,24], focuses on examining factors influencing uptake based on intervention completion data, without asking the participants themselves about their perceptions of the barriers and enablers for use. Understanding perceived barriers for potential users may provide insights into the optimal design and implementation of internet interventions. Thus, research is needed to further investigate the factors that impede the uptake of DMH programs and that are perceived to be the most important barriers preventing their use.

Some of the identified factors influencing the uptake of DMH programs may be modifiable, such as the acceptability of DMH programs [19], whereas some factors such as age or personality factors are not. We may be able to improve uptake by designing interventions that target specific groups that would benefit the most [3,13] or by challenging some of the modifiable barriers before an individual commences an e-mental health program [5]. Several studies have developed and evaluated interventions (termed acceptance facilitation interventions or engagement facilitation interventions) that seek to improve uptake by reducing some of these barriers [5,25,26]. These interventions have primarily focused on increasing acceptance, although some have addressed barriers such as perceived needs, privacy concerns, and knowledge [5,25,26]. Although there is evidence that these interventions can increase the acceptability of internet interventions, a recent randomized controlled trial of an engagement facilitation intervention addressing multiple barriers to engagement found no significant increase in uptake or adherence to the subsequently presented e-mental health program [27], despite aligning the design of the engagement facilitation intervention to consumer views and preferences [23]. The limited success of acceptance and engagement facilitation interventions in changing implementation outcomes suggests that engagement in interventions is highly complex and likely involves a combination of factors that interact to influence behavior. The findings also suggest that consumer preferences do not always align with their eventual behavior. However, given the limited investigation of consumer preferences and priorities to date, more work is needed to better understand the drivers of engagement with DMH programs in both research and practice, particularly in samples that are broadly representative of the general population.

Barriers and Enablers of Engaging With Research Trials

A critical issue with DMH programs is that the findings from controlled research trials are not always consistent with outcomes when programs are implemented in clinical or community settings. Research trials benefit from the inclusion of a diverse range of participants, particularly as representation from a variety of groups increases the ability to generalize the findings to those populations [28]. However, many people may experience barriers to participation in research trials. In the general medical literature, barriers to clinical trial participation have included concerns about safety [29] or negative effects on health [30]; general fear or mistrust of medicine; the burden of trial participation [31]; and lack of access by not being offered participation [32], sometimes because of stringent inclusion criteria [33]. Enablers of participation in research trials include positive aspects of health care providers offering participation, perceived benefits, and altruism [31]. Few studies have specifically focused on factors associated with trial participation in the mental health field, although studies have examined factors influencing participation in partnership or cocreation projects, such as Living Labs [34]. From this literature, factors influencing participation in cocreation research include higher levels of socioeconomic status and more positive attitudes toward health and health care [35]. Other barriers to research engagement may include greater illness severity and functional impairments [36] and as with general health, strict eligibility criteria [37,38]. Stringent exclusion criteria are typical in mental health trials, including the trials of DMH programs, potentially resulting in highly selected samples [39]. The risk with such selection is that the effects observed in trials may not generalize when treatments are applied in real-world settings.

In summary, there is existing evidence of a range of factors influencing the uptake of DMH programs, but potential users’ perceived barriers to uptake are less frequently investigated. It also remains unclear which of these factors are the most influential in individuals’ decisions to engage or not engage with a DMH program. There has been no known previous research on the factors that influence participation in DMH research trials. In addition, previous research on factors associated with engagement has typically relied on convenience samples, which may bias the results because of underrepresentation from certain groups of the population. This study sought to fill these critical research gaps by using an Australian adult sample that was nationally representative in terms of age group, gender, and location (proportionally representing both rurality and states and territories of Australia). By comparing the similarities and differences in the factors that influence the uptake of interventions and factors associated with research participation, it may be possible to better understand the limitations of using trial data when implementing interventions in community and clinical settings.


The primary aims of this study were to identify factors that may influence participation in DMH programs in practice and in research trials of DMH programs (aim 1) and to assess whether these factors are different (aim 2) using survey data from a representative adult sample. Putative factors were selected based on existing evidence of barriers and enablers of engagement with DMH programs and research trials. Identifying these factors in a nationally representative adult sample may inform the targeting of implementation strategies for internet interventions and may be used to identify potential limitations in the ecological validity of research trials. We also explored whether any respondent characteristics influenced how the different factors were ranked (aim 3). Finally, to examine whether the self-rated likelihood of using DMH programs was an adequate proxy for actual uptake of DMH programs, we tested the relationship between participants’ perceptions of likelihood and actual uptake (aim 4), accounting for potential confounding factors.

Ethics Approval

This study was approved by The Australian National University Human Research Ethics Committee (ANU HREC protocol number 2020/593). Participants were paid according to small scheduled payments for completing surveys via Qualtrics Research Services (QRS) in the form of rewards (typically used for gift cards or airline reward programs of a small value, around Aus $2 (US $1.44) to Aus $4 (US $2.88) depending on the selected reward). Data were collected anonymously, and all participants provided written informed consent before commencing the study.

Finding are reported consistently with the Checklist for Reporting Results of Internet E-Surveys (CHERRIES).

Procedure: Recruitment and Data Collection

Participants were recruited through QRS during November 2020 to December 2020 using quota sampling to obtain a sample that was representative in terms of age group, gender, Australian state and territory populations, and remoteness (urban, regional, or rural area classification) to ensure that the sample collected was broadly representative of the Australian community. Participants from a selection of market research panels were invited by QRS to participate in the surveys. To avoid self-selection bias, survey invitations do not include specific details about the contents of the survey and are instead kept very general. After participants clicked the link from QRS, they were directed to our web page displaying the information sheet on the Qualtrics survey platform. Survey data were entered by participants on the Qualtrics platform. The inclusion criteria were that participants must be (1) living in Australia, (2) aged ≥18 years, and (3) able to read and write in English well enough to complete the survey. Participants were required to read the information sheet describing the key aspects of the study (including survey length, data storage and security, voluntary participation, the purpose of the survey, and the research team), agree that they met the inclusion criteria, and then provide consent to participate before completing the survey. Those who were screened as ineligible or did not provide their consent to participate were excluded from the study and provided with relevant mental health resources they may like to access. Participants could withdraw from the study at any time until completion by closing their web browser window, and the survey took approximately 20-30 minutes to complete. None of the questions were mandatory, except for the demographic questions used to determine the representativeness of the sample, and a back button was provided to review responses. Participants were excluded from the final sample if they failed to respond correctly to the 2 quality check items. QRS used IP addresses, cookies, and panel registrations to minimize repeat and poor-quality responses, excluding responses with completion time <7 minutes. If any participant endorsed the suicide screening item of the Patient Health Questionnaire-9 (PHQ-9) by selecting an option other than “not at all,” they were provided with a pop-up that asked them to telephone relevant crisis services (Lifeline, the Suicide Call Back Service, or emergency services).



The usability and technical functionality of the survey were tested by the authors before it was launched. The survey questions were delivered over 15 pages, with the number of questions per page ranging from 1 to 26. Before the section on use of DMH interventions (labeled “internet-based programs”), the following definition was provided: “Internet-based programs are online learning programs or smartphone apps that provide psychological therapy strategies, coping strategies (including mindfulness or relaxation) or support for lifestyle changes to support your mental health and well-being. Internet-based programs teach strategies that are used in face-to-face therapy for reducing symptoms of depression or anxiety.” Similarly, before the section on research participation (labeled “research trials”), a definition was provided: “A research trial is a study that is run by a university, hospital or health provider, testing whether an intervention (such as a pill, device or therapy) works effectively. Participating in a research trial involves having repeated measurements (such as answering surveys or having a medical examination) and being offered a treatment, which may be an active treatment or a control (placebo) treatment. In a research trial, the treatment received by each person is determined by chance – they don't have a choice as to whether they get the active or control treatment.”

Demographic Characteristics

The following demographic characteristics were assessed and included in this study: gender (men, women, and nonbinary), age (18-35, 36-55, and ≥56 years), level of education (high school or less, certificate or diploma, bachelor’s degree, and postgraduate degree), language spoken at home (English and English and other, or other language only), employment status (full time, part time or casual, unemployed, not working owing to study or maternity leave, retirement, etc), and region or area of residence (metropolitan area, regional area, and rural or remote area).

Attitudes and Use of DMH Interventions and Research Participation

We asked participants to rate on a scale of 1 to 5 (strongly disagree=1 to strongly agree=5) the extent to which they agreed with the statements “I am confident in my ability to use the Internet,” and “I want to change the way I think and feel about mental health problems.” We also asked if they had previously used an internet-based program to support their mental health or well-being (yes and no), if they had ever participated in a research trial (yes, maybe or not sure, and no), how likely they would be to participate in a research trial in the future, and how likely they would be to try an internet intervention in 8 hypothetical situations (eg, if a clinician recommended it, out of curiosity, etc) on a 5-point scale of 1 to 5 (extremely unlikely=1 to extremely likely=5). Finally, participants were provided with links to a range of existing DMH programs (Mindspot, myCompass, and MentalHealthOnline), and individuals who clicked on one or more links were identified as an indicator of actual uptake.

Factors Influencing the Use of DMH Programs and Research Participation

We asked participants to rank a list of factors influencing community participation in web-based mental health programs in two settings: (1) use of DMH programs in practice and (2) participation in a research trial of a DMH program. These lists included items such as, “Whether I could be anonymous or maintain my privacy.” The lists were compiled based on the existing evidence of barriers and enablers of engagement in both psychosocial interventions and treatment. Before completing the lists, the participants were provided with brief definitions of internet interventions and research trials. The lists of the factors were the same, although the items were reworded to match the setting. Participants could rank all 15 factors, but we requested that participants rank at least the top 5 factors. Participants ranked at least 5 factors, with most ranking only 5 factors (use of DMH: mean 5.2, SD 1.6; research trials: mean 5.2, SD 1.5), and 13 participants providing no ranking data. To reduce the chances of response bias because of item order, we randomized the presentation order for both lists of factors and hypothetical situations mentioned previously.

Symptom Measures

We used the PHQ-9 [40] to measure depression, the Generalized Anxiety Disorder-7 (GAD-7) [41] to assess anxiety, and the Distress Questionnaire-5 (DQ5) [42] to measure general psychological distress. The first 2 scales assess the frequency of symptoms of major depression (PHQ-9) or generalized anxiety disorder (GAD-7) experienced during the past 2 weeks (4-point scale from not at all=0 to nearly every day=3). Item scores were summed for each scale for the overall severity scores for depression (PHQ-9:9 items; range 0-27) and anxiety (GAD-7:7 items; range 0-21) symptoms. The 5 DQ5 items asked respondents to indicate the frequency of distressing situations, thoughts, and feelings over the previous 30 days using a 5-point scale (1=never, 2=rarely, 3=sometimes, 4=often, and 5=always). Scores were summed and ranged from 5 to 25. For each of these scales, higher scores indicate higher symptom severity, and all have robust psychometric evidence for reliability and validity [42-45]. In this study, all 3 scales had high internal consistency based on Cronbach α of .93 for PHQ-9, .95 for GAD-7, and .94 for DQ5.


The ranked factors influencing the use of DMH programs and DMH research trial participation were reverse scored (eg, ranks 1, 2, and 3 correspond to points 3, 2, and 1, respectively) and summed so that higher scores indicated a higher ranked importance of a factor impacting participation. To compare the rankings of each factor across the 2 settings, the proportion of respondents who rated each factor as one of the top 3 factors influencing their participation was compared using McNemar test with Bonferroni correction for 15 comparisons (α=.003). Logistic regression analyses examined factors associated with high rankings (in the top 3) for the 3 most important factors associated with the 2 outcomes, with Bonferroni correction for the 6 models (α=.008). Candidate predictors of rankings included demographic factors, the likelihood of DMH program use (averaged across all scenarios), the likelihood of research trial participation, and mental health symptom scores. Finally, to examine whether the likelihood of use was related to actual uptake, we tested whether the average likelihood of DMH program use was associated with the past use of a DMH program or uptake of a DMH at the end of the survey, adjusting for demographic characteristics.

Participant Characteristics

Table 1 presents the characteristics of the 1094 participants who completed the survey. The view rate was 31.18% (1094/3509) and the completion rate was 45% (n=3509 entered the survey, of which 117, 3.33% did not consent; 939, 26.76% did not meet demographic quotas; and 1359, 38.73% did not complete the survey or failed reliability checks). By design, the sample was representative of the Australian population in terms of age, gender, and location (both state and territory and remoteness). Although only 7.03% (77/1095) reported using the internet to support their mental health, most (655/1094, 59.87%) reported willingness to use a DMH program if it was recommended by a clinician. Most (783/1094, 71.57%) participants were somewhat or extremely likely to use a DMH program in at least one scenario. Across all scenarios, an average of 42.05% (460/1094) reported that they would be “somewhat likely” or “extremely likely” to use a DMH program. The likelihood of participating in a DMH research trial was similar, with 43.51% (476/1094) reporting that they would be “somewhat likely” or “extremely likely” to participate. Fewer than 17% (186/1094) of the participants reported prior participation in a research trial. There was a moderate correlation between the likelihood of using a DMH program and the likelihood of participating in a DMH research trial (r=0.31).

Table 1. Characteristics of participants included in the study (N=1094).
Sociodemographic characteristics

Age (years), mean (SD)46.49 (18.06)

Age category (years), n (%)

18-35363 (33.2)

36-55349 (31.9)

≥56382 (34.9)

Gender, n (%)

Men529 (48.4)

Women561 (51.3)

Nonbinary4 (0.4)

Highest level of education, n (%)

≤High school295 (27)

Certificate or diploma330 (30.2)

Bachelor’s degree287 (26.2)

Postgraduate degree or diploma182 (16.6)

Employment, n (%)

Full-time350 (32)

Part-time or casual254 (23.2)

Unemployed108 (9.9)

Not working (eg, study, maternity leave, etc)382 (34.9)

Language, n (%)

English946 (86.5)

English and other or other language only148 (13.5)

Location, n (%)

Metropolitan789 (72.1)

Regional200 (18.3)

Rural or remote105 (9.6)
Attitudes and use of internet interventions and research participation

Previously used internet intervention for mental health, n (%)76 (7)

Confidence in using the internet, mean (SD)4.33 (0.83)

Desire to change how think and feel about mental health problems, mean (SD)3.26 (1.09)

Likelihood of using internet interventions, mean (SD)

If a clinician recommended that I try a program3.46 (1.18)

If I was diagnosed with a mental health condition3.11 (1.24)

If I was concerned that I had a mental health condition3.01 (1.22)

If a screening scale indicated I had symptoms of a mental health condition2.98 (1.20)

If I wanted to improve my general well-being2.97 (1.22)

If a friend or colleague recommended a program2.92 (1.18)

Out of curiosity2.68 (1.21)

Likelihood of participating in research trial, mean (SD)3.17 (1.26)

Previously participated in research trial, n (%)

Yes108 (9.9)

Maybe or not sure75 (6.9)

Uptake: clicked on one or more DMHa programs, n (%)51 (4.7)
Mental health symptom measures, mean (SD)

Anxiety (GAD-7b)5.27 (5.94)

Depression (PHQ-9c)6.84 (6.95)

General psychological distress (DQ5d)10.35 (5.46)

aDMH: digital mental health.

bGAD-7: Generalized Anxiety Disorder-7.

cPHQ-9: Patient Health Questionnaire-9.

dDQ5: Distress Questionnaire-5.

Factors Influencing Engagement With DMH Programs in Community and Research Settings

Table 2 presents the perceived factors influencing community participation in DMH programs compared with the factors influencing participation in trials of DMH programs. Trust in the organization delivering the service or trial was ranked at the top for both settings. Other factors that most highly ranked across both settings included anonymity or privacy, adequate information, and the level of need. Shame or embarrassment and other people being aware of participation or use were the lowest ranked for both settings. A direct comparison of the factors is provided in Table 3, based on the proportion of respondents who rated each factor as one of the top 3 factors influencing their participation. Two factors were more likely to be ranked highly for the use of DMH programs in practice compared with participation in DMH research trials: need for anonymity or privacy and access to other care. In contrast, 3 factors were less likely to be ranked highly in the use of DMH compared with research: need for information, effort required, and time required.

To examine whether there were characteristics associated with high rankings (top 3) of the selected factors, logistic regression models were estimated for the 3 highest-ranked items: trust, anonymity or privacy, and perceived need. The results are presented in Tables 4-6. Overall, there was little variation in rankings across demographic groups, intention to use or participate in DMH programs, and mental health status. The only significant associations were that older people were more likely to highly rank the item “Whether I trusted the organisation that delivers the program” than younger people (2% increased odds per year of age); and those who had higher intentions to use DMH programs were more likely to highly rank the item “Whether I thought I needed support for my mental health” (28% increased odds of high endorsement per one-unit increase in intentions).

Table 2. Factors influencing participation in digital mental health (DMH) programs and research trials.
RankUse of DMH programs in practiceParticipation in DMH research trials

1Whether I trusted the organization that delivers the program8200Whether I trusted the organization leading the research trial8260
2Whether I could be anonymous or maintain my privacy7572Having adequate information about the research trial7970
3Whether I thought I needed support for my mental health6836Whether I could be anonymous or maintain my privacy6983
4Whether the program was tailored to my needs6643The effort it takes to participate in the research trial6322
5Having information about whether the program works5971Whether I thought I needed support for my mental health5954
6Whether or not I had good access to other health services (eg, doctor, psychologist)5217The amount of time I have5892
7The effort it takes to do the program4626Whether the research trial was tailored to my needs5751
8Whether I could do the program by myself without anyone helping me4557Whether I had good access to health services (eg, doctor, psychologist)4361
9The amount of time I have4233My awareness of research trials3915
10My comfort with using technology4030The look, feel, and interactivity of the program being tested3572
11The look, feel, and interactivity of the program3946Whether I was assisted by a clinician to participate in the research trial3537
12My awareness of programs that are available3623My comfort with using technology3077
13Whether someone was going to check-in with me to complete the program3057Whether I could participate in the research trial by myself without anyone helping me3075
14Whether people I know liked the program1945The level of shame or embarrassment I would have about participating in a research trial1596
15The level of shame or embarrassment I would have about using the program1923Whether people I know had been in a research trial1480

aRanks reverse scored and cumulated across participants. Scores for each topic were calculated by cumulating the reverse-scored ranks for each of the 15 proposed factors (ie, ranks 1, 2, and 15 correspond to points 15, 14, and 1, respectively) across participants. Higher scores indicate higher ranked importance.

Table 3. Direct comparison of factors ranked in top 3.
FactorRespondents, n (%)McNemar P value

In practiceIn trials
Trust402 (36.7)383 (35).34
Anonymity or privacy378 (34.6)320 (29.3).002a
Need for support327 (29.9)283 (25.9).02
Tailoring307 (28.1)256 (23.4).01
Information about effectiveness264 (24.1)356 (32.5)<.001a
Access to other care250 (22.9)183 (16.7)<.001
Effort required193 (17.6)292 (26.7)<.001
Time available188 (17.2)282 (25.8)<.001
Comfort with technology184 (16.8)148 (13.5).02
Ability to complete without help180 (16.5)139 (12.7).01
Awareness of availability156 (14.3)174 (15.9).26
Look and feel of program150 (13.7)144 (13.2).68
Availability of check-in or support120 (11.0)158 (14.4).01
People I know had used or participated79 (7.2)62 (5.7).12
Level of shame or embarrassment74 (6.8)63 (5.8).29

aItalicized values indicate P<.003.

Table 4. Logistic regression analysis of factor (trust) influencing rankings of top barriers and enablers of using digital mental health (DMH) programs in practice and participation in DMH research trials.


ORa (99% CI)P valueOR (99% CI)P value

Certificate or diploma versus ≤HSb0.864 (0.554-1.348).401.141 (0.727-1.792).45

Bachelor’s versus HS or less0.893 (0.551-1.448).551.450 (0.896-2.346).047

Postgraduate versus HS or less1.231 (0.714-2.121).331.276 (0.731-2.228).26

Part time or casual versus full time0.928 (0.580-1.482).681.233 (0.772-1.970).25

Unemployed versus full time1.054 (0.556-1.997).830.832 (0.422-1.639).48

Not in labor force versus full time0.789 (0.484-1.286).211.304 (0.802-2.121).16
Language: English only versus other1.174 (0.703-1.961).421.218 (0.726-2.043).33

Women versus men1.400 (0.979-2.002).020.982 (0.688-1.403).90

Nonbinary versus men11.560 (0.559-239.202).046.766 (0.327-139.971).10
Age (years)1.017 (1.004-1.031).001c1.010 (0.997-1.023).04
Likelihood of using DMH program0.829 (0.678-1.013).020.972 (0.794-1.191).72
Likelihood of trial participation0.956 (0.830-1.102).421.013 (0.880-1.167).81
PHQ-9d depression score0.998 (0.945-1.055).941.035 (0.980-1.093).11
GAD-7e anxiety score1.016 (0.946-1.090).580.999 (0.931-1.072).98
DQ5f distress score0.967 (0.897-1.041).240.974 (0.905-1.049).37
Constant0.547 (0.145-2.072).240.226 (0.059-0.864).004

aOR: odds ratio.

bHS: high school.

cItalicized values indicate P<.008.

dPHQ-9: Patient Health Questionnaire-9.

eGAD-7: Generalized Anxiety Disorder-7.

fDQ5: Distress Questionnaire-5.

Table 5. Logistic regression analysis of factor (privacy) influencing rankings of top barriers and enablers of using digital mental health (DMH) programs in practice and participation in DMH research trials.


ORa (99% CI)P valueOR (99% CI)P value

Certificate or diploma versus ≤HSb1.011 (0.650-1.571).951.164 (0.731-1.854).40

Bachelor’s versus ≤HS1.027 (0.637-1.655).891.294 (0.786-2.131).18

Postgraduate versus ≤HS0.803 (0.454-1.421).320.808 (0.439-1.484).37

Part time or casual versus full time0.983 (0.608-1.590).930.698 (0.423-1.152).07

Unemployed versus full time1.341 (0.714-2.521).231.054 (0.550-2.018).84

Not in labor force versus full time1.519 (0.933-2.473).031.071 (0.648-1.771).72
Language: English only versus other1.444 (0.846-2.465).081.065 (0.624-1.820).76

Women versus Men1.001 (0.700-1.433).991.291 (0.889-1.875).08

Nonbinary versus men0.449 (0.021-9.615).506.557 (0.317-135.544).11
Age (years)0.995 (0.983-1.008).340.997 (0.983-1.010).51
Likelihood of using DMH program0.919 (0.749-1.126).280.862 (0.697-1.066).07
Likelihood of trial participation1.117 (0.969-1.286).041.049 (0.905-1.215).41
PHQ-9c depression score1.033 (0.979-1.091).121.009 (0.954-1.067).67
GAD-7d anxiety score0.997 (0.930-1.069).910.994 (0.925-1.069).84
DQ5e distress score0.992 (0.921-1.067).771.015 (0.941-1.095).61
Constant0.337 (0.088-1.286).040.424 (0.106-1.700).11

aOR: odds ratio.

bHS: high school.

cPHQ-9: Patient Health Questionnaire-9.

dGAD-7: Generalized Anxiety Disorder-7.

eDQ5: Distress Questionnaire-5.

Table 6. Logistic regression analysis of factor (need) influencing rankings of top barriers and enablers of using digital mental health (DMH) programs in practice and participation in DMH research trials.


ORa (99% CI)P valueOR (99% CI)P value

Certificate or diploma versus ≤HSb1.032 (0.650-1.640).861.063 (0.663-1.703).74

Bachelor’s versus ≤HS0.924 (0.559-1.527).690.696 (0.407-1.189).08

Postgraduate versus ≤HS0.901 (0.502-1.619).650.809 (0.439-1.490).37

Part time or casual versus full time1.078 (0.658-1.767).700.936 (0.551-1.591).75

Unemployed versus full time0.953 (0.477-1.904).860.923 (0.448-1.904).78

Not in labor force versus full time1.027 (0.619-1.706).891.128 (0.665-1.912).56
Language: English only versus other1.389 (0.798-2.418).131.265 (0.705-2.270).30

Women versus men1.069 (0.736-1.551).650.952 (0.645-1.405).74

Nonbinary versus men1.405 (0.067-29.400).770.000 (indeterminate).99
Age (years)1.006 (0.993-1.020).231.006 (0.993-1.019).28
Likelihood of using DMH program1.281 (1.029-1.594).004c1.187 (0.949-1.486).05
Likelihood of trial participation1.110 (0.957-1.288).071.144 (0.980-1.334).03
PHQ-9d depression score0.980 (0.924-1.040).390.945 (0.886-1.007).02
GAD-7e anxiety score0.992 (0.918-1.071).781.038 (0.958-1.124).23
DQ5f distress score0.981 (0.907-1.061).520.990 (0.912-1.075).75
Constant0.120 (0.029-0.495)<.0010.132 (0.030-0.578)<.001

aOR: odds ratio.

bHS: high school.

cItalicized values indicate P<.008.

dPHQ-9: Patient Health Questionnaire-9.

eGAD-7: Generalized Anxiety Disorder-7.

fDQ5: Distress Questionnaire-5.

Relationship Between DMH Use Intentions and Behaviors

Finally, to examine whether the self-rated likelihood of using DMH programs was a good proxy for actual uptake of DMH programs, we estimated 2 logistic regressions: one on past use of DMH programs and one on uptake of a DMH program within the study. The results of these analyses are presented in Table 7. The analyses indicated that use intentions were a strong indicator of both current and past behaviors. Specifically, there was 2.5-fold increase in the odds of engaging with one of the offered DMH programs per one-point increase in intentions and nearly 3-fold increase in the odds of past DMH program use per one-point increase in intentions. Younger age was also significantly associated with the past use of DMH programs, with a 6% increase in the odds of past use per year of age.

Table 7. Logistic regression analyses testing the associations between use intentions and behaviors.

New uptake of DMHa programPrevious use of DMH program

ORb (99% CI)P valueOR (99% CI)P value
Likelihood of using internet intervention2.503 (1.413, 4.434)<.001c2.982 (1.714, 5.187)<.001c

Certificate or diploma versus ≤HSd1.663 (0.615, 4.498).191.182 (0.482, 2.903).63

Bachelor’s degree versus ≤HS0.968 (0.312, 3.002).940.841 (0.338, 2.090).62

Postgraduate degree versus ≤HS0.850 (0.207, 3.487).771.171 (0.387, 3.546).71

Part time or casual versus full time0.967 (0.322, 2.907).940.924 (0.380, 2.251).82

Unemployed versus full time0.924 (0.195, 4.375).901.442 (0.485, 4.290).39

Not in labor force versus full time1.607 (0.548, 4.717).261.648 (0.639, 4.250).17
Language: English only versus other0.680 (0.229, 2.025).361.226 (0.489, 3.074).57
Gender: Male versus other2.023 (0.867, 4.721).030.779 (0.388, 1.562).36
Age (years)1.013 (0.987, 1.040).200.942 (0.918, 0.967)<.001c
Constant0.001 (1.413, 4.434)<.0010.018 (1.714, 5.187)<.001

aDMH: digital mental health.

bOR: odds ratio.

cItalicized values indicate P<.001.

dHS: high school.

Principal Findings

The findings indicate that among a representative Australian adult sample, a few key factors are the common influences of engagement with DMH programs in the context of both use in practice and in research trials. Although these factors may be considered as either “barriers” or “enablers” of uptake, each factor represents both a challenge to clearly articulate the potential difficulties in implementing DMH programs and an opportunity to educate and engage the target audience (ie, both a potential barrier and enabler). The most highly ranked considerations that influence the use of DMH programs were trust in the provider, need for privacy or anonymity, perceived need for support, and information about whether a DMH program is effective.

These outcomes differ somewhat from those of our previous community-based study in which sufficient time and perceived need dominated as key barriers to engagement [23]. Although perceived need was important in this study, the role of trust and need for privacy ranked higher, whereas time constraints had much less influence on use intentions. There are a few possible explanations for the discrepancies. First, we used a ranking approach rather than assigning equal weight to all reported barriers or enablers. Second, our focus was on the uptake or initiation of an intervention rather than on ongoing engagement. In agreement with this, a recent study on university medical students’ attitudes specifically toward the uptake of DMH found that privacy was also a key barrier to engaging in this group [24]. Third, barriers may change over time. For example, public awareness of privacy issues may have increased in recent years in response to problems with electronic medical records or media stories of hacking. Finally, our sample was selected to be representative of the national adult population in Australia in terms of age, gender and location, whereas most previous research on the factors influencing DMH use has relied on convenience samples. It is possible that convenience samples overrepresent people with mental health problem [46] or result in highly educated samples [47] that may have greater existing knowledge of DMH programs. Such biases may skew the preferences and priorities of a sample, leading to conclusions that may neglect the perspectives of underrepresented groups of the community. For example, the mean depression and anxiety symptom scores in this sample were approximately half of those seen in convenience samples [3].

The roles of trust and the need for privacy have been highlighted in previous studies [48-50] and are 2 themes that are likely to intersect [51]. Trust in the provider was the most important consideration both when conducting research trials and when delivering a DMH program in practice. This finding suggests that there may be not only challenges in building trust between providers and end users but also opportunities to promote DMH programs through trusted brands within the mental health sector. Organizations with established reputations for rigor, such as universities, may be trusted more by the public than commercial organizations [52]. Partnerships between stakeholders to ensure that DMH programs meet rigorous standards of safety and efficacy, combined with the reach and infrastructure of the industry and government, may be an important approach for further implementing DMH programs into practice. Privacy and data security have also been identified as key concerns. Clarity around what data are collected and how they are used is likely to be important in promoting uptake, along with emphasizing when programs are delivered anonymously. This information is often lacking or substandard in existing services [48].

The participants also ranked the need for sufficient information on the effectiveness of DMH programs as a key determinant of their use. Marketing DMH programs may benefit from presenting information about the effectiveness of the program in a way that is suitable for diverse audiences [53], without overstating or neglecting existing evidence [23]. Perceived need may be more challenging to address, as it is difficult for programs to engage people experiencing mental health problems if they are unaware that they may benefit from support. Strategies to address this challenge include the implementation of community mental health literacy programs [54], population-based screening with feedback [53,55], and implementation and marketing of DMH programs in diverse clinical and community settings [56]. Novel and effective approaches to research dissemination into the community may also raise awareness of the availability and evidence base of DMH programs. Encouragingly, stigma and embarrassment, along with the perceptions of others, were the least endorsed factors, suggesting that negative attitudes toward people with mental illness and negative attitudes toward treatment may have limited influence on the uptake of digital interventions.

Despite many similarities, there were a few differences in rankings between the use of DMH programs in practice and research. The need for anonymity and privacy was found to be more important for engaging with DMH programs in practice. Having access to other care was also more relevant to DMH in practice. Interestingly, the effort and time required were seen as more important in research trials, indicating that people view research as more burdensome than engaging with a service. This finding may be related to the additional burden of engaging with a research team and completing assessments in the context of a trial. These findings suggest that research cohorts may attract people with greater time and capacity than those who might otherwise engage in DMH programs. Such differences in the profiles of research participants may distort the trial outcomes when DMH programs are implemented in clinical or community settings. Furthermore, information needs were ranked higher in research trials, emphasizing the importance of clearly communicating expectations around participation in trials.

The rankings varied little by demographic characteristics, level of intention to use DMH programs, or mental health symptoms, suggesting that the perceived barriers and enablers to use are relatively homogeneous across the general population. Older adults were more likely to endorse trust as a key concern, suggesting that the reputation of a provider becomes increasingly important when delivering services to older adults. Those who reported a greater likelihood of using DMH programs were more likely to endorse perceived need as a determinant of use, which may indicate that those who are most likely to use programs tend to consider the level of support is needed (eg, no care, informal care, DMH programs, or face-to-face services) based on the severity of their symptoms. However, other demographic factors and mental health symptoms did not influence the relative importance of each factor.

Research on preferences for interventions is sometimes criticized for using behavioral intentions as a proxy for actual behavior. However, we also demonstrated that the likelihood of using a DMH program was strongly associated with both past and current engagement in these programs. These findings suggest that behavioral intentions are likely to be a good proxy for future use, as each 1-point increase in intentions was associated with 2.5-times the odds of engaging with one of the three DMH programs that were offered at the end of the survey. The findings also indicate that people who had engaged in DMH programs in the past had higher intentions for future use, which is reassuring as it suggests a level of satisfaction with past experiences.


This study is one of very few to use a nationally representative adult sample to examine the factors associated with engagement in DMH programs. It is also the first, to our knowledge, to concurrently examine barriers and enablers of using DMH programs in practice and in research trials, providing useful information about why trial outcomes may not translate into practice. However, some limitations of this study should be acknowledged. Although we recruited the sample to be representative of key demographic characteristics, selection biases may still exist, such as those related to access to technology, interest in research, and interest in mental health. Potential participants may have withdrawn (or may have been more inclined to participate) when provided with information about the content of the survey, and some may have been motivated to participate by the small incentive. Furthermore, market research panels may not be representative of the community in terms of other characteristics. In particular, panel members may be more accustomed to research, more familiar with digital applications, and have potentially more favorable attitudes toward research than people who do not engage in such panels. As it is extremely challenging to engage research-averse people in research studies, methods to account for nonparticipation [57] may provide further insight into barriers to service engagement. The study was cross-sectional; therefore, the direction of the effects and causation cannot be inferred. Our indicators of the likelihood of using DMH programs were ad hoc, using hypothetical scenarios that may not reflect actual use or specific programs. Nevertheless, these indicators were strongly related to self-reported past use and the choice to engage in a DMH program at the end of the survey. It is also acknowledged that clicking on the link to a DMH program signifies behavior toward using a program, rather than direct engagement with program content, and factors such as previous experience using DMH programs may have influenced the decision to click on links. Our choice of barriers and enablers associated with the decision to engage in a DMH program was intended to be comprehensive, covering the most commonly endorsed factors identified in previous research. However, there may be other factors that we did not consider. Finally, other unmeasured factors such as personality, access to resources, socioeconomic status, and ethnicity may also influence the observed associations.


The factors that influence engagement in DMH programs are largely consistent in the context of both use in practice and research trials, with some key exceptions. Trust in providers, need for privacy or anonymity, perceived need for support, and information about the effectiveness of DMH programs were commonly endorsed in both settings, whereas time commitment was a greater consideration for research trials than for engaging in DMH programs in practice. Bridging the implementation gap between evidence and the use of DMH programs requires clearly communicating to users the legitimacy of the organization delivering the program. Organizations that deliver DMH programs are often successful at either developing evidence for efficacy and safety or marketing a program but not always both [58]. The findings of this study, in the context of growing literature on the implementation of DMH programs, suggest several additional considerations for optimizing the implementation of DMH programs, including providing transparent information about data use, demonstrating the effectiveness of the program, ensuring that programs are developed around the needs of end users [58], and providing clearer dissemination pathways with better marketing of evidence-based programs in community and clinical settings [56].


This project was supported by the National Health and Medical Research Fund grant # 1138713). The lead author can disseminate the results of this trial without the express permission of the funder. PB was supported by National Health and Medical Research Council (NHMRC) Fellowship 1158707; AC was supported by NHMRC Fellowship 1173146; and FK-L was supported by NHMRC Fellowship 1110371. LF was supported by ARC DECRA DE190101382.

Data Availability

Data are available on request from the primary author.

Authors' Contributions

PB led the design of the study. AG and PB drafted the manuscript and conducted the analyses. All authors contributed to the design, development, and conduct of the survey; critically edited the manuscript; and accepted the final version.

Conflicts of Interest

None declared.

  1. Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, et al. Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. JAMA Psychiatry 2017 Apr 01;74(4):351-359 [FREE Full text] [CrossRef] [Medline]
  2. Batterham PJ, Sunderland M, Calear AL, Davey CG, Christensen H, Teesson M, et al. Developing a roadmap for the translation of e-mental health services for depression. Aust N Z J Psychiatry 2015 Sep;49(9):776-784. [CrossRef] [Medline]
  3. Batterham PJ, Calear AL. Preferences for Internet-based mental health interventions in an adult online sample: findings from an online community survey. JMIR Ment Health 2017 Jun 30;4(2):e26 [FREE Full text] [CrossRef] [Medline]
  4. Musiat P, Goldstone P, Tarrier N. Understanding the acceptability of e-mental health--attitudes and expectations towards computerised self-help treatments for mental health problems. BMC Psychiatry 2014 Apr 11;14:109 [FREE Full text] [CrossRef] [Medline]
  5. Ebert DD, Berking M, Cuijpers P, Lehr D, Pörtner M, Baumeister H. Increasing the acceptance of internet-based mental health interventions in primary care patients with depressive symptoms. A randomized controlled trial. J Affect Disord 2015 May 01;176:9-17. [CrossRef] [Medline]
  6. Apolinário-Hagen J, Harrer M, Kählke F, Fritsche L, Salewski C, Ebert DD. Public attitudes toward guided internet-based therapies: web-based survey study. JMIR Ment Health 2018 May 15;5(2):e10735 [FREE Full text] [CrossRef] [Medline]
  7. Clough BA, Zarean M, Ruane I, Mateo NJ, Aliyeva TA, Casey LM. Going global: do consumer preferences, attitudes, and barriers to using e-mental health services differ across countries? J Ment Health 2019 Feb;28(1):17-25. [CrossRef] [Medline]
  8. Choi I, Sharpe L, Li S, Hunt C. Acceptability of psychological treatment to Chinese- and Caucasian-Australians: internet treatment reduces barriers but face-to-face care is preferred. Soc Psychiatry Psychiatr Epidemiol 2015 Jan;50(1):77-87. [CrossRef] [Medline]
  9. Heneghan C, Goldacre B, Mahtani KR. Why clinical trial outcomes fail to translate into benefits for patients. Trials 2017 Mar 14;18(1):122 [FREE Full text] [CrossRef] [Medline]
  10. Andersson G. Internet interventions: past, present and future. Internet Interv 2018 Jun;12:181-188 [FREE Full text] [CrossRef] [Medline]
  11. Batterham PJ, Neil AL, Bennett K, Griffiths KM, Christensen H. Predictors of adherence among community users of a cognitive behavior therapy website. Patient Prefer Adherence 2008 Feb 02;2:97-105 [FREE Full text] [Medline]
  12. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res 2009 Apr 24;11(2):e13 [FREE Full text] [CrossRef] [Medline]
  13. Farrer LM, Griffiths KM, Christensen H, Mackinnon AJ, Batterham PJ. Predictors of adherence and outcome in internet-based cognitive behavior therapy delivered in a telephone counseling setting. Cogn Ther Res 2014;38(3):358-367. [CrossRef]
  14. Davis MJ, Addis ME. Predictors of attrition from behavioral medicine treatments. Ann Behav Med 1999;21(4):339-349. [CrossRef] [Medline]
  15. Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res 2011 Aug 05;13(3):e52 [FREE Full text] [CrossRef] [Medline]
  16. Donkin L, Glozier N. Motivators and motivations to persist with online psychological interventions: a qualitative study of treatment completers. J Med Internet Res 2012 Jun 22;14(3):e91 [FREE Full text] [CrossRef] [Medline]
  17. Australian Government Response to Contributing Lives, Thriving Communities: Review of Mental Health Programmes and Services. Australian Government Department of Health. 2015.   URL: https:/​/www.​​sites/​default/​files/​response-review-of-mental-health-programmes-and-services.​pdf [accessed 2023-01-06]
  18. Gun SY, Titov N, Andrews G. Acceptability of Internet treatment of anxiety and depression. Australas Psychiatry 2011 Jun;19(3):259-264. [CrossRef] [Medline]
  19. Waller R, Gilbody S. Barriers to the uptake of computerized cognitive behavioural therapy: a systematic review of the quantitative and qualitative evidence. Psychol Med 2009 May;39(5):705-712. [CrossRef] [Medline]
  20. Mohr DC, Siddique J, Ho J, Duffecy J, Jin L, Fokuo JK. Interest in behavioral and psychological treatments delivered face-to-face, by telephone, and by internet. Ann Behav Med 2010 Aug;40(1):89-98 [FREE Full text] [CrossRef] [Medline]
  21. Casey LM, Wright MA, Clough BA. Comparison of perceived barriers and treatment preferences associated with internet-based and face-to-face psychological treatment of depression. Int J Cyber Behav Psychol Learn 2014;4(4):16-22. [CrossRef]
  22. Gulliver A, Calear AL, Sunderland M, Kay-Lambkin F, Farrer LM, Batterham PJ. Predictors of acceptability and engagement in a self-guided online program for depression and anxiety. Internet Interv 2021 Sep;25:100400 [FREE Full text] [CrossRef] [Medline]
  23. Gulliver A, Calear AL, Sunderland M, Kay-Lambkin F, Farrer LM, Banfield M, et al. Consumer-guided development of an engagement-facilitation intervention for increasing uptake and adherence for self-guided Web-based mental health programs: focus groups and online evaluation survey. JMIR Form Res 2020 Oct 29;4(10):e22528 [FREE Full text] [CrossRef] [Medline]
  24. Dederichs M, Weber J, Pischke CR, Angerer P, Apolinário-Hagen J. Exploring medical students' views on digital mental health interventions: a qualitative study. Internet Interv 2021 Sep;25:100398 [FREE Full text] [CrossRef] [Medline]
  25. Baumeister H, Seifferth H, Lin J, Nowoczin L, Lüking M, Ebert D. Impact of an acceptance facilitating intervention on patients' acceptance of internet-based pain interventions: a randomized controlled trial. Clin J Pain 2015 Jun;31(6):528-535. [CrossRef] [Medline]
  26. Baumeister H, Nowoczin L, Lin J, Seifferth H, Seufert J, Laubner K, et al. Impact of an acceptance facilitating intervention on diabetes patients' acceptance of internet-based interventions for depression: a randomized controlled trial. Diabetes Res Clin Pract 2014 Jul;105(1):30-39. [CrossRef] [Medline]
  27. Batterham PJ, Calear AL, Sunderland M, Kay-Lambkin F, Farrer LM, Christensen H, et al. A brief intervention to increase uptake and adherence of an internet-based program for depression and anxiety (enhancing engagement with psychosocial interventions): randomized controlled trial. J Med Internet Res 2021 Jul 27;23(7):e23029 [FREE Full text] [CrossRef] [Medline]
  28. Britton A, McKee M, Black N, McPherson K, Sanderson C, Bain C. Threats to applicability of randomised trials: exclusions and selective participation. J Health Serv Res Policy 1999 Apr;4(2):112-121. [CrossRef] [Medline]
  29. Lewin J, Bell JA, Wang K, Forcina V, Tam S, Srikanthan A, et al. Evaluation of adolescents' and young adults' attitudes toward participation in cancer clinical trials. JCO Oncol Pract 2020 Mar;16(3):e280-e289. [CrossRef] [Medline]
  30. Abdelhafiz AS, Abd ElHafeez S, Khalil MA, Shahrouri M, Alosaim B, Salem RO, et al. Factors influencing participation in COVID-19 clinical trials: a multi-national study. Front Med (Lausanne) 2021 Feb 23;8:608959 [FREE Full text] [CrossRef] [Medline]
  31. Schmotzer GL. Barriers and facilitators to participation of minorities in clinical trials. Ethn Dis 2012;22(2):226-230. [Medline]
  32. Wendler D, Kington R, Madans J, Van Wye G, Christ-Schmidt H, Pratt LA, et al. Are racial and ethnic minorities less willing to participate in health research? PLoS Med 2006 Feb;3(2):e19 [FREE Full text] [CrossRef] [Medline]
  33. Comis RL, Miller JD, Aldigé CR, Krebs L, Stoval E. Public attitudes toward participation in cancer clinical trials. J Clin Oncol 2003 Mar 01;21(5):830-835. [CrossRef] [Medline]
  34. Georges A, Schuurman D, Baccarne B, Coorevits L. User engagement in living lab field trials. Info 2015;17(4):26-39. [CrossRef]
  35. Hearld KR, Hearld LR, Hall AG. Engaging patients as partners in research: factors associated with awareness, interest, and engagement as research partners. SAGE Open Med 2017 Jan 4;5:2050312116686709 [FREE Full text] [CrossRef] [Medline]
  36. Lally J, Watkins R, Nash S, Shetty H, Gardner-Sood P, Smith S, et al. The representativeness of participants with severe mental illness in a psychosocial clinical trial. Front Psychiatry 2018 Dec 4;9:654 [FREE Full text] [CrossRef] [Medline]
  37. Hoertel N, Le Strat Y, Blanco C, Lavaud P, Dubertret C. Generalizability of clinical trial results for generalized anxiety disorder to community samples. Depress Anxiety 2012 Jul;29(7):614-620. [CrossRef] [Medline]
  38. Hoertel N, Le Strat Y, De Maricourt P, Limosin F, Dubertret C. Are subjects in treatment trials of panic disorder representative of patients in routine clinical practice? Results from a national sample. J Affect Disord 2013 Apr 25;146(3):383-389. [CrossRef] [Medline]
  39. Lorenzo-Luaces L, Johns E, Keefe JR. The generalizability of randomized controlled trials of self-guided internet-based cognitive behavioral therapy for depressive symptoms: systematic review and meta-regression analysis. J Med Internet Res 2018 Nov 09;20(11):e10113 [FREE Full text] [CrossRef] [Medline]
  40. Spitzer RL, Kroenke K, Williams JB. Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary care evaluation of mental disorders. Patient health questionnaire. JAMA 1999 Nov 10;282(18):1737-1744. [CrossRef] [Medline]
  41. Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097. [CrossRef] [Medline]
  42. Batterham PJ, Sunderland M, Carragher N, Calear AL, Mackinnon AJ, Slade T. The Distress Questionnaire-5: population screener for psychological distress was more accurate than the K6/K10. J Clin Epidemiol 2016 Mar;71:35-42. [CrossRef] [Medline]
  43. Kroenke K, Spitzer RL, Williams JB, Löwe B. The patient health questionnaire somatic, anxiety, and depressive symptom scales: a systematic review. Gen Hosp Psychiatry 2010;32(4):345-359. [CrossRef] [Medline]
  44. Löwe B, Decker O, Müller S, Brähler E, Schellberg D, Herzog W, et al. Validation and standardization of the Generalized Anxiety Disorder Screener (GAD-7) in the general population. Med Care 2008 Mar;46(3):266-274. [CrossRef] [Medline]
  45. Batterham PJ, Sunderland M, Slade T, Calear AL, Carragher N. Assessing distress in the community: psychometric properties and crosswalk comparison of eight measures of psychological distress. Psychol Med 2018 Jun;48(8):1316-1324. [CrossRef] [Medline]
  46. Batterham PJ. Recruitment of mental health survey participants using Internet advertising: content, characteristics and cost effectiveness. Int J Methods Psychiatr Res 2014 Jun;23(2):184-191 [FREE Full text] [CrossRef] [Medline]
  47. Späth C, Hapke U, Maske U, Schröder J, Moritz S, Berger T, et al. Characteristics of participants in a randomized trial of an Internet intervention for depression (EVIDENT) in comparison to a national sample (DEGS1). Internet Interv 2017 Sep;9:46-50 [FREE Full text] [CrossRef] [Medline]
  48. O'Loughlin K, Neary M, Adkins EC, Schueller SM. Reviewing the data security and privacy policies of mobile apps for depression. Internet Interv 2019 Mar;15:110-115 [FREE Full text] [CrossRef] [Medline]
  49. Gulliver A, Bennett K, Bennett A, Farrer LM, Reynolds J, Griffiths KM. Privacy issues in the development of a virtual mental health clinic for university students: a qualitative study. JMIR Ment Health 2015 Mar 31;2(1):e9 [FREE Full text] [CrossRef] [Medline]
  50. Ritterband LM, Gonder-Frederick LA, Cox DJ, Clifton AD, West RW, Borowitz SM. Internet interventions: in review, in use, and into the future. Prof Psychol Res Pr 2003 Oct;34(5):527-534. [CrossRef]
  51. Iott BE, Campos-Castillo C, Anthony DL. Trust and privacy: how patient trust in providers is related to privacy behaviors and attitudes. AMIA Annu Symp Proc 2020 Mar 4;2019:487-493 [FREE Full text] [Medline]
  52. LaMonica HM, Davenport TA, Roberts AE, Hickie IB. Understanding technology preferences and requirements for health information technologies designed to improve and maintain the mental health and well-being of older adults: participatory design study. JMIR Aging 2021 Jan 06;4(1):e21461 [FREE Full text] [CrossRef] [Medline]
  53. Batterham PJ, Calear AL, Farrer L, Gulliver A, Kurz E. Efficacy of a transdiagnostic self-help internet intervention for reducing depression, anxiety, and suicidal ideation in adults: randomized controlled trial. J Med Internet Res 2021 Jan 22;23(1):e22698 [FREE Full text] [CrossRef] [Medline]
  54. Brijnath B, Protheroe J, Mahtani KR, Antoniades J. Do web-based mental health literacy interventions improve the mental health literacy of adult consumers? Results from a systematic review. J Med Internet Res 2016 Jun 20;18(6):e165 [FREE Full text] [CrossRef] [Medline]
  55. Batterham PJ, Calear AL, Sunderland M, Carragher N, Brewer JL. Online screening and feedback to increase help-seeking for mental health problems: population-based randomised controlled trial. BJPsych Open 2016 Jan;2(1):67-73 [FREE Full text] [CrossRef] [Medline]
  56. Batterham PJ, Gulliver A, Kurz E, Farrer LM, Vis C, Schuurmans J, et al. The effect of dissemination pathways on uptake and relative costs for a transdiagnostic, self-guided internet intervention for reducing depression, anxiety, and suicidal ideation: comparative implementation study. J Med Internet Res 2022 May 06;24(5):e34769 [FREE Full text] [CrossRef] [Medline]
  57. Bergstrand R, Vedin A, Wilhelmsson C, Wilhelmsen L. Bias due to non-participation and heterogenous sub-groups in population surveys. J Chronic Dis 1983;36(10):725-728. [CrossRef] [Medline]
  58. Torous J, Nicholas J, Larsen ME, Firth J, Christensen H. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Ment Health 2018 Aug;21(3):116-119. [CrossRef] [Medline]

DMH: digital mental health
DQ5: Distress Questionnaire-5
GAD-7: Generalized Anxiety Disorder-7
PHQ-9: Patient Health Questionnaire-9
QRS: Qualtrics Research Services

Edited by T Leung; submitted 04.08.22; peer-reviewed by I Choi, H LaMonica; comments to author 15.11.22; revised version received 29.11.22; accepted 30.11.22; published 02.02.23


©Philip Batterham, Amelia Gulliver, Matthew Sunderland, Louise Farrer, Frances Kay-Lambkin, Angelica Trias, Alison Calear. Originally published in the Journal of Medical Internet Research (, 02.02.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.