JMIR Publications

Select Journals for Content Updates

When finished, please click submit.

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

This paper is in the following e-collection/theme issue:

    Review

    Adherence in Internet Interventions for Anxiety and Depression: Systematic Review

    Centre for Mental Health Research, The Australian National University, Canberra, Australia

    Corresponding Author:

    Helen Christensen, PhD, MPsych, BA

    Centre for Mental Health Research

    Australian National University

    Canberra ACT 0200

    Australia

    Phone: +61 2 6125 8409

    Fax:+61 2 6125 0733

    Email:


    ABSTRACT

    Background: Open access websites which deliver cognitive and behavioral interventions for anxiety and depression are characterised by poor adherence. We need to understand more about adherence in order to maximize the impact of Internet-based interventions on the disease burden associated with common mental disorders.

    Objective: The aims of this paper are to review briefly the adherence literature with respect to Internet interventions and to investigate the rates of dropout and compliance in randomized controlled trials of anxiety and depression Web studies.

    Methods: A systematic review of randomized controlled trials using Internet interventions for anxiety and depression was conducted, and data was collected on dropout and adherence, predictors of adherence, and reasons for dropout.

    Results: Relative to reported rates of dropout from open access sites, the present study found that the rates of attrition in randomized controlled trials were lower, ranging from approximately 1 - 50%. Predictors of adherence included disease severity, treatment length, and chronicity. Very few studies formally examined reasons for dropout, and most studies failed to use appropriate statistical techniques to analyze missing data.

    Conclusions: Dropout rates from randomized controlled trials of Web interventions are low relative to dropout from open access websites. The development of theoretical models of adherence is as important in the area of Internet intervention research as it is in the behavioral health literature. Disease-based factors in anxiety and depression need further investigation.

    J Med Internet Res 2009;11(2):e13)

    doi:10.2196/jmir.1194

    KEYWORDS



    Introduction

    Web-based interventions are effective for a range of mental health disorders including depression, panic, post-traumatic stress disorder (PTSD), perceived stress in schizophrenia, stress, insomnia, and eating disorders [1]. While efficacy trials of Web interventions show good-to-excellent levels of adherence, open access websites have been associated with poor adherence and dropout, with substantial numbers of users not completing all Web pages and exiting websites before the full completion of an offered program [2,3]. For example, Farvolden [4] found that only 1% of participants completed a 12-week open access panic program, and Christensen and colleagues [5] reported that less than 1% of users completed all modules in an open access website for depression.

    There is little reason to expect that the rates of adherence to websites offered as open access on the Web would be as strong as those reported for websites which are examined in the context of an efficacy trial. Open access websites provide information and Web content directly to community users at no, or minimal, cost. Data on adherence from these sites is based on the activity of spontaneous users who “visit” these sites, where many users will have no expectation that they will be offered “programs”. In contrast, data from efficacy trials of websites are based on responses from participants who are recruited to the trial on the basis of elevated symptoms; consent in advance of the trial; are provided with information about its parameters (nature and length of the program, etc); and are required to complete pre- and post-intervention surveys. Nevertheless, we need to know more about the basis of adherence and dropout, not least because there is evidence that greater exposure to website content is associated with increased benefit [6,7]. The Internet platform offers the opportunity to yield a rich source of objective data on engagement and dropout, and, consequently, has the potential to inform adherence research generally. High quality, objectively measurable information on treatment compliance can be obtained from logs of page views, resource downloads, time on site, and other indicators of treatment exposure. High volume Internet sites have the potential to investigate the effect of theory-driven modifications on adherence through the use of high throughput online randomized controlled trials (see, for example, Christensen and Mackinnon [8]).

    The present paper has four aims: (1) to undertake a systematic review of the rates of adherence in randomized controlled trials (RCTs) of Internet interventions for anxiety and depression, with the aim of determining rates of attrition in order to confirm that Internet rates of attrition are lower in research trials than open access websites; (2) to collate data from these RCTs to identify predictors of dropout and adherence; (3) to examine the research studies for data on participant’s perceptions of adherence and dropout; and (4) to examine the type of analyses that were used to manage “missingness”, given that dropout from RCTs needs to be considered in every analysis of efficacy. To our knowledge, only two papers have reported rates of dropout from open access websites [4,5], but no systematic review of adherence or dropout from RCTs has been undertaken.

    This paper begins by providing the context for these aims by defining adherence and dropout, briefly reviewing the research strategies used to investigate adherence in both Internet and non-Internet trials, and describing the evidence arising from these strategies. Research indicates that there are differences in the predictors of adherence for different health conditions [9]. Hence, we restrict our review to websites that target anxiety and depression. A brief discussion of approaches to the statistical analysis of dropout is also presented.

    Definitions

    Most definitions of adherence are not well suited to the characteristics of e-interventions. For example, the World Health Organization (WHO) describes adherence as the “extent to which a person’s behavior [...] corresponds with agreed recommendations from a health care provider” [10]. This definition clearly does not transfer readily to the Web environment particularly with respect to interventions that are designed to be offered through open access sites, or to interventions that are predicated on self-help models. In the context of this paper, the term adherence refers to the extent to which individuals experience the content of the Internet intervention. The term dropout is used to describe an individual who fails to complete the research trial protocol associated with an Internet intervention, and thus does not complete trial assessments. These terms correspond reasonably closely to Eysenbach’s terms “non-usage dropout attrition” and “non-usage attrition”, which he applied to the uptake of Internet interventions. Dropout attrition refers to loss of participants from the trial [2]. Non-usage refers to participants’ lack of exposure to the website material. While it is perhaps simpler to use the terms usage and dropout attrition with respect to Web interventions, it is important also to “mainstream” Internet interventions—that is, to provide appraisals of them using terms appropriate to formal non-Internet based trials. For this reason, we use the terms adherence and dropout for the remainder of the article.

    Clearly, dropout and treatment adherence refer to interrelated but conceptually distinct constructs. Individuals may drop out of a trial (fail to complete assessments) but have 100% treatment adherence. This occurs, for example, when users continue to undertake the prescribed program even though they have severed contact with the research or clinical team. Others may complete the protocol fully but adhere to the intervention less than 100% of the time. In this case, participants do not undertake the full Web program, although they may continue to complete all assessments.

    Figure 1. The relationship between dropout and adherencein a two-arm trial
    View this figure

    Figure 1 outlines the potential range of outcomes of dropout and adherence in terms of a two-arm research trial. This represents a simplified analysis because there may be degrees or levels of adherence and dropout. Participants may complete an interim follow-up but not the final follow-up, or miss interim assessments but complete final assessments, and so on (ie, participants are classified as dropouts at various stages of the study).

    The figure serves as a schematic to highlight important questions, such as how adherence is measured if participants drop out informally or formally by withdrawing their consent. Dropout increases progressively with program length [11], so direct comparisons of percentage dropout are not appropriate unless the length of the programs are roughly equivalent. Within the research trial context, dropouts have been subclassified as no shows, (those who do not proceed to the consent or treatment stage and do not complete assessments), early dropouts (those who drop out relatively early in a program and complete only one or very few assessments), or late dropouts [12]. The term early completer has been used to refer to those who benefit from the intervention but do not complete the protocol. In the Web context, these participants have been referred to “e-attainers” [8].

    Strategies to Studying Adherence

    Three general approaches are undertaken to investigate adherence in both Web and non-Web environments. The first approach involves using correlational or regression analyses within trials to establish associations between adherence and various personality, demographic, and service delivery factors. Common variables investigated include aspects of service delivery, therapist factors, rewards and incentives, program duration, the nature of the medium of delivery, and personality factors, including expectations, self-determination, self-efficacy, or support from partners and friends (Davis and Addis [12], page 347). Disease-specific effects such as disease severity predict adherence, with a high level of emotional distress leading to early dropout [12]. Demographic variables, such as age, socioeconomic status, education, and marital status typically do not predict patient adherence across a range of health conditions [13]. For depression, the results from 14 epidemiological studies failed to indicate any clear predictors of adherence to medication regimes, although adverse side effects reduced adherence [14]. The amount of variance explained by these investigations is low. Moreover, these categories do not take into account the unique type and range of adherence variables associated with Internet delivery, and additional work is needed to investigate “computer factors” rather than therapist factors. With respect to e-health applications, one example of this correlational approach is a study of 82,000 users of an open access website for depression [15].

    A second approach involves the use of post-test questionnaires to obtain retrospective analyses of people’s perceptions of trial participation, barriers to the use of the treatment, and other factors. An example of this approach is Ritterband’s follow-up interview which has been used to identify barriers to seeking out and using a paediatric website [16].

    A third approach involves the experimental manipulation of variables believed to be causal in promoting adherence. A recent review summarized the effects of 38 systematic reviews of the effectiveness of adherence interventions across a range of disease conditions and intervention types published between 1990 and 2005 [9]. Interventions to improve adherence were classified into four types: technical solutions (such as simplifying doses); behavioral interventions; educational programs; and social support interventions. This review revealed that less frequent medication schedules (fewer but higher dose tablets) increased adherence (for most targeted disorders, with the exception of depression, see Yildiz et al [17]), as did behavioral interventions which provided reminders, used monitoring, or introduced rewards. Educational interventions were successful when patients were trained in cognitive problem solving or when they were taught motivational techniques. However, knowledge alone was not successful. Little evidence was found for the effectiveness of social support interventions. A number of studies investigated adherence to anxiety and depression interventions [9]. These interventions were complex, involving many components of collaborative care. Consequently, the specific components that were critical in improving adherence were difficult to identify [14], although collaborative care produced better adherence than educational interventions directed at the provider [10,18]. To date, in the e-health field, there has been little experimental manipulation of factors likely to increase adherence in e-health trials.

    Approaches Missing Data Arising From Dropout

    Intention to treat analyses (ITT), where all participants in the trial are analyzed regardless of whether they drop out, is recommended for publication in most large-scale research studies, although the extent to which this approach is undertaken in RCTs of Internet trials is not known. ITT analyses take into account bias arising from selective attrition and hence are preferred over completer analyses, where only those completing the protocol are analyzed. Missing data for ITT approaches can be managed in a number of ways, including through the use of last observation carried forward (LOCF) imputation. However, more advanced methods which include the use of multiple imputation [19,20] and maximum-likelihood based methods [21], are more likely to yield valid outcomes. The use of ITT and methods to manage missing data is examined in the present review.


    Methods

    Study Selection

    Relevant studies were identified using the methodology employed in our previous systematic reviews of RCTs of mental health Internet interventions [1,22]. The databases of PubMed, PsycInfo, and Cochrane Register Randomized Controlled Trials were searched using the key terms “Internet OR Web” together with search strategies designed to capture randomized controlled trials. Criteria for inclusion of a study in the current review were that it (1) involved a self-help website for a depressive or anxiety disorder; (2) tested the efficacy of a self-help psychoeducational or skills training intervention; (3) employed a randomized controlled trial design; and (4) incorporated a control group that was not subjected to an active treatment intervention. Only peer reviewed published articles were included in the analysis. Dissertations and published poster abstracts were excluded. Because our previous reviews collected information prior to 2007, we updated the search to include papers published before February 2009. For this update, a total of 1177 abstracts were retrieved from the searches conducted in PubMed, PsycInfo, and Cochrane Controlled Trials.  Of these, 1154 papers were excluded because they did not describe a self-help website for a depression or anxiety disorder, 2 were excluded because the intervention they described did not test the efficacy of a psychoeducational or skills training program, 3 were excluded because they did not employ a randomized controlled trial design, 7 were excluded because they used a control group that was subjected to an active treatment, and 2 were excluded because they were not published as a peer reviewed paper.  The remaining 9 studies met the criteria for inclusion in the review, and we added them to the original 14 studies.

    Coding of Study Characteristics

    Sample Size

    The number of participants in the study intervention was recorded.

    Dropout

    Dropout was defined as the number of individuals who failed to complete the research protocol. Typically, these figures were derived from the trial flow diagram. Thus, dropouts were those who failed to complete post-treatment or follow-up assessments once they had been accepted into the trial.

    Adherence

    Adherence was defined as an indicator of the extent to which individuals used the material on the website. Information on adherence was collected and reported with respect to logons, time on site, and number of modules attempted.

    Predictors of Adherence or Dropout

    We also recorded any reported association between a predictor (gender, severity) and any dropout or adherence measure.

    Reason for Dropout

    Any data on the reason for dropout was recorded.

    Types of Statistical Analyses

    Analyses were classified as either completer or intention to treat (ITT), with the method used to handle missing data noted.


    Results

    There were 8 trials of depression interventions, 1 trial of a depression, anxiety, and stress intervention, 1 trial of a generalized anxiety disorder (GAD) intervention, 5 trials of panic disorder (PD) interventions, 4 trials of social phobia (SP) interventions, and 4 trials of Post Traumatic Stress Disorder (PTSD) interventions (Table 1).

    Table 1. Summary of included randomized controlled trials of Internet interventions for anxiety and depression
    View this table

    Rates of Dropout/Non-completion of Study Protocol

    Completion of protocol rates for depression sites ranged from a low of 43% [23] to a high of 99% [29], with some trials indicating poorer retention after a longer follow-up [26]. All studies reported lower rates of completion in the experimental intervention group relative to the control with the exception of Spek et al [30]. The one GAD trial reported a 6-month follow-up retention of 44% in the experimental group [34]. Trials for PD reported high rates of retention—approximately 80 - 90% for the experimental group, but these were based on small numbers of participants, and rates of dropout were often not reported separately for experimental and control conditions. Rates of completion for the SP interventions were approximately 90% at 12-month follow-up. Rates for PTSD ranged from 87% at post-treatment [47] to 47% at 6 week follow-up [48].

    Adherence

    Adherence data were reported using indications such as number of log ons, duration of Web exposure, number of modules or exercises completed, and number of postings on bulletin boards. Although rates varied considerably, adherence to the complete online treatment was approximately 50 - 70% for depression sites and 50% for the sole GAD intervention [34]. Rates of adherence to the PD interventions were reported as high as 80 - 100% [36,37]. The SP trials reported 70 - 90%, and one of the PTSD trials reported a rate of 64% [48].

    Predictors of Adherence

    For depression, predictors found to be associated with increased adherence were lower baseline rates of depression, younger age, and poorer knowledge of psychological treatments. Education or quality of life, when measured, did not predict adherence. For GAD, lower symptom levels predicted better adherence. Data for PD trials were scant. One trial of a PTSD intervention reported higher adherence with treatment for women, older persons, those who lived with a partner, and those less experienced with a computer.

    Self-Reported Reason for Dropout

    Only one study conducted a formal survey of the reasons for dropout [48]. However, the following were mentioned as reasons for dropout in the Internet intervention group or, where separate data were not provided, in the group of participants as a whole: time constraints [23,30,31,32,34,36,37,41,42,43], lack of motivation [39,40,43], technical or computer-access problems [42,46,48], depressive episode or physical illness [39,40], the lack of face-to-face contact [48], preference for taking medication [40], perceived lack of treatment effectiveness [39,43,44,47,48], improvement in condition [29,32,44,47], and burden of the program [23,48].

    Methods to Analyze Missing Data

    For depression, four approaches to missing data were used: analysis of completers only [23, 29]; intention to treat (ITT) using last observation carried forward (LOCF) [6,24,25,26]; mixed models with maximum likelihood estimation (REML) [27,28,32], the latter being one of the best of the approaches and standard good practice [49]; and multiple imputation [30,31], also a recommended strategy [50]. For GAD, a completer analysis was conducted. Panic disorder studies reported two approaches: four studies used ITT with LOCF [36,37,39,40] and one analyzed completers alone [38]. All of the social phobia studies utilized LOCF. Three PTSD studies used completer analyses [45,47,48], and one study used ITT with LOCF [46].


    Discussion

    Findings

    Relative to reported rates of dropout from open access sites, the present study found that the rates of attrition in RCTs were lower, ranging from a high loss of 50% to a low of 1% over various follow-up periods. Treatment adherence was relatively high, at over 50%. These rates are relatively similar to those in randomized controlled trials of non-Internet-based interventions for generalized anxiety disorder and depression, with a recent review suggesting attrition rates are about 15% on average for GAD, but the rate of dropout ranged from 0 - 50% [51]. Our findings suggest that there is nothing particularly non-adherent about an Internet intervention per se when delivered in the context of a randomized controlled trial. However, these findings confirm that dropout is much less dramatic than that associated with open access websites. As such, the findings clearly articulate the need to compare rates of adherence for open access interventions against appropriate benchmarks. In our view, the rates of adherence for open access websites should be compared to rates of adherence reported for traditional health services provided by practitioners face to face (Meichenbaum and Turk [52], page 25). Where reported, these data show that adherence rates are high in face-to-face treatment as well, with as many as 70% of patients missing by a third session, and hypothetical attrition curves indicating that almost 100% of users are non-adherent after 10 sessions. Stress, exercise, or smoking programs have estimated discontinuation rates of between 20 - 80% (see Turk and Meichenbaum [13], page 249), while anti-depressant medication is discontinued by approximately 40 - 80% (see Sabate [10], page 66).

    The findings from our review of RCTs also need to be compared to other recent work on rates of adherence in Web treatments, including a recent review of barriers to the uptake of computerized cognitive behavior programs [53]. This review differs substantially from ours in that it used an integrative methodology (combining both qualitative and quantitative work), reported work up to July 2005 only, reviewed computer-based interventions in addition to Internet-based ones, and focused on CBT style interventions only. Its focus was also substantially different because it covered acceptability and satisfaction in addition to dropout. This review reported that a medium of 83% of participants completed the study (ie, did not dropout) and a medium of only 56% completed a course of the program in data from quantitative studies. Although these rates cannot be compared formally, they appear to be slightly lower than those of the present review. The medium dropout rate of interventions from the depression studies was 60%, while the adherence level ranged between 38 - 78%, depending on which outcome measure was used.

    In our study, predictors of adherence were similar to previously identified factors [9], including disease severity, treatment length, and chronicity. Very few studies formally examined reasons for dropout, and it was noted that personal circumstances “played a major role, including travel (for those studies based around a clinic computer)” (see Waller and Gilbody [53], page 3). Most studies also failed to use appropriate statistical techniques to analyze missing data.

    Limitations of the Study

    Measures of adherence to websites did vary across studies, and we acknowledge that the use of different methods (log ons vs modules completed, etc) will yield different measures of adherence, and that these measures will not necessarily correlate strongly. Website design will be another important factor in determining the type and richness of particular outcome measures. Further research is needed to determine whether a universal indicator of adherence using diverse measures could be developed. For example, a “percent” of adherence might be a useful approach. Both a strength and a limitation of the present study was its focus on anxiety and depression websites. Although beyond the scope of the present paper, there is a clear need to consolidate information about reliable predictors of adherence across other physical and mental disorders and diseases, and to identify both disease-specific and generic predictors.

    The focus of this paper was to examine adherence in RCTs. It was not possible to compare directly the rates of adherence between open access services and trial-based Web interventions. To our knowledge the open access websites reported in the introduction are the only ones for which there is published data. With further publication of data for open access sites, it may be possible to undertake a formal review of predictors and rates. Once sufficient trials and evaluations have been conducted within open access sites and websites used as part of RCT trials, techniques to develop appropriate quantitative comparisons between efficacy and effectiveness studies could be systematically employed to compare these rates (see Hunsley and Mash [54]).

    Implications for Future Work

    The findings from our review reinforce conclusions that have been drawn from traditional intervention research. Little is known about the specific component factors that improve adherence in health interventions. Research within this area is essentially atheoretical, and a coherent approach is required. Given the importance of adherence research, and the unique advantage of Web-based data collection for analyzing adherence, we suggest a potential research agenda to advance this area.

    A first step requires the adoption of a theoretical approach to the understanding of dropout and adherence. The framework adopted by WHO [10] identifies five dimensions to pursue: health system factors, socioeconomic factors, therapy-related factors, condition-related factors, and patient-related factors. Historically, the emphasis has been on patient factors. For example, according to Davis and Addis, “What is needed are theories which link specific client characteristics and treatment processes onto attrition” [12] (page 347). However, there is now recognition that health systems factors seem critical. A substantial body of research on depression interventions in primary care emphasizes the importance of case management and continuity of care for efficacy and adherence [55]. These findings, together with the overall greater adherence rates achieved within the context of RCTs, point to the potential benefits of incorporating simple procedures such as monitoring and follow-up to increase adherence. This in turn suggests that attention to behavior theory/modification approaches may yield the greatest benefits for increasing adherence to open access websites. Information on the effectiveness of types, frequency, and size of rewards, as well as information on appropriate reinforcement schedules, is likely to be highly useful in developing comprehensive adherence programs. The lessons learned within research contexts for improving adherence to trial protocols [51] might be profitably employed in the design of better treatment delivery systems in community practice, although recommendations such as “if in doubt, screen out” are counter-productive to the aims of open access websites, which aim to reach individuals who are not yet committed to a treatment program. Research from the Internet intervention field already suggests that substantial gains might be achieved by using email tracking. Clarke et al [28], when comparing the outcomes of two trials of the Overcoming Depression on the Internet (ODIN) website, reported that reminders (both telephone and email) were likely to be the crucial factor in determining retention (and improvement). Studies of established Internet-based treatment programs indicate that high rates of adherence are indeed possible if case management and continuity of care principles are followed [56]. It is not yet known whether tailoring improves adherence in mental health, although this is often promoted as the cornerstone of health promotion campaigns, and it forms a rich area for potential investigation.

    Other authors have pointed to the potential of a range of theoretical models other than those based on behavior theory/modification to inform programs for increasing adherence at an individual level, although these focus to a greater degree on person, rather than systems, factors. The health belief model [57] attempts to predict behavior on the basis of a person’s perception of the risks associated with a health condition, as well as beliefs about the costs, potential side effects/difficulties, and benefits of treatment. The protection motivation theory, the theory of reasoned action, the theory of planned behavior, the social-cognitive theory, and models based on self-efficacy [57] have also been proposed as theoretical models that might inform adherence practice.

    A second area likely to advance our understanding of website adherence involves research into the methods by which technology engages users. Eysenbach [2] has cited specific factors such as usability and other technological factors that will have an impact on adherence. This research agenda potentially covers a range of areas: (1) the generic ways in which humans interact with the technology associated with the Internet (such as frequency of use); (2) the specific methods used by individuals to interact with intervention programs as realized on a website (such as skipping sections, completing online assessments); and (3) the means by which users engage preferentially with certain names or brands of website. The latter includes research into the nature of trust (see, for example, Corritore et al [58]). An example of (2), above, is research into the effectiveness of various forms of the presentation of multimedia (see, for example, Sun and Cheng [59]). The investigation of the ways in which humans interact with computers and the Internet (as in (1), above) is also a potentially interesting area for future research. Many users report forming attachments to their computers (an observation that can be confirmed by undertaking a quick search on Google with the phrase “I love my computer”), and a better understanding of the “computer therapeutic alliance” might well be justified.

    Internet interventions have a number of unique features that may impact on adherence rates. These features include the ease with which the interventions can be accessed, the expectations of users, the level of contact with a health professional, and the presence of rewards or motivators. Although evidence is lacking, Internet delivery may increase adherence relative to face-to-face interventions for individual users who respond to the interactivity, tailoring, and online rewards associated with some websites. One hypothesis worthy of investigation is that Internet delivery creates a technology “alliance”, reflecting the attraction or attachment which develops between people and electronic gadgets and computers. Moreover, the hypothesis that websites attract people who prefer treatment delivered anonymously, prefer distal contact, or are housebound because of mental or physical disability requires testing.

    A third issue that warrants further detailed investigation is the role of disease factors and their influence on treatment uptake and maintenance. The cognitive and emotional characteristics of individuals with depression (or anxiety) are likely to impact their choice of treatment, their treatment uptake, and their rates of adherence. As part of their condition, individuals with depression may believe that they do not deserve treatment or that their treatment is unlikely to be effective and, as a consequence, they may be more likely to drop out. Depressed individuals may simply not be able to face using a computer. Intervention programs that directly address “cognitive dysfunctional thoughts” about treatment outcomes may produce better adherence and outcomes.

    This review also identified a number of methodological improvements that are needed to advance the area. Often terms such as adherence, compliance, attrition, and dropout are not operationalized and are used interchangeably. Many studies fail to measure adherence to treatment. Statistical approaches to handling missing data are limited. Use of mixed models with REML approaches to missingness, rather than the use of biased methods such as LOCF or limited methods such as completer analyses, are to be encouraged [49,50]. There may be a need for reanalysis of research trials which are already published.

    Finally, it is appropriate to consider the implications of these findings for identifying ways of reducing the high attrition rates on open access websites. Based on the data from RCTs, it seems likely that adherence to open access sites might immediately be improved if users were to consent to the use of automated reminders and messages. It is likely that the use of automated reminders will be successful across a range of interventions, not just those directed at anxiety and depression. Research investigating the acceptability and effectiveness of such a tracking procedure for open access sites should be accorded a high level of priority.

    Acknowledgments

    Thanks are due to Agus Salim, for helpful discussion of models of missingness. Phil Batterham formatted the manuscript and provided assistance with referencing. HC is supported by a NHMRC Principal Research Fellowship 525411. KG is supported by NHMRC Research Fellowship 525413. The comments of anonymous reviewers of an earlier version of the manuscript are gratefully acknowledged.

    Conflicts of Interest

    HC and KG have developed online psychoeducational and automatic psychological skills training programs (eg, MoodGYM, BluePages Depression Information, e-couch).

    References

    1. Griffiths KM, Christensen H. Review of randomized controlled trials of Internet interventions for mental disorders and related conditions. Clin Psychol 2006;10(1):16-29. [CrossRef]
    2. Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [Medline] [CrossRef]
    3. Glasgow RE. eHealth evaluation and dissemination research. Am J Prev Med 2007 May;32(5 Suppl):S119-S126. [Medline] [CrossRef]
    4. Farvolden P, Denisoff E, Selby P, Bagby RM, Rudy L. Usage and longitudinal effectiveness of a Web-based self-help cognitive behavioral therapy program for panic disorder. J Med Internet Res 2005;7(1):e7 [FREE Full text] [Medline] [CrossRef]
    5. Christensen H, Griffiths KM, Korten AE, Brittliffe K, Groves C. A comparison of changes in anxiety and depression symptoms of spontaneous users and trial participants of a cognitive behavior therapy website. J Med Internet Res 2004 Dec 22;6(4):e46 [FREE Full text] [Medline] [CrossRef]
    6. Christensen H, Griffiths KM, Jorm AF. Delivering interventions for depression by using the internet: randomised controlled trial. BMJ 2004 Jan 31;328(7434):265 [FREE Full text] [Medline] [CrossRef]
    7. Christensen H, Griffiths KM, Mackinnon AJ, Brittliffe K. Online randomized controlled trial of brief and full cognitive behaviour therapy for depression. Psychol Med 2006 Dec;36(12):1737-1746. [Medline] [CrossRef]
    8. Christensen H, Mackinnon A. The law of attrition revisited. J Med Internet Res 2006;8(3):e20; author reply e21 [FREE Full text] [Medline] [CrossRef]
    9. van Dulmen S, Sluijs E, van Dijk L, de Ridder D, Heerdink R, Bensing J. Patient adherence to medical treatment: a review of reviews. BMC Health Serv Res 2007;7(1):55 [FREE Full text] [Medline] [CrossRef]
    10. Sabate E. World Health Organization. Adherence to Long-term Therapies: Evidence for Action. 2003   URL: http://www.who.int/chp/knowledge/publications/adherence_full_report.pdf [accessed 29 Mar 2009] [WebCite Cache]
    11. Christensen H, Griffiths K, Groves C, Korten A. Free range users and one hit wonders: community users of an Internet-based cognitive behaviour therapy program. Aust N Z J Psychiatry 2006 Jan;40(1):59-62. [Medline] [CrossRef]
    12. Davis MJ, Addis ME. Predictors of attrition from behavioral medicine treatments. Ann Behav Med 1999;21(4):339-349. [Medline] [CrossRef]
    13. Turk DC, Meichenbaum D. Adherence to self-care regimens: the patient’s perspective. In Sweet JJ, Rozensky RH, Tovian SM, editors. Handbook of Clinical Psychology in Medical Settings. New York, NY: Plenum Press; 1991:249-266.
    14. Pampallona S, Bollini P, Tibaldi G, Kupelnick B, Munizza C. Patient adherence in the treatment of depression. Br J Psychiatry 2002 Feb;180(2):104-109 [FREE Full text] [Medline] [CrossRef]
    15. Batterham PJ, Neil AL, Bennett K, Griffiths KM, Christensen H. Predictors of adherence among community users of a cognitive behavior therapy website. Patient Preference and Adherence 2008;2:97-105.
    16. Ritterband LM, Borowitz S, Cox DJ, Kovatchev B, Walker LS, Lucas V, et al. Using the internet to provide information prescriptions. Pediatrics 2005 Nov;116(5):e643-e647 [FREE Full text] [Medline] [CrossRef]
    17. Yildiz A, Pauler DK, Sachs GS. Rates of study completion with single versus split daily dosing of antidepressants: a meta-analysis. J Affect Disord 2004 Feb;78(2):157-162. [Medline] [CrossRef]
    18. Vergouwen ACM, Bakker A, Katon WJ, Verheij TJ, Koerselman F. Improving adherence to antidepressants: a systematic review of interventions. J Clin Psychiatry 2003 Dec;64(12):1415-1420. [Medline]
    19. Little R, Rubin DB. Statistical Analysis with Missing Data. New York, NY: J. Wiley & Sons; 1987.
    20. Rubin DB. Multiple Imputation for Nonresponse in Surveys. New York, NY: J. Wiley & Sons; 1987.
    21. Verbeke G, Molenberghs G. Linear Mixed Models for Longitudinal Data. New York, NY: Springer; 2000.
    22. Griffiths K, Farrer L, Christensen H. Clickety-click: e-mental health train on track. Australas Psychiatry 2007 Apr;15(2):100-108. [Medline] [CrossRef]
    23. Andersson G, Strömgren T, Ström L, Lyttkens L. Randomized controlled trial of internet-based cognitive behavior therapy for distress associated with tinnitus. Psychosom Med 2002;64(5):810-816 [FREE Full text] [Medline] [CrossRef]
    24. Andersson G, Bergström J, Holländare F, Carlbring P, Kaldo V, Ekselius L. Internet-based self-help for depression: randomised controlled trial. Br J Psychiatry 2005 Nov;187(5):456-461 [FREE Full text] [Medline] [CrossRef]
    25. Griffiths KM, Christensen H, Jorm AF, Evans K, Groves C. Effect of web-based depression literacy and cognitive-behavioural therapy interventions on stigmatising attitudes to depression: randomised controlled trial. Br J Psychiatry 2004 Oct;185(4):342-349 [FREE Full text] [Medline] [CrossRef]
    26. Mackinnon A, Griffiths KM, Christensen H. Comparative randomised trial of online cognitive-behavioural therapy and an information website for depression: 12-month outcomes. Br J Psychiatry 2008 Feb;192(2):130-134 [FREE Full text] [Medline] [CrossRef]
    27. Clarke G, Reid Ed, Eubanks D, O'Connor E, DeBar LL, Kelleher C, et al. Overcoming depression on the Internet (ODIN): a randomized controlled trial of an Internet depression skills intervention program. J Med Internet Res 2002;4(3):E14. [CrossRef] [Medline]
    28. Clarke G, Reid Ed, Eubanks D, O'Connor E, DeBar LL, Kelleher C, et al. Overcoming depression on the Internet (ODIN) (2): a randomized trial of a self-help depression skills program with reminders. J Med Internet Res 2005;7(2):e16. [Medline] [CrossRef]
    29. Patten SB. Prevention of depressive symptoms through the use of distance technologies. Psychiatr Serv 2003 Mar;54(3):396-398 [FREE Full text] [Medline] [CrossRef]
    30. Spek V, Nyklícek I, Smits N, Cuijpers P, Riper H, Keyzer J, et al. Internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years old: a randomized controlled clinical trial. Psychol Med 2007 Dec;37(12):1797-1806. [Medline] [CrossRef]
    31. Spek V, Cuijpers P, Nyklícek I, Smits N, Riper H, Keyzer J, et al. One-year follow-up results of a randomized controlled clinical trial on internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years. Psychol Med 2008 May;38(5):635-639. [Medline] [CrossRef]
    32. Warmerdam L, van Straten A, Twisk J, Riper H, Cuijpers P. Internet-based treatment for adults with depressive symptoms: randomized controlled trial. J Med Internet Res 2008;10(4):e44 [FREE Full text] [Medline] [CrossRef]
    33. van Straten A, Cuijpers P, Smits N. Effectiveness of a web-based self-help intervention for symptoms of depression, anxiety, and stress: randomized controlled trial. J Med Internet Res 2008;10(1):e7 [FREE Full text] [Medline] [CrossRef]
    34. Kenardy J, McCafferty K, Rosa V. Internet-delivered indicated prevention for anxiety disorders: a randomized controlled trial. Behavioural and Cognitive Psychotherapy 2003;31(3):279-289. [CrossRef]
    35. Kenardy J, McCafferty K, Rosa V. Internet-delivered indicated prevention for anxiety disorders: six-month follow-up. Clin Psychol 2006;10(1):39-42. [CrossRef]
    36. Carlbring P, Westling BE, Ljungstrand P, Ekselius L, Andersson G. Treatment of panic disorder via the Internet: a randomized trial of a self-help program. Behav Ther 2001;32:751-764. [CrossRef]
    37. Carlbring P, Bohman S, Brunt S, Buhrman M, Westling BE, Ekselius L, et al. Remote treatment of panic disorder: a randomized trial of internet-based cognitive behavior therapy supplemented with telephone calls. Am J Psychiatry 2006 Dec;163(12):2119-2125 [FREE Full text] [Medline] [CrossRef]
    38. Klein B, Richards D. A brief Internet-based treatment for panic disorder. Behavioural and Cognitive Psychotherapy 2001;29(1):113-117. [CrossRef]
    39. Klein B, Richards JC, Austin DW. Efficacy of internet therapy for panic disorder. J Behav Ther Exp Psychiatry 2006 Sep;37(3):213-238. [Medline] [CrossRef]
    40. Richards JC, Klein B, Austin DW. Internet cognitive behavioural therapy for panic disorder. Does the inclusion of stress management information improve end-state functioning? Clin Psychol 2006;10(1):2-15. [CrossRef]
    41. Andersson G, Carlbring P, Holmström A, Sparthan E, Furmark T, Nilsson-Ihrfelt E, et al. Internet-based self-help with therapist feedback and in vivo group exposure for social phobia: a randomized controlled trial. J Consult Clin Psychol 2006 Aug;74(4):677-686. [Medline] [CrossRef]
    42. Carlbring P, Gunnarsdóttir M, Hedensjö L, Andersson G, Ekselius L, Furmark T. Treatment of social phobia: randomised trial of internet-delivered cognitive-behavioural therapy with telephone support. Br J Psychiatry 2007 Feb;190(2):123-128 [FREE Full text] [Medline] [CrossRef]
    43. Titov N, Andrews G, Schwencke G, Drobny J, Einstein D. Shyness 1: distance treatment of social phobia over the Internet. Aust N Z J Psychiatry 2008 Jul;42(7):585-594. [Medline]
    44. Titov N, Andrews G, Schwencke G. Shyness 2: treating social phobia online: replication and extension. Aust N Z J Psychiatry 2008 Jul;42(7):595-605. [Medline]
    45. Hirai M, Clum GA. An Internet-based self-change program for traumatic event related fear, distress, and maladaptive coping. J Trauma Stress 2005 Dec;18(6):631-636. [Medline] [CrossRef]
    46. Knaevelsrud C, Maercker A. Internet-based treatment for PTSD reduces distress and facilitates the development of a strong therapeutic alliance: a randomized controlled clinical trial. BMC Psychiatry 2007;7(1):13 [FREE Full text] [Medline] [CrossRef]
    47. Lange A, van de Ven JP, Schrieken B, Emmelkamp PM. Interapy, treatment of posttraumatic stress through the Internet: a controlled trial. J Behav Ther Exp Psychiatry 2001 Jun;32(2):73-90. [Medline] [CrossRef]
    48. Lange A, Rietdijk D, Hudcovicova M, van de Ven JP, Schrieken B, Emmelkamp PMG. Interapy: a controlled randomized trial of the standardized treatment of posttraumatic stress through the internet. J Consult Clin Psychol 2003 Oct;71(5):901-909. [Medline] [CrossRef]
    49. Houck PR, Mazumdar S, Koru-Sengul T, Tang G, Mulsant BH, Pollock BG, et al. Estimating treatment effects from longitudinal clinical trial data with missing values: comparative analyses using different methods. Psychiatry Res 2004 Dec 15;129(2):209-215. [Medline] [CrossRef]
    50. Schafer JL. Analysis of Incomplete Multivariate Data. London, UK: Chapman and Hall; 1997.
    51. Hunot V, Churchill R, Silva de Lima M, Teixerira V. Psychological therapies for generalised anxiety disorder. Cochrane Database Syst Rev 2007;1 CD001848.
    52. Meichenbaum D, Turk DC. Facilitating Treatment Adherence: A Practitioner’s Guidebook. New York, NY: Plenum Press; 1987.
    53. Waller R, Gilbody S. Barriers to the uptake of computerized cognitive behavioural therapy: a systematic review of the quantitative and qualitative evidence. Psychol Med 2009 May;39(5):705-712. [Medline] [CrossRef]
    54. Hunsley J, Mash EJ. Evidence-based assessment. Annu Rev Clin Psychol 2007;3(1):29-51. [Medline] [CrossRef]
    55. Bower P, Gilbody S, Richards D, Fletcher J, Sutton A. Collaborative care for depression in primary care. Making sense of a complex intervention: systematic review and meta-regression. Br J Psychiatry 2006 Dec;189(6):484-493 [FREE Full text] [Medline] [CrossRef]
    56. Robertson L, Smith M, Castle D, Tannenbaum D. Using the Internet to enhance the treatment of depression. Australas Psychiatry 2006 Dec;14(4):413-417. [Medline] [CrossRef]
    57. Brawley LR, Culos-Reed SN. Studying adherence to therapeutic regimens: overview, theories, recommendations. Control Clin Trials 2000;21(5 Suppl):156S-163S.
    58. Corritore CL, Kracher B, Wiedenbeck S. On-line trust: concepts, evolving themes, a model. Int J Hum Comput Stud 2003;58(6):737-758. [CrossRef]
    59. Sun PC, Cheng HK. The design of instructional multimedia in e-Learning: a media richness theory-based approach. Computers and Education 2007;49(3):662-667. [CrossRef]


    Abbreviations

    CBT: cognitive behavioural therapy
    GAD: generalized anxiety disorder
    ITT: intention to treat
    LLM: linear mixed modelling
    LOCF: last observation carried forward
    MAR: missing at random
    MI: multiple imputation
    ML: maximum likelihood methods
    NMAR: not missing at random
    PD: panic disorder
    PTSD: post-traumatic stress disorder
    RCT: randomized controlled trials
    REML: restricted maximum likelihood estimation
    SP: social phobia
    WHO: World Health Organization


    Edited by G Eysenbach; submitted 08.01.09; peer-reviewed by B Danaher, P Cuijpers; comments to author 29.01.09; revised version received 11.03.09; accepted 17.03.09; published 24.04.09

    © Christensen H, Griffiths KM, Farrer L. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 24.04.2009.  

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.