JMIR Publications

Select Journals for Content Updates

When finished, please click submit.

This paper is in the following e-collection/theme issue:

    Original Paper

    Development and Initial Evaluation of an Internet-Based Support System for Face-to-Face Cognitive Behavior Therapy: A Proof of Concept Study

    1Psychology, Department of Behavioral Sciences and Learning, Linköping University, Linköping, Sweden

    2Psykologpartners W&W AB, Linköping, Sweden

    3Psychiatry Section, Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden

    *all authors contributed equally

    Corresponding Author:

    Kristoffer NT Månsson, MSc

    Psychology

    Department of Behavioral Sciences and Learning

    Linköping University

    Building I:3 415

    Campus Valla

    Linköping,

    Sweden

    Phone: 46 (0)70 5085082

    Fax:46 (0)13 282145

    Email:


    ABSTRACT

    Background: Evidence-based psychological treatments, such as cognitive behavior therapy (CBT), have been found to be effective in treating several anxiety and mood disorders. Nevertheless, issues regarding adherence are common, such as poor patient compliance on homework assignments and therapists’ drifting from strictly evidence-based CBT. The development of Internet-delivered CBT (ICBT) has been intensive in the past decade and results show that guided ICBT can be as effective as face-to-face CBT but also indicate a need to integrate the two forms of CBT delivery.

    Objective: In this study, we developed and tested a new treatment format in which ICBT and face-to-face therapy were blended. We designed a support system accessible via the Internet (using a computer or an Apple iPad) for patients and therapists delivering CBT face-to-face. The support system included basic CBT components and a library of interventions gathered from existing ICBT manuals.

    Methods: The study involved 15 patients with mild to moderate anxiety or depression (or both). Eight therapists conducted the treatments. All participants were interviewed after the nine-week intervention. Further, patients provided self-reports on clinical measures pre- and post-trial, as well as at a 12-month follow-up.

    Results: A reduction was found in symptom scores across all measures. The reliable change index ranged from 60% to 87% for depression and anxiety. Large effect sizes (Cohen’s d) ranging from 1.62 (CI 95% 0.59-2.66) to 2.43 (CI 95% 1.12-3.74) were found. There were no missing data and no treatment dropouts. In addition, the results had been maintained at the 12-month follow-up. Qualitative interviews revealed that the users perceived the support system as beneficial.

    Conclusions: The results suggest that modern information technology can effectively blend with face-to-face treatments and be used to facilitate communication and structure in therapy, thus reducing therapist drift.

    J Med Internet Res 2013;15(12):e280)

    doi:10.2196/jmir.3031

    KEYWORDS



    Introduction

    The lifetime prevalence of mood and anxiety disorders is high, ranging from 20.8% to 28.8% [1], and these disorders are associated with significantly reduced quality of life in sufferers [2]. According to the World Health Organization (WHO), by the year 2030 depression is expected to be the largest contributor to disease burden [3]. Thus, there is a need for cost-effective and accessible treatments. Evidence-based treatments exist and both pharmacotherapy and psychological treatments (mainly cognitive behavior therapy; CBT) have established evidence bases [4]. However, access to CBT, often the preferred treatment option [5], is limited; thus, a gap remains between demand for treatment and actual provision [6]. Moreover, evidence-based psychological treatments do not help all patients and issues regarding adherence to therapy are common [7]. Dropout from treatment, poor engagement, and incomplete adherence to homework assignments are all issues that can reduce the treatment’s effectiveness [8]. Further, therapist drift from evidence-based CBT (toward supportive therapy) has been highlighted as a potential obstacle to effective CBT [9]. Hence, ways to optimize the delivery of CBT, improve response rates, and increase the treatment’s availability represent important goals for the field.

    Research on Internet-delivered cognitive behavior therapy (ICBT) has been intensive in the past decade and studies indicate that clinician-guided ICBT can be as effective as face-to-face CBT for some diagnoses [10,11]. ICBT has been developed for several mood and anxiety disorders [12,13] and has also been implemented in regular clinical psychiatric settings [14]. Several clear advantages can be noted when comparing ICBT with traditional, face-to-face CBT [15]. Obviously, less time is needed when the treatment is delivered via the Internet [16]; hence, more patients can be reached at a lower cost. In addition, the rationale and content of ICBT adhere to treatment manuals closely and, even though there is a role for the therapist in guided ICBT [17], this treatment format is probably less susceptible to therapist drift. On the other hand, some problems with ICBT should be noted. Therapies with minimal support may lead to increased dropout rates (particularly in unguided ICBT) [18], and some patients probably need more support than ICBT provides in order to adhere to the therapy [19]. Face-to-face meetings with patients have the advantage of providing additional clinically relevant information that can be used for diagnostic purposes and individual case management. Moreover, when patients need support in order to understand therapeutic techniques, face-to-face contact is beneficial (as a treatment technique can be illustrated directly during the session).

    Based on our research experience examining both treatment-delivery methods—face-to-face and via the Internet—we sought to merge the two to create a blended treatment format. Previous studies have combined computerized manuals with face-to-face treatment [20,21], used technological adjuncts in therapy [22], or used the Internet after treatment completion to help prevent relapse [23,24]. To our knowledge, this is the first study to merge ICBT and face-to-face CBT in a system used by both the therapist and the patient. Accordingly, we constructed a Web platform accessible via personal computer and Apple iPad and blended ICBT with traditional face-to-face CBT. This case series pilot study aimed to explore clinical outcomes on measures of anxiety, depression, and quality of life when Internet-delivered support was provided to patients and their therapists. In addition, we also examined user experiences and the ways that the support system was used and perceived. We hypothesized that the support system would contribute positively to adherence and that outcomes would align with those achieved in face-to-face CBT. This study was designed as a proof-of-concept study, generating ideas for further research.


    Methods

    Procedure

    The local ethics committee approved the study (ID: 2011/456-31). Fifteen participants fulfilling all inclusion criteria were notified via email of their acceptance and gave written informed consent prior to treatment. The treatment duration was between eight and nine weeks. In combination with the last session, a qualitative interview was conducted with the patient. On this occasion, each patient was also directed to a website where he or she was to fill out self-report questionnaires representing the clinical outcome measures. At post-treatment, seven therapists participated in a focus group discussing user experiences. One individual interview was conducted with another therapist. In addition, long-term follow-up clinical data were obtained at a 12-month follow-up with each patient.

    The Support System

    The platform (support system) utilized in the study was conceived and designed by the first and the last authors, KM and GA. A programmer wrote the code and used the open-source code language PHP and MySQL to construct the platform, which was accessible via personal computer or Apple iPad through an encrypted secure sockets layer (SSL) connection to the Internet. The support system was not developed as a mobile phone application. Users were assigned personal log-in IDs via email and an additional temporary password was sent via SMS (short message service) at each attempt to log on (this feature was implemented in order to increase the level of security). The platform was to be used at home and in the clinical setting where treatment was provided. The platform was set up so that the user would receive part of the information that would have been provided in a face-to-face treatment session—in other words, there were no computerized treatment components and the main part of the system consisted of material that in a face-to-face session would have been presented on printed paper or verbally (eg, goal setting). The therapists and patients did not access the same content in the platform; therapists controlled which support resources and information patients had access to. The platform contained some basic components of CBT, such as scheduling visits, keeping an agenda, and setting goals. The platform also included a library with both text and media resources used in psychoeducation and as homework assignments. These resources were compiled primarily from prior studies on ICBT for anxiety [25] and depression [26]; they were presented not as discrete treatments but rather as part of the face-to-face treatment (ie, as handouts). Use of text material in face-to-face treatment has been found to be common among CBT practitioners [27] and we regarded this procedure as aligning with standard CBT. The digital library contained supplemental information on CBT, such as behavioral activation, activity scheduling, exposure therapy, common cognitive biases, and maintenance via safety behaviors. We also provided some audio files, such as relaxation instructions. In addition, the platform included common questionnaires and forms used in homework assignments, such as guides to creating a fear hierarchy and keeping daily thought records and sleep diaries.

    In addition, the users could communicate via an internal message system and the therapists could communicate via mobile phone SMS. However, communication between users was essentially made within the support system. Users could also create memos about topics they wished to remember or discuss in therapy and they were able to upload and share personal files. The Web platform was built to give support to both therapists and patients in the delivery of face-to-face CBT. For further details, see Figure 1 and Multimedia Appendix 1.

    Figure 1. Screenshot of the Internet-based support system.
    View this figure

    Recruitment

    The study involved 23 participants in total: 15 patients and 8 therapists. Patients were recruited mainly via local media advertising and information distributed on a university campus. Participants declared their interest via a research project website and provided informed consent. Therapists were recruited separately via email.

    Participants

    The patients’ mean age was 43 years old (SD 15) and ages ranged from 22 to 70 years. A majority of participating patients (60%, 9/15) were currently university students or previously had been. Table 1 presents the demographic characteristics and Table 2 the distribution of psychiatric diagnoses. Of the eight therapists who were involved in the study, seven had recently received basic psychotherapy training as part of the final year of their five-year clinical psychology program. One therapist was in his first year of clinical practice as a psychologist. The mean time of experience in CBT under supervision was 14 months (range 4-26 months). All the therapists volunteered and did not receive any compensation.

    A total of 26 patients were interviewed via telephone using the structured diagnostic psychiatric interview for Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) and International Classification of Diseases (ICD-10): Mini-International Neuropsychiatric Interview (MINI [28]). Two psychology students in their final term conducted the clinical interviews. Both students had experience with diagnostic interviews and basic training in CBT. After the interviews, participants were instructed to fill out self-reports measuring anxiety, depression, quality of life, and alcohol consumption via a secured and encrypted website.

    The following participant inclusion criteria were applied in the study: (1) over 18 years of age, (2) no history of ongoing suicidal attempts or current tendency toward self-harm, (3) no current alcohol or drug addiction, (4) no current contact with a physician administering pharmacotherapy that required monitoring, (5) if the patient was on pharmacotherapy, the dose had been stable for at least one month before the study began and the participant agreed to keep the dosage unchanged during the study or inform the therapist if a change was needed, (6) not receiving other concurrent psychological treatment, (7) not diagnosed with severe major depression disorder according to MADRS-S (Montgomery Åsberg Depression Rating Scale − self-report) (cut-off at >30 points), (8) reported <4 on MADRS-S item 9 (about suicidal ideations), (9) not suffering from a severe psychiatric condition that could interfere with the treatment (eg, bipolar disorder or schizophrenia, as reported in the clinical interview), and (10) access to the Internet via a computer. A senior researcher (licensed psychologist and psychotherapist) reviewed participants to determine their eligibility for inclusion.

    Table 1. Demographic characteristics (N=15).
    View this table
    Table 2. The distribution of diagnoses according to the MINI psychiatric interviews (N=15).
    View this table

    Cognitive Behavior Therapy

    The individualized CBT was delivered during a period of eight to nine weeks. The treatments were based on generic CBT components, such as case conceptualization, behavior analysis, agenda setting, goal setting, psychoeducation, and homework assignments. Therapists received clinical group supervision from a licensed clinical psychologist during four scheduled appointments during the trial. In addition, on four occasions the therapists and the research project manager met and discussed experiences and feedback regarding technical issues. Each therapist and patient pair individually decided how to use the platform during the treatment—for instance, during therapy sessions and in additional contact between sessions.

    Clinical Measures

    We used the Beck Anxiety Inventory (BAI) [29] and the Generalized Anxiety Disorder Screener-7 (GAD-7) [30] to measure anxiety. The Montgomery Åsberg Depression Rating Scale-self-rating version (MADRS-S) [31] and the Patient Health Questionnaire-9 (PHQ-9) [32] were used to measure symptoms of depression. In addition, the Quality of Life Inventory (QOLI) [33] was administered. All clinical measures have established good psychometric properties when administered via the Internet [34,35]. We also used the Alcohol Use Disorder Identifications Test (AUDIT) [36] for screening purposes but not as an outcome measure.

    Statistical Analysis

    Data were analyzed with SPSS version 21. All the statistical analyses were intent-to-treat and statistical significance was set at P<.05. Differences between pre- and post-treatment measures and pre-treatment and long-term follow-up measures of depression, anxiety, and quality of life were examined using paired sample t tests. Within-group effect sizes were calculated based on the pooled standard deviation, expressed as Cohen’s d.

    We used a reliable change index (RC) in accordance with Jacobson and Truax [37] to determine the proportion of participants who showed treatment improvements from pre- to post-treatment and from pre-treatment to long-term follow-up. As suggested by Lambert and Ogles [38], we used internal consistency as a measure of reliability. The following values were used in the calculation of RC: Cronbach alpha=.92 for BAI, GAD-7, and PHQ-9 [29,30,39]; Cronbach alpha=.87 for MADRS-S [40]; and Cronbach alpha=.83 for QOLI [33].

    Qualitative Analysis

    This part of the study was based on content analysis in order to analyze the data [41]. Individual interviews and the focus group were recorded and transcribed. All interviews were recorded using a portable digital recorder. The purpose of the interviews was to identify user experiences of the platform used in the treatment. Meaningful units from the text based on the research question were coded and organized into categories. Sentences that reflected opinions, such as adjectives and words that expressed an emotive experience were selected. Meaningful units and user experiences were sorted as positive, negative, and neutral. Topics that were mentioned frequently and explicitly became the basis for the categorization. Each patient interview (n=15) was merged and overarching themes were created (see Table 3). Overarching themes were formulated in an ongoing discussion between two researchers, facilitating the development of supplementary views and critical perspectives. The same procedure of qualitative analysis was applied in analyzing both the focus group discussions and the individual therapists’ comments.

    Table 3. Overarching themes identified in the qualitative interviews with patients and therapists.
    View this table

    Results

    Clinical Outcome Measures

    This study had no missing data and no treatment dropouts. Statistically significant main effects were obtained for all clinical outcome measures, pre- to post-treatment, t14=4.25 to 7.25, all P’s<.001, and pre- to long-term follow-up, t14=3.53 to 6.20, all P’s<.05 (see also Table 4 and Figures 2-4). Moreover, comparing mean values post-treatment to long-term follow-up were nonsignificant: t14=0.73 to 1.55, all P’s>.142. Large within-group effect sizes (Cohen’s d) were observed on all clinical outcome measures, with pre- to post-effect sizes ranging from 1.26-2.43 and long-term follow-up between 1.08-1.94 (Table 4). Reliable change as defined by Jacobson and Truax [34] was observed in a majority of the patients, ranging from 60% to 87% on depression and anxiety measures and from 53% to 80% at long-term follow-up (Table 4). Reliable change on quality of life was 40% at post-treatment and 60% at 12-month follow-up.

    Table 4. Mean, SD, effect size, and reliable change index at pre-treatment, post-treatment, and at long-term follow-up to treatment.
    View this table
    Figure 2. Histograms showing clinical measures of anxiety. Mean values at pre-treatment, post-treatment, and at 12-month follow-up. Error bars represent the standard deviation.
    View this figure
    Figure 3. Histograms showing clinical measures of depression. Mean values at pre-treatment, post-treatment, and at 12-month follow-up. Error bars represent the standard deviation.
    View this figure
    Figure 4. Histogram showing clinical measures on quality of life. Mean values at pre-treatment, post-treatment, and at 12-month follow-up. Error bars represent the standard deviation.
    View this figure

    Patients’ User Experiences

    The patients reported that they had used the various featured components of the support system. A majority said that the platform was easy to use; however, two patients with less computer experience had used the platform only sporadically. No patient regarded the iPad as a disturbing feature of the therapy. Thirteen patients reported that they had carried out all or almost all their homework assignments; two patients reported that they had carried out more than half their assignments. The availability of assignments on the Internet was described as easy, convenient, and modern, and patients indicated that it prevented avoidance of feared situations (eg, in exposure therapy tasks). About half the patients appreciated having online feedback between face-to-face sessions. One patient had more frequent contact with the therapist during a period of challenging homework assignments. Some also found the process of writing about their problems to be helpful. Most of the patients expressed largely positive opinions about sharing information and homework over the Internet. One-third of the patients also reported that they spent additional time reading other information on the Internet related to the therapy. The content in the library was perceived as a supplement to the treatment, promoting theoretical understanding and enhanced knowledge. Several patients highlighted the use of the platform as a memory aid. One patient described the educational advantages of obtaining information through several media, such as speech, writing, and images. Another reported advantage of the Internet-based platform was the patients’ ability to return to previous assignments and reflect on their experience of change, which was perceived as motivating. A few patients indicated that the treatment contributed to increased participation and responsibility for their treatment and that both the therapist and patient collaboratively created the homework assignments with the help of the support system.

    The treatment was said to facilitate the generalization of treatment to the patients’ everyday lives. Several patients perceived the treatment duration as longer and more extensive than they had expected a nine-week intervention to be. About one-third of the patients expressed ambivalence when it came to continued work on their own. One patient also suggested developing this concept so that it could be used after treatment in order to maintain treatment gains.

    Therapists’ User Experiences

    The self-reported estimation of time spent with each therapy and week ranged between 15 and 180 minutes (mean 77, SD 57). Seven therapists (87.5%, 7/8) estimated that they spent less or as much time in these treatments, compared to treatments without the platform. The therapists accessed the platform on average seven times per patient per week. However, this number also includes the pre-treatment phase, during which the therapists were introduced to the support system. The therapists confirmed that they had used all the components of the platform. Internal messages and correspondence via SMS were frequently used. Messages contained clarifications and reformulations of homework assignments, providing support, encouragement, and answers to patients’ questions. Correspondence via the platform was also used to remind patients about assignments. This resulted in the use of the platform when therapy sessions were cancelled. The therapists were able to help the patients relatively rapidly instead of waiting until the next appointment. This may have contributed to the therapy’s continuity, allowing therapists to prepare the face-to-face sessions based on the results of a patient’s weekly assignments. One therapist also highlighted that the platform made it possible to focus more on the relational aspects of treatment during sessions. A majority of the therapists highlighted these possibilities as major benefits. Using the platform also contributed to structure in therapy. The structure was obtained when both therapist and patient could agree on and set an agenda, goals, and the central focus of homework assignments. The library was considered a source of knowledge and therapy-skill enhancement; it supported therapists in choosing among interventions and was helpful when therapists presented the treatment rationale. However, this easy access also led therapists to rely more on the content of the library material and to seek fewer other sources for help and supervision. The iPad was used to take notes, to draw, and to take pictures and videos during therapy sessions that could later be shared via the platform. Using the iPad during face-to-face sessions could potentially be clumsy and problematic, creating a barrier between therapist and patient, in particular if the therapist’s technical experience was limited. But using the iPad to take pictures of drawings on the wallboard was highlighted as beneficial. The patients’ level of computer experience was mentioned as important by the therapists, even if technology was not seen as a problem for a particular patient.


    Discussion

    Principal Findings

    The aim of this study was to investigate the combination of face-to-face CBT and an Internet-based support system. Overall, the therapies led to improvements on clinical measures and the platform was perceived as helpful. Clinical outcome data showed significant main effects and reliable changes on all measures and these results had been maintained at 12-month follow-up. The findings on the whole align with previous findings regarding computerized support in combination with face-to-face CBT [42]. The patients’ user experiences show that the platform and the iPad were regarded as beneficial. Several patients stated that they gained more from the treatment than they had originally expected to during the nine-week period. However, this might also be explained by the therapists’ level of involvement and may not directly relate to the availability of the platform. Obviously, some therapists were involved in the research process, but all the therapists had limited clinical experience and therefore it is reasonable to assume that the support system benefitted the delivery of the therapy. Indeed, this is a research question on its own, given that online therapist education is only in its infancy [43,44] and that direct support for novice therapists via the Internet is even less well researched. We also hypothesize that this support system might be a useful instrument for supervising therapists. If psychotherapy supervision focuses on enhancing interpersonal therapist skills, the support system, which emphasizes the importance of formulation and homework assignments, can serve as a complement in supervision.

    Therapists found several major benefits to delivering treatment in this format—for instance, improved between-session communication and ease of sharing treatment-related information. These inputs highlight some important implications and advantages over traditional face-to-face treatments. Replacing missed sessions with Internet-delivered interventions adds continuity and the treatment thus becomes less vulnerable to factors such as physical health and logistical problems, such as travel distance. Further, this may have implications for structuring treatment, for it demonstrates the importance of the therapists being keen to set the agenda and it shows that homework assignments must be formulated clearly to provide clear, educative information. This study had no missing data, nor any dropouts from treatment. Hence, it is plausible that communication via the Internet between therapy sessions mitigates dropout rates. Further, our results suggest that Internet-based treatment delivery might decrease tendencies toward therapist drift [9] and might increase adherence to CBT treatment manuals.

    Limitations

    Among the limitations of the present study is that the sample size was low, thus reducing statistical power. We did not include a wait list or an active control group and therefore had no control for time and repeated testing, which limits the generalizability of the findings. Also, the study design does not allow us to explore different factors that explain the outcome of the study. It is possible that certain features in the system, ie, stand-alone SMS communication between therapy sessions, could achieve similar clinical outcome results.

    It is noteworthy that two therapists in the study were also involved in the research process and evaluation of the system. Even if our procedure is more biased than if independent researchers had conducted the interviews, the qualitative method, content analysis, provided us with the possibility of reviewing the comments and views of the participants.

    The therapists in this study had relatively little experience delivering CBT and the included patients were recruited via media advertisement. Therefore, this method might not be equally applicable in a representative clinical setting. Moreover, some possible obstacles to the support platform emerged in this study. Limited computer experience is likely to make this treatment delivery format less efficient in some situations; indeed, not all the patients found the platform very easy to use. However, only two participants reported having limited computer experience and we did not include any additional evaluative questions. This implies that future studies should address the issue of computer experience in greater detail.

    Conclusions

    In sum, this study indicates that Internet-based support can be used without compromising the effects of CBT and that it in fact provides a vehicle for communication and for the structure of the therapy. This may be important for new therapists in training, but it is also potentially valuable to experienced therapists, who can have a tendency to drift away from the treatment manuals. Future studies should investigate the relative merits of adding a support system to face-to-face therapy and whether doing so facilitates retention and long-term learning. Studies are also needed to investigate the use of Internet-delivered support in more regular clinical settings with practicing CBT therapists.

    Acknowledgments

    GA conceived the idea for the project, contributed to the manuscript, and received funding support for the project (professor’s contract). KM contributed by designing the platform, managing the project, and contributing to the manuscript. MD participated as a clinical supervisor. ESR and EG contributed as therapists, conducted the pre- and post-treatment clinical interviews, and made major contributions to the manuscript. We would like to thank the therapists who volunteered to take part in the study: Daniel Goine, Frida Gustafsson, Nils Johansson, Peter Molander, Pontus Jonsson, and Sahar Gaveli. Finally, we gratefully acknowledge the work of our technician and programmer, Fredrik Nordell.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Screenshots from the therapist’s view of the platform. When the patient enters the support system, the view shows only information that is relevant to him or her. The therapist controls the visibility of this content.

    PDF File (Adobe PDF File), 1MB

    References

    1. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry 2005 Jun;62(6):593-602. [CrossRef] [Medline]
    2. Saarni SI, Suvisaari J, Sintonen H, Pirkola S, Koskinen S, Aromaa A, et al. Impact of psychiatric disorders on health-related quality of life: general population survey. Br J Psychiatry 2007 Apr;190:326-332 [FREE Full text] [CrossRef] [Medline]
    3. World Health Organization. The global burden of disease: 2004 update. 2008.   URL: http://www.who.int/healthinfo/global_burden_disease/GBD_report_2004update_full.pdf [accessed 2013-10-07] [WebCite Cache]
    4. Cuijpers P, Sijbrandij M, Koole SL, Andersson G, Beekman AT, Reynolds CF. The efficacy of psychotherapy and pharmacotherapy in treating depressive and anxiety disorders: a meta-analysis of direct comparisons. World Psychiatry 2013 Jun;12(2):137-148 [FREE Full text] [CrossRef] [Medline]
    5. van Schaik DJ, Klijn AF, van Hout HP, van Marwijk HW, Beekman AT, de Haan M, et al. Patients' preferences in the treatment of depressive disorder in primary care. Gen Hosp Psychiatry 2004;26(3):184-189. [CrossRef] [Medline]
    6. Kohn R, Saxena S, Levav I, Saraceno B. The treatment gap in mental health care. Bull World Health Organ 2004;82(11):858-866.
    7. Haynes RB, McDonald HP, Garg AX. Helping patients follow prescribed treatment: clinical applications. JAMA 2002 Dec 11;288(22):2880-2883. [Medline]
    8. Burns DD, Nolen-Hoeksema S. Coping styles, homework compliance, and the effectiveness of cognitive-behavioral therapy. J Consult Clin Psychol 1991 Apr;59(2):305-311. [Medline]
    9. Waller G. Evidence-based treatment and therapist drift. Behav Res Ther 2009 Feb;47(2):119-127. [CrossRef] [Medline]
    10. Andersson G. Guided internet treatment for anxiety disorders. As effective as face-to-face therapies? Stud Health Technol Inform 2012;181:3-7. [Medline]
    11. Cuijpers P, Donker T, van Straten A, Li J, Andersson G. Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychol Med 2010 Dec;40(12):1943-1957. [CrossRef] [Medline]
    12. Andersson G, Carlbring P, Ljótsson B, Hedman E. Guided Internet-Based CBT for Common Mental Disorders. J Contemp Psychother 2013 May 21;43(4):223-233. [CrossRef]
    13. Hedman E, Ljótsson B, Lindefors N. Cognitive behavior therapy via the Internet: a systematic review of applications, clinical efficacy and cost-effectiveness. Expert Rev Pharmacoecon Outcomes Res 2012 Dec;12(6):745-764. [CrossRef] [Medline]
    14. Andersson G, Hedman E. Effectiveness of guided Internet-based cognitive behavior therapy in regular clinical settings. Verhaltenstherapie 2013;23(3):140-148. [CrossRef]
    15. Andersson G. The promise and pitfalls of the internet for cognitive behavioral therapy. BMC Med 2010;8:82 [FREE Full text] [CrossRef] [Medline]
    16. Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis. PLoS One 2010 Oct;5(10):e13196 [FREE Full text] [CrossRef] [Medline]
    17. Paxling B, Lundgren S, Norman A, Almlöv J, Carlbring P, Cuijpers P, et al. Therapist behaviours in internet-delivered cognitive behaviour therapy: analyses of e-mail correspondence in the treatment of generalized anxiety disorder. Behav Cogn Psychother 2013 May;41(3):280-289. [CrossRef] [Medline]
    18. Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [CrossRef] [Medline]
    19. Spek V, Cuijpers P, Nyklícek I, Riper H, Keyzer J, Pop V. Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis. Psychol Med 2007 Mar;37(3):319-328. [CrossRef] [Medline]
    20. Craske MG, Rose RD, Lang A, Welch SS, Campbell-Sills L, Sullivan G, et al. Computer-assisted delivery of cognitive behavioral therapy for anxiety disorders in primary-care settings. Depress Anxiety 2009;26(3):235-242 [FREE Full text] [CrossRef] [Medline]
    21. Wright JH, Wright AS, Albano AM, Basco MR, Goldsmith LJ, Raffield T, et al. Computer-assisted cognitive therapy for depression: maintaining efficacy while reducing therapist time. Am J Psychiatry 2005 Jun;162(6):1158-1164. [CrossRef] [Medline]
    22. Clough BA, Casey LM. Technological adjuncts to increase adherence to therapy: a review. Clin Psychol Rev 2011 Jul;31(5):697-710. [CrossRef] [Medline]
    23. Ebert D, Tarnowski T, Gollwitzer M, Sieland B, Berking M. A transdiagnostic internet-based maintenance treatment enhances the stability of outcome after inpatient cognitive behavioral therapy: a randomized controlled trial. Psychother Psychosom 2013;82(4):246-256. [CrossRef] [Medline]
    24. Holländare F, Johnsson S, Randestad M, Tillfors M, Carlbring P, Andersson G, et al. Randomized trial of Internet-based relapse prevention for partially remitted depression. Acta Psychiatr Scand 2011 Oct;124(4):285-294. [CrossRef] [Medline]
    25. Furmark T, Holmström A, Sparthan E, Carlbring P, Andersson G. Social fobi - Effektiv hjälp med kognitiv beteendeterapi Social Anxiety - Effective treatment using cognitive behavioural therapy. 1st edition. Lund: Liber; 2006.
    26. Andersson G, Bergström J, Holländare F, Carlbring P, Kaldo V, Ekselius L. Internet-based self-help for depression: randomised controlled trial. Br J Psychiatry 2005 Nov;187:456-461 [FREE Full text] [CrossRef] [Medline]
    27. Keeley H, Williams C, Shapiro DA. A United Kingdom survey of accredited cognitive behaviour therapists' attitudes towards and use of structured self-help materials. Behav. Cognit. Psychother 2002 Apr 16;30(02):193-203. [CrossRef]
    28. Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. The Mini-International Neuropsychiatric Interview (MINI): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry 1998;59 Suppl 20:22-33;quiz 34. [Medline]
    29. Beck AT, Epstein N, Brown G, Steer RA. An inventory for measuring clinical anxiety: psychometric properties. J Consult Clin Psychol 1988 Dec;56(6):893-897. [Medline]
    30. Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097. [CrossRef] [Medline]
    31. Svanborg P, Asberg M. A new self-rating scale for depression and anxiety states based on the Comprehensive Psychopathological Rating Scale. Acta Psychiatr Scand 1994 Jan;89(1):21-28. [Medline]
    32. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [Medline]
    33. Frisch MB, Cornell J, Villanueva M, Retzlaff PJ. Clinical validation of the Quality of Life Inventory. A measure of life satisfaction for use in treatment planning and outcome assessment. Psychological Assessment 1992;4(1):92-101. [CrossRef]
    34. Hedman E, Ljótsson B, Rück C, Furmark T, Carlbring P, Lindefors N, et al. Internet administration of self-report measures commonly used in research on social anxiety disorder: A psychometric evaluation. Computers in Human Behavior 2010 Jul;26(4):736-740. [CrossRef]
    35. Thorndike FP, Carlbring P, Smyth FL, Magee JC, Gonder-Frederick L, Ost L, et al. Web-based measurement: Effect of completing single or multiple items per webpage. Computers in Human Behavior 2009 Mar;25(2):393-401. [CrossRef]
    36. Saunders JB, Aasland OG, Babor TF, de la Fuente JR, Grant M. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO Collaborative Project on Early Detection of Persons with Harmful Alcohol Consumption--II. Addiction 1993 Jun;88(6):791-804. [Medline]
    37. Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 1991 Feb;59(1):12-19. [Medline]
    38. Lambert MJ, Ogles BM. Using clinical significance in psychotherapy outcome research: the need for a common procedure and validity data. Psychother Res 2009 Jul;19(4-5):493-501. [CrossRef] [Medline]
    39. Cameron IM, Crawford JR, Lawton K, Reid IC. Psychometric comparison of PHQ-9 and HADS for measuring depression severity in primary care. Br J Gen Pract 2008 Jan;58(546):32-36 [FREE Full text] [CrossRef] [Medline]
    40. Carlbring P, Brunt S, Bohman S, Austin D, Richards J, Öst L, et al. Internet vs. paper and pencil administration of questionnaires commonly used in panic/agoraphobia research. Computers in Human Behavior 2007 May;23(3):1421-1434. [CrossRef]
    41. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005 Nov;15(9):1277-1288. [CrossRef] [Medline]
    42. Roy-Byrne P, Craske MG, Sullivan G, Rose RD, Edlund MJ, Lang AJ, et al. Delivery of evidence-based treatment for multiple anxiety disorders in primary care: a randomized controlled trial. JAMA 2010 May 19;303(19):1921-1928 [FREE Full text] [CrossRef] [Medline]
    43. Kobak KA, Craske MG, Rose RD, Wolitsky-Taylor K. Web-based therapist training on cognitive behavior therapy for anxiety disorders: a pilot study. Psychotherapy (Chic) 2013 Jun;50(2):235-247 [FREE Full text] [CrossRef] [Medline]
    44. Sholomskas DE, Syracuse-Siewert G, Rounsaville BJ, Ball SA, Nuro KF, Carroll KM. We don't train in vain: a dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. J Consult Clin Psychol 2005 Feb;73(1):106-115 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    AUDIT: Alcohol Use Disorder Identifications Test
    BAI: Beck Anxiety Inventory
    CBT: cognitive behavior therapy
    DSM-IV: Diagnostic and Statistical Manual of Mental Disorders, 4th Edition
    GAD-7: Generalized Anxiety Disorder−seven item version
    ICBT: Internet-based cognitive behavior therapy
    ICD-10: International Statistical Classification of Diseases and Related Health Problems, version 10
    MADRS-S: Montgomery Åsberg Depression Rating Scale−self-report version
    MySQL: My Structured Query Language
    PHP: hypertext preprocessor
    PHQ-9: Patient Health Questionnaire−nine-item version
    QOLI: Quality of Life Inventory
    RC: reliable change index
    RCT: randomized controlled trial
    SMS: short message service
    SNRI: serotonin and norepinephrine reuptake inhibitors
    SSL: secure sockets layer
    SSRI: selective serotonin reuptake inhibitors
    WHO: World Health Organization


    Edited by D Mohr; submitted 17.10.13; peer-reviewed by J Wright, L Palmili, K Kobak; comments to author 04.11.13; revised version received 19.11.13; accepted 21.11.13; published 10.12.13

    ©Kristoffer NT Månsson, Erica Skagius Ruiz, Elisabet Gervind, Mats Dahlin, Gerhard Andersson. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 09.12.2013.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.