Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, December 24 through Wednesday, December 26 inclusive. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 05.02.19 in Vol 21, No 2 (2019): February

Preprints (earlier versions) of this paper are available at, first published May 25, 2018.

This paper is in the following e-collection/theme issue:


    Design and Delivery Features That May Improve the Use of Internet-Based Cognitive Behavioral Therapy for Children and Adolescents With Anxiety: A Realist Literature Synthesis With a Persuasive Systems Design Perspective

    1Department of Pediatrics, University of Alberta, Edmonton, AB, Canada

    2Centre for Research in Family Health, IWK Health Centre, Halifax, NS, Canada

    3Department of Psychology, Dalhousie University, Halifax, NS, Canada

    4Department of Pediatrics, Dalhousie University, Halifax, NS, Canada

    5Deparment of Psychiatry, Dalhousie University, Halifax, NS, Canada

    6Department of Community of Health and Epidemiology, Dalhousie University, Halifax, NS, Canada

    7Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada

    Corresponding Author:

    Amanda S Newton, PhD

    Department of Pediatrics

    University of Alberta

    3-526 Edmonton Clinic Health Academy

    11405 - 87 Avenue

    Edmonton, AB, T6G 1C9


    Phone: 1 780 248 5581



    Background: Internet-based cognitive behavioral therapy (iCBT) is a persuasive system as its design combines therapeutic content, technological features, and interactions between the user and the program to reduce anxiety for children and adolescents. How iCBT is designed and delivered differs across programs. Although iCBT is considered an effective approach for treating child and adolescent anxiety, rates of program use (eg, module completion) are highly variable for reasons that are not clear. As the extent to which users complete a program can impact anxiety outcomes, understanding what iCBT design and delivery features improve program use is critical for optimizing treatment effects.

    Objective: The objectives of this study were to use a realist synthesis approach to explore the design and delivery features of iCBT for children and adolescents with anxiety as described in the literature and to examine their relationship to program use outcomes.

    Methods: A search of published and gray literature was conducted up to November 2017. Prespecified inclusion criteria identified research studies, study protocols, and program websites on iCBT for child and adolescent anxiety. Literature was critically appraised for relevance and methodological rigor. The persuasive systems design (PSD) model, a comprehensive framework for designing and evaluating persuasive systems, was used to guide data extraction. iCBT program features were grouped under 4 PSD categories—Primary task support, Dialogue support, System credibility support, and Social support. iCBT design (PSD Mechanisms) and delivery features (Context of use) were linked to program use (Outcomes) using meta-ethnographic methods; these relationships were described as Context-Mechanism-Outcome configurations. For our configurations, we identified key PSD features and delivery contexts that generated moderate-to-high program use based on moderate-to-high quality evidence found across multiple iCBT programs.

    Results: A total of 44 documents detailing 10 iCBT programs were included. Seven iCBT programs had at least one document that scored high for relevance; most studies were of moderate-to-high methodological rigor. We developed 5 configurations that highlighted 8 PSD features (Tailoring, Personalization [Primary task supports]; Rewards, Reminders, Social role [Dialogue supports]; and Trustworthiness, Expertise, Authority [System credibility supports]) associated with moderate-to-high program use. Important features of delivery Context were adjunct support (a face-to-face, Web- or email-based communications component) and whether programs targeted the prevention or treatment of anxiety. Incorporating multiple PSD features may have additive or synergistic effects on program use.

    Conclusions: The Context-Mechanism-Outcome configurations we developed suggest that, when delivered with adjunct support, certain PSD features contribute to moderate-to-high use of iCBT prevention and treatment programs for children and adolescents with anxiety. Standardization of the definition and measurement of program use, formal testing of individual and combined PSD features, and use of real-world design and testing methods are important next steps to improving how we develop and deliver increasingly useful treatments to target users.

    J Med Internet Res 2019;21(2):e11128





    Cognitive behavioral therapy (CBT) is recommended as a first-line treatment for children and adolescents with anxiety [1-4]. Trained mental health professionals have traditionally delivered CBT, but there is an increasing interest in the internet as a delivery platform to circumvent the multiple barriers to receiving in-person treatment. These barriers can include direct and incidental costs to families, lack of trained deliverers, and inconvenient service times and locations [5]. Internet-based CBT (iCBT) is also proposed to preserve adolescent autonomy, appeal to user preferences, reduce health care system costs, and improve the time it takes to receive treatment [6-8]. iCBT is recognized as an important treatment option [9-11] to meet the increasing demands of a population where anxiety is often undetected and untreated, but with whom early access to care can improve long-term outcomes [3,12-14].

    iCBT uses technological features (ie, multimedia and email) to deliver treatment content through the Web or a software application to generate interactions between the user and the program [15]. In this way, iCBT aims to reinforce, change, or shape attitudinal or behavioral health outcomes and aligns with the concept of a persuasive system [16-18]. Although recent efforts have been made to provide guidance on the design and delivery features of iCBT [19], considerable differences exist across programs both in terms of their features and the health outcomes they produce. Recent systematic reviews and meta-analyses have found a range of iCBT programs to be effective at reducing anxiety in children and adolescents [9,10,20,21]; however, poor and highly variable rates of completion can be found across programs (up to 50% of participants not reaching the end of a program) [5,7,10,21,22]. The term program use captures the various, typically objective, outcomes used across studies (ie, adherence, compliance, and number of program activities or homework completed) that describe the extent to which users interact with a program.

    Understanding the factors that influence iCBT program use is important as there have been indications that greater program use is associated with better outcomes [23,24]. Studies of iCBT in children and adolescents with anxiety [25,26] have found that certain participant demographics (eg, gender, age, location, and anxiety severity) and delivery features (eg, parental or therapist support) relate to or predict program use. However, the relationship between technological design features of a program, the ability of those features to enhance the persuasiveness of a program, and actual iCBT program use by children and adolescents with anxiety, has received minimal attention in the literature.


    Recognizing iCBT as a persuasive system, we conducted a realist synthesis to examine the technological design and program delivery features of iCBT for children and adolescents with anxiety in order to document their potential relation to persuading program use. The realist synthesis approach provided a framework to answer 2 main questions: (1) what design and delivery components (technological features, treatment content, and interactions) are described for iCBT programs for children and adolescents with anxiety? and (2) what components may explain reported program use outcomes?


    Study Design

    This realist synthesis was conducted using steps recommended by Pawson and Tilley [27,28] and is reported in accordance with the Realist and Meta-narrative Evidence Synthesis: Evolving Standards (RAMESES II) [29]. Realist synthesis is theory-driven in that the synthesis searches for and refines explanations of intervention effects by asking: “What works, for whom, and in what circumstances?” [27]. In this synthesis, we examined the relationships that exist between the therapeutic and technological features of iCBT (Mechanisms) and program use (Outcomes), and the program delivery formats and interactions (Contexts) that support them. We expressed these relationships as Context-Mechanism-Outcome configurations.

    Context-Mechanism-Outcome Configuration Development

    We began the synthesis by developing candidate Context-Mechanism-Outcome configurations for how iCBT programs may work. The development process consisted of brainstorming activities, with the research team reviewing literature on human-technology interaction and studies of iCBT programs for anxiety to identify relevant and pre-existing theories, models, or frameworks to work from. The persuasive systems design (PSD) model [17] emerged as a key framework for understanding how iCBT, as a persuasive system, was intended to work, and we used this model to develop the initial list of Context-Mechanism-Outcome configurations. The model describes 28 PSD (technological) features, subdivided across the following 4 categories, which can be implemented by programs to guide the user toward their health-related goal: (1) Primary task support, (2) Dialogue support, (3) System credibility support, and (4) Social support.

    Using the PSD model, we identified which PSD features (Mechanisms) might be associated with iCBT program use (Outcomes) to formulate Mechanism-Outcome dyads. We then hypothesized which program delivery formats, interactions, and conditions for use (Context) might promote the occurrence of the Mechanism-Outcome interactions. Together, these steps led to the generation of 5 candidate Context-Mechanism-Outcome configurations (Multimedia Appendix 1) [30-34]. The configurations were as comprehensive and justifiable as possible, referencing literature that supported their development and inclusion in the list. These configurations would undergo refinement and testing during the analysis stage of the synthesis, whereby we used evidence from the literature to validate their explanatory usefulness and applicability for answering our research questions.

    Literature Search

    We used 3 main strategies to identify literature for iCBT programs. The first search strategy involved an information specialist conducting a systematic and comprehensive search of 8 electronic databases: MEDLINE, EMBASE, ERIC, PsycINFO, CINAHL, Cochrane Library, ProQuest Dissertations & Theses Global, and PubMed for the period 1990 to 2017. The second strategy involved a manual search using Google, an internet search engine, and gray literature repositories (Association for Computing Machinery Digital Library, Open Grey, Institute of Electrical and Electronics Engineers Digital Library, and Canadian Agency for Drugs and Technologies in Health) to identify conference proceedings, program evaluations, and government or technical reports. For both search strategies, MeSH terms and text words were based on mental health condition (anxiety and phobias), intervention modality (internet-based and mobile app), intervention type (prevention and treatment), and therapeutic approach (CBT; Multimedia Appendix 2). The third strategy involved manually searching the table of contents in the Journal of Medical Internet Research, Internet Interventions, Journal of Cyber Therapy & Rehabilitation, and Journal of Telemedicine and Telecare, and a review of reference lists of included documents and reviews (eg, systematic reviews).

    We employed the search strategies on an iterative and recurrent basis until November 2017 to ensure the review was up to date and inclusive. Before discontinuing the literature search, a test of saturation was applied to the search strategies, which involved verifying that further searching would not yield any new results [35].

    Literature Selection

    In this study, 2 independent, trained reviewers (authors ADR and LW) screened the identified documents for eligibility using a 2-stage approach. During this process, reviewer discrepancies were resolved by consensus or third party arbitration (author ASN). At stage 1, all documents were screened for eligibility using the title and abstract. During this stage, we randomly selected 100 citations to assess inter-rater agreement for inclusion or exclusion decisions; Cohen kappa was 0.74 between raters, reflecting substantial agreement [36]. All documents that were screened “yes, include” or “unsure to include” moved to stage 2. At stage 2, the full text of documents was reviewed by 1 reviewer (ADR), in consultation with another (ASN), with a resulting decision to either include or exclude a document from the synthesis.

    For an iCBT program to be included in this synthesis, supporting documents needed to be published in English and provide information on treatment Context, program design (PSD) and delivery features, and program use Outcomes. Each document did not need to provide details on all 3, but all 3 needed to be represented in the total documents for an iCBT program. In addition, at least one published study on the iCBT program needed to be available for inclusion so that we could assess the methodological quality of the study providing program use outcome data.

    Intervention studies (eg, clinical trials) were eligible for inclusion if they evaluated iCBT anxiety programs with children (aged <14 years) or adolescents (aged 12-19 years). As some iCBT programs were designed for and evaluated with participants from a broader age range (eg, programs also geared toward young adults), only those studies that provided separate data for participants aged ≤19 years were included. We also required that the type of iCBT program under evaluation be designed for an anxiety disorder(s) or anxiety symptoms associated with a disorder as classified according to the Diagnostic and Statistical Manual of Mental Disorders, fifth edition [37] such as social phobia (social anxiety disorder), generalized anxiety disorder, panic disorder, separation anxiety disorder, or specific phobia. Transdiagnostic programs (ie, programs designed for anxiety plus another diagnosis such as depression) were also eligible for inclusion. We also required that the iCBT program consisted of modules designed for use by the child or adolescent (and not solely delivered to or facilitated by a parent or therapist) as child or adolescent program use was our outcome of interest. Theoretical papers, mixed-methods and qualitative studies, and policy or implementation documents were also eligible if they included a focus on how an iCBT program was proposed to work.

    Literature Appraisal

    Documents were assessed for relevance and rigor based on consensus between 2 reviewers (authors ADR and LW). Relevance was assessed based on the level of contribution a document provided for an iCBT program in 3 domains: (1) underpinning theory and/or the context and sequence for delivery (Context), (2) PSD features (Mechanism), and (3) program use outcomes (Outcomes). The level of contribution for each domain was rated low if little or no information was provided, medium if some information was provided, and high if information was well described. Exemplar documents with a high level of contribution across the 3 domains informed decision rules for the rating of all other documents.

    The methodological quality (rigor) of research studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [38,39]. The MMAT is a reliable, efficient, and valid tool that provides different sections for assessing studies of qualitative, randomized, nonrandomized studies, descriptive studies, and mixed-method designs [38-40]. Multiple publications of the same study (thesis + journal publication) received the same MMAT score. MMAT scores could range from 0% to 100%, with a greater score indicating that more quality criteria were met.

    Data Extraction and Coding

    Documents for each iCBT program were grouped together during data extraction and coding. Two reviewers (ADR and LW) cross-referenced extraction and coding decisions, with a random subset of 10 documents; the remaining documents were coded by 1 reviewer (ADR). In addition to document characteristics (type and year of publication), iCBT program data were extracted for the following:

    • Participants: eligibility criteria and participant demographics.
    • iCBT program Context: number of modules, module workflow and sequence, delivery setting, adjunct support, and level of prevention according to the Institute of Medicine model [41].
    • Theory or proposed Mechanisms: program features, content and components, including PSD features, and information on why the iCBT program was designed a certain way or how the program was proposed to work.
    • Program use Outcomes: information related to how many Web-based modules, exercises, or activities were completed by users or how many users completed certain aspects of the program, measured at posttreatment.
    Context and Mechanism Data

    Adjunct support details were coded when human-derived technological or therapeutic communication was provided to users to complement iCBT program delivery.

    Therapeutic content in programs was coded according to the 5 main CBT components found in the American Academy of Child and Adolescent Psychiatry (AACAP) practice parameter [42]: psychoeducation, somatic management skills, cognitive restructuring, exposure methods, and relapse prevention.

    As most authors did not use PSD terminology or concepts, program descriptions were coded as PSD features using a codebook and glossary [17] (Multimedia Appendix 3). PSD features were coded when they were executed by the technology (intrinsic to the design and delivery of the internet-based program) and not by human action (eg, congratulatory comments provided in person by a parent or teacher), which is in line with the use of the PSD model by others [15]. When available, suggested mediators and moderators of program use were extracted, as was information on partial or full Context-Mechanism-Outcome configurations, as discussed by the original authors of the included documents.

    Outcome Data

    We found inconsistent and heterogeneous measurement and reporting of program use Outcomes—such as program adherence, compliance, and completion. These limitations have been noted by others [22].

    For each study, program use Outcomes are reported as published by original authors and are collectively referred to in this study using the umbrella term program use. As no applicable cut-off scores have yet been established, when possible, Outcomes were converted into percentages (based on quartiles) to assist with interpreting program use. We used the following parameters to summarize program use: high use (≥75%), moderate use (50%-74%), low use (25%-49%), or very low use (≤24%). Study dropout or attrition data were not included in the analysis because these data may not directly reflect program use (eg, program completion), but rather rates of study participants who did not fulfill the research protocol (eg, filling out questionnaires) [15]. Corresponding authors were contacted to provide clarity and completeness of unclear or unreported information and to ensure accurate application of the PSD model for coding iCBT program features. An author for each of the included programs responded to our requests (n=10).

    Data Analysis and Synthesis Process

    We used a multistep approach to data analysis that was structured according to Pawson’s techniques [27,28] and meta-ethnography [43,44]. The first step involved determining recurrent patterns or themes (demi-regularities) across documents for each iCBT program for delivery Context, PSD features and program Mechanisms, and Outcomes related to program use. The purpose was to use evidence from the literature to (1) identify PSD Mechanisms in each program most frequently associated with the program use Outcomes to refine the candidate Mechanism-Outcome dyads, and (2) incorporate delivery Context of each iCBT program into the dyads to refine the overall Context-Mechanism-Outcome configurations. Context-Mechanism-Outcome configurations that were supported by evidence from at least two different iCBT programs progressed to the next step of analysis.

    The second step in the analysis involved reciprocal translation analysis, a meta-ethnographic technique that involved reviewing the Context-Mechanism-Outcome configurations across iCBT programs [44,45]. Configurations that were found to have mixed (ie, more heterogeneous support with no larger trend) or confounding evidence across programs, or could not be refined by better describing or recombining the Context, Mechanism or Outcome factors, did not progress to the next stage of analysis. What remained were configurations that provided the best support, across multiple programs, to explain the relationship of design and delivery components of iCBT with program use.

    In the final step, we used lines-of-argument synthesis, a theorizing technique [44] that involved assessing how well each Context-Mechanism-Outcome configuration could explain why the same PSD Mechanism(s), operating in different iCBT program Contexts, might result in particular program use Outcomes. We took into consideration the quality and quantity of evidence supporting the configuration and held a meeting to discuss, amend, and finalize configurations with individuals from across Canada with expertise in electronic health interventions. Configuration refinement continued until we felt that it reflected a pattern that would remain consistent despite differences in small- or large-scale details across iCBT programs. At that point in time, we considered the configuration to be adequately developed.


    Included Documents

    The literature search and selection progress are presented in Figure 1. The search strategy yielded 11,511 unique documents for stage 1 screening, after duplicates were removed. Of these, 801 documents underwent stage 2 screening. In total, 44 documents detailing 10 iCBT programs were included in the realist synthesis. Documents were published studies (n=20), theses (n=5), published or registered protocols for trials (n=12), study or program websites (n=6), and a study flyer (n=1).

    Characteristics of Internet-Based Cognitive Behavioral Therapy for Anxiety in Children and Adolescents

    Program and Participant Characteristics

    Table 1 presents an overview of iCBT program and user characteristics. The majority of child and adolescent users were white, English speakers, of middle-to-high socioeconomic status, who lived in urban centers with both biological parents. Programs designed to treat an anxiety disorder tended to be longer in duration (included more modules) than prevention-based programs. Treatment-based programs were delivered in the community (some included occasional health care clinic visits) and involved weekly Web- or email-based therapist interaction and parent-dedicated modules. Prevention-based programs were often provided in schools, with a teacher facilitating program administration. Most iCBT programs were adaptations of previously developed mental health prevention or treatment resources [46-55].

    Figure 1. Flow diagram of the literature search and selection process. iCBT: internet-based cognitive behavioral therapy.
    View this figure
    Table 1. Overview of the internet-based cognitive behavioral therapy user, program, and delivery characteristics.
    View this table
    Figure 2. Overview of the persuasive systems design features across the 10 internet-based cognitive behavioral therapy programs included in the synthesis.
    View this figure
    Cognitive Behavioral Therapy Components and Persuasive Systems Design Features

    All programs described themselves as CBT-based and contained at least 3 of the 5 AACAP-recommended CBT components (most commonly psychoeducation, somatic management skills, and cognitive restructuring). The workflow of the programs presented the more foundational and simpler treatment components (eg, psychoeducation and symptom identification) before more challenging skills (eg, desensitization or exposure). Many programs also integrated interpersonal therapy skills [57], such as assertiveness training and problem solving, to reduce environmental stressors and enhance social support.

    The frequency of the PSD features used in the iCBT programs is shown in Figure 2. All programs incorporated Reduction and Tunneling (Primary task supports) to regulate the pace and sequence by which users moved through the program. Self-monitoring, a Primary task support, was also used in all programs to track users’ progress or mood over time. A total of 8 programs used Rehearsal by providing recommended or required Web-based homework or practice activities. Moreover, 7 programs provided a Personalized review (eg, progress report and feedback) of homework or module content before the next module was accessible to users. Most programs used Tailoring (n=6) or Personalization (n=7) (Primary task supports) to adapt the content to users’ characteristics such as their primary anxiety concern, age, or name. Incorporating a Social role, such as a Web-based or virtual therapist or cartoon guide, was one of the most commonly used Dialogue supports (n=8). In addition, 9 of the 10 programs had program content, technology, and interaction features that were relatable and appealing to target users (Similarity and Liking features). As all programs were part of a research study, Trustworthiness and Surface credibility (System credibility supports) were considered inherent to their design (ie, programs were ad-free, not marketed for commercial use, and accessed through a secure platform), although few documents explicitly reported this. Authority (System credibility support) was incorporated when Web- or email-based therapist support was provided. Social support features were seldom used among programs.

    Level of Contribution and Methodological Quality

    Details of the quality appraisal are provided in Multimedia Appendix 4 [58-75]. Documents tended to provide the greatest level of contribution to understanding program Context and Outcomes with relatively few details available for understanding program Mechanisms. A total of 7 iCBT programs had at least one document that scored high for level of contribution for program Context, Mechanisms, and Outcomes. We evaluated 29 research studies (found in 25 documents) for methodological rigor. A total of 19 studies met all 4 MMAT criteria (100%), 7 met 3 criteria (75%), and 3 met 2 criteria (50%). Lower ratings reflected an unclear description of: processes for recruitment, selection, randomization, or allocation; how group differences (if any) were controlled for; the percentage of outcome data obtained; or withdrawal or dropout rates were not within acceptable limits.

    Summary of Key Internet-Based Cognitive Behavioral Therapy Contexts, Mechanisms, and Outcomes

    Tables 2 and 3 present an overview of the delivery Context and PSD Mechanisms that were most frequently or consistently associated with moderate-to-high program use Outcomes across iCBT programs. Contexts most indicative of program use were the adjunct support person and the communication approach (eg, Web [the program platform or internal messaging system], email, or in-person) provided to the user—both of which typically varied based on the level of prevention the program was designed for. The type of adjunct support also depended on the age of program users. Children generally received the most extensive adjunct support (ie, therapist and parent), and program use was often greater among this age group than among adolescents. PSD features identified as having a high value to encouraging program use were Tailoring and Personalization (Primary task supports); Social role, Reminders, and Rewards (Dialogue supports); and Authority, Expertise, and Trustworthiness (System credibility supports). Program use Outcomes most commonly reported either the total proportion or average proportion of program modules, homework, or activities completed by users. More than half of the Outcomes indicated high or moderate program use.

    When finalizing our proposed configurations, we considered both what program Contexts and Mechanisms were combined (using Tables 2 and 3) and in what way and for what purpose they may have been combined (using excerpts from individual documents; see Table 4) to explain how moderate-to-high program use Outcomes were generated.

    Proposed Internet-Based Cognitive Behavioral Therapy Contexts and Mechanisms for Moderate-to-High Program Use

    Key Contexts and PSD Mechanisms that may have led to moderate-to-high iCBT program use Outcomes are described in Table 4 alongside examples from contributing programs.

    Context-Mechanism-Outcome Configuration 1: Tailoring and Personalization

    Evidence from 8 iCBT programs suggested that indicated prevention and treatment programs that provided adjunct support along with Tailoring and Personalization resulted in greater program use. Studies supporting this configuration had a mean MMAT score of 90.2%.

    Tailoring and Personalization were used to provide users with more information to increase the individualized feel of the program and portray that the program knew and could meet the user’s needs. Programs most commonly tailored content (ie, therapeutic elements and examples) based on the user’s age or mental health condition (eg, specific phobic and social anxiety). User information was often collected by the adjunct support person during study enrollment (ie, part of eligibility screening). Personalization was a feature that could be initiated through program automation (ie, user’s name appeared on the home screen; his or her pronouns) or via program communications (ie, individualized weekly emails and secure messages). The adjunct support person acted as an extension of the Tailored or Personalized program experience through their contact with users within (eg, by providing personalized feedback on Web-based homework [50,76,83]) or outside of the iCBT program (eg, by tailoring in-person session objectives [55]). Programs with a combination of Tailoring and Personalization reported some of the highest program use Outcomes.

    Context-Mechanism-Outcome Configuration 2: Reminders

    Evidence from 6 iCBT programs suggested that indicated prevention and treatment programs that provided adjunct therapist support along with Reminders also resulted in greater program use. Studies supporting this configuration had a mean MMAT score of 92.1%.

    The programs contributing to this Context-Mechanism-Outcome configuration involved multiple modules or activities (the number of modules ranged from 6-11 or app use ranged from daily-weekly activities); therefore, users were required to log into the program over numerous instances. Reminders were used to encourage the user to take program action, either by promoting skills practice [46] or by “prompt[ing] participants who [were] late in completing a module” [78] to access the newly available content. Programs provided generic and automatized Reminders through email or the Web-based platform. Reminders were also embedded in the regular, electronic, personalized communications (eg, feedback and progress check-ins) sent by the adjunct therapist to the child or adolescent. If users remained absent from the program beyond the recommended treatment schedule (eg, longer than 1 week), the adjunct therapist provided additional in-person or telephone follow-up, encouraging users to access the next available module.

    Context-Mechanism-Outcome Configuration 3: Rewards

    Evidence from 4 iCBT programs suggested that programs and mobile apps that provided adjunct support along with Rewards resulted in greater program use. Studies supporting this configuration had a mean MMAT score of 85.0%.

    Table 2. An overview of the delivery Context and persuasive systems design features that may explain program use Outcomes across internet-based cognitive behavioral therapy treatment programs.
    View this table

    A total of 3 iCBT programs regularly incorporated Rewards into modules to encourage ongoing program use and promote the completion of essential treatment exercises [46,47,51]. The iCBT program for dental anxiety opted for a final Reward and presented users with a virtual diploma at the end of their treatment [45]. Rewards were also used as a proxy to track program progress, including completion of exposure activities [46,47,51]. Unlike computer-based programs, the open and flexible design approach to mobile apps gave users the option to select what treatment content and tasks they wanted to access and when. Progressive reward incentives were used to persuade users to complete more of the app’s content and critical components. In the REACH for success app, a cartoon character provided regular guidance and feedback to users and entertained them with animations following task completion (Reward) [47]. In the SmartCAT app, a point system tied to external prizes (Reward) was a feature managed by the adjunct therapist [46]. In-person sessions with a therapist or parent also provided positive reinforcement of program use (ie, Praise, Rewards); sessions were also used to instruct users on how to incorporate Rewards into their anxiety management activities outside of the program [48,51].

    Context-Mechanism-Outcome Configuration 4: Therapist, Social Role, Authority, Expertise, and Trustworthiness

    Evidence from 6 iCBT programs suggested that indicated prevention and treatment programs with adjunct Web- or email-based therapist support that also provided a Social role component, and conveyed Authority, Expertise, Trustworthiness, resulted in greater program use. Studies supporting this configuration had a mean MMAT score of 91.3%.

    The Social role component of iCBT programs was often fulfilled by a therapist or coach (who received specialized training with the program but may not have been a licensed psychologist). Therapists engaged in regular, Web- or email-based communication with the user and served 2 roles by (1) facilitating program delivery by providing technical support and answering users’ questions and (2) promoting program completion by providing reminders and encouragement, reinforcing program concepts, and ensuring accurate comprehension and application of CBT skills. Together, the Social role feature and therapist emails complemented (had overlap with) other PSD features such as Reminders, Praise, and Suggestion. Therapists had secure access to users’ written responses or logged data so that they could send specific communications to users, demonstrating therapists’ credibility and competence with both the therapeutic process and individual responsiveness (Authority, Expertise, and Trustworthiness). Moreover, 3 studies measured child-reported therapeutic alliance with their iCBT program therapist and found it to be strong and program use to be high [76,81]. One of these studies correlated therapeutic alliance with program use and found a significant, positive relationship [81].

    Context-Mechanism-Outcome Configuration 5: Therapist + Parent, Social Role, Authority, Expertise, and Trustworthiness

    Evidence from 5 treatment programs suggested that iCBT programs with adjunct therapist and parent support that also included a Social role component, and conveyed Authority, Expertise, and Trustworthiness, had greater program use. Studies supporting this configuration had a mean MMAT score of 90.8%.

    Table 3. An overview of the delivery Context and persuasive systems design features that may explain program use Outcomes across internet-based cognitive behavioral therapy indicated prevention and universal prevention programs.
    View this table
    Table 4. Configuration summaries of the key Contexts and persuasive systems design Mechanisms that may have led to moderate-to-high program use Outcomes.
    View this table

    Treatment-based iCBT programs were designed to be child-parent combined or parent-supported child-based interventions. Combined interventions required parents to complete parent-specific modules (eg, psychoeducation, relaxation training, problem solving, and modeling adaptive behaviors) either before or alongside their child as they completed their own child-directed modules. In parent-supported interventions, parents may have also been responsible for explaining program instructions and assisting their children with their modules [50,54,76], coaching or supporting their child with in vivo exposure or practice activities [46,48,50], and overseeing their child’s treatment schedule [54]. The support the adjunct therapist provided to children was also extended to parents. Parents had the opportunity to ask clarifying questions, receive expert advice, and troubleshoot difficulties with their child’s iCBT progress with the program therapist. Studies found that both parent and child ratings of therapeutic alliance and program use were high [76,81]. One study correlated therapeutic alliance and program use and found significant, positive relationships for both parents and children [81]. Anderson et al [81] hypothesized that therapist emails may have contributed to fostering a strong therapeutic alliance.


    Principal Findings

    The role of technological, persuasive components in iCBT programs is an understudied aspect of program design and evaluation. The extent of iCBT program use may be a fundamental indication of its persuasiveness and its potential to assist the user with their goals of the program [15,88-91]. This realist synthesis identified 5 possible relationships as to how the use of specific PSD features (technological Mechanisms), supported by some key user and delivery features (Context), may generate moderate-to-high program use (Outcomes) in iCBT for children and adolescents with anxiety.

    The 5 Context-Mechanism-Outcome configurations provide support for several persuasive strategies to improve iCBT program use: Tailoring and Personalization as Primary task supports; Rewards, Reminders and Social role features in programs serving as Dialogue supports; and Trustworthiness, Expertise, and Authority features serving as System credibility supports for a program. Traditionally, PSD features that stimulate human-computer communication, such as Dialogue supports, have been the most widely used and studied features for improving program use [6,15,89,92,93]. However, this synthesis demonstrated that having credibility (System credibility supports) and supporting users in completing their target behavior (Primary task supports) may also promote moderate-to-high program use. We hypothesize that using multiple PSD features, both within and across the different support categories, may produce additive or synergistic effects on program use; however, there was insufficient evidence available for our analysis to explain the impact of PSD feature combinations that involved more than a few features (and proposed functions) at a time. This is because the authors of the original studies included in our review typically discussed or formally tested the relationship of only 1 or 2 PSD features and program use at a time. Therefore, our configurations present the fewest possible PSD features that could be linked to higher program use (ie, we uncoupled features as much as possible—an approach that may make testing of their effects more efficient in future studies).

    Moreover, we suspect that not all PSD features may have equal influence on program use. Depending on the program, some PSD features may be necessary for program use (part of the basic requirements or foundational design framework of iCBT), whereas others may be complementary to program use (have an impact by enhancing the design framework of iCBT); although both types of features together may influence program use. In this realist synthesis, all 10 iCBT programs described a purposeful design that incorporated Reduction and Tunneling (Primary task supports) and Similarity and Liking (Dialogue supports) to create a logical, incremental, relevant, and aesthetically pleasing experience for users—these may be the necessary PSD features for iCBT for children and adolescents with anxiety. The PSD features described in our configurations are hypothesized to be complementary—building on the persuasiveness of necessary PSD features to further improve or optimize iCBT program use.

    A meta-analysis of PSD features used in internet-based interventions for mental health demonstrated that determining the amount and type of persuasive principles to include may be a delicate balance, as some principles seem to work together, whereas when other principles co-occur, they may have an unapparent or diminishing effect [90]. As was found for this synthesis, it is not necessarily the number of PSD features used in iCBT, but it is their proposed purpose or implementation that is particularly critical for optimizing program use outcomes [90]. For example, when comparing 2 indicated prevention programs, we observed that the internet cognitive behavioral skills-based program [51] had fewer PSD features than the individually tailored iCBT program [55], although the former reported greater program use. At this time, our understanding of how to best bring together PSD features, such as Personalized Reminders [93], in the design and delivery of iCBT for child and adolescent anxiety is limited. Therefore, further research on the theory, function, quality, and effectiveness of individual PSD features is needed to deliberately use and combine them for idealized treatment outcomes. Moreover, involving target users in the (co)design and testing of treatments is recommended to improve the acceptability, feasibility, and effectiveness of iCBT with children and adolescents [19,94,95]. These participatory research efforts may provide important guidance on the usefulness and functionality of select (PSD) features of iCBT programs in the real world and from the user’s perspective [96]; therefore, facilitating greater program use [97].

    In this synthesis, 3 important potential relationships were identified: (1) adjunct support seemed to improve program use even when input or support was minimal (eg, in-person, classroom-based program administration with no treatment advice given) or when it was provided by a nonexpert (eg, teacher) [24,25]; (2) users of treatment programs demonstrated higher program use than users of universal prevention programs; and (3) a trend for increased program use among programs for children (more so than for adolescents) was identified. Within these relationships, multiple contextual aspects or user characteristics may have also had an additive or synergistic effect on program use. For example, the level of expertise the adjunct support person had (eg, teacher vs therapist) and the degree of their guidance or therapeutic involvement (eg, in-person program administration vs personalized feedback emails) increased from prevention-based to treatment programs. In the literature, little is known about how much, when, and what type of support is necessary for enhancing program use and efficacy [98,99]. Although some evidence suggests that layperson support is as effective as clinician support [100], this synthesis suggests that the person providing support as well as the intensity of their support activities (frequency; inclusion of therapeutic elements) may have a noteworthy effect on program use. The nature of the role adjunct support plays in iCBT program use is also unclear. It has been suggested that adjunct support may leverage the advantages of therapeutic alliance [101,102], which might include principles of persuasion (eg, users feel the need to respond to social cues [18]), it may establish process expectations and social accountability [103], or it may develop a sense of legitimacy or credibility of the program [103] (see Santarossa et al’s study [104] for further suggestions). Programs that had both therapist and parent involvement may have (1) reinforced child’s understanding of and confidence in treatment content and (2) increased the child’s interaction with the program by creating a perceived sense of cooperation (shared goals) and accountability toward treatment progress. In this realist synthesis, adjunct support may have been used to complement or replace the use of some PSD features in iCBT, particularly Dialogue supports. For example, in-person therapist sessions or telephone calls provided opportunities for Reminders, Personalized feedback, or Praise to be conveyed to users [49,50,54,55]. Consideration of how and when to provide adjunct support is critical when preparing for the implementation and integration of iCBT within routine practice, such as allotting for therapist time, making changes in clinical workflow, and when conducting economic analyses.

    Future Directions

    This realist synthesis not only provides support for incorporating some of the well-studied and highly used PSD features into iCBT (ie, Reminders [93,105,106]) but also draws attention to underutilized features that can be incorporated in the designs of new treatments. For example, Rewards were only occasionally used by iCBT programs included in this study (4 out of 10 programs) but are more commonly used techniques for increasing program use in internet-based interventions targeting physical activity or dietary behaviors [107]. Recent efforts into improving the gamification of technology-based CBT for pediatric mental health (see SPARX [108]), where incorporating game-design elements such as Praise and Rewards are regularly used to enhance program use and engagement [109], demonstrate the potential benefits that the use of these features may have.

    Although this synthesis and other recent reviews have been helpful for identifying PSD features of interest for improving program use of internet-based interventions [15,89,90], the next step is to formally isolate and evaluate the effectiveness of these PSD Mechanisms in producing optimal program use. This synthesis suggests 8 features that may be a priority for further examination. Modeling, factorial designs or the multiphase optimization strategy [110-112] (see also Baker et al’s study [113] for other suggestions) can be used to evaluate the best set (individual or combination) of program features to use under different conditions (eg, delivery setting and start or end of treatment). Studies with multiple, active treatment arms would also allow the examination of the comparative effects of select program features [114] or in different delivery contexts (ie, varying the type of adjunct support). From this synthesis, only 3 studies of 3 different programs conducted these comparisons. These studies provided important insights into the impact of delivery medium [49], type of adjunct support [24], and delivery location [25] on iCBT program use. Qualitative studies or self-report data would also provide meaningful information on the factors affecting program use from child and adolescent or health care provider standpoints. Another important line of inquiry relates to defining and measuring program use to ensure its validity and reliability for future studies. Designing studies that incorporate in vivo, objective measurements or automatic data capture of program use [115,116] could improve our awareness of program use predictors beyond user demographics (ie, age), for example, to actual usage behaviors (ie, number of Web pages viewed). This method would allow for iCBT program use to not only be measured at end of the intervention but also throughout the program access period to assess usage patterns over time [15], when certain design or delivery features may be more or less activated or present.

    Strengths and Challenges of Realist Synthesis

    This is the first study to examine PSD features as they relate to program use in iCBT for children and adolescents with anxiety. A strength of this synthesis is the inclusion of diverse and high-quality evidence (ie, MMAT scores>75% [38]) from both the published and gray literature. Approaching our research questions using a single theoretical framework (ie, the PSD model) allowed for systematic and incremental accumulation of knowledge about how iCBT may work from a trackable, technological perspective.

    The lack of operationalization of how PSD features and aspects of Context and program use Outcomes were defined, described, and measured by authors affected our data extraction and coding strategies. As adherence to recent recommendations [19] and reporting guidelines, such as Consolidated Standards of Reporting Trials-electronic health (CONSORT-EHEALTH) [117], become mandatory for publication, the opportunity to identify the active ingredients of iCBT will improve. Clarifying PSD features with original authors was an attempt to mitigate the potential bias that lies in coding technological program features and interpreting the papers using the PSD model [92]. However, few details about the time or the quality of communication by the adjunct support person(s) were available, limiting our understanding of the important role this contextual feature played in program use. For reasons of inclusivity, we described the heterogeneous outcomes using the umbrella term program use. Adoption of a recent standardized definition and calculation of adherence [118] can clarify what is meant by specific program use terms and allow for comparisons of outcomes across programs. iCBT programs’ widespread implementation and ability to meet the health goals of users will involve an understanding of the expectations and actuality of program use in the real world (ie, using true effectiveness studies or formative program evaluation), and setting benchmarks for an effective dose in different delivery settings. Finally, like others [119-121], we recognized the outcomes of persuasive systems depended on multiple factors, many of which were not examined in this synthesis. However, it was rare to have information on treatment or technology preferences of users (eg, early completers [22,122]), their psychological characteristics or cognitions (eg, motivation, personality, expectations, and treatment perceptions [123]), or personal circumstances (eg, program access [124])—factors that are also considered critical to program use and could be used to construct and validate more intricate Context-Mechanism-Outcome configurations.


    The Context-Mechanism-Outcome configurations identified by this realist synthesis provide an initial understanding of how, why, and for whom iCBT programs for children and adolescents with anxiety work from a persuasive systems’ perspective. Appreciating that the effectiveness of iCBT programs may hinge on whether and to what extent programs are used, this study is an important step toward successfully implementing and integrating iCBT into routine clinical care. Recognizing that multiple PSD features are incorporated in iCBT program designs and that individual features may affect each other differently, further knowledge and testing of the purpose and function of these features will help determine the number and combination to use in certain delivery contexts (eg, adjunct support included; level of prevention a program is designed for). As PSD features are modifiable, iCBT program designers and developers can look to create more persuasive programs that promote greater use and improved treatment outcomes.


    The authors of this study would like to thank the authors who responded to our request for more information about their iCBT program. The authors would also like to thank Robin Featherstone for developing the search strategy and conducting the electronic database search and Dr Purnima Sundar for reviewing the manuscript for clarity of content. This study was supported by a Canadian Institutes of Health Research (Knowledge Synthesis Grant CIHR: KRS2014) awarded to ASN (Principal Investigator) and an Alberta Innovates - Graduate Studentship in Health awarded to ADR (Trainee). During the conduct of this review, ASN and LH held CIHR new investigator awards and PM holds a Tier 1 Canada Research Chair in Child Health.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Candidate Context-Mechanism-Outcome configurations.

    PDF File (Adobe PDF File), 62KB

    Multimedia Appendix 2

    Document electronic search strategy.

    PDF File (Adobe PDF File), 8KB

    Multimedia Appendix 3

    The persuasive systems design (PSD) model.

    PDF File (Adobe PDF File), 98KB

    Multimedia Appendix 4

    The level of contribution and methodological quality of documents for the included internet-based cognitive behavioral therapy programs.

    PDF File (Adobe PDF File), 64KB


    1. Arnberg FK, Linton SJ, Hultcrantz M, Heintz E, Jonsson U. Internet-delivered psychological treatments for mood and anxiety disorders: a systematic review of their efficacy, safety, and cost-effectiveness. PLoS One 2014 May;9(5):e98118 [FREE Full text] [CrossRef] [Medline]
    2. Compton SN, March JS, Brent D, Albano AM, Weersing R, Curry J. Cognitive-behavioral psychotherapy for anxiety and depressive disorders in children and adolescents: an evidence-based medicine review. J Am Acad Child Adolesc Psychiatry 2004 Aug;43(8):930-959. [Medline]
    3. Albano A, Kendall P. Cognitive behavioural therapy for children and adolescents with anxiety disorders: clinical research advances. Int Rev Psychiatry 2009 Jul 11;14(2):129-134. [CrossRef]
    4. Seligman LD, Ollendick TH. Cognitive-behavioral therapy for anxiety disorders in youth. Child Adolesc Psychiatr Clin N Am 2011 Apr;20(2):217-238 [FREE Full text] [CrossRef] [Medline]
    5. Clarke AM, Kuosmanen T, Barry MM. A systematic review of online youth mental health promotion and prevention interventions. J Youth Adolesc 2015 Jan;44(1):90-113. [CrossRef] [Medline]
    6. Calear AL, Christensen H. Review of internet-based prevention and treatment programs for anxiety and depression in children and adolescents. Med J Aust 2010;192(11 Suppl):S12-S14. [Medline]
    7. Richardson T, Stallard P, Velleman S. Computerised cognitive behavioural therapy for the prevention and treatment of depression and anxiety in children and adolescents: a systematic review. Clin Child Fam Psychol Rev 2010 Sep;13(3):275-290. [CrossRef] [Medline]
    8. Skinner H, Biscope S, Poland B, Goldberg E. How adolescents use technology for health information: implications for health professionals from focus group studies. J Med Internet Res 2003 Dec 18;5(4):e32 [FREE Full text] [CrossRef] [Medline]
    9. Podina IR, Mogoase C, David D, Szentagotai A, Dobrean A. A meta-analysis on the efficacy of technology mediated CBT for anxious children and adolescents. J Rat-Emo Cognitive-Behav Ther 2015 Nov 17;34(1):31-50. [CrossRef]
    10. Vigerland S, Lenhard F, Bonnert M, Lalouni M, Hedman E, Ahlen J, et al. Internet-delivered cognitive behavior therapy for children and adolescents: a systematic review and meta-analysis. Clin Psychol Rev 2016 Dec;50:1-10 [FREE Full text] [CrossRef] [Medline]
    11. Spence S, March S, Vigerland S, Serlachius E. Internet-based therapies for child adolescent emotional behavioral problems. Guid internet-based Treat psychiatry Cham: Springer International Publishing; 2016:197-217.
    12. Essau CA, Lewinsohn PM, Olaya B, Seeley JR. Anxiety disorders in adolescents and psychosocial outcomes at age 30. J Affect Disord 2014 Jul;163:125-132 [FREE Full text] [CrossRef] [Medline]
    13. Neil AL, Christensen H. Efficacy and effectiveness of school-based prevention and early intervention programs for anxiety. Clin Psychol Rev 2009 Apr;29(3):208-215. [CrossRef] [Medline]
    14. Donovan CL, Spence SH. Prevention of childhood anxiety disorders. Clin Psychol Rev 2000 Jun;20(4):509-531. [Medline]
    15. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012 Nov;14(6):e152 [FREE Full text] [CrossRef] [Medline]
    16. Chatterjee S, Price A. Healthy living with persuasive technologies: framework, issues, and challenges. J Am Med Inform Assoc 2009 Mar;16(2):171-178 [FREE Full text] [CrossRef] [Medline]
    17. Oinas-Kukkonen H, Harjumaa M. A Systematic Framework for Designing and Evaluating Persuasive Systems. In: PERSUASIVE 2008: Persuasive Technology. Berlin, heidelberg: Springer; 2008 Presented at: International Conference on Persuasive Technology; 2008; Oulu, Finland p. 164-176. [CrossRef]
    18. Fogg BJ. Persuasive technology: using computers to change what we think and do. Ubiquity 2002 Dec 01;2002(5):-. [CrossRef]
    19. Hill C, Creswell C, Vigerland S, Nauta MH, March S, Donovan C, et al. Navigating the development and dissemination of internet cognitive behavioral therapy (iCBT) for anxiety disorders in children and young people: a consensus statement with recommendations from the #iCBTLorentz Workshop Group. Internet Interv 2018 Jun;12:1-10 [FREE Full text] [CrossRef] [Medline]
    20. Pennant ME, Loucas CE, Whittington C, Creswell C, Fonagy P, Fuggle P, et al. Computerised therapies for anxiety and depression in children and young people: a systematic review and meta-analysis. Behav Res Ther 2015 Apr;67:1-18. [CrossRef] [Medline]
    21. Rooksby M, Elouafkaoui P, Humphris G, Clarkson J, Freeman R. Internet-assisted delivery of cognitive behavioural therapy (CBT) for childhood anxiety: systematic review and meta-analysis. J Anxiety Disord 2015 Jan;29:83-92. [CrossRef] [Medline]
    22. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res 2009;11(2):e13 [FREE Full text] [CrossRef] [Medline]
    23. Spence SH, Donovan CL, March S, Kenardy JA, Hearn CS. Generic versus disorder specific cognitive behavior therapy for social anxiety disorder in youth: a randomized controlled trial using internet delivery. Behav Res Ther 2017 Dec;90:41-57. [CrossRef] [Medline]
    24. Calear AL, Batterham PJ, Poyser CT, Mackinnon AJ, Griffiths KM, Christensen H. Cluster randomised controlled trial of the e-couch Anxiety and Worry program in schools. J Affect Disord 2016 May 15;196:210-217. [CrossRef] [Medline]
    25. Neil AL, Batterham P, Christensen H, Bennett K, Griffiths KM. Predictors of adherence by adolescents to a cognitive behavior therapy website in school and community-based settings. J Med Internet Res 2009;11(1):e6 [FREE Full text] [CrossRef] [Medline]
    26. Calear A, Christensen H, Mackinnon A, Griffiths K. Adherence to the MoodGYM program: outcomes and predictors for an adolescent school-based population. J Affect Disord 2013 May;147(1-3):338-344. [CrossRef] [Medline]
    27. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 2005 Jul;10(Suppl 1):21-34. [CrossRef] [Medline]
    28. Pawson R, Tilley N. Realistic Evaluation. London: Sage; 1997.
    29. Wong G, Westhorp G, Manzano A, Greenhalgh J, Jagosh J, Greenhalgh T. RAMESES II reporting standards for realist evaluations. BMC Med 2016 Dec;14(1):96 [FREE Full text] [CrossRef] [Medline]
    30. Crutzen R, Cyr D, de Vries NK. The role of user control in adherence to and knowledge gained from a website: randomized comparison between a tunneled version and a freedom-of-choice version. J Med Internet Res 2012 Mar 09;14(2):e45 [FREE Full text] [CrossRef] [Medline]
    31. van Gemert-Pijnen JE, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011 Dec 05;13(4):e111 [FREE Full text] [CrossRef] [Medline]
    32. Ludden GD, van Rompay TJ, Kelders SM, van Gemert-Pijnen JE. How to increase reach and adherence of web-based interventions: a design research viewpoint. J Med Internet Res 2015 Jul 10;17(7):e172 [FREE Full text] [CrossRef] [Medline]
    33. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010 Feb 17;12(1):e4 [FREE Full text] [CrossRef] [Medline]
    34. Gasser R, Brodbeck D, Degen M, Luthiger J, Wyss R, Reichlin S. Persuasiveness of a mobile lifestyle coaching application using social facilitation. Berlin, Heidelberg: Springer; 2006 May 18 Presented at: International Conference on Persuasive Technology; May 18, 2006; Eindhoven, The Netherlands p. 27-38. [CrossRef]
    35. Walker JL. The use of saturation in qualitative research. Can J Cardiovasc Nurs 2012 Apr;22(2):37-41. [Medline]
    36. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med 2012 Oct;22(3):276-282 [FREE Full text] [Medline]
    37. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 5th Edition: DSM-5. Washington: American Psychiatric Pub; 2013.
    38. Pluye P, Gagnon M, Griffiths F, Johnson-Lafleur J. A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews. Int J Nurs Stud 2009 Apr;46(4):529-546. [CrossRef] [Medline]
    39. Pluye P, Robert E, Cargo M, Bartlett G. Mixed Methods Appraisal Tool. 2011 Jan 19. Proposal: A mixed methods appraisal tool for systematic mixed studies reviews   URL: http:/​/mixedmethodsappraisaltoolpublic.​​w/​file/​fetch/​84371689/​MMAT%202011%20criteria%20and%20tutorial%202011-06-29updated2014.​08.​21.​pdf [WebCite Cache]
    40. Pace R, Pluye P, Bartlett G, Macaulay AC, Salsberg J, Jagosh J, et al. Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. Int J Nurs Stud 2012 Jan;49(1):47-53. [CrossRef] [Medline]
    41. Institute of Medicine. In: Mrazek PJ, Haggerty RJ, editors. Reducing Risks for Mental Disorders: Frontiers for Preventive Intervention Research. Washington: The National Academies Press; 1994.
    42. Connolly SD, Bernstein GA, Work Group on Quality Issues. Practice parameter for the assessment and treatment of children and adolescents with anxiety disorders. J Am Acad Child Adolesc Psychiatry 2007 Feb;46(2):267-283. [CrossRef] [Medline]
    43. Atkins S, Lewin S, Smith H, Engel M, Fretheim A, Volmink J. Conducting a meta-ethnography of qualitative literature: lessons learnt. BMC Med Res Methodol 2008 Apr 16;8(1):21 [FREE Full text] [CrossRef] [Medline]
    44. Noblit GW, Hare RD. Meta-Ethnography: Synthesizing qualitative studies. Newbury Park, California: SAGE; 1988.
    45. Byng R, Norman I, Redfern S. Using realistic evaluation to evaluate a practice-level intervention to improve primary healthcare for patients with long-term mental illness. Evaluation 2016 Jul 24;11(1):69-93. [CrossRef]
    46. Pramana G, Parmanto B, Kendall PC, Silk JS. The SmartCAT: an m-health platform for ecological momentary intervention in child anxiety treatment. Telemed J E Health 2014 May;20(5):419-427 [FREE Full text] [CrossRef] [Medline]
    47. Patwardhan M. Assessing the impact of usability design features of an mHealth app on clinical protocol compliance using a mixed methods approach. Arizona State University: ProQuest Dissertations Publishing; 2016.   URL: [accessed 2018-11-28] [WebCite Cache]
    48. Shahnavaz S. Karolinska Institutet. 2016 Dec 02. Cognitive behavioral therapy for children and adolescents with dental phobia   URL: https:/​/openarchive.​​xmlui/​bitstream/​handle/​10616/​45319/​Thesis_Shervin_Shahnavaz.​pdf?sequence=5&isAllowed=y
    49. Spence SH, Holmes JM, March S, Lipp OV. The feasibility and outcome of clinic plus internet delivery of cognitive-behavior therapy for childhood anxiety. J Consult Clin Psychol 2006 Jun;74(3):614-621. [CrossRef] [Medline]
    50. Vigerland S, Thulin U, Ljótsson B, Svirsky L, Ost L, Lindefors N, et al. Internet-delivered CBT for children with specific phobia: a pilot study. Cogn Behav Ther 2013;42(4):303-314. [CrossRef] [Medline]
    51. Keller ML. An Internet Cognitive-Behavioral Skills-Based Program for Child Anxiety. Charleston, SC: Proquest, Umi Dissertation Publishing; 2011.
    52. Calear AL, Christensen H, Griffiths KM, Mackinnon A. The Y-Worri Project: study protocol for a randomised controlled trial. Trials 2013 Mar 19;14:76 [FREE Full text] [CrossRef] [Medline]
    53. Calear AL, Christensen H, Mackinnon A, Griffiths KM, O'Kearney R. The YouthMood Project: a cluster randomized controlled trial of an online cognitive behavioral program with adolescents. J Consult Clin Psychol 2009 Dec;77(6):1021-1032. [CrossRef] [Medline]
    54. Vigerland S. Karolinska Institutet. 2015. Internet-delivered CBT for children with anxiety disorders: effect and prerequisites for implementation within public health care   URL: https:/​/openarchive.​​xmlui/​bitstream/​handle/​10616/​44877/​Thesis_Sarah__Vigerland.​pdf?sequence=1&isAllowed=y
    55. Silfvernagel K, Gren-Landell M, Emanuelsson M, Carlbring P, Andersson G. Individually tailored internet-based cognitive behavior therapy for adolescents with anxiety disorders: a pilot effectiveness study. Internet Interv 2015 Sep;2(3):297-302. [CrossRef]
    56. Calear AL, Batterham PJ, Griffiths KM, Christensen H. Generalized anxiety disorder stigma in adolescents: personal and perceived stigma levels and predictors. Stigma and Health 2017 Aug;2(3):208-215. [CrossRef]
    57. Lipsitz JD, Markowitz JC. Mechanisms of change in interpersonal therapy (IPT). Clin Psychol Rev 2013 Dec;33(8):1134-1147 [FREE Full text] [CrossRef] [Medline]
    58. Spence S. Australian New Zealand Clinical Trials Registry (ANZCTR). 2011. Factors predicting therapy outcome in online cognitive behaviour therapy (with minimal therapist assistance) in the treatment of anxiety disorders among children and adolescents   URL: [accessed 2017-11-15]
    59. Spence S. Australian New Zealand Clinical Trials Registry (ANZCTR). 2011. A comparison of the effects of online versus clinic-based delivery of CBT in reducing anxiety disorders and anxiety symptoms among clinically anxious adolescents   URL: [accessed 2017-11-15]
    60. Stasiak K. Australian New Zealand Clinical Trials Registry (ANZCTR). 2012. Open trial of the effectiveness of BRAVE - an internet-based cognitive behavioural therapy for children with anxiety following the Canterbury earthquakes   URL:
    61. Spence SH. The University of Queensland. 2017 Jan 01. BRAVE-Online - Helping young people overcome anxiety   URL: [accessed 2018-11-28] [WebCite Cache]
    62. Shahnavaz S. 2015. Internet-based cognitive behavioral therapy for children with dental anxiety   URL: [WebCite Cache]
    63. Serlachius E. Internet-delivered cognitive behavior therapy (CBT) for children age 8-12 years with anxiety disorders   URL: [WebCite Cache]
    64. Serlachius E. 2014. Internet-delivered treatment for children with anxiety disorders in a rural area; An open trial in a clinical setting   URL: [WebCite Cache]
    65. Serlachius E. 2016. Internet-based CBT for children with anxiety disorders: implementation in clinical settings   URL: [WebCite Cache]
    66. Vigerland S. 2016. BIP in Jämtland Härjedalen: increased access to CBT within regular health care in northern Sweden   URL: [WebCite Cache]
    67. Stockholm’s Child and Adolescent Psychiatry (BUP). 2013. [About BUP]   URL: [WebCite Cache]
    68. Silk J. 2014. Smartphones to enhance the treatment of childhood anxiety (Smart-CAT)   URL: [WebCite Cache]
    69. Parmanto B, Silk J, Pramana G. The Office of the National Coordinator for Health Information Technology (ONC). 2013. SmartCAT: Smartphone-enhanced Child Anxiety Treatment   URL: https:/​/behavioralhealth.​​submissions/​17038-smartcat-smartphone-enhanced-child-anxiety-treatment [accessed 2018-03-13] [WebCite Cache]
    70. Keller M. Anxiety and Depression Research Center at UCLA. 2009. Internet skills-based program for child anxiety   URL: [accessed 2018-11-28] [WebCite Cache]
    71. Keller M, Craske M. Anxiety and Depression Research Center at UCLA. 2009. Internet cognitive-behavioral skills-based program study flyer   URL: [WebCite Cache]
    72. Calear A. Australian New Zealand Clinical Trials Registry (ANZCTR). 2010. Effect of the e-coucAnxiety and Worry program on generalised anxiety disorder in an adolescent school-based population   URL: [WebCite Cache]
    73. Griffiths K, Tayler G, Christensen H. Australian National University. 2017. e-couch Self Help   URL: [WebCite Cache]
    74. Christensen H, Griffiths K. ehub Health. 2017. MoodGYM CBT training program   URL: [WebCite Cache]
    75. Spence S. Australian New Zealand Clinical Trials Registry (ANZCTR). 2011. Is disorder-specific CBT more effective than transdiagnostic CBT in reducing social phobia in the online-delivered treatment of socially phobic children and adolescents?   URL: [accessed 2017-11-15]
    76. Spence S, Donovan C, March S, Gamble A, Anderson R, Prosser S, et al. Online CBT in the treatment of child and adolescent anxiety disorders: issues in the development of BRAVE-ONLINE and two case illustrations. Behav Cognit Psychother 2008 Jun 25;36(04):411-430. [CrossRef]
    77. March S. Investigation of an internet-based treatment for childhood anxiety (BRAVE for Children-ONLINE): An evaluation of efficacy and factors associated with treatment outcome. St. Lucia, Queensland: The University of Queensland; 2008.
    78. March S, Spence SH, Donovan CL. The efficacy of an internet-based cognitive-behavioral therapy intervention for child anxiety disorders. J Pediatr Psychol 2009 Jun;34(5):474-487 [FREE Full text] [CrossRef] [Medline]
    79. Stasiak K, Merry SN, Frampton C, Moor S. Delivering solid treatments on shaky ground: feasibility study of an online therapy for child anxiety in the aftermath of a natural disaster. Psychother Res 2018 Jul;28(4):643-653. [CrossRef] [Medline]
    80. Spence SH, Donovan CL, March S, Gamble A, Anderson RE, Prosser S, et al. A randomized controlled trial of online versus clinic-based CBT for adolescent anxiety. J Consult Clin Psychol 2011 Oct;79(5):629-642. [CrossRef] [Medline]
    81. Anderson RE, Spence SH, Donovan CL, March S, Prosser S, Kenardy J. Working alliance in online cognitive behavior therapy for anxiety disorders in youth: comparison with clinic delivery and its role in predicting outcome. J Med Internet Res 2012;14(3):e88 [FREE Full text] [CrossRef] [Medline]
    82. Conaughton RJ, Donovan CL, March S. Efficacy of an internet-based CBT program for children with comorbid High Functioning Autism Spectrum Disorder and anxiety: a randomised controlled trial. J Affect Disord 2017 Aug;218:260-268. [CrossRef]
    83. Vigerland S, Ljótsson B, Thulin U, Öst L, Andersson G, Serlachius E. Internet-delivered cognitive behavioural therapy for children with anxiety disorders: a randomised controlled trial. Behav Res Ther 2016 Jan;76:47-56 [FREE Full text] [CrossRef] [Medline]
    84. Vigerland S, Serlachius E, Thulin U, Andersson G, Larsson J, Ljótsson B. Long-term outcomes and predictors of internet-delivered cognitive behavioral therapy for childhood anxiety disorders. Behav Res Ther 2017 Dec;90:67-75. [CrossRef] [Medline]
    85. Jolstedt M, Ljótsson B, Fredlander S, Tedgård T, Hallberg A, Ekeljung A, et al. Implementation of internet-delivered CBT for children with anxiety disorders in a rural area: a feasibility trial. Internet Interv 2018 Jun 01;12:121-129. [CrossRef]
    86. Silfvernagel K. Individually tailored internet-based cognitive behavioural therapy for adolescents, young adults and older adults with anxiety. Linköping, Sweden: Linköping University; 2017.
    87. Calear AL, Christensen H, Brewer J, Mackinnon A, Griffiths KM. A pilot randomized controlled trial of the e-couch anxiety and worry program in schools. Internet Interv 2016 Nov;6:1-5. [CrossRef]
    88. Lentferink AJ, Oldenhuis HK, de Groot M, Polstra L, Velthuijsen H, van Gemert-Pijnen JE. Key components in eHealth interventions combining self-tracking and persuasive eCoaching to promote a healthier lifestyle: a scoping review. J Med Internet Res 2017 Aug 01;19(8):e277 [FREE Full text] [CrossRef] [Medline]
    89. Wozney L, Huguet A, Bennett K, Radomski AD, Hartling L, Dyson M, et al. How do eHealth programs for adolescents with depression work? A realist review of persuasive system design components in internet-based psychological therapies. J Med Internet Res 2017 Aug 09;19(8):e266 [FREE Full text] [CrossRef] [Medline]
    90. Wildeboer G, Kelders SM, van Gemert-Pijnen JE. The relationship between persuasive technology principles, adherence and effect of web-based interventions for mental health: a meta-analysis. Int J Med Inform 2016 Dec;96:71-85. [CrossRef] [Medline]
    91. Baumel A, Yom-Tov E. Predicting user adherence to behavioral eHealth interventions in the real world: examining which aspects of intervention design matter most. Transl Behav Med 2018 Sep 08;8(5):793-798. [CrossRef] [Medline]
    92. Lehto T, Oinas-Kukkonen H. Persuasive features in web-based alcohol and smoking interventions: a systematic review of the literature. J Med Internet Res 2011;13(3):e46 [FREE Full text] [CrossRef] [Medline]
    93. Langrial SU, Al Araimi FA. Promoting Pediatric Healthcare through Persuasive Information Systems: A Qualitative Study. 2017 Presented at: PACIS 2017 Proceedings; 2017; Langkawi, Malaysia p. 30.
    94. Anderson R. New MRC guidance on evaluating complex interventions. Br Med J 2008 Oct 22;337:a1937. [Medline]
    95. Orlowski SK, Lawn S, Venning A, Winsall M, Jones GM, Wyld K, et al. Participatory research as one piece of the puzzle: a systematic review of consumer involvement in design of technology-based youth mental health and well-being interventions. JMIR Hum Factors 2015;2(2):e12 [FREE Full text] [CrossRef] [Medline]
    96. Rooksby J, Rouncefield M, Sommerville I. Testing in the wild: the social and organisational dimensions of real world practice. Comput Supported Coop Work 2009 Sep 3;18(5-6):559-580. [CrossRef]
    97. Howe D, Batchelor S, Coates D, Cashman E. Nine key principles to guide youth mental health: development of service models in New South Wales. Early Interv Psychiatry 2014 May;8(2):190-197. [CrossRef] [Medline]
    98. Andersson G, Carlbring P, Berger T, Almlöv J, Cuijpers P. What makes Internet therapy work? Cogn Behav Ther 2009 Jan;38(Suppl 1):55-60. [CrossRef] [Medline]
    99. Schueller SM, Tomasino KN, Mohr DC. Integrating human support into behavioral intervention technologies: the efficiency model of support. Clin Psychol Sci Pract 2016 Nov 17;24(1):27-45. [CrossRef]
    100. Titov N, Andrews G, Davies M, McIntyre K, Robinson E, Solley K. Internet treatment for depression: a randomized controlled trial comparing clinician vs. technician assistance. PLoS One 2010;5(6):e10939 [FREE Full text] [CrossRef] [Medline]
    101. Martin DJ, Garske JP, Davis MK. Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review. J Consult Clin Psychol 2000 Jun;68(3):438-450. [Medline]
    102. Howgego IM, Yellowlees P, Owen C, Meldrum L, Dark F. The therapeutic alliance: the key to effective patient outcome? A descriptive review of the evidence in community mental health case management. Aust N Z J Psychiatry 2003 Apr;37(2):169-183. [Medline]
    103. Mohr DC, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res 2011;13(1):e30 [FREE Full text] [CrossRef] [Medline]
    104. Santarossa S, Kane D, Senn CY, Woodruff SJ. Exploring the role of in-person components for online health behavior change interventions: can a digital person-to-person component suffice? J Med Internet Res 2018 Apr 11;20(4):e144 [FREE Full text] [CrossRef] [Medline]
    105. Fry JP, Neff RA. Periodic prompts and reminders in health promotion and health behavior interventions: systematic review. J Med Internet Res 2009;11(2):e16 [FREE Full text] [CrossRef] [Medline]
    106. O'Leary K, Liu L, Mcclure J, Ralston J, Pratt W. Persuasive reminders for health self-management. 2016 Presented at: AMIA Annu Symp Proc American Medical Informatics Association; 2016; Chicago, Illinois p. 994-1003.
    107. Lewis ZH, Swartz MC, Lyons EJ. What's the point?: A review of reward systems implemented in gamification interventions. Games Health J 2016 Apr;5(2):93-99. [CrossRef] [Medline]
    108. Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Lucassen MF. The effectiveness of SPARX, a computerised self help intervention for adolescents seeking help for depression: randomised controlled non-inferiority trial. Br Med J 2012;344:e2598 [FREE Full text] [Medline]
    109. Brown M, O'Neill N, van Woerden H, Eslambolchilar P, Jones M, John A. Gamification and adherence to web-based mental health interventions: a systematic review. JMIR Ment Health 2016;3(3):e39 [FREE Full text] [CrossRef] [Medline]
    110. Collins LM, Dziak JJ, Kugler KC, Trail JB. Factorial experiments: efficient tools for evaluation of intervention components. Am J Prev Med 2014 Oct;47(4):498-504 [FREE Full text] [CrossRef] [Medline]
    111. Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med 2007 May;32(5 Suppl):S112-S118 [FREE Full text] [CrossRef] [Medline]
    112. Collins LM, Baker TB, Mermelstein RJ, Piper ME, Jorenby DE, Smith SS, et al. The multiphase optimization strategy for engineering effective tobacco use interventions. Ann Behav Med 2011 Apr;41(2):208-226 [FREE Full text] [CrossRef] [Medline]
    113. Baker TB, Gustafson DH, Shah D. How can research keep up with eHealth? Ten strategies for increasing the timeliness and usefulness of eHealth research. J Med Internet Res 2014;16(2):e36 [FREE Full text] [CrossRef] [Medline]
    114. Alfonsson S, Olsson E, Hursti T. The effects of therapist support and treatment presentation on the clinical outcomes of an Internet based applied relaxation program. Internet Interv 2015 Sep 01;2(3):289-296. [CrossRef]
    115. Rocha A, Camacho R, Ruwaard J, Riper H. Using multi-relational data mining to discriminate blended therapy efficiency on patients based on log data. Internet Interv 2018 Jun 01;12:176-180 [FREE Full text] [CrossRef] [Medline]
    116. Moller AC, Merchant G, Conroy DE, West R, Hekler E, Kugler KC, et al. Applying and advancing behavior change theories and techniques in the context of a digital health revolution: proposals for more effectively realizing untapped potential. J Behav Med 2017 Feb;40(1):85-98. [CrossRef] [Medline]
    117. Eysenbach G. CONSORT-EHEALTH: implementation of a checklist for authors and editors to improve reporting of web-based and mobile randomized controlled trials. Stud Health Technol Inform 2013;192:657-661. [Medline]
    118. Sieverink F, Kelders SM, van Gemert-Pijnen JE. Clarifying the concept of adherence to eHealth technology: systematic review on when usage becomes adherence. J Med Internet Res 2017 Dec 06;19(12):e402 [FREE Full text] [CrossRef] [Medline]
    119. Halko S, Kientz JA. Personality and persuasive technology: an exploratory study on health-promoting mobile applications. Berlin, Heidelberg: Springer; 2010 Presented at: International conference on persuasive technology; June 7, 2010; Copenhagen, Denmark p. 150-161.
    120. Karanam Y, Filko L, Kaser L, Alotaibi H, Makhsoom E, Voida S. Motivational affordances and personality types in personal informatics. 2014 Sep 13 Presented at: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication; 2014; Seattle, WA p. 79-82. [CrossRef]
    121. Xu Y, Poole E, Miller A, Eiriksdottir E, Kestranek D, Catrambone R, et al. This is not a one-horse race: understanding player types in multiplayer pervasive health games for youth. : ACM; 2012 Feb 11 Presented at: Proceedings of the ACM 2012 conference on computer supported cooperative work; February 11-15, 2012; Seattle, WA p. 843-852. [CrossRef]
    122. Christensen H, Mackinnon A. The law of attrition revisited. J Med Internet Res 2006 Sep 29;8(3):e20 [FREE Full text] [CrossRef] [Medline]
    123. Melville KM, Casey LM, Kavanagh DJ. Dropout from Internet-based treatment for psychological disorders. Br J Clin Psychol 2010 Nov;49(Pt 4):455-471. [CrossRef] [Medline]
    124. Waller R, Gilbody S. Barriers to the uptake of computerized cognitive behavioural therapy: a systematic review of the quantitative and qualitative evidence. Psychol Med 2009 May;39(5):705-712. [CrossRef] [Medline]


    AACAP: American Academy of Child and Adolescent Psychiatry
    CBT: cognitive behavioral therapy
    CIHR: Canadian Institutes of Health Research
    CONSORT-EHEALTH: Consolidated Standards of Reporting Trials-electronic health
    iCBT: internet-based cognitive behavioral therapy
    MMAT: mixed method appraisal tool
    PSD: persuasive system design

    Edited by G Eysenbach; submitted 25.05.18; peer-reviewed by S Kelders, J Povey; comments to author 05.07.18; revised version received 26.09.18; accepted 20.10.18; published 05.02.19

    ©Ashley D Radomski, Lori Wozney, Patrick McGrath, Anna Huguet, Lisa Hartling, Michele P Dyson, Kathryn Bennett, Amanda S Newton. Originally published in the Journal of Medical Internet Research (, 05.02.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.