Cognitive behavioral therapy (CBT) is recommended as a first-line treatment for children and adolescents with anxiety [1-4]. Trained mental health professionals have traditionally delivered CBT, but there is an increasing interest in the internet as a delivery platform to circumvent the multiple barriers to receiving in-person treatment. These barriers can include direct and incidental costs to families, lack of trained deliverers, and inconvenient service times and locations [5]. Internet-based CBT (iCBT) is also proposed to preserve adolescent autonomy, appeal to user preferences, reduce health care system costs, and improve the time it takes to receive treatment [6-8]. iCBT is recognized as an important treatment option [9-11] to meet the increasing demands of a population where anxiety is often undetected and untreated, but with whom early access to care can improve long-term outcomes [3,12-14].

iCBT uses technological features (ie, multimedia and email) to deliver treatment content through the Web or a software application to generate interactions between the user and the program [15]. In this way, iCBT aims to reinforce, change, or shape attitudinal or behavioral health outcomes and aligns with the concept of a persuasive system [16-18]. Although recent efforts have been made to provide guidance on the design and delivery features of iCBT [19], considerable differences exist across programs both in terms of their features and the health outcomes they produce. Recent systematic reviews and meta-analyses have found a range of iCBT programs to be effective at reducing anxiety in children and adolescents [9,10,20,21]; however, poor and highly variable rates of completion can be found across programs (up to 50% of participants not reaching the end of a program) [5,7,10,21,22]. The term program use captures the various, typically objective, outcomes used across studies (ie, adherence, compliance, and number of program activities or homework completed) that describe the extent to which users interact with a program.

Understanding the factors that influence iCBT program use is important as there have been indications that greater program use is associated with better outcomes [23,24]. Studies of iCBT in children and adolescents with anxiety [25,26] have found that certain participant demographics (eg, gender, age, location, and anxiety severity) and delivery features (eg, parental or therapist support) relate to or predict program use. However, the relationship between technological design features of a program, the ability of those features to enhance the persuasiveness of a program, and actual iCBT program use by children and adolescents with anxiety, has received minimal attention in the literature.

Recognizing iCBT as a persuasive system, we conducted a realist synthesis to examine the technological design and program delivery features of iCBT for children and adolescents with anxiety in order to document their potential relation to persuading program use. The realist synthesis approach provided a framework to answer 2 main questions: (1) what design and delivery components (technological features, treatment content, and interactions) are described for iCBT programs for children and adolescents with anxiety? and (2) what components may explain reported program use outcomes?

This realist synthesis was conducted using steps recommended by Pawson and Tilley [27,28] and is reported in accordance with the Realist and Meta-narrative Evidence Synthesis: Evolving Standards (RAMESES II) [29]. Realist synthesis is theory-driven in that the synthesis searches for and refines explanations of intervention effects by asking: “What works, for whom, and in what circumstances?” [27]. In this synthesis, we examined the relationships that exist between the therapeutic and technological features of iCBT (Mechanisms) and program use (Outcomes), and the program delivery formats and interactions (Contexts) that support them. We expressed these relationships as Context-Mechanism-Outcome configurations.

We began the synthesis by developing candidate Context-Mechanism-Outcome configurations for how iCBT programs may work. The development process consisted of brainstorming activities, with the research team reviewing literature on human-technology interaction and studies of iCBT programs for anxiety to identify relevant and pre-existing theories, models, or frameworks to work from. The persuasive systems design (PSD) model [17] emerged as a key framework for understanding how iCBT, as a persuasive system, was intended to work, and we used this model to develop the initial list of Context-Mechanism-Outcome configurations. The model describes 28 PSD (technological) features, subdivided across the following 4 categories, which can be implemented by programs to guide the user toward their health-related goal: (1) Primary task support, (2) Dialogue support, (3) System credibility support, and (4) Social support.

Using the PSD model, we identified which PSD features (Mechanisms) might be associated with iCBT program use (Outcomes) to formulate Mechanism-Outcome dyads. We then hypothesized which program delivery formats, interactions, and conditions for use (Context) might promote the occurrence of the Mechanism-Outcome interactions. Together, these steps led to the generation of 5 candidate Context-Mechanism-Outcome configurations (Multimedia Appendix 1) [30-34]. The configurations were as comprehensive and justifiable as possible, referencing literature that supported their development and inclusion in the list. These configurations would undergo refinement and testing during the analysis stage of the synthesis, whereby we used evidence from the literature to validate their explanatory usefulness and applicability for answering our research questions.

We used 3 main strategies to identify literature for iCBT programs. The first search strategy involved an information specialist conducting a systematic and comprehensive search of 8 electronic databases: MEDLINE, EMBASE, ERIC, PsycINFO, CINAHL, Cochrane Library, ProQuest Dissertations & Theses Global, and PubMed for the period 1990 to 2017. The second strategy involved a manual search using Google, an internet search engine, and gray literature repositories (Association for Computing Machinery Digital Library, Open Grey, Institute of Electrical and Electronics Engineers Digital Library, and Canadian Agency for Drugs and Technologies in Health) to identify conference proceedings, program evaluations, and government or technical reports. For both search strategies, MeSH terms and text words were based on mental health condition (anxiety and phobias), intervention modality (internet-based and mobile app), intervention type (prevention and treatment), and therapeutic approach (CBT; Multimedia Appendix 2). The third strategy involved manually searching the table of contents in the Journal of Medical Internet Research, Internet Interventions, Journal of Cyber Therapy & Rehabilitation, and Journal of Telemedicine and Telecare, and a review of reference lists of included documents and reviews (eg, systematic reviews).

We employed the search strategies on an iterative and recurrent basis until November 2017 to ensure the review was up to date and inclusive. Before discontinuing the literature search, a test of saturation was applied to the search strategies, which involved verifying that further searching would not yield any new results [35].

In this study, 2 independent, trained reviewers (authors ADR and LW) screened the identified documents for eligibility using a 2-stage approach. During this process, reviewer discrepancies were resolved by consensus or third party arbitration (author ASN). At stage 1, all documents were screened for eligibility using the title and abstract. During this stage, we randomly selected 100 citations to assess inter-rater agreement for inclusion or exclusion decisions; Cohen kappa was 0.74 between raters, reflecting substantial agreement [36]. All documents that were screened “yes, include” or “unsure to include” moved to stage 2. At stage 2, the full text of documents was reviewed by 1 reviewer (ADR), in consultation with another (ASN), with a resulting decision to either include or exclude a document from the synthesis.

For an iCBT program to be included in this synthesis, supporting documents needed to be published in English and provide information on treatment Context, program design (PSD) and delivery features, and program use Outcomes. Each document did not need to provide details on all 3, but all 3 needed to be represented in the total documents for an iCBT program. In addition, at least one published study on the iCBT program needed to be available for inclusion so that we could assess the methodological quality of the study providing program use outcome data.

Intervention studies (eg, clinical trials) were eligible for inclusion if they evaluated iCBT anxiety programs with children (aged <14 years) or adolescents (aged 12-19 years). As some iCBT programs were designed for and evaluated with participants from a broader age range (eg, programs also geared toward young adults), only those studies that provided separate data for participants aged ≤19 years were included. We also required that the type of iCBT program under evaluation be designed for an anxiety disorder(s) or anxiety symptoms associated with a disorder as classified according to the Diagnostic and Statistical Manual of Mental Disorders, fifth edition [37] such as social phobia (social anxiety disorder), generalized anxiety disorder, panic disorder, separation anxiety disorder, or specific phobia. Transdiagnostic programs (ie, programs designed for anxiety plus another diagnosis such as depression) were also eligible for inclusion. We also required that the iCBT program consisted of modules designed for use by the child or adolescent (and not solely delivered to or facilitated by a parent or therapist) as child or adolescent program use was our outcome of interest. Theoretical papers, mixed-methods and qualitative studies, and policy or implementation documents were also eligible if they included a focus on how an iCBT program was proposed to work.

Documents were assessed for relevance and rigor based on consensus between 2 reviewers (authors ADR and LW). Relevance was assessed based on the level of contribution a document provided for an iCBT program in 3 domains: (1) underpinning theory and/or the context and sequence for delivery (Context), (2) PSD features (Mechanism), and (3) program use outcomes (Outcomes). The level of contribution for each domain was rated low if little or no information was provided, medium if some information was provided, and high if information was well described. Exemplar documents with a high level of contribution across the 3 domains informed decision rules for the rating of all other documents.

The methodological quality (rigor) of research studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [38,39]. The MMAT is a reliable, efficient, and valid tool that provides different sections for assessing studies of qualitative, randomized, nonrandomized studies, descriptive studies, and mixed-method designs [38-40]. Multiple publications of the same study (thesis + journal publication) received the same MMAT score. MMAT scores could range from 0% to 100%, with a greater score indicating that more quality criteria were met.

Documents for each iCBT program were grouped together during data extraction and coding. Two reviewers (ADR and LW) cross-referenced extraction and coding decisions, with a random subset of 10 documents; the remaining documents were coded by 1 reviewer (ADR). In addition to document characteristics (type and year of publication), iCBT program data were extracted for the following:

We used a multistep approach to data analysis that was structured according to Pawson’s techniques [27,28] and meta-ethnography [43,44]. The first step involved determining recurrent patterns or themes (demi-regularities) across documents for each iCBT program for delivery Context, PSD features and program Mechanisms, and Outcomes related to program use. The purpose was to use evidence from the literature to (1) identify PSD Mechanisms in each program most frequently associated with the program use Outcomes to refine the candidate Mechanism-Outcome dyads, and (2) incorporate delivery Context of each iCBT program into the dyads to refine the overall Context-Mechanism-Outcome configurations. Context-Mechanism-Outcome configurations that were supported by evidence from at least two different iCBT programs progressed to the next step of analysis.

The second step in the analysis involved reciprocal translation analysis, a meta-ethnographic technique that involved reviewing the Context-Mechanism-Outcome configurations across iCBT programs [44,45]. Configurations that were found to have mixed (ie, more heterogeneous support with no larger trend) or confounding evidence across programs, or could not be refined by better describing or recombining the Context, Mechanism or Outcome factors, did not progress to the next stage of analysis. What remained were configurations that provided the best support, across multiple programs, to explain the relationship of design and delivery components of iCBT with program use.

In the final step, we used lines-of-argument synthesis, a theorizing technique [44] that involved assessing how well each Context-Mechanism-Outcome configuration could explain why the same PSD Mechanism(s), operating in different iCBT program Contexts, might result in particular program use Outcomes. We took into consideration the quality and quantity of evidence supporting the configuration and held a meeting to discuss, amend, and finalize configurations with individuals from across Canada with expertise in electronic health interventions. Configuration refinement continued until we felt that it reflected a pattern that would remain consistent despite differences in small- or large-scale details across iCBT programs. At that point in time, we considered the configuration to be adequately developed.

The literature search and selection progress are presented in Figure 1. The search strategy yielded 11,511 unique documents for stage 1 screening, after duplicates were removed. Of these, 801 documents underwent stage 2 screening. In total, 44 documents detailing 10 iCBT programs were included in the realist synthesis. Documents were published studies (n=20), theses (n=5), published or registered protocols for trials (n=12), study or program websites (n=6), and a study flyer (n=1).

Details of the quality appraisal are provided in Multimedia Appendix 4 [58-75]. Documents tended to provide the greatest level of contribution to understanding program Context and Outcomes with relatively few details available for understanding program Mechanisms. A total of 7 iCBT programs had at least one document that scored high for level of contribution for program Context, Mechanisms, and Outcomes. We evaluated 29 research studies (found in 25 documents) for methodological rigor. A total of 19 studies met all 4 MMAT criteria (100%), 7 met 3 criteria (75%), and 3 met 2 criteria (50%). Lower ratings reflected an unclear description of: processes for recruitment, selection, randomization, or allocation; how group differences (if any) were controlled for; the percentage of outcome data obtained; or withdrawal or dropout rates were not within acceptable limits.

Tables 2 and 3 present an overview of the delivery Context and PSD Mechanisms that were most frequently or consistently associated with moderate-to-high program use Outcomes across iCBT programs. Contexts most indicative of program use were the adjunct support person and the communication approach (eg, Web [the program platform or internal messaging system], email, or in-person) provided to the user—both of which typically varied based on the level of prevention the program was designed for. The type of adjunct support also depended on the age of program users. Children generally received the most extensive adjunct support (ie, therapist and parent), and program use was often greater among this age group than among adolescents. PSD features identified as having a high value to encouraging program use were Tailoring and Personalization (Primary task supports); Social role, Reminders, and Rewards (Dialogue supports); and Authority, Expertise, and Trustworthiness (System credibility supports). Program use Outcomes most commonly reported either the total proportion or average proportion of program modules, homework, or activities completed by users. More than half of the Outcomes indicated high or moderate program use.

When finalizing our proposed configurations, we considered both what program Contexts and Mechanisms were combined (using Tables 2 and 3) and in what way and for what purpose they may have been combined (using excerpts from individual documents; see Table 4) to explain how moderate-to-high program use Outcomes were generated.

Key Contexts and PSD Mechanisms that may have led to moderate-to-high iCBT program use Outcomes are described in Table 4 alongside examples from contributing programs.

The role of technological, persuasive components in iCBT programs is an understudied aspect of program design and evaluation. The extent of iCBT program use may be a fundamental indication of its persuasiveness and its potential to assist the user with their goals of the program [15,88-91]. This realist synthesis identified 5 possible relationships as to how the use of specific PSD features (technological Mechanisms), supported by some key user and delivery features (Context), may generate moderate-to-high program use (Outcomes) in iCBT for children and adolescents with anxiety.

The 5 Context-Mechanism-Outcome configurations provide support for several persuasive strategies to improve iCBT program use: Tailoring and Personalization as Primary task supports; Rewards, Reminders and Social role features in programs serving as Dialogue supports; and Trustworthiness, Expertise, and Authority features serving as System credibility supports for a program. Traditionally, PSD features that stimulate human-computer communication, such as Dialogue supports, have been the most widely used and studied features for improving program use [6,15,89,92,93]. However, this synthesis demonstrated that having credibility (System credibility supports) and supporting users in completing their target behavior (Primary task supports) may also promote moderate-to-high program use. We hypothesize that using multiple PSD features, both within and across the different support categories, may produce additive or synergistic effects on program use; however, there was insufficient evidence available for our analysis to explain the impact of PSD feature combinations that involved more than a few features (and proposed functions) at a time. This is because the authors of the original studies included in our review typically discussed or formally tested the relationship of only 1 or 2 PSD features and program use at a time. Therefore, our configurations present the fewest possible PSD features that could be linked to higher program use (ie, we uncoupled features as much as possible—an approach that may make testing of their effects more efficient in future studies).

Moreover, we suspect that not all PSD features may have equal influence on program use. Depending on the program, some PSD features may be necessary for program use (part of the basic requirements or foundational design framework of iCBT), whereas others may be complementary to program use (have an impact by enhancing the design framework of iCBT); although both types of features together may influence program use. In this realist synthesis, all 10 iCBT programs described a purposeful design that incorporated Reduction and Tunneling (Primary task supports) and Similarity and Liking (Dialogue supports) to create a logical, incremental, relevant, and aesthetically pleasing experience for users—these may be the necessary PSD features for iCBT for children and adolescents with anxiety. The PSD features described in our configurations are hypothesized to be complementary—building on the persuasiveness of necessary PSD features to further improve or optimize iCBT program use.

A meta-analysis of PSD features used in internet-based interventions for mental health demonstrated that determining the amount and type of persuasive principles to include may be a delicate balance, as some principles seem to work together, whereas when other principles co-occur, they may have an unapparent or diminishing effect [90]. As was found for this synthesis, it is not necessarily the number of PSD features used in iCBT, but it is their proposed purpose or implementation that is particularly critical for optimizing program use outcomes [90]. For example, when comparing 2 indicated prevention programs, we observed that the internet cognitive behavioral skills-based program [51] had fewer PSD features than the individually tailored iCBT program [55], although the former reported greater program use. At this time, our understanding of how to best bring together PSD features, such as Personalized Reminders [93], in the design and delivery of iCBT for child and adolescent anxiety is limited. Therefore, further research on the theory, function, quality, and effectiveness of individual PSD features is needed to deliberately use and combine them for idealized treatment outcomes. Moreover, involving target users in the (co)design and testing of treatments is recommended to improve the acceptability, feasibility, and effectiveness of iCBT with children and adolescents [19,94,95]. These participatory research efforts may provide important guidance on the usefulness and functionality of select (PSD) features of iCBT programs in the real world and from the user’s perspective [96]; therefore, facilitating greater program use [97].

In this synthesis, 3 important potential relationships were identified: (1) adjunct support seemed to improve program use even when input or support was minimal (eg, in-person, classroom-based program administration with no treatment advice given) or when it was provided by a nonexpert (eg, teacher) [24,25]; (2) users of treatment programs demonstrated higher program use than users of universal prevention programs; and (3) a trend for increased program use among programs for children (more so than for adolescents) was identified. Within these relationships, multiple contextual aspects or user characteristics may have also had an additive or synergistic effect on program use. For example, the level of expertise the adjunct support person had (eg, teacher vs therapist) and the degree of their guidance or therapeutic involvement (eg, in-person program administration vs personalized feedback emails) increased from prevention-based to treatment programs. In the literature, little is known about how much, when, and what type of support is necessary for enhancing program use and efficacy [98,99]. Although some evidence suggests that layperson support is as effective as clinician support [100], this synthesis suggests that the person providing support as well as the intensity of their support activities (frequency; inclusion of therapeutic elements) may have a noteworthy effect on program use. The nature of the role adjunct support plays in iCBT program use is also unclear. It has been suggested that adjunct support may leverage the advantages of therapeutic alliance [101,102], which might include principles of persuasion (eg, users feel the need to respond to social cues [18]), it may establish process expectations and social accountability [103], or it may develop a sense of legitimacy or credibility of the program [103] (see Santarossa et al’s study [104] for further suggestions). Programs that had both therapist and parent involvement may have (1) reinforced child’s understanding of and confidence in treatment content and (2) increased the child’s interaction with the program by creating a perceived sense of cooperation (shared goals) and accountability toward treatment progress. In this realist synthesis, adjunct support may have been used to complement or replace the use of some PSD features in iCBT, particularly Dialogue supports. For example, in-person therapist sessions or telephone calls provided opportunities for Reminders, Personalized feedback, or Praise to be conveyed to users [49,50,54,55]. Consideration of how and when to provide adjunct support is critical when preparing for the implementation and integration of iCBT within routine practice, such as allotting for therapist time, making changes in clinical workflow, and when conducting economic analyses.

This realist synthesis not only provides support for incorporating some of the well-studied and highly used PSD features into iCBT (ie, Reminders [93,105,106]) but also draws attention to underutilized features that can be incorporated in the designs of new treatments. For example, Rewards were only occasionally used by iCBT programs included in this study (4 out of 10 programs) but are more commonly used techniques for increasing program use in internet-based interventions targeting physical activity or dietary behaviors [107]. Recent efforts into improving the gamification of technology-based CBT for pediatric mental health (see SPARX [108]), where incorporating game-design elements such as Praise and Rewards are regularly used to enhance program use and engagement [109], demonstrate the potential benefits that the use of these features may have.

Although this synthesis and other recent reviews have been helpful for identifying PSD features of interest for improving program use of internet-based interventions [15,89,90], the next step is to formally isolate and evaluate the effectiveness of these PSD Mechanisms in producing optimal program use. This synthesis suggests 8 features that may be a priority for further examination. Modeling, factorial designs or the multiphase optimization strategy [110-112] (see also Baker et al’s study [113] for other suggestions) can be used to evaluate the best set (individual or combination) of program features to use under different conditions (eg, delivery setting and start or end of treatment). Studies with multiple, active treatment arms would also allow the examination of the comparative effects of select program features [114] or in different delivery contexts (ie, varying the type of adjunct support). From this synthesis, only 3 studies of 3 different programs conducted these comparisons. These studies provided important insights into the impact of delivery medium [49], type of adjunct support [24], and delivery location [25] on iCBT program use. Qualitative studies or self-report data would also provide meaningful information on the factors affecting program use from child and adolescent or health care provider standpoints. Another important line of inquiry relates to defining and measuring program use to ensure its validity and reliability for future studies. Designing studies that incorporate in vivo, objective measurements or automatic data capture of program use [115,116] could improve our awareness of program use predictors beyond user demographics (ie, age), for example, to actual usage behaviors (ie, number of Web pages viewed). This method would allow for iCBT program use to not only be measured at end of the intervention but also throughout the program access period to assess usage patterns over time [15], when certain design or delivery features may be more or less activated or present.

This is the first study to examine PSD features as they relate to program use in iCBT for children and adolescents with anxiety. A strength of this synthesis is the inclusion of diverse and high-quality evidence (ie, MMAT scores>75% [38]) from both the published and gray literature. Approaching our research questions using a single theoretical framework (ie, the PSD model) allowed for systematic and incremental accumulation of knowledge about how iCBT may work from a trackable, technological perspective.

The lack of operationalization of how PSD features and aspects of Context and program use Outcomes were defined, described, and measured by authors affected our data extraction and coding strategies. As adherence to recent recommendations [19] and reporting guidelines, such as Consolidated Standards of Reporting Trials-electronic health (CONSORT-EHEALTH) [117], become mandatory for publication, the opportunity to identify the active ingredients of iCBT will improve. Clarifying PSD features with original authors was an attempt to mitigate the potential bias that lies in coding technological program features and interpreting the papers using the PSD model [92]. However, few details about the time or the quality of communication by the adjunct support person(s) were available, limiting our understanding of the important role this contextual feature played in program use. For reasons of inclusivity, we described the heterogeneous outcomes using the umbrella term program use. Adoption of a recent standardized definition and calculation of adherence [118] can clarify what is meant by specific program use terms and allow for comparisons of outcomes across programs. iCBT programs’ widespread implementation and ability to meet the health goals of users will involve an understanding of the expectations and actuality of program use in the real world (ie, using true effectiveness studies or formative program evaluation), and setting benchmarks for an effective dose in different delivery settings. Finally, like others [119-121], we recognized the outcomes of persuasive systems depended on multiple factors, many of which were not examined in this synthesis. However, it was rare to have information on treatment or technology preferences of users (eg, early completers [22,122]), their psychological characteristics or cognitions (eg, motivation, personality, expectations, and treatment perceptions [123]), or personal circumstances (eg, program access [124])—factors that are also considered critical to program use and could be used to construct and validate more intricate Context-Mechanism-Outcome configurations.

The Context-Mechanism-Outcome configurations identified by this realist synthesis provide an initial understanding of how, why, and for whom iCBT programs for children and adolescents with anxiety work from a persuasive systems’ perspective. Appreciating that the effectiveness of iCBT programs may hinge on whether and to what extent programs are used, this study is an important step toward successfully implementing and integrating iCBT into routine clinical care. Recognizing that multiple PSD features are incorporated in iCBT program designs and that individual features may affect each other differently, further knowledge and testing of the purpose and function of these features will help determine the number and combination to use in certain delivery contexts (eg, adjunct support included; level of prevention a program is designed for). As PSD features are modifiable, iCBT program designers and developers can look to create more persuasive programs that promote greater use and improved treatment outcomes.