This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Internet-based cognitive behavioral therapy (iCBT) for child and adolescent anxiety has demonstrated efficacy in randomized controlled trials, but it has not yet been examined when disseminated as a public health intervention. If effective, iCBT programs could be a promising first-step, low-intensity intervention that can be easily accessed by young people.
The objective of our study was to examine the feasibility and acceptability of a publicly available online, self-help iCBT program (BRAVE Self-Help) through exploration of program adherence, satisfaction, and changes in anxiety.
This study was an open trial involving the analysis of data collected from 4425 children and adolescents aged 7-17 years who presented with elevated anxiety at registration (baseline) for the iCBT program that was delivered through an open-access portal with no professional support. We assessed the program satisfaction via a satisfaction scale and measured adherence via the number of completed sessions. In addition, anxiety severity was assessed via scores on the Children’s Anxiety Scale, 8-item (CAS-8) at four time points: baseline, Session 4, Session 7, and Session 10.
Participants reported moderate satisfaction with the program and 30% completed three or more sessions. Statistically significant reductions in anxiety were evident across all time points for both children and adolescents. For users who completed six or more sessions, there was an average 4-point improvement in CAS-8 scores (Cohen
Participant feedback was positive, and the program was acceptable to most young people. Furthermore, significant and meaningful reductions in anxiety symptoms were achieved by many children and adolescents participating in this completely open-access and self-directed iCBT program. Our results suggest that online self-help CBT may offer a feasible and acceptable first step for service delivery to children and adolescents with anxiety.
Half of all lifetime mental health disorders begin before the age of 14 years [
The current Australian federal government recommendations encourage primary prevention and early intervention across the life span through easy-to-access first-line responses, particularly for children [
BRAVE Self-Help is an online, open-access, self-help intervention for child and adolescent anxiety that addresses the needs of anxious Australian youth using a digital, low-intensity, and population-level model. The BRAVE Self-Help initiative was initially supported by
There has been relatively little research examining the implementation of iCBT programs for child and adolescent anxiety in real-world clinical and community contexts (eg, outside university-based research trials); furthermore, there are no formalized guidelines for a large-scale dissemination [
Despite substantial challenges involved in determining the outcomes of an open-access, real-world service, we examined the impact of the BRAVE Self-Help intervention through a feasibility and acceptability approach. As the program was developed and intended for youth with anxiety, we were specifically interested in those children and adolescents who reported elevated levels of anxiety at enrollment into the self-help program. Our primary aim was to determine the feasibility and acceptability of BRAVE Self-Help when disseminated nationally through open access. Specifically, we evaluated the level of adherence to and satisfaction with the program as well as the extent to which anxiety symptoms changed over the course of the program for those with elevated symptoms. We hypothesized that children and adolescents who adhered to the program would show significant reductions in their self-reported anxiety from baseline to Sessions 4, 7, and 10. We also expected high satisfaction among participants of the program.
Participants were 4425 anxious young people aged 7-17 years (1473 children aged 7-12 years; 2952 adolescents aged 13-17 years) with mean (SD) age 12.95 (2.97) years; there were 66.39% (2938/4425) females and 31.77% (1406/4425) males, and 1.84% (81/4425) participants identified as another gender category. In terms of residence, 57.45% (2542/4425) participants resided in major cities, with 23.35% (1033/4425) from Inner Regional Australia, 11.21% (496/4425) from Outer Regional Australia, and 2.55% (113/4425) living in remote or very remote Australia (241/4425 [5.44%] provided data that could not be accurately coded).
All participants registered for the program through a website accessible only to Australian families. Then, participants were directed to the program in several ways through (1) self-referral and internet searching, (2) referral from health or education professionals, (3) links hosted on several Australian mental health information sites (eg, Reach Out and Beacon), and (4) direct links from the
To be included in this study, participants were required to have enrolled in the BRAVE Self-Help program between July 1, 2014, and June 30, 2016. We monitored the program progress for participants through to November 17, 2016. All participants (including those registered at the end of the recruitment period) were monitored over a 20-week period, which was sufficient to complete the ten sessions. Participants who were enrolled outside of the 2-year recruitment period were excluded from this study. As this online program is open-access, young people do not need a referral to register and begin the sessions. Besides, there are no set inclusion criteria for enrollment in and access to the program, and thus, users are able to access the program for prevention, early intervention, or treatment purposes. When registering, users are not required to demonstrate symptomatic levels of anxiety, and the program is completely open access and self-sought. In this study, however, participants were included only if they demonstrated elevated anxiety at the baseline above a predetermined criterion (≥84th percentile or
There are two versions of the program, one for children aged 7-12 years and another for teenagers aged 13-17 years; participants selected which version they wish to complete. Informed consent (and parental consent in the case of children aged <16 years) was required prior to beginning the program, and it was obtained during the Web-based registration process. Participation was voluntary, and young people were made aware that they could cease using the program at any time, without consequence. The study protocol approval was obtained from the ethics committees of the University of Queensland (UQ), University of Southern Queensland, and Griffith University. Furthermore, data were stored on secure servers hosted by UQ. Participants were not provided with any reimbursements for participation.
The BRAVE-ONLINE program for youth anxiety when delivered with minimal therapist assistance has been described elsewhere [
In addition to the removal of any therapist contact, the following changes were made to the existing intervention. First, we created a new infrastructure to surround the existing program and provide clearer navigation to resources for users. This included additional home pages, new graphics, demonstration videos, frequently asked questions, and dedicated sections for resources and key program components (eg, relaxation room and exposure section). Second, we integrated relaxation activities into the program via the dedicated relaxation room, where users could live stream relaxation or download relaxation recordings (or transcripts) to add to their music libraries. Third, the exposure hierarchy (BRAVE ladder) was integrated into the new home page infrastructure in a dedicated section and additional tools were created to ensure that young participants were able to build their exposure hierarchy effectively without a therapist. This included easy creation of steps and rewards, the ability to move steps around, and the capacity to check-off steps when complete. Fourth, in addition to the already included automatic email reminders and session completion messages, the self-help program also included automatic alert messages that were sent to the users if they reported anxiety scores in the clinical range. Finally, we integrated a self-registration system into the program that required young people to register for the program and provide a contact email address and name or pseudonym. Registration also included the provision of consent and an explanation of the monitoring and reporting features included in the program. All users were able to first trial the program in a 20-minute guest access before registering. Young people were able to register for and complete the program on their own, via a personal computer or mobile device. Program sessions were conducted in a prescribed sequence, although participants could progress at their own pace. Furthermore, automatic email reminders were sent if participants had not completed the next session within 1 week.
In this study, we aimed to disseminate the intervention as widely as possible on a national scale; thus, we implemented assessment procedures conducive to this goal. The advisory committee identified administering an exhaustive assessment battery as a potential deterrent for young people registering and completing the program in this self-directed manner. Thus, to minimize participant burden and the potential barriers to participation, we administered a limited assessment battery, including one brief measure of anxiety, along with basic demographic characteristics and a brief satisfaction scale.
The progression of participants through the program. CAS-8: Children’s Anxiety Scale, 8-item.
Demographic data (eg, age, gender, and postcode) were collected when participants created their account to access the program. Postcodes were categorized according to the Australian Standard Geographic Classification system [
We measured anxiety symptom severity using CAS-8 [
Session completion (adherence) was operationalized as the number of program sessions completed and was automatically recorded by the program. We measured both satisfaction and acceptability via a 5-item scale based on a satisfaction questionnaire administered in previous trials of the BRAVE Program [
Given the self-help nature of this intervention, the program was designed to incorporate checks on anxiety, alerts regarding participants who were experiencing high levels of anxiety, and the provision of appropriate referral information. In this study, any child or adolescent scoring in the clinical ranges of anxiety at any assessment point was sent an automatic message that alerted the person to his or her high score and encouraged him or her to seek further support from additional sources, including family, friends, and professional services. The message also included contact details for crisis lines and services. We calculated the proportion of children and adolescents receiving email alerts at the four different assessment points.
We used IBM Statistics 24 and MPlus 8 for statistical analyses. Descriptive data for satisfaction item and total means were evaluated and presented for the total sample as well as child and adolescent subsamples. Examples of feedback are provided as well. Furthermore, descriptive data for program adherence (number of sessions completed) were evaluated and reported for the total sample as well as according to child and adolescent samples. In terms of safety alerts, the proportion of participants receiving email alerts at each time point was calculated. Baseline differences in anxiety severity, age, gender, and geographic location between the participants who completed less than three sessions and those who completed three or more sessions were examined using
First, we examined the mean change in raw anxiety scores across the program. Following the procedure of Rickwood et al [
Given the large amount of missing data from participants who failed to provide data at all four assessment points, we also analyzed the full sample through latent growth curve modeling (LGCM) to confirm the findings of the completer analysis. LGCM is accepted as a suitable framework for use in the evaluation of efficacy in psychological interventions [
The second method for analyzing changes in anxiety included utilization of the Reliable Change Index (RCI) [
The third method of assessing change in anxiety was through examination of the proportion of individual youth cases crossing the clinical threshold [
Using the last provided satisfaction scores for each participant, the mean total satisfaction rating was 17.72 (SD 5.16) out of a maximum 25. The mean satisfaction ratings for the individual items are provided in
In terms of the open feedback item,
As indicated in
Of note, a large proportion of registered participants (958/4425, 21.65%) did not go on to complete the first session. A further 48.05% (2126/4425) of registered participants completed only one or two sessions of the program. However, 30.31% (1341/4425) participants went on to complete three or more sessions, with 1095 of these providing assessment data for at least two time points. The average number of sessions completed for all registered participants (including those who did not start the program) was 2.21 (SD 2.44).
The mean satisfaction ratings for individual satisfaction items.
Feedback comments from participants.
For those participants who provided at least two data points (and, therefore, had completed at least three sessions), the average number of sessions completed was 5.69 (SD 2.35) out of 10. On average, child users completed 5.77 sessions (SD 2.45), while adolescent users completed 5.62 sessions (SD 2.25). A significant negative correlation was evident between the number of sessions completed and final CAS-8 scores (
Changes in the CAS-8 score over time are provided for the child program participants in
A visual summary of how many of the registered participants (N=4425) completed how many sessions of BRAVE Self-Help during the 20-week period (including participants who only provided one assessment point).
Changes in the mean anxiety scores according to the number of sessions completed for child program users. CAS-8: Children’s Anxiety Scale, 8-item.
Changes in the mean anxiety scores according to the number of sessions completed for adolescent program users. CAS-8: Children’s Anxiety Scale, 8-item.
The mean CAS-8 scores over time and the results of ANOVAs and effect sizes are presented in
The results for LGCMs confirm the findings of the completer sample and demonstrate that anxiety decreased significantly over time. Estimated means, SDs, and effect sizes for the CAS-8 in the child and adolescent samples are provided in
The proportion of child and adolescent users demonstrating statistically reliable (gender-adjusted) improvement, deterioration, or no change according to sessions completed is presented in
Changes in the mean anxiety scores from a user’s baseline to final CAS-8 score. CAS-8: Children’s Anxiety Scale, 8-item.
Estimated Children Anxiety Scale, 8-item, means, SDs, and effect sizes from growth curve analyses.
Group | Baseline, mean (SD) | Session 4 | Session 7 | Session 10 | |||
Mean (SD) | Cohen |
Mean (SD) | Cohen |
Mean (SD) | Cohen |
||
Children | 14.09 (3.06) | 11.55 (4.59) | 0.60 | 10.24 (4.96) | 0.82 | 9.57 (5.17) | 0.86 |
Adolescents | 15.83 (3.36) | 13.51 (4.70) | 0.60 | 12.42 (5.47) | 0.74 | 11.53(5.77) | 0.89 |
aEffect size from the baseline to Session 4.
bEffect size from the baseline to Session 7.
cEffect size from the baseline to Session 10.
The proportion of child and adolescent users crossing the “elevated” and “clinical” thresholds at the different time points (and according to sessions completed) is provided in
For those who initially demonstrated “clinical” levels of anxiety, 59.1% (309/523) demonstrated an improvement into the “nonclinical” range by their final recorded score. We observed no differences in the proportion of children (134/217, 61.8%) and adolescents (175/306, 57.2%) achieving this level of improvement (
The proportion of young people receiving email alerts for the presence of clinical-level anxiety was higher among adolescents than among children across all four time points and showed reduction over time for both groups. At the baseline (registration), 59.96% (1770/2952) of adolescents and 38.15% (562/1473) of children received email alerts, with this difference being statistically significant (
This study reports on the feasibility and acceptability of a free, self-help iCBT intervention, offered nationally in Australia, and highlights the potential benefits, yet significant challenges evident, with this type of service delivery. The BRAVE Self-Help initiative was designed to provide an evidence-based intervention to anxious Australian children and adolescents while minimizing user burden and barriers to receipt of treatment (eg, cost, accessibility, stigma, and privacy). This study examined the feasibility and acceptability of this approach by evaluating satisfaction with, and adherence to, the program as well as changes in anxiety symptoms. Although there was no control comparison condition or trial methodology, the methodological approach to analysis was comprehensive, with data being collected on multiple occasions and the effects assessed using multiple methods of evaluating changes in anxiety. Furthermore, both satisfaction and adherence data provided further information about the acceptability of the open-access intervention.
The BRAVE Self-Help program is extremely comprehensive as all specialist techniques are incorporated into the content and prominent examples and opportunities for skill rehearsal are integrated within and between sessions. In addition, the content is interactive, engaging, and age appropriate, and our previous research has demonstrated high consumer acceptability and satisfaction [
Qualitative feedback indicated that many users were able to obtain benefits from the program, and potential improvements were noted by others. In particular, the intervention might benefit from accompanying app-based features, either for the entire program or for specific intervention components such as exposure. Furthermore, when delivered in this self-directed format, sessions might need further refinement to decrease the length, increase the use of videos and acceptable graphics, and reduce monitoring and reminder systems (unless requested, or demonstrated as being important in stimulating engagement). Interestingly, the need for sessions to be personalized was raised, despite the intervention being run as a self-help program without any professional contact. Thus, there are opportunities to implement innovative technology-based methods for achieving treatment personalization (eg, utilizing algorithms to present personalized treatment content or messages based on previous session responses) in such open-access, self-help online interventions.
Despite positive feedback and satisfaction with the program, there was a noticeable variation in the degree of program adherence across users. The fact that around 21% (958/4425) of participants did not go on to complete any of the sessions and a further 48% (2126/4425) completed only two sessions or less indicates that the program was either not acceptable or potentially not useful to a substantial proportion of people. These rates of session progression are not dissimilar to those found in other large-scale, open-access eHealth interventions such as MoodGym (in a sample of 82,159 participants, 63% completed no modules, 27% completed only 1 module, and 10% completed 2 or more modules out of 5 modules) [
In all likelihood, these results may suggest that services that are widely or publicly available attract users from varying contexts and backgrounds, who will subsequently engage very differently with the programs than those in strictly controlled RCTs. With respect to the BRAVE Self-Help initiative, given the significant change observed in users completing only three sessions, it is possible that several participants engaged in the program until they obtained the benefit they needed, which might have occurred early on in the program. For others, such as those who did not commence the program at all, it is possible that their expectations were not aligned with what the program had to offer and that this only became apparent after registering. For others still, it may be possible that a self-directed program with little guidance was simply not enough to sustain engagement. In this study, we were unable to determine the reasons for nonadherence, and this will be important for future studies. Nonetheless, while the program was acceptable, the low rates of adherence suggest that such approaches will not be sufficient for or acceptable to all.
The results of this study demonstrate that self-reported anxiety decreased significantly over time, with effects being greater as the number of sessions completed by youth increased. These findings were confirmed through the completer analyses (N=1095) and growth curve modeling, which utilized all eligible participants registering for the program (N=4425). Improvements were evident for both children and adolescents, although adolescents showed slightly less improvement on some outcome indicators. Based on the assessment point at the beginning of Session 4, results showed that around 43% of children and around one-third of adolescents had recovered (no longer experienced elevated anxiety). For those with clinical levels of anxiety, over half (57%) of children and just under half (45%) of adolescents were no longer in the clinical range after completing three sessions (ie, showed response). Importantly, even greater reductions were evident for child and adolescent users who completed more sessions, such that of those who completed nine sessions, over two-thirds (70%) were no longer in the clinical range (ie, showed response) and almost half (47%) no longer demonstrated elevated anxiety (ie, showed recovery). Furthermore, of all participants who completed nine sessions, around half (54%) achieved statistically reliable change as indicated by RCI. Thus, we observed substantial reductions in anxiety across multiple measures among users of a freely available, evidence-based online intervention.
Although not directly comparable, as diagnostic status was not determined by clinical interview in this study, the results can be compared with those of the previous RCTs of the therapist-assisted BRAVE-ONLINE program. The demonstrated effect sizes for reductions in anxiety from the baseline to Session 9 in this study (Cohen
The effect sizes for BRAVE Self-Help presented in this study are somewhat smaller than those of a pilot implementation study (N=20) conducted very recently by Jolstedt et al [
It is often argued that low-intensity interventions such as iCBT programs are only effective with professional support. However, the results of this study demonstrate that self-help online interventions may be effective for many young people if they complete at least three sessions of the program. Indeed, the improvements demonstrated by participants in this open-access, self-help program are significant and reveal that a meaningful change is feasible without therapist support. The finding that a substantial proportion of young people can achieve clinically meaningful improvements through such an intervention in a self-help format has significant implications for models of service delivery. In addition, substantial therapist time and cost savings may be afforded by such self-help interventions, contributing to an increase in the overall efficiency of youth mental health services. Furthermore, providing evidence-based services via online self-help may reach more young people at a far lower cost than face-to-face service delivery models. It would seem that if young people who would benefit from self-help programs are accurately identified, more costly resources (eg, face-to-face therapist sessions) could be reserved for those young people who need them the most.
Another implication of the findings relates to the treatment dose or magnitude of change in early sessions. Specifically, it is worth noting that the highest magnitude of change was evident following the completion of the first six sessions. Thus, the results of this study demonstrate that even a small dose of self-help treatment may be effective for some young people and would perhaps bring about a change equivalent to that from an extended program. These findings are also consistent with those of Chu et al [
This open dissemination study has some design limitations. There was no control group against which to determine whether changes simply reflect spontaneous recovery, regression to the mean, or nonspecific intervention effects. In addition, there was no initial interview to confirm the clinical diagnosis, instead a single informant and a single measure was relied upon. Although this measure is unable to provide a comprehensive overview of anxious symptomatology type and intensity, it does allow for examination of the magnitude of anxiety symptom change, which was the objective of this study. Furthermore, it was necessary to minimize participant burden in an open trial of this nature where participation was completely voluntary and self-sought.
We also note that some analyses in this study were limited to those individuals who completed at least two assessment points and, thus, had completed at least three program sessions. While this setting provides a fair evaluation of the outcome for those who completed at least a proportion of the program, it does not consider those who initially enrolled and decided not to continue at all with the program. Perhaps, such young people were not ready to participate in active treatment or they might require alternative treatment modalities. In fact, in this study, those who completed less than three sessions were more likely to reside in non-metropolitan areas, were older and had a higher anxiety severity. It is possible that low-intensity iCBT programs may be more accessible or preferred by youth who live in major cities. Also, younger children may be more likely than adolescents to complete iCBT programs with parental assistance, which may bring about higher adherence to the program. Furthermore, the finding that youth with lower levels of anxiety adhered more to the program is consistent with the objectives of low-intensity iCBT interventions and supports the notion that iCBT for youth could be useful as a first step in intervention.
In terms of sample representativeness, all participants in this study demonstrated elevated anxiety > 84th percentile on the CAS-8. Thus, all participants were experiencing anxiety at a level that was interfering with their lives; however, only half of the participants demonstrated “clinical” levels of anxiety (>94th percentile). Thus, the sample was somewhat less severe than those examined in previous RCTs of iCBT for child anxiety [
Despite its limitations, the findings of this study are encouraging and certainly justify further research in an RCT or similar design to confirm the effectiveness of this type of intervention. Historically, there has been a lack of formalized approaches to the dissemination of iCBT interventions [
This online, self-help program for anxiety has the capacity to reach greater numbers of young people compared with programs that require therapist contact. The results of this open trial demonstrate moderate to high program acceptability when delivered in this way and show its potential feasibility in bringing about clinically and statistically meaningful reductions in anxiety for children and adolescents. Greater reductions were evident for those who completed more sessions, although significant improvements were most evident in the first six sessions. Overall, self-help iCBT is a potentially feasible and acceptable approach for delivering evidence-based interventions through a public health delivery model.
A summary of analysis of variance (ANOVA) and effect sizes.
A reliable change over time.
The proportion of participants crossing the "elevated" and "clinical" thresholds.
The proportion of clinical participants moving from "clinical" to "elevated" and "normal" anxiety levels.
analysis of variance
Children’s Anxiety Scale, 8-item
cognitive behavioral therapy
internet-based cognitive behavioral therapy
latent growth curve modeling
Reliable Change Index
randomized controlled trials
University of Queensland
The authors wish to acknowledge the funders of this project,
SM, SHS, and CLD acknowledge that although intellectual property for BRAVE-ONLINE is owned by UniQuest/the University of Queensland, they may potentially benefit from royalties related to the program.