Published on 28.10.20 in Vol 22, No 10 (2020): October
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/16802, first published Oct 28, 2019.
24-Month Outcomes of Primary Care Web-Based Depression Prevention Intervention in Adolescents: Randomized Clinical Trial
Background: Adolescent depression carries a high burden of disease worldwide, but access to care for this population is limited. Prevention is one solution to curtail the negative consequences of adolescent depression. Internet interventions to prevent adolescent depression can overcome barriers to access, but few studies examine long-term outcomes.
Objective: This study compares CATCH-IT (Competent Adulthood Transition with Cognitive Behavioral Humanistic and Interpersonal Training), an internet-based intervention, to a general health education active control for depression onset at 12 and 24 months in adolescents presenting to primary care settings.
Methods: A 2-site randomized trial, blinded to the principal investigators and assessors, was conducted comparing Competent Adulthood Transition with Cognitive Behavioral Humanistic and Interpersonal Training to health education to prevent depressive episodes in 369 adolescents (193 youths were randomly assigned to Competent Adulthood Transition with Cognitive Behavioral Humanistic and Interpersonal Training and 176 to health education) with subthreshold depressive symptoms or prior depressive episodes. Participants were recruited from primary care settings in the United States. The primary outcome was the occurrence of a depressive episode, determined by the Depression Symptom Rating. The secondary outcome was functioning, measured by the Global Assessment Scale.
Results: In intention-to-treat analyses, the adjusted hazard ratio favoring Competent Adulthood Transition with Cognitive Behavioral Humanistic and Interpersonal Training for first depressive episode was not statistically significant at 12 months (hazard ratio 0.77, 95% CI 0.42-1.40, P=.39) and 24 months (hazard ratio 0.87, 95% CI 0.52-1.47, P=.61). Competent Adulthood Transition with Cognitive Behavioral Humanistic and Interpersonal Training provided preventive benefit for first depressive episode for those with mild hopelessness or at least moderate paternal monitoring at baseline. Global Assessment Scale scores improved comparably in both groups (intention-to-treat).
Conclusions: A technology-based intervention for adolescent depression prevention implemented in primary care did not have additional benefit at 12 or 24 months. Further research is necessary to determine whether internet interventions have long-term benefit.
Trial Registration: ClinicalTrials.gov NCT01893749; http://clinicaltrials.gov/ct2/show/NCT01893749.
J Med Internet Res 2020;22(10):e16802
As many as 7%-13% of adolescents living in the United States and other high-income countries experience minor or major depressive episodes each year . Adolescents with depressive episodes, compared to those without depression, have a higher incidence of social adjustment challenges [ , ] and are at risk for future suicide attempts and recurrent depressive episodes throughout life [ ]. Unfortunately, only a minority of depressed adolescents receive treatment [ ]. The negative consequences of untreated depression, delayed treatment, and extensive lack of access to treatment call for evidence-based preventive interventions that can be disseminated widely.
Recent systematic reviews and meta-analyses [, ] found that evidence-based interventions (eg, cognitive behavioral therapy) for adolescent depression prevention and treatment could be successfully delivered via the internet [ ], indicating their potential to increase access to effective preventive programs. While digital interventions can circumvent geographic barriers, time constraints, and stigma by allowing users to access interventions from their personal devices, their effects on adolescent depression prevention are mixed [ , ], and more intensive interventions with human support or guidance may have superior results [ , ].
To this point, most digital-based preventive interventions have primarily focused on short-term reduction in depressive symptoms and have not focused on how the interventions impact the risk of developing depressive episodes over time. In a review  examining 83 studies targeting depressive outcomes in youth, of which 8 were delivered online, only one-third (32/83) specifically focused on reducing the risk of depressive episodes rather than symptoms; small effect size reductions of depressive symptoms were found postintervention, and there was a modest absolute risk reduction of depressive episodes [ ]. This limitation, in part, results from a lack of prospective studies examining the long-term outcomes necessary to detect depression onset [ , ]. A prior randomized clinical trial of CATCH-IT [ - ], the internet-based depression prevention intervention evaluated in this study, demonstrated lower depressive symptoms at 6, 12, and 30 months, relative to baseline. Despite these long-term outcomes, because randomization was based on primary care delivery models of CATCH-IT, all participants in that study received CATCH-IT rather than an active comparator.
This randomized clinical trial compared the efficacy of the CATCH-IT intervention to an online general health education program in preventing the onset of depressive episodes at 12 and 24 months in an intermediate to high-risk group of adolescents recruited from primary care settings . We previously reported that at 6 months, CATCH-IT demonstrated preventive benefit against depressive episodes compared to that offered by health education for adolescents with higher depressive symptoms at baseline but not for the entire sample [ ]. Given our prior 6-month results, showing significant effects for moderation based on baseline adolescent characteristics and documenting increased benefit of psychotherapy after termination (sleeper effects) [ - ], we hypothesized that a depression prevention benefit might emerge for the CATCH-IT group, and significant moderation relationships (in prespecified theory-based covariates) might be demonstrated in intention-to-treat analysis at 12 and 24 months [ ].
Study Design and Setting
To evaluate the efficacy of CATCH-IT in a scalable setting, we used a hybrid type 1 efficacy-implementation design in which we tested the efficacy of the clinical intervention while simultaneously collecting information for large-scale implementation . Focus groups tested the prototype of the system for usability and accessibility in urban minority adolescents. Adolescent participants welcomed the platform and expressed preference for an online program that is engaging and private with some face-to-face interaction to express their feelings.
The trial model was implemented in 8 major US health systems (31 primary care sites including more than 41,000 adolescents) in a population health approach (screen all youth, offer intervention, assessment, refer those in need of treatment) with over 1200 primary care staff consenting and being trained. Data were collected from urban and suburban clinics located in Boston, Massachusetts and surrounding areas, and in Chicago, Illinois and surrounding areas, including northern Indiana. Participating primary care clinics were recruited by study staff and health care providers at the study sites. Research staff worked closely with office nursing staff conducting in-person adolescent screening and primary health care providers trained in motivational interviewing [, ].
A more detailed description of our protocol, which follows the CONSORT statement, and a detailed description of the study design and implementation process appear elsewhere [, ]. Institutional review board approval was received from the institutional review board of record (University of Illinois at Chicago) and local institutional review boards. A data safety and monitoring board reviewed the trial and results twice per year.
Eligible adolescents were aged 13 to 18 years and had either an elevated score on the Center for Epidemiologic Studies Depression (CES-D) scale  (8-17 on 10-item scale at screening or ≥16 on a 20-item scale at baseline, indicating clinically significant risk for depression) [ ] or a prior history of a major depressive episode or dysthymia [ , ].
Adolescents were excluded if they had a current DSM-IV (Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition) diagnosis of major depressive disorder or were currently being treated for major depressive disorder. Adolescents with schizophrenia, psychosis, bipolar disorder, significant reading impairment or developmental disability, current drug or alcohol abuse, imminent risk of suicide, or who were in treatment for serious medical conditions were also excluded.
Participant Recruitment and Enrollment
Adolescents were recruited from February 2012 to July 2016 via screening, posted flyers, recruitment letters (sent to all families) and by information offered directly to families by their primary care physician during clinic visits. Initial screening for risk of depression was administered either during a face-to-face interview or over the phone using the Patient Health Questionnaire-2 [, ]. A positive screen was the presence of any cardinal depression symptom (depressed mood, irritability, or anhedonia over the prior 2 weeks). Eligible participants and parents were then referred for an enrollment assessment at the office of their primary care physician. During this appointment, eligibility was formally confirmed through a full assessment (using Kiddie Schedule for Affective Disorders Scale, KSADS; CES-D10), accessibility to a computer and internet was confirmed, and baseline informed consent was obtained (see for CONSORT diagram). Participants were provided with the study website URL and personal log-in information.
We previously published a report  on the implementation of this study in 30 participating clinics with 42,310 adolescents (data were not available on adolescents from 1 additional clinic). A total of 369 adolescents were randomized into the intervention or control program. The average reach (the proportion of target audience exposed to the intervention) for screening across the Chicago study sites was 0.216, and the average reach value for enrollment across all study sites was 0.181. We describe the effect of internal barriers to recruiting adolescents (eg, screening adolescents, transferring screening results to the research team) on the screening and enrollment process by reporting reach; these internal barriers, which were within the practices, are related to the implementation of universal screening [ ].
Randomization and Masking
Random assignment to CATCH-IT or health education was achieved via a computer-generated sequence blocked by site and time of entry. Participants were also stratified according to their gender, age (13-14 years; 15-18 years) [, ], and the severity of their risk of depressive episode (current CES-D score or prior episode of major depressive disorder). Allocation to CATCH-IT versus health education was 1:1. Although condition assignment was kept from participants, families, clinicians, and investigators until all initial assessments and preliminary data gathering were completed, it was not possible to blind the participants to their assignment, nor could the primary care clinician be blinded, as CATCH-IT implementation required clinicians to perform 3 motivational interviews over the course of the study. However, the 2 arms were presented to adolescents and parents as different approaches to improving adolescent health (eg, coping skills training or physical health improvement). Assessors were blinded for the duration of the study, and investigators were blinded until all 12-month data were collected.
The CATCH-IT Intervention
CATCH-IT, an internet-based depression prevention intervention delivered in primary care settings, sought to address gaps in the field of adolescent depression prevention: (1) the implementation of a low cost and low burden intervention in primary care where youth and their families usually receive care and (2) the efficacy of an intervention in reducing diagnosed depressive episodes (versus only symptom reduction) across a 24-month follow-up period [, ]. CATCH-IT, originally developed in the United States, incorporates cognitive behavioral therapy and interpersonal psychotherapy and was developed to have high levels of interactivity using previously published methods [ , ].
CATCH-IT integrates a set of sequential online modules with a coaching component consisting of direct interviews and periodic phone calls. There are 15 adolescent modules (14 modules on depression and an optional anxiety module with relaxation technique) designed to develop coping strategies protective against depression and 5 parent modules  (described in prior publications in detail) [ , , ]. Each module takes approximately 15-20 minutes. Adapted from the Coping with Depression Adolescent Course, in combination with behavioral activation strategies (for example, behavioral scheduling of pleasurable activities) [ ] and techniques used in interpersonal psychotherapy (for example, effective social problem solving, building and engaging social support) [ ], CATCH-IT encourages the user to adopt a proactive approach toward mental health maintenance. Motivational support includes 3 formal motivational interviews delivered at baseline, 2 months, and 12 months by the participant’s primary care physician, and 1 to 3 coaching phone calls by study staff. A separate CATCH-IT parent program, based on Beardslee and Gladstone’s Preventive Intervention Project [ ], included 4 modules that provided information about youth depression and ways to promote resilience in adolescents, and included a separate module for parents who think they may be depressed.
The Health Education Intervention
Health education, a 14-module online health educational tool, was used as an active control. Health education addresses adolescent health, wellness, and general safety. Module 14 discusses the identification of mood symptoms, the importance of seeking treatment for mental health concerns, and stigma associated with mental health treatment that might act as a barrier towards seeking professional help ; 4 modules for parents were included.
Shared Elements Across Both Treatment Arms
Both intervention arms provided care consistent with Guidelines for Adolescent Depression in Primary Care (GLAD-PC) , closely related to the chronic care model. This included screening adolescents for depression; training clinicians in the identification, diagnosis, and treatment of depression; establishing a treatment plan; creating referral relationships; and developing a safety plan [ ]. Up to 3 check-in calls were made during the first 3 weeks of intervention to ensure participants had gained access to the website. We provide estimates of nonintervention time related to assessment and follow-up. These were developed based on total number of calls and sampling of call duration (40 randomly selected files to determine mean duration of calls and assessments).
The primary outcome was occurrence of first depressive episode, where a Depression Symptom Rating (DSR) score indicating at least subthreshold major depression (DSR≥3) was considered a depressive episode. The secondary outcome was mental health functioning measured by GAS scores (: Table S10).
At baseline and during follow-up assessments, demographic information, educational status, and institution were collected from the participants and their families.
Fidelity and Adherence
Fidelity to the intervention was based on completion of the motivation interviews, quality of the motivation interviews as rated by the Motivational Interview Treatment Integrity (4.2.1) which ranges from 1 (low quality) to 5 (high quality)  and number of characters typed into the CATCH-IT website. We also measured modules completed and total time on the website by adolescents and parents. Additionally, at the 2-year time point, we asked 11 adolescents and parents 4 questions regarding satisfaction with the study and intervention.
The 2-question screener was based on the Patient Health Questionnaire-Adolescent . The K-SADS [ , ], a semistructured interview assessing current and lifetime psychiatric diagnoses in participants under age 18, was administered with parents and adolescents by licensed mental health professionals [ , ]. DSR scores were obtained from the K-SADS and the Kiddie Longitudinal Interval Follow-up Evaluation [ ], which was used to identify each week of the follow-up interval’s onset and offset of diagnoses based on recall, and GAS ratings were ascertained at each assessment. The K-SADS and GAS were completed at baseline and at 2, 6, 12, and 24 months. To test for robustness of findings, we examined outcomes using DSR cut-points of ≥3, ≥4, and ≥5, indicating probable major depressive episode [ ]. We selected DSR≥3 (at least subthreshold major depressive episode) versus DSR≥4 (probable or definite major depressive episode) as our primary outcome due to the lower than expected occurrence of DSR≥4 in the entire sample, identified during data safety monitoring board reviews of data. For assessment of presence of current or prior episodes (secondary versus primary prevention), we conducted an exploratory analysis of whether presence or absence of DSR≥3 or ≥4 affected subsequent outcomes.
Covariates in Moderation Analyses
We have proposed and published the technology-based behavioral vaccine model as an integrated conceptual framework for understanding the prevention of depression across the lifespan through internet-based interventions in community settings [, , ]. Based on this model, we propose moderators within this conceptualization: (1) a life course schedule that is theory-driven and that includes booster doses operationalized as patient or participant factors influencing response over the life course, including demographics such as gender, race, ethnicity, age, parental education, and site; vulnerability factors (CES-D10, Adolescent Life Events Questionnaire, Beck Hopelessness Scale, Child Report of Parental Behavior Inventory); child comorbid psychopathology (Child Anxiety Related Emotional Disorders; Disruptive Behaviors Disorder Scale; substance abuse, using CRAFFT [Car, Relax, Alone, Forget, Friends, Trouble]); and parent psychopathology (CES-D10); (2) effective components of information and training to encode responses to future threats that can then be deployed at some future points using information (operationalized as internet modules completed, time on site); (3) a motivational framework to boost response to behavior prescription (intrinsic motivation operationalized as Theory of Planned Behavior Scale, Transtheoretical Model Scale) as well as personal relevance (Sociocultural Relevance Scale); and (4) a structured implementation strategy as, to intervene effectively, we must address or provide proper conditions for successful participation in the intervention such as positive primary care relationships (operationalized Physician Relationship Scale) [ , , , ].
We estimated incidence rates by calculating the number of depressive episodes per 10,000 person-weeks of follow-up. Kaplan-Meier curves were used to estimate the time to first episode distribution for each intervention under 6 different treatment allocations: intention-to-treat, modified intention-to-treat (CATCH-IT adolescents who completed the baseline motivational interview), as treated (visited website at least once), per-protocol 2, per-protocol 4, and per-protocol 7 (adolescent and parent completed a total of 2, 4, 7 modules, respectively) (: Table S1-2). Cox proportional hazard regression was used to estimate the hazard ratio comparing CATCH-IT to health education, adjusted for covariates. We present hazard ratios and coefficients adjusted for gender, ethnicity (Hispanic or non-Hispanic), race (White or non-White), baseline age, site (Boston or Chicago), and baseline CES-D10 score. We conducted exploratory analysis of the moderating effects of theory-based covariates by including interaction terms in the Cox models. We conducted the analysis of CES-D10 as a moderator in both the full sample and the subsample with elevated baseline CES-D10 scores. To examine whether the sizable number of moderator analyses we tested reveal significant effects more than by chance, we plotted all the ordered P values that were observed against what would be observed by chance in a quantile–quantile plot ( : Figure S1). We used linear mixed-effect growth models with random intercept and slope to examine differences between groups in change over time in GAS. We transformed the time scale in our growth models toward linearity by using a logarithm transformation of time. For the exploratory analysis of the impact of prior episodes on outcomes, standard descriptive statistics were used. Original sample size calculations to achieve 80% power were based on randomization of 200 into each arm and 28% incidence in the control condition at 12 months, a constant hazard ratio of 0.62, and total attrition of 36% of the sample [ ]. Analyses were conducted using R (version 3.3.1), SAS (version 9.4; SAS Institute), and Mplus (version 8) [ ].
Of the 369 participants, 193 were randomized to CATCH-IT, and 176 were randomized to health education (); depressive episode assessments (K-SADS) were available for 278 participants (75.3%) at 12 months and 181 (49.1%) at 24 months or greater ( : Table S3). The mean time observed was 524.8 days (SD 314.8 days) from baseline to final K-SADS. Predictors of missing episode assessment data were consistent at 12 and 24 months and included Chicago site, CATCH-IT group, older age, and lower parental education ( : Table S4-S5).
Adolescents were 13 to 18 years (mean 15.4 years, SD 1.5); 62% (233/3654) of adolescents had at least a prior subthreshold depressive episode (DSR≥3), and 40% (147/3654) had at least a prior probable depressive episode (DSR≥4). At baseline, the mean CES-D10 (0-30 scale) score was 9.4 (SD 4.6), the mean GAS score was 78.1 (SD 9.4), and the mean score on the Beck Hopelessness Scale was 4.7 (SD 3.6). Of the 369 participants, 16 (4.3%) adolescents reported recurrent thoughts (subthreshold or threshold) of death at baseline, 2 (0.5%) adolescents reported subthreshold suicidal ideation, and 2 (0.5%) adolescents reported subthreshold nonsuicidal self-harm thoughts. For parents at baseline, the mean CES-D10 score was 6.4 (SD 5.0), and 22.9% (72/314) were at least moderately depressed (CES-D20≥16). Adolescents were diverse in self-reported race and ethnicity: 159/369 (43.1%) identified as non-Hispanic White; 94/369 (25.5%) identified as non-Hispanic Black; and 77/369 (20.9%) identified as Hispanic. The parents of about half of the adolescents (mothers: 144/359, 40.1%; fathers: 157/336, 46.7%) had earned a high school diploma or less, and 39.4% (143/363) were not married. Relative to Chicago, the Boston site had more adolescents who identified as White (P<.001), a lower percentage of ethnic minority adolescents (P=.01), greater levels of parent education (P=.03 for mothers and P<.001 for fathers), more parents who were married at baseline (P=.03), and parents who were older (P<.001) (). A complete description of the cohort data at baseline is provided in a prior publication [ ].
Fidelity and Adherence
Tests of fidelity and adherence have been described elsewhere  and are shown in and : Tables S11-S12. The median number of adolescent and parent modules completed was 4 in the CATCH-IT group and 8 in the health education group. Number of characters typed was used to measure active use of the modules, with adolescents and parents (combined) in the CATCH-IT arm typing a mean of 3713 (median 1899) characters. Adolescents spent more time on the CATCH-IT site than on the health education site (CATCH-IT: median 39.6 minutes; health education: median 8.4 minutes) but completed more modules on health education (CATCH-IT: median 1.0; health education: median 4.0). For CATCH-IT, 14 adolescents (14/193, 7.3%) completed all 15 modules; for health education, 74 adolescents (74/176, 42.0%) completed all 14 modules.
The median number of motivational interviews completed was 2 for both adolescents and parents, with 94.8% (183/193) of adolescents completing at least 1 and 34.2% (66/193) completing all 3 (and : Table S11). Motivational interviews were assessed for fidelity by 2 trained raters using the Motivation Interview Treatment Integrity 4.2.1 coding manual. The mean length of motivation interview was 7.7 minutes (SD 4.0), and mean lengths of technical and relational global ratings were 3.0 (SD 0.5) and 2.9 (SD 0.6), respectively ( : Table S11). Based on a log recording telephone and email contacts with Chicago participants ( : Table S12), a mean of 17.0 (SD 8.9) and 14.1 (SD 6.5) contacts were made with CATCH-IT and health education participants, respectively. We estimated mean total time required to complete assessment and follow-up common to both arms with GLAD-PC model and total time specific for the intervention (101 minutes for assessment and follow-up common to both arms, 171 minutes for CATCH-IT and 31 minutes for health education, and in total contact time, assessment and follow-up plus intervention time, 272 minutes versus 132 minutes for CATCH-IT and health education, respectively) ( : Table S13). With regard to the end of study satisfaction questions, adolescents noted they valued the periodic assessment calls above all other study elements, regardless of study arm.
|Outcomes||CATCH-ITa, mean (SD)||Health education, mean (SD)||CATCH-IT, median (IQR)||Health education, median (IQR)||P valueb|
|Modules completed||3.4 (4.7)||6.8 (6.5)||1 (4)||4 (14)||.003|
|Total minutes on site||100.2 (143.1)||22.8 (31.0)||39.6 (149.2)||8.4 (35.1)||<.001|
|Days visited site||3.7 (4.5)||1.4 (1.6)||2 (4)||1 (2)||<.001|
|Total characters typed||3071 (4572)||N/Ad||923 (4469)||N/A||N/A|
|Modules completed||2.1 (2.0)||2.2 (1.9)||2 (4)||4 (4)||.80|
|Total minutes on site||32.6 (37.3)||8.6 (10.0)||22.4 (51.9)||5.6 (14.9)||<.001|
|Days visited site||1.6 (1.6)||0.9 (1.1)||1 (2)||1 (1)||<.001|
|Total characters typed||716 (977)||N/A||101 (1205)||N/A||N/A|
|Adolescents and parents|
|Modules completed||5.3 (5.8)||8.8 (7.3)||4 (8)||8 (17)||<.001|
|Total minutes on site||130.6 (157.9)||30.6 (35.6)||75.8 (192.2)||18.9 (40.8)||<.001|
|Days visited site||5.2 (5.2)||2.2 (2.2)||4 (6)||2 (2)||<.001|
|Total characters typed||3713 (4932)||1899 (5792)|
|Adolescents||2.0 (0.9)||N/A||2 (2)||N/A||N/A|
|Parents||1.9 (0.9)||N/A||2 (2)||N/A||N/A|
|Adolescents and parents||3.8 (1.8)||N/A||4 (4)||N/A||N/A|
|Participant contactsg||17 (9)||N/A||18 (13)||N/A||.004|
aCATCH-IT: Competent Adulthood Transition with Cognitive Behavioral Humanistic and Interpersonal Training.
bMedians were compared using Wilcoxon rank-sum test.
cn=193 for the CATCH-IT group; n=176 for the health education group.
dN/A: not applicable.
en=165 for the CATCH-IT group; n=157for the health education group.
fCATCH-IT adolescents and parents were offered 3 motivational interviews; seefor additional information.
gn=248 telephone calls and emails with Chicago participants, including administrative contacts such as scheduling calls in addition to coaching calls, safety checks, and motivational interview calls. Seefor additional information.
Time to Depressive Episode
In intention-to-treat analyses, the hazard ratio favored CATCH-IT for occurrence of first depressive episode but was statistically significant neither at 12 months (hazard ratio 0.77, 95% CI 0.42-1.40, P=.39;: Table S1) nor at 24 months (hazard ratio 0.87, 95% CI 0.52-1.47, P=.61, : Table S2). For per-protocol 2 analysis (at least 2 modules completed by adolescent and parent combined in either arm), the effect of CATCH-IT on depressive episode prevention was improved but remained nonsignificant at 12 months (hazard ratio 0.65, 95% CI 0.34-1.23, P=.18, : Table S1) and 24 months (hazard ratio 0.73, 95% CI 0.41-1.28, P=.27; : Table S2). Annual incidence of first at least subthreshold depressive episode (DSR≥3) was 12.7% for CATCH-IT and 13.8% for health education; incidence of probable major depressive episode (DSR≥4) was 4.6% for CATCH-IT and 5.8% for health education ( : Table S6).
Functional Status Improvement
GAS scores improved in each group from baseline to 24 months (: Table S17). For GAS with time logarithm transformed, the slope was 2.964 (SE 0.285; P<.001), and adjusted change from baseline to 24 months was 9.5 (SE 0.9) for CATCH-IT, compared to a slope of 3.076 (SE 0.278; P<.001) and change of 9.9 (SE 0.9) for health education. There was no significant difference in slopes between groups (P=.78) ( : Table S17).
Through 12 months, within the group of adolescents who enrolled based on elevated CES-D (321/369, 86.9%) with or without a prior depressive episode, those with higher baseline CES-D10 scores exhibited a marginally stronger preventive effect of CATCH-IT on time to event (DSR≥3) relative to those with lower baseline scores (0.15 decline in the logarithm of hazard ratio for each CES-D10 point increase, P=.054) (: Table S7, ). Those who also visited the website at least once (as treated) (253/369, 68.6%) showed a similar effect (0.18 decline in the logarithm of hazard ratio for each CES-D10 point increase, P=.03). Baseline CES-D10 was no longer a significant moderator at 24 months, either in the sample as a whole or in the subgroups tested. However, 2 additional moderators were identified as significant on time to event through 24 months (DSR≥3). Specifically, there was a complex interaction between baseline hopelessness (270/369, 73.2%) whereby, at lower Beck Hopelessness Scale scores, CATCH-IT demonstrated a preventive effect, but results favored health education at higher scores (P=.04) ( : Table S8, ). Similarly, there was a complex interaction between baseline paternal monitoring (170/369, 46.1%) and group assignment, such that at higher levels of paternal monitoring at baseline, CATCH-IT demonstrated preventive effects on time to event (DSR≥3) but favored health education at lower paternal monitoring (P=.048, : Table S9, ). Site, race, ethnicity, and gender did not moderate outcomes. We conducted an overall comparison of the 12-month and 24-month moderation effects to see whether there was any signal evident beyond what one would expect from a null distribution. : Figure S1 provides a comparison of all the ordered P values against this standard. The moderation effects for hopelessness and paternal monitoring did not stand out against the null distribution. No GAS moderation analyses yielded statistically significant results after time transformation (all P>.05 after logarithm transformation, : Table S18).
Assessment of presence of current or prior episodes (secondary versus primary prevention) showed there was no meaningful variation in the presence of current or past DSR≥3 or ≥4 across groups at baseline (: Table S14). With the main outcome threshold of DSR≥3, about two-thirds of the sample could be characterized as secondary prevention and one-third primary prevention ( : Table S14). However, most of the episodes of DSR≥3 during follow-up were new episodes; only 4 episodes were present or persisting at 2 months in each study arm ( : Table S15). Adolescents with past or current episodes at baseline were statistically equally likely to have a follow-up episode regardless of group assignment. For adolescents with a current or past episode with DSR≥3 at baseline, 17.9% of CATCH-IT and 19.8% of health education participants had an episode during the follow-up period ( : Table S16).
This hybrid Type 1 effectiveness-implementation clinical trial  conducted in primary care practice settings (31 sites) with a diverse sample is the only study, to our knowledge, with over 12 months of follow-up for adolescents enrolled in an internet-based study [ , , ]. At 12 and 24 months postintervention using Cox proportional hazard regression analysis, we found that CATCH-IT was not superior to health education in preventing the onset of depressive episodes. The annual incidence of major depressive episodes was fairly low for both groups (DSR≥4: 4.6% CATCH-IT, 5.8% health education; DSR≥3: 13.7% CATCH-IT, 12.7%). This compares favorably with the 12-month incidence of major depressive episodes (DSR≥4) in the United States, which is 8.3% [ ] in general samples and >15% in similar risk–adjusted samples; these rates are roughly half the rates for minor plus major depressive episodes in high-risk samples (DSR≥3) [ , , ]. Moderated analyses suggested that adolescents who enroll with higher CES-D scores, lower levels of hopelessness, and higher paternal monitoring may obtain preventive benefit from CATCH-IT for 12 months, and possibly for as long as two years postenrollment.
There are several potential explanations for why this trial failed to show a main effect of the CATCH-IT intervention. First, the incidence of depressive episodes was much lower than anticipated for this high-risk sample, and remained lower than expected even after lowering the threshold of primary outcome from probable depressive episode (DSR ≥4) to subthreshold depressive episode (DSR≥3). An important factor may be the amount of assessment follow-up time common to both arms (150 minutes) within the framework of the GLAD-PC guidelines , which was nearly equal to the total intervention time for CATCH-IT (170 minutes), rather than any particular effect of the health education intervention (30 minutes). It is possible that this level of assessment and follow-up suppressed the level of depressive episodes in both arms, depriving the CATCH-IT intervention of some of the potential to prevent episodes. The demonstrated intervention of a Chinese language CATCH-IT at 12 months which had only self-assessments but did have an attention control is consistent with this explanation [ ], as is the significant improvement in functional status over time in both groups. Another explanation would be that, in fact, the active control reduced between group differences since most adolescent studies with active controls have not demonstrated between group differences [ , ]. The recruitment of slightly less than the target sample size (92%), higher attrition (>50% versus 36% anticipated), weaker intervention effect (hazard ratio 0.87 versus predicted hazard ratio 0.62), and lower dose (hazard ratio more favorable for those with higher dose) could all play roles in the nonsignificant results in intention-to-treat analyses. Perhaps a dose threshold of 2-4 modules (including cognitive restructuring, behavioral activation) is essential, as occurred in the Chinese CATCH-IT trial [ ] and other studies [ - ]. Additionally, the fact that some teens seem to respond more favorably to the health education intervention (those with no current symptoms), while others to CATCH-IT (those with symptoms), reduced between group differences.
Many have called for examination of moderators for treatment tailoring and to contribute to future meta-analyses parsing out differential treatment outcomes . Moderated results suggest that CATCH-IT may benefit a significant portion of youth, such as groups of youth with elevated depressive symptoms. For example, adolescents who were enrolled with no current depressed mood but with a prior depressive episode fared worse with CATCH-IT, while those with depressed mood may benefit for as long as 12 months. Similarly, adolescents with lower hopelessness may benefit from CATCH-IT, while those with higher levels may not benefit from such cognitive behavioral therapy–based interventions [ ]. It is possible that the context of social support may matter, such that CATCH-IT, which requires youth effort (eg, completion of cognitive behavioral therapy homework), may offer preventive benefit for those who report adequate paternal monitoring, consistent with Mohr’s supportive accountability model [ ]. One might speculate that more paternal involvement in the health education intervention may be overbearing because less accountability is needed. While female gender predicted higher risk of episodes, results did not vary by gender nor did they vary by ethnicity. The results of this trial along with those from prior trials [ ] suggest that interventions can prevent depressive episode onset and worsening of depressive symptoms in youth subpopulations.
This study has important limitations and strengths. While it lacked an inactive or wait-list control, wait-list control is less ethical, as effective interventions are available. We cannot deconstruct the effects of motivation interview versus the online modules, but depressive symptoms decreased for adolescents who used CATCH-IT even without a motivation interview in our pilot study . Differential loss to follow-up for those in the CATCH-IT group and at the Chicago site is concerning. However, predictors of increased risk of episodes, such as depressed mood or female gender, did not predict loss to follow-up, and make it unlikely that hazard ratios or incidence of episodes were affected by dropout patterns. Measurement of depressive episodes is challenging, and measurement issues could explain lower than expected rates. However, both sites were trained by an experienced evaluation team with methods used in prior clinical trials [ ]. Despite these limitations, this study has several strengths. The standardized implementation across several primary care settings supports feasibility of this model of care, though it will be important to determine potential logistical barriers such as check-in calls to participants. The examination of depressive episodes, long-term follow up, active control, and blinding of outcome assessors address methodological limitations of prior studies.
Future investigators should exercise caution in the design of low-intensity technology-based preventive interventions when considering active controls and extensive assessment batteries. Rather than an active control, an informational brochure may be a better option for a control; likewise, rather than using the KSADS to gather diagnostic information, the much shorter Mini-International Neuropsychiatric Interview for Children and Adolescents could be considered. Similarly, careful stakeholder-grounded intervention design (additional incentives or complementary design features, for example, online discussion groups or animation) may be needed to induce adolescent-parent pairs to complete more modules and thus strengthen the preventive effect to ensure minimum dose levels in the target population. Moderated results suggest that life course factors (depressed mood) and family factors (paternal monitoring) may be related to the potential for efficacy beyond 6 months. For example, a general health promotion model intervention may be more appropriate for adolescents at risk for a major depressive episode but currently with minimal symptoms, or conversely, CATCH-IT may not benefit adolescents with high levels of hopelessness and low levels of paternal monitoring. In short, we have a great deal to learn about how to implement technology-based depression prevention interventions and for whom they will work, in primary care and community settings. Future research should investigate how to best tailor online interventions to the characteristics of teens at risk of depression in order to optimize outcomes and prevent the development of major depression.
In memoriam, CB passed away on August 2, 2019. An internationally renowned pioneer of community mental health, CB had a lifelong commitment to the south side of Chicago. This paper, which is one of his more than 500 scholarly texts, adds to his seminal work on the power of resilience in the face of adversity and his advocacy of prevention. CB was a role model for many, including the other authors of this paper who will strive to carry on his legacy. CB mentored the senior author for 16 years; without the insight, support, and wisdom of CB, this project could never have been completed. CB is sorely missed.
Research reported in this article was supported by the National Institute of Mental Health of the National Institutes of Health under award number R01MH090035. The implementation process was developed with funding from Robert Wood Johnson Foundation. The funders had no involvement in study design, data collection and analysis, and preparation and submission of the manuscript. The interest of Asok K Ray, MD FRCS (Edin), Purnima Ray, and Malika Ray in this manuscript and research is appreciated.
Ethical bodies that approved this study include institutional review boards from Wellesley College, University of Illinois, Advocate Health Care, Franciscan St. Mary, Northwestern, and NorthShore University Health System.
Conflicts of Interest
Supplemental tables and figure.DOCX File , 191 KB
- Bromet E, Andrade LH, Hwang I, Sampson NA, Alonso J, de Girolamo G, et al. Cross-national epidemiology of DSM-IV major depressive episode. BMC Med 2011 Jul 26;9:90 [FREE Full text] [CrossRef] [Medline]
- Janssens KAM, Rosmalen JGM, Ormel J, van Oort FVA, Oldehinkel AJ. Anxiety and depression are risk factors rather than consequences of functional somatic symptoms in a general population of adolescents: the TRAILS study. J Child Psychol Psychiatry 2010 Mar;51(3):304-312. [CrossRef] [Medline]
- Pettit JW, Lewinsohn PM, Roberts RE, Seeley JR, Monteith L. The long-term course of depression: development of an empirical index and identification of early adult outcomes. Psychol Med 2009 Mar;39(3):403-412 [FREE Full text] [CrossRef] [Medline]
- Weissman MM, Wolk S, Goldstein RB, Moreau D, Adams P, Greenwald S, et al. Depressed adolescents grown up. JAMA 1999 May 12;281(18):1707-1713. [CrossRef] [Medline]
- Rushton J, Bruckman D, Kelleher K. Primary care referral of children with psychosocial problems. Arch Pediatr Adolesc Med 2002 Jun;156(6):592-598. [CrossRef] [Medline]
- Hollis C, Falconer CJ, Martin JL, Whittington C, Stockton S, Glazebrook C, et al. Annual Research Review: Digital health interventions for children and young people with mental health problems - a systematic and meta-review. J Child Psychol Psychiatry 2017 Apr;58(4):474-503. [CrossRef] [Medline]
- Clarke AM, Kuosmanen T, Barry MM. A systematic review of online youth mental health promotion and prevention interventions. J Youth Adolesc 2015 Jan;44(1):90-113. [CrossRef] [Medline]
- Mohr DC, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res 2011 Mar 10;13(1):e30 [FREE Full text] [CrossRef] [Medline]
- Hetrick SE, Cox GR, Witt KG, Bir JJ, Merry SN. Cognitive behavioural therapy (CBT), third-wave CBT and interpersonal therapy (IPT) based interventions for preventing depression in children and adolescents. Cochrane Database Syst Rev 2016 Aug 09(8):CD003380. [CrossRef] [Medline]
- Abuwalla Z, Clark MD, Burke B, Tannenbaum V, Patel S, Mitacek R, et al. Long-term telemental health prevention interventions for youth: A rapid review. Internet Interv 2018 Mar;11:20-29 [FREE Full text] [CrossRef] [Medline]
- Van Voorhees BW, Fogel J, Reinecke MA, Gladstone T, Stuart S, Gollan J, et al. Randomized clinical trial of an Internet-based depression prevention program for adolescents (Project CATCH-IT) in primary care: 12-week outcomes. J Dev Behav Pediatr 2009 Feb;30(1):23-37 [FREE Full text] [CrossRef] [Medline]
- Saulsberry A, Marko-Holguin M, Blomeke K, Hinkle C, Fogel J, Gladstone T, et al. Randomized clinical trial of a primary care internet-based intervention to prevent adolescent depression: one-year outcomes. J Can Acad Child Adolesc Psychiatry 2013 May;22(2):106-117 [FREE Full text] [Medline]
- Richards K, Marko-Holguin M, Fogel J, Anker L, Ronayne J, Van Voorhees BW. Randomized clinical trial of an internet-based intervention to prevent adolescent depression in a primary care setting (CATCH-IT): 2.5-year outcomes. J Evid Based Psychother 2016 Sep;16(2):113-134 [FREE Full text] [Medline]
- Gladstone TG, Marko-Holguin M, Rothberg P, Nidetz J, Diehl A, DeFrino DT, et al. An internet-based adolescent depression preventive intervention: study protocol for a randomized control trial. Trials 2015 May 01;16:203 [FREE Full text] [CrossRef] [Medline]
- Gladstone T, Terrizzi D, Stinson A, Nidetz J, Canel J, Ching E, et al. Effect of internet-based cognitive behavioral humanistic and interpersonal training vs. internet-based general health education on adolescent depression in primary care: a randomized clinical trial. JAMA Netw Open 2018 Nov;1(7):1-15 [FREE Full text] [CrossRef] [Medline]
- Carroll KM, Ball SA, Martino S, Nich C, Babuscio TA, Rounsaville BJ. Enduring effects of a computer-assisted training program for cognitive behavioral therapy: a 6-month follow-up of CBT4CBT. Drug Alcohol Depend 2009 Feb 01;100(1-2):178-181 [FREE Full text] [CrossRef] [Medline]
- Hartmann A, von Wietersheim J, Weiss H, Zeeck A. Patterns of symptom change in major depression: classification and clustering of long term courses. Psychiatry Res 2018 Sep;267:480-489. [CrossRef] [Medline]
- von Brachel R, Hirschfeld G, Berner A, Willutzki U, Teismann T, Cwik JC, et al. Long-term effectiveness of cognitive behavioral therapy in routine outpatient care: a 5- to 20-year follow-up study. Psychother Psychosom 2019;88(4):225-235. [CrossRef] [Medline]
- Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 2012 Mar;50(3):217-226 [FREE Full text] [CrossRef] [Medline]
- Mahoney N, Gladstone T, DeFrino D, Stinson A, Nidetz J, Canel J, et al. Prevention of adolescent depression in primary care: barriers and relational work solutions. Calif J Health Promot 2017 Aug;15(2):1-12 [FREE Full text] [Medline]
- Radloff LS. The use of the Center for Epidemiologic Studies Depression Scale in adolescents and young adults. J Youth Adolescence 1991 Apr;20(2):149-166. [CrossRef]
- Vilagut G, Forero CG, Barbaglia G, Alonso J. Screening for depression in the general population with the Center for Epidemiologic Studies Depression (CES-D): a systematic review with meta-analysis. PLoS One 2016;11(5):e0155431 [FREE Full text] [CrossRef] [Medline]
- Van Voorhees BW, Paunesku D, Gollan J, Kuwabara S, Reinecke M, Basu A. Predicting future risk of depressive episode in adolescents: the Chicago Adolescent Depression Risk Assessment (CADRA). Ann Fam Med 2008;6(6):503-511. [CrossRef] [Medline]
- Johnson JG, Harris ES, Spitzer RL, Williams JBW. The patient health questionnaire for adolescents: validation of an instrument for the assessment of mental disorders among adolescent primary care patients. J Adolesc Health 2002 Mar;30(3):196-204. [CrossRef] [Medline]
- Pastore DR, Juszczak L, Fisher MM, Friedman SB. School-based health center utilization: a survey of users and nonusers. Arch Pediatr Adolesc Med 1998 Aug;152(8):763-767. [CrossRef] [Medline]
- Piantadosi S. Clinical Trials: A Methodologic Perspective. New York, NY: John Wiley & Sons; 2017.
- Gagne R, Briggs L, Wager W. Principles of Instructional Design. Fort Worth, TX: Harcourt Brace Jovanovich College Publishers; 1992.
- Van Voorhees BW, Watson N, Bridges JFP, Fogel J, Galas J, Kramer C, et al. Development and pilot study of a marketing strategy for primary care/internet-based depression prevention intervention for adolescents (the CATCH-IT intervention). Prim Care Companion J Clin Psychiatry 2010;12(3):e1-e9 [FREE Full text] [CrossRef] [Medline]
- Van Voorhees BW, Gladstone T, Cordel S, Marko-Holguin M, Beardslee W, Kuwabara S, et al. Development of a technology-based behavioral vaccine to prevent adolescent depression: A health system integration model. Internet Interv 2015 Sep;2(3):303-313 [FREE Full text] [CrossRef] [Medline]
- Jacobson N, Martell C, Dimidjian S. Behavioral activation treatment for depression: returning to contextual roots. Clinical Psychology-Science and Practice 2001;8(3):255-270. [CrossRef]
- Robertson M. Interpersonal Psychotherapy-A Clinician's Guide. London: CRC Press; 2003.
- Beardslee WR, Gladstone TRG, Wright EJ, Cooper AB. A family-based approach to the prevention of depressive symptoms in children at risk: evidence of parental and child change. Pediatrics 2003 Aug;112(2):e119-e131. [CrossRef] [Medline]
- Van Voorhees BW, Fogel J, Houston TK, Cooper LA, Wang NY, Ford DE. Beliefs and attitudes associated with the intention to not accept the diagnosis of depression among young adults. Ann Fam Med 2005;3(1):38-46. [CrossRef] [Medline]
- Zuckerbrot RA, Cheung A, Jensen PS, Stein REK, Laraque D, GLAD-PC STEERING GROUP. Guidelines for Adolescent Depression in Primary Care (GLAD-PC):part i. practice preparation, identification, assessment, and initial management. Pediatrics 2018 Dec;141(3):e20174081. [CrossRef] [Medline]
- Moyers TB, Rowell LN, Manuel JK, Ernst D, Houck JM. The Motivational Interviewing Treatment Integrity Code (MITI 4): rationale, preliminary reliability and validity. J Subst Abuse Treat 2016 Jun;65:36-42 [FREE Full text] [CrossRef] [Medline]
- Kaufman J, Birmaher B, Brent D, Rao U, Flynn C, Moreci P, et al. Schedule for Affective Disorders and Schizophrenia for School-Age Children-Present and Lifetime Version (K-SADS-PL): initial reliability and validity data. J Am Acad Child Adolesc Psychiatry 1997 Jul;36(7):980-988. [CrossRef] [Medline]
- Kaufman J, Birmaher B, Brent DA, Ryan ND, Rao U. K-SADS-PL. J Am Acad Child Adolesc Psychiatry 2000 Oct;39(10):1208. [CrossRef] [Medline]
- Keller MB, Lavori PW, Friedman B, Nielsen E, Endicott J, McDonald-Scott P, et al. The Longitudinal Interval Follow-up Evaluation. A comprehensive method for assessing outcome in prospective longitudinal studies. Arch Gen Psychiatry 1987 Jun;44(6):540-548. [CrossRef] [Medline]
- Garber J, Clarke GN, Weersing VR, Beardslee WR, Brent DA, Gladstone TRG, et al. Prevention of depression in at-risk adolescents: a randomized controlled trial. JAMA 2009 Jun 03;301(21):2215-2224 [FREE Full text] [CrossRef] [Medline]
- Van Voorhees BW, Mahoney N, Mazo R, Barrera AZ, Siemer CP, Gladstone TRG, et al. Internet-based depression prevention over the life course: a call for behavioral vaccines. Psychiatr Clin North Am 2011 Mar;34(1):167-183 [FREE Full text] [CrossRef] [Medline]
- Van Voorhees BW, Vanderplough-Booth K, Fogel J, Gladstone T, Bell C, Stuart S, et al. Integrative internet-based depression prevention for adolescents: a randomized clinical trial in primary care for vulnerability and protective factors. J Can Acad Child Adolesc Psychiatry 2008 Nov;17(4):184-196 [FREE Full text] [Medline]
- Muthén LK, Muthén BO. Statistical analysis with latent variables. Mplus. URL: https://statmodel.com/download/usersguide/MplusUserGuideVer_8.pdf [accessed 2020-09-01]
- Booth KVP, Paunesku D, Msall M, Fogel J, Van Voorhees BW. Using population attributable risk to help target preventive interventions for adolescent depression. Int J Adolesc Med Health 2008;20(3):307-319. [CrossRef] [Medline]
- Kruger J, Kim P, Iyer V, Marko-Holguin M, Fogel J, DeFrino D, et al. Evaluation of protective and vulnerability factors for depression following an internet-based intervention to prevent depression in at-risk adolescents. International Journal of Mental Health Promotion 2017 Apr 19;19(2):69-84. [CrossRef]
- Ip P, Chim D, Chan KL, Li TMH, Ho FKW, Van Voorhees BW, et al. Effectiveness of a culturally attuned Internet-based depression prevention program for Chinese adolescents: A randomized controlled trial. Depress Anxiety 2016 Dec;33(12):1123-1131. [CrossRef] [Medline]
- Rasing SPA, Creemers DHM, Janssens JMAM, Scholte RHJ. Depression and anxiety prevention based on cognitive behavioral therapy for at-risk adolescents: a meta-analytic review. Front Psychol 2017;8:1066 [FREE Full text] [CrossRef] [Medline]
- Weersing VR, Jeffreys M, Do MT, Schwartz KTG, Bolano C. Evidence base update of psychosocial treatments for child and adolescent depression. J Clin Child Adolesc Psychol 2017;46(1):11-43 [FREE Full text] [CrossRef] [Medline]
- Clarke G, Reid E, Eubanks D, O'Connor E, DeBar LL, Kelleher C, et al. Overcoming depression on the Internet (ODIN): a randomized controlled trial of an internet depression skills intervention program. J Med Internet Res 2002 Dec;4(3):E14 [FREE Full text] [CrossRef] [Medline]
- Christensen H, Griffiths K, Groves C, Korten A. Free range users and one hit wonders: community users of an Internet-based cognitive behaviour therapy program. Aust N Z J Psychiatry 2006 Jan;40(1):59-62. [CrossRef] [Medline]
- Gellatly J, Bower P, Hennessy S, Richards D, Gilbody S, Lovell K. What makes self-help interventions effective in the management of depressive symptoms? meta-analysis and meta-regression. Psychol Med 2007 Sep;37(9):1217-1228. [CrossRef] [Medline]
- Ebert DD, Buntrock C, Reins JA, Zimmermann J, Cuijpers P. Efficacy and moderators of psychological interventions in treating subclinical symptoms of depression and preventing major depressive disorder onsets: protocol for an individual patient data meta-analysis of randomised controlled trials. BMJ Open 2018 Mar 16;8(3):e018582. [CrossRef] [Medline]
- Raes F, Williams JMG, Hermans D. Reducing cognitive vulnerability to depression: a preliminary investigation of MEmory Specificity Training (MEST) in inpatients with depressive symptomatology. J Behav Ther Exp Psychiatry 2009 Mar;40(1):24-38. [CrossRef] [Medline]
|CATCH-IT: Competent Adulthood Transition with Cognitive Behavioral Humanistic and Interpersonal Training|
|CES-D: Center for Epidemiologic Studies - Depression|
|CONSORT: Consolidated Standards of Reporting Trials|
|DSR: Depression Symptom Rating|
|GAS: Global Assessment Scale|
|GLAD-PC: Guidelines for Adolescent Depression in Primary Care|
|K-SADS: Kiddie Schedule for Affective Disorders Scale|
Edited by G Eysenbach; submitted 28.10.19; peer-reviewed by B Zima, C Moberg, M Deady; comments to author 03.03.20; revised version received 28.07.20; accepted 16.08.20; published 28.10.20
©Benjamin Van Voorhees, Tracy R G Gladstone, Kunmi Sobowale, C Hendricks Brown, David A Aaby, Daniela A Terrizzi, Jason Canel, Eumene Ching, Anita D Berry, James Cantorna, Milton Eder, William Beardslee, Marian Fitzgibbon, Monika Marko-Holguin, Linda Schiffer, Miae Lee, Sarah A de Forest, Emily E Sykes, Jennifer H Suor, Theodore J Crawford, Katie L Burkhouse, Brady C Goodwin, Carl Bell. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 28.10.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.