This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Smoking is one of the largest contributors to the global burden of disease. Internet interventions have been shown to reduce smoking rates successfully. However, improved methods of evaluating effectiveness need to be developed for large-scale Internet intervention trials.
To illustrate a method to interpret outcomes of large-scale, fully automated, worldwide Internet intervention trials.
A fully automated, international, Internet-based smoking cessation randomized controlled trial was conducted in Spanish and English, with 16,430 smokers from 165 countries. The randomized controlled trial replicated a published efficacy trial in which, to reduce follow-up attrition, 1000 smokers were followed up by phone if they did not provide online follow-up data.
The 7-day self-reported abstinence rates ranged from 36.18% (2239/6189) at 1 month to 41.34% (1361/3292) at 12 months based on observed data. Given high rates of attrition in this fully automated trial, when participants unreachable at follow-up were presumed to be smoking, the abstinence rates ranged from 13.63% (2239/16.430) at 1 month to 8.28% (1361/16,430) at 12 months. We address the problem of interpreting results with high follow-up attrition rates and propose a solution based on a smaller study with intensive phone follow-up.
Internet-based smoking cessation interventions can help large numbers of smokers quit. Large-scale international outcome studies can be successfully implemented using automated Internet sites. Interpretation of the studies’ results can be aided by extrapolating from results obtained from subsamples that are followed up by phone or similar cohort maintenance methods.
ClinicalTrials.gov NCT00721786; http://clinicaltrials.gov/ct2/show/NCT00721786 (Archived by WebCite at http://www.webcitation.org/63mhoXYPw)
One billion tobacco-related deaths are projected for the 21st century, 80% of which will occur in low- and middle-income countries [
Internet-based interventions have several advantages, including time and cost effectiveness, almost unlimited scalability, increased intervention fidelity, ease of updates and expansions to conform to the most up-to-date research, and the ability to make them available across the world. With such penetration, even small improvements in likelihoods of quitting smoking can have profound effects on public health relative to the cost of the intervention (see the RE-AIM framework [
A key benefit of Internet interventions is the automation of delivery. However, with few exceptions [
The progression of Internet intervention studies to address a particular health problem at a worldwide level generally begins with face-to-face clinical trials. The interventions developed at this stage are then adapted for delivery via the Web. Online randomized controlled trials with strong cohort maintenance efforts, such as using staff to send personalized email or to make phone calls to reach participants who do not respond to automated follow-up assessments, can provide estimates of outcome that approximate traditional methods. We suggest that the next step ought to be very-large-scale randomized trials, conducted in a fully automated fashion, to reflect as closely as possible the routine dissemination of Internet interventions that can be made available to anyone in the world, with minimal staffing. However, such large-scale trials generally cannot afford individual live follow-up. This report presents a method that may help researchers in the field to estimate effectiveness data of self-help automated interventions.
The motivation needed to enter traditional face-to-face trials is high: people either actively seek them out, respond to an advertisement by calling and visiting a clinic, or are directly recruited from preexisting registries based on demographic, behavioral, or clinical factors. In contrast, those signing up for a Web-based trial generally do so via a Web search and clicking on a link. Of the thousands who visit the website, few will elect to join, fewer still will make adequate use of the intervention, and only a minority will respond to automated follow-up invitations. The difference in effort involved to enter an Internet trial versus a face-to-face trial makes comparisons between the two problematic. Website visitors are more akin to persons reading an advertisement for a trial, most of whom will not actually call or visit the study clinic. Those filling out an online eligibility questionnaire are similar to those calling a phone number to inquire more about a traditional outcome study. Signing up for an online trial takes little effort; although many online participants are likely curious about the Internet trial, they may not be as committed to participating as those signing up after traveling to a study clinic. Once people enter into the study, it is extremely easy for them to drop out of an Internet trial, since there has been no direct personal contact with study staff. Researchers in the field need to reconsider how best to interpret findings that involve large attrition to systematically study the effectiveness of Internet interventions as they would be routinely used in practice, rather than as part of a well-staffed randomized controlled trial.
Attrition is a recognized concern in Internet trials [
The usual strategy for determining quit rates in a cessation trial is the
The outcomes of cessation trials therefore largely depend on the completeness of follow-up data. For example, suppose the
With live follow-up (eg, phone calls), it is possible for geographically limited Internet trials to obtain follow-up rates of up to 78%, which is comparable with face-to-face trials [
We propose one possible model of structuring an Internet trial that may help assess effectiveness of a trial once efficacy is established. In 2009, Muñoz and colleagues reported on the outcome of a Web-based smoking cessation trial conducted in Spanish and English (n = 1000) [
Conducting a smaller and logistically feasible live follow-up trial followed by or concurrently with a larger fully automated trial can address the concerns of cost versus scope mentioned previously. The goal of the current study was therefore to illustrate the use of this approach in interpreting the outcomes of a fully automated trial. We used the outcomes of the Muñoz et al [
Recruitment procedures were the same as described elsewhere [
Study procedures are described in detail elsewhere [
The only difference from the procedures described in the 2009 [
As in the 2009 trial [
The 4 arms (conditions) of the trial were the following:
1. A noninteractive, static smoking cessation guide (Guía para dejar de fumar [
2. Condition 1, plus individually timed email messages: preprogrammed emails with links to sections of smoking cessation guide timed to quit date [
3. Condition 2, plus an 8-session cognitive–behavioral mood management course (based on Lewinsohn et al [
4. Condition 3, plus a virtual participant-driven, unmoderated support group (an asynchronous bulletin board).
We retained three specific hypotheses regarding the outcome of the intervention from the 2009 [
A
A
The
The
The
To test hypotheses 1 (condition 1 will result in worse outcomes) and 3 (conditions with mood management will result in better outcomes), we conducted repeated binary logistic regressions. The quit rates were predicted from the intervention condition assignment (1 versus others for hypothesis 1; 1 and 2 versus 3 and 4 for hypothesis 3), covarying participant demographic characteristics (gender, age, education, and race), language (English or Spanish), depression (CES-D score and presence of current or past MDE), and level of addiction (FTND). We conducted these analyses twice: once with the M=S assumption, and the other with observed data (without the M=S assumption).
To test hypothesis 2—that intervention conditions would yield incrementally better outcomes—we constructed binary logistic regression models, predicting the 7-day quit rate at 1, 3, 6, and 12 months. The model predictors were the same as those used for repeated measures analyses, described above. As above, these analyses were conducted twice: once with the M=S assumption, and the other with observed data (without the M=S assumption).
Due to the considerable size of the sample (n = 16,430), we elected to report significance only if we obtained a
Participants were 16,430 smokers (3332 English- and 13,098 Spanish-speaking), aged 18 to 84 (mean 36.2, SD 10.7), from 165 countries. The three most-represented countries for English speakers were the United States (n = 1251), India (n = 358), and South Africa (n = 306). The three most-represented Spanish-speaking countries were Spain (n = 4341), Argentina (n = 2513), and Mexico (n = 2100). Just over half of participants were men (8638/16,349, 52.84%), and most were well educated (12,628/16,379, 77.10% with at least some college education), gainfully employed (12,960/16,415, 78.95% at least part-time), and married or living as married (8846/16,403, 53.93%).
Participants reported having smoked for 20.6 years, on average (SD 10.9), smoking on average 1 pack per day (mean 19.6, SD 9.9 cigarettes). The average age at first cigarette was 15.6 (SD 3.2) years, and the average age of smoking regularly (first 5 packs) was 18.6 (SD 4.3). The average level of nicotine dependence, as measured by the FTND, was 5.2 (SD 2.5), indicating moderate dependence, and similar to face-to-face smoking cessation trials [
Participant characteristics for each condition are shown in
Participant characteristics, by conditiona
Condition 1 |
Condition 2 |
Condition 3 |
Condition 4 |
|
|
Male, n (%) | 2168/4102 (52.85%) | 2150/4080 (52.70%) | 2165/4088 (52.96%) | 2155/4079 (52.83%) | 1.00 |
Age (years), mean (SD) | 36.1 (11.4) | 36.3 (11.8) | 36.5 (14.5) | 36.4 (13.6) | .47 |
Some college or more, n (%) | 3167/4107 (77.11%) | 3141/4088 (76.83%) | 3173/4091 (77.56%) | 3147/4093 (76.89%) | .93 |
White, n (%) | 2803/4086 (68.60%) | 2801/4069 (68.84%) | 2796/4076 (68.60%) | 2802/4075 (68.76%) | .37 |
Spanish-speaking, n(%) | 3284/4118 (79.75%) | 3263/4097 (79.64%) | 3275/4110 (79.68%) | 3276/4105 (79.81%) | 1.00 |
Employed, n (%) | 3217/4115 (78.18%) | 3237/4091 (79.12%) | 3268/4109 (79.53%) | 3238/4100 (78.98%) | .36 |
Married or partnered, n (%) | 2206/4112 (53.65%) | 2209/4089 (54.02%) | 2234/4103 (54.45%) | 2197/4099 (53.60%) | .86 |
CES-Dc score, mean (SD) | 16.9 (12.1) | 17.0 (12.0) | 16.8 (12.4) | 16.9 (12.4) | .74 |
Current or past major depressive episode, n (%) | 1276/4109 (31.05%) | 1275/4091 (31.17%) | 1280/4103 (31.20%) | 1277/4101 (31.14%) | 1.00 |
Cigarettes/day, mean (SD) | 19.4 (9.9) | 19.8 (10.2) | 19.5 (10.1) | 19.6 (9.7) | .36 |
Age started smoking (years), mean (SD) | 15.5 (3.2) | 15.6 (3.2) | 15.5 (3.2) | 15.6 (3.4) | .46 |
Age regular smoker (years), mean (SD) | 18.6 (4.4) | 18.5 (4.0) | 18.6 (4.3) | 18.7 (4.4) | .19 |
Years smoked, mean (SD) | 20.5 (10.8) | 20.6 (10.9) | 20.7 (10.9) | 20.6 (11.0) | .81 |
FTNDd score, mean (SD) | 5.2 (2.5) | 5.3 (2.5) | 5.2 (2.5) | 5.2 (2.5) | .41 |
a Conditions were as follows: condition 1: a noninteractive smoking cessation guide, cigarette counter, and an online journal; condition 2: condition 1, plus individually timed email messages; condition 3: condition 2, plus an 8-session cognitive–behavioral mood management course; and condition 4: condition 3, plus a virtual participant-driven support group.
b
c Center for Epidemiologic Studies Depression scale.
d Fagerström Test for Nicotine Dependence.
The progression of participants through the study is outlined in
The current study relied solely on automated emailed reminders to obtain follow-up data. For month 1 follow-up, 6563/16,430 (40.0%) participants provided data. This number was reduced to 4992/16,430 (30.38%), 3813/16,430 (23.21%), and 3606/16,430 (21.95%) for follow-ups at months 3, 6, and 12, respectively. These numbers were comparable with those obtained in the earlier [
CONSORT diagram for progression of participants through the fully automated Internet stop smoking trial.
Based on observed data, 1 month after enrollment, 36.18% (2239/6189) reported not having smoked in the past 7 days (
Overall self-reported abstinence rates (% quit) in an online sample of 16,430 consented smokers
1-month follow-up | 3-month follow-up | 6-month follow-up | 12-month follow-up | ||||||
Completed follow-ups, n (%) | 6563/16,430 (39.95%) | 4992/16,430 (30.38%) | 3813/16,430 (23.21%) | 3606/16,430 (21.95%) | |||||
7 days | 30 days | 7 days | 30 days | 7 days | 30 days | 7 days | 30 days | ||
|
|||||||||
n | 2239/6189 | 1640/6182 | 1797/4566 | 1465/4562 | 1478/3508 | 1243/3504 | 1361/3292 | 1211/3286 | |
% | 36.18% | 26.53% | 39.36% | 32.11% | 42.13% | 35.47% | 41.34% | 36.85% | |
|
|||||||||
n | 2239/16,430 | 1640/16,430 | 1797/16,430 | 1465/16,430 | 1478/16,430 | 1243/16,430 | 1361/16,430 | 1211/16,430 | |
% | 13.63% | 9.98% | 10.94% | 8.92% | 9.00% | 7.57% | 8.28% | 7.37% |
a Missing observations are presumed to be smoking.
We noted several differences between treatment conditions (
Observing the quit rates, it is clear that hypothesis 2—that conditions would result in incremental improvements in quit rates—is not supported. To test the two other hypotheses, we conducted repeated-measures logistic regressions, with the same covariates as in the simple logistic regressions above. Hypothesis 1 was largely supported. With observed data, condition 1 resulted in lower quit rates than conditions 2, 3, and 4 (Wald χ2
1 = 30.1,
7-day quit rates (n, %) by intervention condition
Conditiona |
|
|||||
1 (cessation guide) | 2 (1 + email |
3 (2 + mood |
4 (3 + virtual group) | |||
|
||||||
Month 1 | 526/1912 (27.51%) | 611/1578 (38.72%) | 550/1371 (40.12%) | 552/1328 (41.57%) | <.001b | |
Month 3 | 427/1175 (36.34%) | 467/1156 (40.40%) | 428/1132 (37.81%) | 475/1103 (43.06%) | .02 | |
Month 6 | 327/827 (39.54%) | 395/893 (44.23%) | 342/845 (40.47%) | 414/943 (43.90%) | .12 | |
Month 12 | 306/730 (41.92%) | 355/833 (42.62%) | 314/845 (37.16%) | 386/884 (43.67%) | .06 | |
|
||||||
Month 1 | 526/4118 (12.77%) | 611/4097 (14.91%) | 550/4110 (13.38%) | 552/4105 (13.45%) | .03 | |
Month 3 | 427/4118 (10.37%) | 467/4097 (11.40%) | 428/4110 (10.41%) | 475/4105 (11.57%) | .20 | |
Month 6 | 327/4118 (7.94%) | 395/4097 (9.64%) | 342/4110 (8.32%) | 414/4105 (10.09%) | .002b | |
Month 12 | 306/4118 (7.43%) | 355/4097 (8.66%) | 314/4110 (7.64%) | 386/4105 (9.40%) | .005b |
a Conditions were as follows: 1: a noninteractive smoking cessation guide, cigarette counter, and an online journal; 2: condition 1, plus individually timed email messages; 3: condition 2, plus an 8-session cognitive–behavioral mood management course; and condition 4: condition 3, plus a virtual participant-driven support group.
b Significant, controlling for demographic characteristics (gender, age, education, race), language of the intervention (English or Spanish), level of addiction (Fagerström Test for Nicotine Dependence [FTND] score), and depression (Center for Epidemiologic Studies Depression scale [CES-D] score and presence of current or past major depressive episodes).
For the current trial, the quit rate at 12 months was 8%, assuming M=S. However, the
We can approximate the true quit rate by using the rates from the earlier [
The 2009 [
The interval can be narrowed down further. The observed quit rate for the current study at 12 months is 41%; in the 2009 [
Most likely quit rate range extrapolated from the current trial and an identical trial with live follow-up (Muñoz et al [
In this paper we have highlighted the problems of attrition in international Internet trials, especially in the context of the M=S convention, and offered a way to reconcile the demands of needing to employ costly means of follow-up with the advantages that the breadth of a very-large-scale automated trial allows. By referencing the identically conducted trial, with the only difference being live follow-up for those who did not respond to automated email reminders, we estimated the true quit rate for the current trial to lie between 21% and 30% of participants. We also found that more complex versions of the intervention resulted in better cessation rates than a static online smoking cessation guide, suggesting that some level of complexity and personalization may be helpful in Internet interventions.
Internet interventions are a relatively new form of health-promoting behavior change interventions that are likely to grow considerably due to the benefits of reach and cost effectiveness. To ensure that these interventions are improving health outcomes, they must be tested to ensure a strong evidence base. Indeed, Internet-based interventions are increasingly evidence based [
One of the most significant benefits of Internet interventions is their cost effectiveness due to sustainability and nonconsumability. The cost of creating an effective Internet site for smoking cessation is relatively modest, about US $55,000 in our case. Conducting the randomized trials to evaluate its effectiveness costs much more. Once the original efficacy trial with live follow-up was completed, however, leaving the site open to conduct the fully automated randomized trial reported here was relatively inexpensive. We estimate that it cost about US $120 per participant who reported quitting successfully by at least one follow-up point. If the nicotine patch had been used, assuming a 20% quit rate with the patch, at a cost of US $3.91 per day [
The benefit of Internet intervention trials may be undervalued if methods for their evaluations underestimate their effects. Though sophisticated statistical procedures that may tackle missing data exist (eg, multiple imputation), they may not be accurate when the proportion of missing data is very large, as is often the case in fully automated Internet studies. Therefore, we have proposed a hybrid solution that includes conducting a smaller trial with aggressive follow-up using live methods (eg, phone calls) to assess for an intervention’s efficacy followed by a larger naturalistic trial to assess effectiveness. This method would allow for the rigorous testing of efficacy by ensuring high follow-up rates with a smaller trial. The larger trial would then allow one to take advantage of the wide reach and automated nature of Internet interventions to assess for the overall impact of the study by extrapolating results from the smaller trial onto a larger sample. Thus, the 2009 [
There are limitations to our study and the way we have used our proposed 2-step method. In both studies, smoking was assessed via self-report rather than biomedical validation measures; however, this is the recommended approach in large-scale community trials [
Internet-based interventions for health problems are becoming increasingly popular due to their enormous reach and cost effectiveness: no other medium permits conducting a randomized trial of an empirically supported intervention for over 16,000 individuals across 165 countries at such low cost. However, in testing these interventions via randomized controlled trials, particularly when assessing dichotomous outcomes, it is necessary to develop new methods of analysis that are able to fully reflect the true impact and effectiveness of large-scale, international public health Internet interventions.
Future directions involve carrying out outcome studies that are more generalizable to how Internet interventions would be used outside of a strict randomized trial context. Specifically, users of such sites are likely to pick and choose among intervention elements provided by the sites. Thus, the next step after randomized trials ought to be participant preference trials, in which users are provided access to all elements of the interventions that were found to be reasonably effective within a randomization context, and allowed to use the elements they prefer. Our team is conducting such a study, which we believe would best estimate the effectiveness of a self-help automated Internet intervention that would be made available at no charge to anyone in the world who wanted to use it.
Researchers in the Internet intervention field should consider adopting this approach, namely a progression of studies, from strict efficacy randomized trials (with live follow-ups to reduce attrition), to fully automated randomized trials (to approximate how a self-help site would be used), proceeding to participant preference effectiveness studies (in which all elements tested in the earlier randomized trials are made available to all participants). Such an approach would contribute to the use of evidence-based Internet interventions to reduce health disparities worldwide [
Center for Epidemiologic Studies Depression scale
confidence interval
Fagerström Test for Nicotine Dependence
last observation carried forward
missing observations presumed quit
missing observations presumed smoking
major depressive episode
This research was supported by grant 13RT-0050 from the Tobacco-Related Disease Research Program, by an infrastructure grant from the University of California Committee on Latino Research to the University of California/San Francisco General Hospital Latino Mental Health Research Program (LMHRP; Muñoz, Principal Investigator), by grants from the Tobacco Research Network Program, Fogarty International Center, National Institute of Drug Abuse (No.TW05935, Pérez-Stable, Principal Investigator), and from the National Cancer Institute for Redes en Acción (U01CA86117, Pérez-Stable, Principal Investigator), National Institutes of Health, by NIMH training grant to Adrian Aguilera (MH 5T32MH018261-27; Patricia Arean, Principal Investigator), and by NIMH grant 5K08MH091501 (Leykin, Principal Investigator). The authors thank the Center for Health and Community (Nancy Adler, Director) for providing office space and additional resources. Special thanks go to Google, Inc for awarding the UCSF/SFGH LMHRP an AdWords grant (Muñoz, Principal Investigator), which provided us with the ability to recruit smokers worldwide using Google sponsored links.
None declared.
CONSORT-EHEALTH V1.6 checklist [