Internet-based interventions have several advantages, including time and cost effectiveness, almost unlimited scalability, increased intervention fidelity, ease of updates and expansions to conform to the most up-to-date research, and the ability to make them available across the world. With such penetration, even small improvements in likelihoods of quitting smoking can have profound effects on public health relative to the cost of the intervention (see the RE-AIM framework [5-11]). Reports indicate that, in fact, Internet interventions can obtain quit rates comparable with those of other treatment modalities [12-19].

A key benefit of Internet interventions is the automation of delivery. However, with few exceptions [13,18], most trials rely on live personnel to conduct follow-up assessments via phone or email contacts with participants. Insofar as interaction with live personnel may affect outcomes, these trials depart from the fully automated framework. The two main reasons for sacrificing the benefits of automation and fidelity of administration, while incurring considerable costs to conduct live follow-up, are attrition and the analytical convention of imputing smoking status to missing data when presenting cessation trial outcomes.

The progression of Internet intervention studies to address a particular health problem at a worldwide level generally begins with face-to-face clinical trials. The interventions developed at this stage are then adapted for delivery via the Web. Online randomized controlled trials with strong cohort maintenance efforts, such as using staff to send personalized email or to make phone calls to reach participants who do not respond to automated follow-up assessments, can provide estimates of outcome that approximate traditional methods. We suggest that the next step ought to be very-large-scale randomized trials, conducted in a fully automated fashion, to reflect as closely as possible the routine dissemination of Internet interventions that can be made available to anyone in the world, with minimal staffing. However, such large-scale trials generally cannot afford individual live follow-up. This report presents a method that may help researchers in the field to estimate effectiveness data of self-help automated interventions.

The motivation needed to enter traditional face-to-face trials is high: people either actively seek them out, respond to an advertisement by calling and visiting a clinic, or are directly recruited from preexisting registries based on demographic, behavioral, or clinical factors. In contrast, those signing up for a Web-based trial generally do so via a Web search and clicking on a link. Of the thousands who visit the website, few will elect to join, fewer still will make adequate use of the intervention, and only a minority will respond to automated follow-up invitations. The difference in effort involved to enter an Internet trial versus a face-to-face trial makes comparisons between the two problematic. Website visitors are more akin to persons reading an advertisement for a trial, most of whom will not actually call or visit the study clinic. Those filling out an online eligibility questionnaire are similar to those calling a phone number to inquire more about a traditional outcome study. Signing up for an online trial takes little effort; although many online participants are likely curious about the Internet trial, they may not be as committed to participating as those signing up after traveling to a study clinic. Once people enter into the study, it is extremely easy for them to drop out of an Internet trial, since there has been no direct personal contact with study staff. Researchers in the field need to reconsider how best to interpret findings that involve large attrition to systematically study the effectiveness of Internet interventions as they would be routinely used in practice, rather than as part of a well-staffed randomized controlled trial.

Attrition is a recognized concern in Internet trials [20], which affects interpretation of results in two ways: (1) if most participants drop out, the remaining sample is highly self-selected and may not be representative of the original visitors, and (2) if participants do not complete the intervention, but respond to follow-up, the outcome data won’t represent the intervention’s potential. Of course, there are also parallels in face-to-face trials: participants are also highly self-selected and not a representative sample of all who have the disorder being treated.

The usual strategy for determining quit rates in a cessation trial is the missing = smoking (M=S) convention, a variant of the intent-to-treat analysis, which presumes that all participants unreachable at follow-up are smoking. This is similar to the last observation carried forward (LOCF) convention; however, the M=S convention is more conservative, for two reasons. First, the LOCF convention permits responded-to-treatment observations to be carried forward as well as nonresponse, whereas M=S presumes that every dropout is a treatment failure. Second, because cessation trial outcomes are dichotomous (quit vs did not quit), the degree of response/nonresponse (eg, fewer cigarettes) cannot be captured by M=S.

The outcomes of cessation trials therefore largely depend on the completeness of follow-up data. For example, suppose the true quit rate for a hypothetical intervention is 20%. Three trials assessing the effect of that intervention with follow-up rates of 100%, 70%, and 40% would yield M=S outcomes of 20%, 14%, and 8%, respectively, prompting widely differing conclusions about efficacy. Since 14%–22% can be expected with a nicotine patch [4] and 4%–8% can be obtained with a placebo patch, the possible M=S outcome implications vary significantly. Because automated Internet trials have inherently high dropout rates, the M=S convention may be more reflective of follow-up success than of treatment efficacy.

With live follow-up (eg, phone calls), it is possible for geographically limited Internet trials to obtain follow-up rates of up to 78%, which is comparable with face-to-face trials [21]. However, live follow-ups for large-scale, worldwide Internet trials are costly and logistically difficult. Conversely, allowing the logistical limitations of live follow-up procedures to constrain the number of participants compromises scope and reach, limiting the public health applications of an Internet trial.

We propose one possible model of structuring an Internet trial that may help assess effectiveness of a trial once efficacy is established. In 2009, Muñoz and colleagues reported on the outcome of a Web-based smoking cessation trial conducted in Spanish and English (n = 1000) [14]. Live follow-up was used with those who did not provide data after an automated reminder, obtaining follow-up rates of 68% at 12-month follow-up. At 12 months, 20% of Spanish- and 21% of English-speaking participants were no longer smoking (M=S). After random allocation of the first 1000 participants, live follow-up ended, but the rest of the online intervention study was left exactly the same, with the goal of conducting a larger trial to demonstrate the demand for and the reach of the intervention as delivered in a fully automated format. Here, we report on the results of the fully automated portion of that trial.

Conducting a smaller and logistically feasible live follow-up trial followed by or concurrently with a larger fully automated trial can address the concerns of cost versus scope mentioned previously. The goal of the current study was therefore to illustrate the use of this approach in interpreting the outcomes of a fully automated trial. We used the outcomes of the Muñoz et al [14] trial to interpret the results of the current fully automated trial, and we tested three hypotheses to determine whether a more complex intervention would increase quit rates.

Recruitment procedures were the same as described elsewhere [14]. Google AdWords ads were the main means for recruiting participants. Eligible participants were 18+ years of age, smoking 5+ cigarettes per day, with regular (1+ times/week) access to email and Internet, intending to quit in the next month. Of the visitors screened for eligibility, 16,475/78,623 (20.95%) were ineligible, 1052/78,623 (1.34%) were <18 years old, 2738/78,623 (3.48%) smoked <5 cigarettes/day, 9875/78,623 (12.56%) were not ready to quit, and 4646/78,623 (5.91%) had no email address. Participants were not paid for their participation in the study. Participants were recruited from November 2005 to September 2009.

Study procedures are described in detail elsewhere [14]. Briefly, visitors to the site completed brief demographics and eligibility questionnaires. Eligible participants viewed and e-signed a consent document, which detailed the study procedures, including randomization. Consenting participants completed baseline questionnaires. To select out one-time visitors, participants were asked to return 3 times over the next 7 days and report the number of cigarettes smoked. Those meeting this requirement set their quit dates, were automatically randomly assigned to 1 of 4 conditions, and were given access to the website. Participants were sent automated follow-up assessment emails at 1, 3, 6, and 12 months after their quit date. Only self-reported smoking data were gathered, for three reasons: (1) biochemical verification was not feasible for a very large worldwide trial, (2) the fully automated nature of the trial precluded additional participant contact, and (3) there is growing evidence that self-report is sufficient for nonintensive interventions [22-24].

The only difference from the procedures described in the 2009 [14] paper, wherein research assistants either called or emailed those who did not submit follow-up data after the automated email contact, was that in the present study we did not use live follow-up.

As in the 2009 trial [14], participants were randomly assigned to 1 of 4 conditions. The website used a preprogrammed algorithm for random assignment using stratified randomization, with gender and history of major depressive episodes (MDEs) as stratification variables. Condition 1 contained the most basic elements, and conditions 2–4 incrementally added further elements.

The 4 arms (conditions) of the trial were the following:

1. A noninteractive, static smoking cessation guide (Guía para dejar de fumar [25-27]), a cigarette counter, and an online journal.

2. Condition 1, plus individually timed email messages: preprogrammed emails with links to sections of smoking cessation guide timed to quit date [28].

3. Condition 2, plus an 8-session cognitive–behavioral mood management course (based on Lewinsohn et al [29]).

4. Condition 3, plus a virtual participant-driven, unmoderated support group (an asynchronous bulletin board).

We retained three specific hypotheses regarding the outcome of the intervention from the 2009 [14] study and tested them in the fully automated sample: (1) conditions 2, 3, and 4 will outperform condition 1, (2) condition 4 will obtain the best quit rates, followed by condition 3, followed by condition 2, followed by condition 1, and (3) conditions 3 and 4 (containing mood management) will outperform conditions 1 and 2.

A d emographic questionnaire included age, gender, race/ethnicity, education, income, and marital status.

A s moking questionnaire included age when the participant started smoking, age when smoking regularly, number of cigarettes per day, confidence in quitting, and smoking exposure.

The Fagerström Test for Nicotine Dependence (FTND) [30] is a commonly used 6-item test of nicotine dependence, with a range from 0 to 10.

The MDE Screener (Mood Screener) [31] screens for the presence of the 9 symptoms of current and past MDEs according to the Diagnostic and Statistical Manual of Mental Disorders, 4th edition, as well as for criterion C (significant impairment in functioning). This instrument has been shown to have good agreement with the PRIME-MD and with clinician-administered interviews [32-34].

The Center for Epidemiologic Studies Depression scale (CES-D) [35] is a 20-item self-report scale designed to measure the current level of depressive symptoms.

To test hypotheses 1 (condition 1 will result in worse outcomes) and 3 (conditions with mood management will result in better outcomes), we conducted repeated binary logistic regressions. The quit rates were predicted from the intervention condition assignment (1 versus others for hypothesis 1; 1 and 2 versus 3 and 4 for hypothesis 3), covarying participant demographic characteristics (gender, age, education, and race), language (English or Spanish), depression (CES-D score and presence of current or past MDE), and level of addiction (FTND). We conducted these analyses twice: once with the M=S assumption, and the other with observed data (without the M=S assumption).

To test hypothesis 2—that intervention conditions would yield incrementally better outcomes—we constructed binary logistic regression models, predicting the 7-day quit rate at 1, 3, 6, and 12 months. The model predictors were the same as those used for repeated measures analyses, described above. As above, these analyses were conducted twice: once with the M=S assumption, and the other with observed data (without the M=S assumption).

Due to the considerable size of the sample (n = 16,430), we elected to report significance only if we obtained a P value less than .01, to reduce type I error.

Participants were 16,430 smokers (3332 English- and 13,098 Spanish-speaking), aged 18 to 84 (mean 36.2, SD 10.7), from 165 countries. The three most-represented countries for English speakers were the United States (n = 1251), India (n = 358), and South Africa (n = 306). The three most-represented Spanish-speaking countries were Spain (n = 4341), Argentina (n = 2513), and Mexico (n = 2100). Just over half of participants were men (8638/16,349, 52.84%), and most were well educated (12,628/16,379, 77.10% with at least some college education), gainfully employed (12,960/16,415, 78.95% at least part-time), and married or living as married (8846/16,403, 53.93%).

Participants reported having smoked for 20.6 years, on average (SD 10.9), smoking on average 1 pack per day (mean 19.6, SD 9.9 cigarettes). The average age at first cigarette was 15.6 (SD 3.2) years, and the average age of smoking regularly (first 5 packs) was 18.6 (SD 4.3). The average level of nicotine dependence, as measured by the FTND, was 5.2 (SD 2.5), indicating moderate dependence, and similar to face-to-face smoking cessation trials [36-39].

Participant characteristics for each condition are shown in Table 1.

The progression of participants through the study is outlined in Figure 1. Of the over 150,000 participants who visited our website, 78,623 provided enough data to evaluate their eligibility, 28,703 signed consent, and 16,430 completed baseline assessments and the washout period and underwent random assignment.

The current study relied solely on automated emailed reminders to obtain follow-up data. For month 1 follow-up, 6563/16,430 (40.0%) participants provided data. This number was reduced to 4992/16,430 (30.38%), 3813/16,430 (23.21%), and 3606/16,430 (21.95%) for follow-ups at months 3, 6, and 12, respectively. These numbers were comparable with those obtained in the earlier [14] study, where 38%, 30%, 27%, and 23% of participants who never received any live follow-up returned at months 1, 3, 6, and 12, respectively.

Based on observed data, 1 month after enrollment, 36.18% (2239/6189) reported not having smoked in the past 7 days (Table 2). A similar proportion of participants reported 7-day abstinence at months 3 (1797/4566, 39.36%), 6 (1478/3508, 42.13%), and 12 (1361/3292, 41.34%). A somewhat smaller proportion of participants reported not having smoked in the past 30 days (1640/6182, 26.53%; 1465/4562, 32.11%; 1243/3504, 35.47%; and 1211/3286, 36.85% at 1, 3, 6, and 12 months, respectively). Using the M=S convention, the cessation rates observed in this study were modest. The 7-day abstinence rates at 1, 3, 6, and 12 months, respectively, were 13.63% (2239/16,430), 10.94% (1797/16,430), 9.00% (1478/16,430), and 8.28% (1361/16,430). The respective 30-day abstinence rates were 9.98% (1640/16,430), 8.92% (1465/16,430), 7.57% (1243/16,430), and 7.37% (1211/16,430). In the 45 months of this intervention being available online, 3,489 individuals reported having quit for at least 7 days (about 18 per week over the course of the study), and 2,786 individuals (about 14 per week) reported having quit for at least 30 days.

We noted several differences between treatment conditions (Table 3). With observed data (ie, without the M=S assumption), at 1-month follow-up, condition 1 performed significantly poorer than all other conditions, in partial support for hypothesis 1 (Wald χ² ₃ = 80.7, P < .001). No significant differences between conditions were noted at months 3 (Wald χ² ₃ = 10.3, P = .02), 6 (Wald χ² ₃ = 5.8, P = .12), and 12 (Wald χ² ₃ = 7.5, P = .06). With M=S analyses, significant differences were observed at months 6 (Wald χ² ₃ = 14.8, P = .002) and 12 (Wald χ² ₃ = 13.0, P = .005), such that conditions 2 and 4 outperformed conditions 1 and 3.

Observing the quit rates, it is clear that hypothesis 2—that conditions would result in incremental improvements in quit rates—is not supported. To test the two other hypotheses, we conducted repeated-measures logistic regressions, with the same covariates as in the simple logistic regressions above. Hypothesis 1 was largely supported. With observed data, condition 1 resulted in lower quit rates than conditions 2, 3, and 4 (Wald χ² ₁ = 30.1, P < .001, beta = –.28, 95% confidence interval [CI], –.39 to –.18). A similar result was observed with M=S data, though the result did not cross the significance level set for this study (Wald χ² ₁ = 6.1, P = .01, beta = –.12, 95% CI –.21 to –.02). Hypothesis 3—that mood management conditions would result in higher quit rates—was supported only with observed data (Wald χ² ₁ = 9.5, P = .002, beta = –.14, 95% CI –.23 to –.05), but not with the M=S data (Wald χ² ₁ = .0, P = .96, beta = .00, 95% CI –.08 to .78).

For the current trial, the quit rate at 12 months was 8%, assuming M=S. However, the true quit rate is unknown: 8% is clearly an underestimate, because the M=S is only an assumption and is highly conservative. As a thought experiment, if the opposite assumption is made—that all those with missing data have quit (M=Quit)—then the quit rate would be 88% (8% observed quit + 80% missing). The M=S is an underestimate of the true quit rate, but M=Quit is clearly an overestimate. Though we can say with confidence that the true quit rate resides between 8% and 88%, this is not very informative (Figure 2).

We can approximate the true quit rate by using the rates from the earlier [14] trial. Because the 2009 trial used exactly the same intervention (plus live follow-up), similar quit rates can be expected. To determine whether the two cohorts are similar, we used binary logistic regression with the same covariates as in the previous models to compare the two most similar subgroups in the two trials: those in the 2009 study who completed all four follow-ups after only an automated email reminder (and were thus never exposed to live follow-up), and those in the current study who also completed all four follow-ups. For these subgroups, the 12-month quit rates were not significantly different (46/96, 48% in the 2009 study vs 714/1326, 54% in the present study; Wald χ² ₁ = 1.6, P = .21, odds ratio = 0.75, 95% CI 0.48–1.17), which provides additional credence for the ability to extrapolate results of the current study in the context of the 2009 trial. Only two covariates crossed the significant threshold. One was depression history (Wald χ² ₂ = 13.8, P = .001), with participants without a history of depression appearing to quit at higher rates (603/1043, 57.8%) than those with past (92/213, 43.2%) or current (62/156, 40%) depression. The other was FTND score, with those scoring higher being less likely to quit (Wald χ² ₁ = 7.0, P = .01, odds ratio = 0.94, 95% CI 0.90–0.99).

The 2009 [14] study obtained an average quit rate of 21% at 12 months assuming M=S, or, assuming M=Quit, 52%. The true rate for the current study therefore most likely resides between 21% and 52%, which is considerably more informative than the 8%–88% interval (Figure 2).

The interval can be narrowed down further. The observed quit rate for the current study at 12 months is 41%; in the 2009 [14], it was 30%. However, that 30% was based on both automated and live follow-up responders, and the reported quit rates of automated responders were about 70% higher than that of live responders (16.3% vs 9.8%, respectively, across all follow-ups). The current study’s observed quit rate (41%) is therefore an overestimate of the true quit rate, as everyone who provided data did so with automated follow-up. The most likely conclusion about the true quit rate in the current study is that the upper bound is 30% (observed quit rate in the 2009 study) and the lower bound is 21%, as illustrated in Figure 2.