This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Participant engagement influences treatment effectiveness, but it is unknown which intervention design features increase treatment engagement for online smoking cessation programs.
We explored the effects of 4 design features (ie, factors) on early engagement with an Internet-based, motivational smoking cessation program.
Smokers (N=1865) were recruited from a large health care organization to participate in an online intervention study, regardless of their interest in quitting smoking. The program was intended to answer smokers’ questions about quitting in an effort to motivate and support cessation. Consistent with the screening phase in the multiphase optimization strategy (MOST), we used a 2-level, full-factorial design. Each person was randomized to 1 of 2 levels of each factor, including message tone (prescriptive vs motivational), navigation autonomy (dictated vs not), proactive email reminders (yes vs no), and inclusion of personally tailored testimonials (yes vs no). The effects of each factor level on program engagement during the first 2 months of enrollment were compared, including number of visits to the website resulting in intervention content views (as opposed to supplemental content views), number of intervention content areas viewed, number of intervention content pages viewed, and duration of time spent viewing this content, as applicable to each factor.
Adjusting for baseline readiness to quit, persons who received content written in a prescriptive tone made the same number of visits to the website as persons receiving content in a motivational tone, but viewed 1.17 times as many content areas (95% CI 1.08-1.28;
Using a prescriptive message tone, dictating content viewing order, and sending reminder emails each resulted in greater program engagement relative to the contrasting level of each experimental factor. The results require replication, but suggest that a more directive interaction style may be preferable for online cessation programs.
clinicaltrials.gov NCT00992264; http://clinicaltrials.gov/ct2/show/NCT00992264 (Archived by WebCite at http://www.webcitation.org/6F7H7lr3P)
Smoking remains a leading cause of death and disability, accounting for approximately 1 in 5 deaths each year in the United States [
Although greater program engagement does not automatically mean a program is more effective (in fact, people may not return to the program because it was effective in helping them change their behavior), some level of intervention exposure is clearly important for an intervention to have its intended effect. Research has consistently shown a dose-response effect for smoking cessation interventions, including Internet-based programs [
Consistent with the initial screening phase of the multiphase optimization strategy (MOST) for treatment development [
For this study,
Each design factor was chosen based on empirical or theoretical evidence for its effects on smoking cessation or because its treatment effects are unclear. For example, research suggests that interventions grounded in the principles of motivational interviewing can be effective across a range of health risk behaviors, including smoking abstinence [
In the current study, we hypothesized that each of the experimental factor levels would also have a differential effect on our 4 measures of engagement. Participants randomized to receive online intervention content written in a prescriptive tone (as opposed to a motivational tone) would find the content less acceptable; therefore, they would view fewer content areas and Web pages, spend less time reviewing the content, and may return to the site less often. Similar hypotheses were made for people who were required to view content in a prespecified (dictated) order based on their stage of change, as opposed to being allowed to navigate the site freely, based on their interests. These assumptions are consistent with people’s desire for autonomy as described in self-determination theory [
The study design and methods, including an extensive overview of each of the experimental factor choices, their theoretical rationale, and how each was operationalized in the Questions about Quitting (Q2) intervention is available elsewhere [
This study was a collaboration between Group Health Research Institute in Seattle, Washington and the University of Michigan Center for Health Communications Research in Ann Arbor, Michigan. Participants were recruited from Group Health, a large, regional nonprofit health plan in Washington State. All research materials (intervention materials, surveys, and protocols) were approved by the institutional review boards at Group Health Research Institute and the University of Michigan. The study is registered with clinicaltrials.gov (NCT00992264). Data reported in this paper were collected between May 2010 and December 2011.
Consistent with the initial phase of the MOST framework [
A study invitation letter was sent to adult likely smokers identified from automated health plan records. The study program was described as providing information and guidance to help people decide if quitting was right for them and how to quit if and when they decided to do so. The goal was to recruit smokers interested in quitting, as well as those with no interest in quitting.
Individuals interested in learning more about the program were provided a unique log-in access code in the invitation letter and were directed to the study website where they were screened for eligibility, provided consent, and were enrolled online. People were eligible if they were aged 18 years or older, a current member of Group Health, smoked 100 cigarettes in their lifetime, smoked even a puff in the past 7 days, smoked an average of at least 5 cigarettes per day, were not currently enrolled in a smoking cessation program or taking medication to stop smoking, had access to the Internet for personal use, were willing to check their email at least once a week, were comfortable reading and writing in English, had no visual impairments that prevented reading text on a computer screen, and were comfortable using a computer and the Internet.
After providing online consent, participants completed a baseline assessment online and then were randomized to an intervention arm using an automated algorithm. Half of all participants were randomized to each contrasting factor level and assignment to each intervention group was stratified by participants’ readiness to quit smoking at baseline (no interest in quitting in the next 6 months, interested in quitting in the next 6 months but not the next month, or interested in quitting in the next month). Following randomization, participants could immediately access their personalized intervention program following the baseline assessment and were encouraged to return to the site as often as they wanted. Because enrollment required log-in using a preassigned log-in code, it was not possible for participants to enroll in the study more than once. Participants were blinded to their group assignment.
The program was developed through an iterative and interactive design process. The final design and layout was informed by focus group testing with smokers. Intervention content was written by experts in behavioral science at the Group Health Research Institute and University of Michigan’s Center for Health Communications Research (CHCR). The personalized intervention content was tailored using the nonproprietary Michigan Tailoring System, developed by researchers in the CHCR. Additional detail on the program design and content are available in McClure et al [
The intervention was delivered via the Internet. Participants were told they would receive an individually tailored program designed to answer their questions about quitting smoking and to help them make a decision about whether and how to stop smoking, but they were not told any specifics about the treatment arms before or after accessing the intervention. The intervention included a combination of core intervention content and additional special feature content. The core content was accessible from the main page and organized in 3 main drop-down headers or content areas, each targeting smokers at different stages of readiness to quit smoking, specifically those not ready to quit, those ready to quit, and those who already quit (see sample screenshot in
Participants could view the Q2 program as often as they liked and they were encouraged to return to the website in the future. Upon return, if more than 24 hours had elapsed since their last visit, participants were asked to restate their readiness to quit smoking and the content was retailored to reflect their current smoking status and interest in quitting. The basic intervention layout, number of pages, and substantive core content remained unchanged, but the text was refreshed to reflect the change in participants’ current motivation for quitting or smoking status. The intent was to ensure that the program content remained responsive to individuals’ current needs.
Example screenshot of Questions about Quitting (Q2) layout.
Participants were randomized to receive content written in either a prescriptive or motivational tone. Prescriptive messaging was written in a didactic tone and clearly advised smokers to quit smoking and how to achieve this goal. Motivational messaging was written in a tone consistent with the key principles of motivational interviewing (express empathy, develop discrepancy, roll with resistance, support autonomy and self-efficacy) [
Half of the participants could freely view content on the website in any order they wished. The other half of the participants, in the dictated navigation arm, were required to first view content matched to their baseline readiness to quit and to view the content in a prespecified order. After this content was seen, they were then free to navigate the site.
Participants were also randomized to receive weekly proactive email reminders or not. Email messages were standardized across all individuals and encouraged participants to return to the Q2 website to view the optional special feature content. However, we did not track special feature page views because it was not part of the core intervention. Additionally, not all participants had access to this content at the same time. For those whose navigation of the site was dictated based on their initial readiness to quit, access to this optional content was only available after they viewed all Web pages in their initial stage-appropriate content area.
Participants were randomized to receive 3 highly tailored testimonials designed to promote their self-efficacy for quitting or to not receive these testimonials. Testimonials were tailored on each individual’s stage of change, level of nicotine dependence, prior use of pharmacotherapy for nicotine dependence, depression history, perceived risks and benefits of quitting smoking, and their self-efficacy for quitting. One testimonial was included at the end of each of the 3 core content sections. Testimonials were designed to support self-efficacy for quitting by providing personally tailored information and modeling appropriate quitting behaviors. Information was presented in an interview format with a smoker or former smoker. Because not all participants received the extra testimonial content, tracking data from these pages, including time spent viewing this content, were excluded from the analyses to normalize the engagement metrics across treatment arms.
Self-report data were collected at baseline using an online survey. This data included demographics; current smoking status; number of cigarettes smoked per day; stage of readiness to quit smoking; nicotine dependence assessed with the Fagerstrom Test of Nicotine Dependence [
Automated tracking data were collected each time participants visited the website. This data included the date and time each participant visited the website and individual date/time stamps each time a content page was accessed or left.
Intervention exposure was defined as any exposure to the core intervention content. Visits to the main page and special features were excluded. Engagement was defined by: (1) the number of unique visits to the website during which the core intervention content was viewed, (2) the number of unique treatment core content sections viewed (out of a possible 3), (3) the number of times individual pages (core content subsections) were viewed, and (4) the cumulative duration of minutes spent viewing the core intervention content. Sessions automatically timed out after 30 minutes of inactivity or ended when individuals left the website (eg, logged out, closed their browser, or visited a different website).
Data were monitored over the course of the study to ensure participants were appropriately randomized, baseline data were collected, and automated user statistics on program use were being appropriately captured.
The analytic sample included all individuals, regardless of exposure to the core intervention content, to take advantage of the balancing effect of randomization on all covariates, measured and unmeasured. As a result, any observed differences between the randomized treatment groups are because of differences in the effect of the interventions. If we had limited the analyses to only those individuals who observed some content, this would restrict the sample based on a posttreatment outcome. That is, differences in observed program engagement levels between groups could be due to treatment effects or imbalances in covariates between the treatment groups. To further complicate matters, 1 of the factor levels (receipt of proactive emails) directly affected the probability that an individual viewed any content; limiting the analytic sample only to those who saw some content when comparing the 2 levels of this factor would bias the results.
Descriptive statistics were used to characterize the study sample based on data collected during the baseline survey. To assess engagement with the website content, we examined website-tracking data for each participant. We calculated means, standard deviations, medians, and interquartile ranges for each count-based outcome measure. We compared the number of visits to the website in which an individual viewed core intervention content between the 2 levels of each of the 4 factors using Poisson regression models that adjusted for initial readiness to quit smoking because this was a stratification variable in the randomization process. Similar Poisson models were used to estimate the effect of random factor level assignment on the number of content areas visited, ranging from zero to 3. Estimates obtained from Poisson models are generally interpretable as incidence rate ratios, but in the context of an experiment like ours in which all subjects shared a common period of exposure, estimates can equivalently be interpreted as the ratio of mean event counts among the exposed to that of the unexposed group.
The distributions of the number of individual page views and of the cumulative number of minutes spent viewing intervention content each had a larger proportion of zeros than expected from a Poisson distribution. Due to the inflated number of zeros we used zero-inflated Poisson (ZIP) models to estimate the effects of the factors on these 2 measures [
A total of 683 page views timed out automatically after 30 minutes of inactivity. Among page views that did not time out, most views were significantly shorter than 30 minutes, suggesting it was unlikely that all timed-out sessions truly reflected 30 minutes of time spent viewing these pages. Thus, we treated the true viewing time for these page views as missing values and imputed the viewing time for these page views using a chained equation, multiple imputation procedure [
To investigate whether the effects of the random factor assignments may have differed by initial readiness to quit, we refitted each of the regression models described previously with the inclusion of interaction terms between the factors of interest and a categorical variable indicating initial readiness to quit smoking. Joint tests of the set of interaction terms within each model fit were conducted using a Wald test statistic with 2 degrees of freedom calculated to assess the significance of interactions.
Tracking data management was conducted using SAS software version 9.2 (SAS Institute, Inc, Cary, NC, USA) and all analyses, including multiple imputations, were conducted using Stata version 12 (StataCorp LP, College Station, TX, USA).
Demographic characteristics of the enrolled sample (N=1865) are presented in
A total of 690 of 1865 enrolled participants (37.00%) failed to view any of the core intervention content within 2 months after joining the study, whereas 1175 participants (63.00%) viewed at least some core content during this period. Participants who failed to view any core content differed significantly with regard to their baseline readiness to quit (
Participants viewed the core content on a total of 1691 separate visits, resulting in 6592 unique content page views. On average, participants who accessed the core intervention made 1.4 visits (median 1, range 1-11) to view this content, viewed an average of 1.4 of the 3 core content areas (median 1, range 1-3), and viewed on average 5.6 total core content pages (median 4, range 1-53). After imputing duration of timed-out visits, the average cumulative time accrued viewing the core intervention content was 12.3 minutes (median 7.0, range 0.10-180). Of the 3 core content areas, the pages designed for people ready to quit were viewed most often. Content designed for people not yet ready to quit was viewed second most often, followed by the content designed for people who have already quit.
Engagement outcomes by factor level are presented in detail in
Baseline characteristics of enrolled participantsa (N=1865).
Characteristics | Participants | |
|
|
|
|
Female | 1178 (63.16) |
|
|
|
|
White, non-Hispanic | 1534 (82.25) |
|
|
|
|
High school or less | 524 (28.10) |
|
Some college | 944 (50.62) |
|
College degree or higher | 396 (21.23) |
|
|
|
|
Employed | 1287 (69.00) |
|
|
|
|
Married/partnered | 1052 (56.41) |
|
|
|
|
In next 30 days | 815 (43.70) |
|
In next 6 months, but not in next 30 days | 816 (43.75) |
|
Not thinking of quitting | 234 (12.55) |
Years smoked, mean (SD) | 24.9 (14.2) | |
Age (years), mean (SD) | 44.2 (14.7) | |
Nicotine dependence (FTND)b, mean (SD) | 4.2 (2.2) | |
|
|
|
|
Motivation for quitting | 7.4 (2.5) |
|
Self-efficacy for quitting | 5.4 (2.6) |
|
Importance of quitting | 7.6 (2.6) |
a Complete data were available on all baseline outcomes, with 1 missing value each for race and education.
b FTND: Fagerstrom Test of Nicotine Dependence.
Recruitment flow and allocation of participants.
Comparison of engagement metrics by factor level.a
Factor level | Prescriptive message tone | Proactive emails | Dictated navigation | Tailored testimonialb | ||||
|
Ratio of means |
|
Ratio of means |
|
Ratio of means |
|
Ratio of means |
|
Website visits | 1.00 |
.93 | 1.20 |
<.001 | 1.02 |
.70 | 0.92 |
.11 |
Content areas viewed | 1.17 |
<.001 | 1.08 |
.10 | 0.80 |
<.001 | 0.96 |
.30 |
Content page views | 1.15 |
.009 | 1.58 |
<.001 | 1.17 |
.003 |
|
|
Cumulative duration | 0.87 |
.06 | 1.51 |
<.001 | 1.37 |
<.001 |
|
|
a Point estimates represent ratio of means between each contrasting factor level and are adjusted for baseline stage of change. Results reflect the effect of each factor level (prescriptive tone, email reminders, dictated navigation, and testimonials) relative to those who did not receive the stated factor. Effects of randomized factors on website visits and content areas viewed were estimated with Poisson regression models, and effects on page views and duration were estimated using zero-inflated Poisson regression models.
b Content page views and duration of time spent viewing content were not examined for those in the testimonial factor because these individuals had more content pages to view containing more material.
Secondary analyses investigated the interaction between baseline readiness to quit (a measure of motivation) and each of the factor levels, to determine if participants with different levels of readiness to quit at enrollment engaged differently with the core Q2 program (results not shown). Out of 17 tests for interaction, only 1 was statistically significant; the prescriptive tone resulted in significantly less cumulative viewing time (ratio of means 0.54, 95% CI 0.36-0.83) among those with no interest in quitting in the next 6 months, but had no significant effect on the viewing times of those in interested in quitting in the next month (ratio of means 0.83, 95% CI 0.66-1.04) or next 6 months (ratio of means 1.03, 95% CI 0.82-1.29). These 3 estimates, each specific to a level of readiness to quit smoking, differed significantly from one another (
Program engagement is critical for any intervention to be effective, but promoting program engagement is a particularly important issue in Web-based interventions because treatment exposure is dependent on the motivation and self-direction of the individual user. In order to maximize the effectiveness of future Internet-based smoking treatment programs, we need a better understanding of how to engage smokers in these programs and, in particular, how to promote engagement with the most critical core program elements designed to motivate and promote behavior change. The current study provides insight into these issues by comparing the relative effects of 2 contrasting levels of each experimental design feature (factor): message tone (prescriptive vs motivational), navigation autonomy (dictated vs not), proactive email reminders (yes vs no), and inclusion of personally tailored testimonials (yes vs no). We sought to determine if one level promoted greater treatment engagement than the other within each factor.
We found that using a prescriptive message tone, dictating the order content was viewed, and sending email prompts had the greatest effects on early program engagement among a population-based sample of smokers at varying stages of readiness to quit smoking. Each of these increased the total number of core page views. Cumulative exposure to the core content was also increased by dictating navigation order and sending emails. Although the prescriptive tone was not statistically significant at the .05 level, the effect estimate was greater than 1 (
The inclusion of tailored testimonials did not have an effect on Web visits or the number of content areas viewed, although we did not expect it would. The primary goal of this factor was to promote smoking cessation through enhancing self-efficacy and modeling appropriate cessation-relevant behaviors, so its real impact is expected to be observed in future analyses examining long-term cessation and treatment utilization (the primary study outcomes). Also, persons whose navigation autonomy was dictated saw fewer total content areas, which was expected because they had to view all content in their stage-matched content area before gaining access to the other 2 content areas. This barrier likely deterred exposure to more content areas.
The findings suggest that using a prescriptive message tone, dictating the order in which content is viewed to match smokers’ initial level of interest in quitting, and sending weekly email prompts may increase online program engagement. This directive approach is somewhat counterintuitive for an intervention intended to motivate persons to quit smoking. Motivational interviewing suggests that people who are not ready to take action may respond better to counseling which is less directive and recognizes their ambivalence for change and autonomy to make their own decisions [
It is noteworthy that one-third of participants failed to view any of the core intervention content during the first 2 months of the study. The reason for this is not evident, but motivation for quitting could play a role. Overall, people who failed to view core content were less likely to be ready to quit smoking in the next 30 days compared to those who viewed the content. Future planned analyses will explore how those who viewed the intervention differed from those who did not and whether these individuals failed to ever view the core intervention content or simply delayed their viewing. All participants have access to the Q2 program for a full year after enrollment.
Several caveats should be considered when interpreting these results. First, the findings might look different if we had examined tracking data for the testimonial pages and special features. However, including these would inappropriately skew the effects of the factor levels on engagement because exposure to these program elements varied by treatment arm. Also, the special feature content was not considered part of the core smoking cessation intervention. Thus, our more conservative approach is justified. The findings might also look different if engagement was observed over a longer period of time. We chose to measure engagement over the first 2 months of enrollment because this would seem to be a critical time. If one fails to engage with the program within 2 months after making an effort to enroll, it may be that they will not engage at all. We will be able to address this in future analyses when 1-year follow-up data are available. Next, the average cumulative exposure duration would be higher if we had not imputed missing values for each page view that timed out after 30 minutes of inactivity. However, we believe it preferable to treat this information as missing and use multiple imputations to accommodate this missing data in the analyses than to potentially overestimate this important outcome and artificially inflate exposure. Additionally, although it is tempting to interpret the interaction results as evidence that the prescriptive tone was less effective among people with no interest in quitting smoking, caution must be used in drawing this conclusion since this was the only significant interaction out of 17 and we did not adjust for multiple comparisons. Finally, we should caution readers not to interpret the results as an evaluation of motivational interviewing per se, which is a specific counseling technique. We can only comment on the application of several key principles of motivational interviewing when applied in a Web-based program not the full complement of motivational interviewing skills, which would be difficult to simulate outside an actual counseling session. Thus, we consider this an evaluation of a motivational message tone grounded in motivational interviewing principles.
There are limitations with this study. For one, it is not clear if the results will generalize to other Internet-based treatment programs since engagement is associated with the specific content of an intervention. But because we focused on design principles such as message tone, navigation autonomy, and use of proactive emails, it will be possible for others to apply these same strategies to future programs and test their effects. Similarly, we do not know if the results will generalize to other smokers, particularly uninsured minority males. All smokers in the current study had medical insurance (at least at the time of enrollment), most were female (63%), and most were white (82%). However, enrolling a higher proportion of female and white smokers is consistent with findings from other population-based, online cessation trials [
The study has several distinct strengths. Chief among these, the study systematically explores how the design of a public health smoking intervention influences smokers’ interaction with the program. Other strengths include the large study sample (N=1865) which included a broad spectrum of smokers with differing levels of motivation to quit, use of a rigorous study design grounded in the MOST methodological framework, use of automated tracking data to confirm individual exposure to the website at the level of each individual Web page and time spent viewing specific pages, and use of sophisticated imputation methods to account for time spent online without overinterpreting cumulative exposure time based on time-out parameters.
The results of the current study provide important insight about how to design a population-based, online smoking cessation intervention. Ultimately, it will be important to see what effect each of the experimental factors has on long-term cessation outcomes, but the current study suggests that taking a directive intervention approach, including a prescriptive message tone, dictated site navigation, and proactive email outreach may be useful for increasing program engagement particularly in population-based interventions targeting smokers with varying levels of motivation for quitting. Future research should seek to replicate these findings. Moreover, more methodologically rigorous science should seek to systematically elucidate the optimal strategies for maximizing the effectiveness of online behavioral intervention programs.
multiphase optimization strategy
Questions about Quitting (intervention)
zero-inflated Poisson
This research was funded by the National Cancer Institute (R01 CA138598, J McClure, PI). We are grateful to the contributions of the many study team members at Group Health Research Institute and the University of Michigan. This includes the Group Health Research Institute Survey Research Program, Melanie Mamiya, Roy Pardee, and Rebecca Hubbard for their assistance designing and fielding the study, and Annie Shaffer for her assistance with the manuscript preparation. We also thank the Center for Health Communications Research at the University of Michigan, including Kate Barr, Janine Konkel, Mike Nowak, Ian Moore, Sarah Pennewell, and Ed Saunders for their contributions to the development and conduct of the Q2 trial. The intervention evaluated in this study was developed by researchers at the Group Health Research Institute and University of Michigan.
None declared.
CONSORT-EHEALTH checklist V1.6.2 [