Published research on the use of Web-based behavior change programs is growing rapidly. One of the observations characterized as problematic in these studies is that participants often make relatively few website visits and spend only a brief time accessing the program. Properly structured websites permit the unobtrusive measurement of the ways in which participants access (are exposed to) program content. Research on participant exposure to Web-based programs is not merely of interest to technologists, but represents an important opportunity to better understand the broader theme of program engagement and to guide the development of more effective interventions.
The current paper seeks to provide working definitions and describe initial patterns of various measures of participant exposure to ChewFree.com, a large randomized controlled trial of a Web-based program for smokeless tobacco cessation.
We examined measures of participant exposure to either an Enhanced condition Web-based program (interactive, tailored, and rich-media program) or a Basic condition control website (static, text-based material). Specific measures focused on email prompting, participant visits (number, duration, and pattern of use over time), and Web page viewing (number of views, types of pages viewed, and Web forum postings).
Participants in the ChewFree.com Enhanced condition made more visits and spent more time accessing their assigned website than did participants assigned to the Basic condition website. In addition, exposure data demonstrated that Basic condition users thoroughly accessed program content, indicating that the condition provided a meaningful, face-valid control to the Enhanced condition.
We recommend that researchers conducting evaluations of Web-based interventions consider the collection and analysis of exposure measures in the broader context of program engagement in order to assess whether participants obtain sufficient exposure to relevant program content.
One of the common findings of research on Web-based behavior change programs is that participants spend only a relatively meager amount of time accessing their online intervention [
This paper describes participant exposure to a two-arm randomized controlled trial of Web-based programs designed to assist adults in quitting smokeless tobacco (either snuff or chewing tobacco). Following a brief program description, we present a set of unobtrusive measures of website exposure [
We designed the ChewFree trial to compare the efficacy of two smokeless tobacco cessation websites: Basic and Enhanced.The Basic condition, which represented a subset of the content presented in the Enhanced condition, offered a printable self-help smokeless tobacco cessation booklet, printable cessation resources (eg, describing the use of herbal snuff products, nicotine replacement products), and annotated links to other recommended websites for tobacco cessation. The Enhanced condition offered a tailored and interactive Web-based program that included text-based information (health and behavioral strategies focused on quitting and preventing relapse), video-based testimonials, printable resources, interactive activities, annotated links to other website resources, and two Web forums (a "Talk with Others" social support forum, and an "Ask an Expert" forum for submitting questions to project staff).
ChewFree.com intervention components were based largely on Bandura's Social Cognitive Theory [
The Basic and Enhanced Web-based programs offered smokeless tobacco cessation assistance using markedly different information architectures [
Basic condition (excerpt of Enough Snuff guide)
Enhanced condition (excerpt showing video narration by a smokeless tobacco cessation expert to accompany the Personal Quitting Assistant, or PQA)
Content was presented using text, graphics, activities, and two types of videos: a video expert guide who narrated key portions of content that was also presented as text, and video testimonials of smokeless tobacco users whose presentations supported the recommendations of the program. The narration videos were automatically launched (ie, they did not require user selection) for users with high bandwidth connections, but they were not displayed automatically to participants with dial-up access [
We designed the Enhanced condition to be attractive by offering a broad spectrum of content tailored to the interests and the smokeless tobacco use/abstinence status of each participant. For example, participants who were preparing to quit were encouraged to review program content focused on Planning to Quit, whereas those participants who indicated that they had quit using smokeless tobacco were encouraged to review content on Staying Quit. In addition, the intervention used multiple methods for delivering content along with engaging activities. Compared to the Basic condition, we predicted that the Enhanced condition would encourage participants to visit more often and for longer periods of time—especially during the first several weeks post-enrollment when attempts to quit and related lapse/relapse experiences would most likely occur.
Participants were recruited using a multifaceted marketing campaign that included (1) thematic promotional "releases" to print and broadcast media, (2) Google ads, (3) placement of a link on other websites, (4) limited purchase of paid advertising, (5) direct mailings to smokeless tobacco users, and (6) targeted mailings to health care and tobacco control professionals. This campaign resulted in more than 23500 visits to the ChewFree.com recruitment website from distinct IP addresses over a 1-year period, which yielded 2523 eligible smokeless tobacco users who completed the registration process and enrolled in the ChewFree.com smokeless tobacco cessation research project [
There is no single universally accepted measure for assessing participant exposure to a Web-based program. Computer-delivered content lends itself well to unobtrusive monitoring of usage patterns. As noted by Peterson [
For the present paper, we examined data from participants as of January 10, 2006, which, for most participants, represented approximately 12 months after enrollment (mean = 367.1 days, SD = 116.9; no significant differences between conditions). The minimum number of days since enrollment was 181 days and, in each case, the scheduled date of the 6-month follow-up assessment had elapsed.
|
Percentage of participants sent treatment-related email prompts |
|
Number of visits |
Aggregate duration of visits | |
Number of daily visits post-enrollment | |
Number of days of program access post-enrollment | |
|
Overall number of Web page views |
Specific Web page views (selected smokeless tobacco cessation content) | |
Web forum postings |
Participants in the Enhanced condition received a variety of email prompts during the study that were not related to assessments. These prompts fell into three categories:
1. Intervention: Participants were sent up to three email messages prior to their quit date, tailored to their chosen method of quitting (cold turkey, nicotine fading, brand switching, blending), and one message on their quit date.
2. Support: We sent three supportive emails timed at fixed intervals after the participant's self-reported quit date.
3. Re-engagement: Participants who failed to log in on a regular basis were typically sent multiple tailored email messages encouraging them to resume accessing the program.
Typical measures of visit data include number of visits per participant per condition and both average and total visit duration. We programmed the ChewFree website to record the date/time stamp of the start and end of each participant visit (also referred to as a "session") and for each Web page the participant viewed during each visit. These date/time stamps allowed us to examine both the number of unique visits per participant and session duration.
Because participants were able to abruptly end their use of the program by closing their browser window, there were occasions when we did not capture the date/time data for the end of the session. To analyze these instances, we conservatively approximated the end of the visit by using the date/time of the last Web page that had been accessed before the abrupt end of the session. In addition, we followed the operational definition for visit expiration recommended by Peterson [
Participants in both conditions were required to complete an online baseline assessment prior to accessing the program. In addition, all participants received email reminders to complete online follow-up assessments at 6 weeks, 3 months, and 6 months. The email prompt contained a link that caused the log-in page of the Web-based program to appear, followed by presentation of the online assessment. At the end of each assessment, users were returned to their respective website, at which time they were free to explore the website and review its contents. When counting distinct visits that involved program content review, we excluded those visits associated with online assessments unless the participant also explored website content.
We focused our analysis on aggregate duration (collapsed across visits) because we were concerned with the overall amount of participant exposure to the program. Although we did not choose to do so for purposes of this paper, we could also have examined the changing patterns in the duration of individual visits over time.
We examined the time course for each participant visit by calculating the number of days in which a visit occurred since the date the participant completed the baseline assessment and formally began the study. It is important to note that at the end of the baseline assessment, each participant was automatically presented with the home page of the condition to which he/she was randomly assigned. If, following the end of the assessment, a participant continued to explore the Web-based content, then that event was counted as a unique visit and assigned a value of zero (since zero days had elapsed since the end of the baseline assessment). If a participant had multiple visits on any given day, then this analysis counted each of those visits in the total for that day (ie, participants could have multiple visits per day). We limited our analysis to those visits in which Web-based program content pages were accessed.
In addition to measuring the number of visits per day, we used Kaplan-Meier survival analyses [
Finer grained within-visit analyses focused on participants' viewing of Web pages that presented specific content designed to encourage smokeless tobacco cessation. Because we recorded the date/time of each Web page viewed during each visit, it was possible to calculate the percentage of participants who viewed specific types of Web pages (using the participant sample in each condition as the denominator). For example, we were able to measure the extent to which participants in either condition accessed a ChewFree.com Web page that provided links to other websites offering smokeless tobacco cessation information and assistance (eg, the National Cancer Institute, the National Spit Tobacco Education Program, and the Oral Health America Foundation).
In the Enhanced condition, we also measured participants' use of ChewFree.com Web pages that offered more interactive features, including whether they viewed pages that automatically played video testimonials, whether they accessed a Web page that offered a print feature (and triggered a print dialog box), and whether they listed the names of people whom they believed could offer useful support for smokeless tobacco cessation. And although this paper focuses on exposure rather than on outcome results, we also report on the extent to which participants in the Enhanced condition used the Web page designed to help them choose a quit date for stopping the use of smokeless tobacco.
Finally, we captured data on the extent to which participants in the Enhanced condition used the available peer Web forum ("Talk with Others") or expert forum ("Ask an Expert"). Forum use was logged into the database when participants posted messages, either by creating a new message or responding to an existing message. Unfortunately, we did not track passive viewing of the forum messages, nor did we collect data that would allow us to calculate the amount of time spent viewing forum content.
Analysis revealed that 63.3% of participants (760/1220) in the Enhanced condition set a quit date and were sent a program-generated series of tailored email prompts associated with preparing to quit. After having been sent at least one of these emails, 10.7% of these participants (81/760) requested to opt out of receiving further emails. A total of 40.7% of participants (488/1220) who reported having quit using smokeless tobacco during the course of the program were eligible to be sent a series of emails supportive of continued abstinence. However, the number of participants who were sent these supportive emails was reduced to 34.8% (425/1220) because 63 had opted out of receiving program-generated emails. Enhanced condition participants who had not exercised the opt-out option (90%; 1079/1200) were also scheduled to receive emails at 7, 30, and 60 days since last log-in, encouraging them to re-engage with the site. We plan to conduct future analyses to assess the relation between the automated email prompts, website usage, and outcome results.
Our initial analysis showed that 0.6% of participants (7/1260) in the Enhanced condition and 0.8% of participants (10/1263) in the Basic condition never visited their assigned website after completing the baseline assessment and becoming enrolled. An additional 3.7% of participants (47/1260) in the Enhanced condition and 5.9% of participants (74/1263) in the Basic condition returned following enrollment but did so only to complete online assessments. These individuals never viewed any Web pages that contained smokeless tobacco cessation content. Removing these participants from our analyses reduced the sample to 2375 participants (1200 in Enhanced condition; 1175 in Basic condition) for whom visit duration could be measured (
Rather than being normally distributed, the observed patterns of website visit frequency and duration displayed a significantly positive- or right-skewed distribution, with most cases having occurred at lower values (more frequent and longer visits occurring soon after enrollment). We used the nonparametric Mann-Whitney
Visit details by participant
|
|
||||
|
|
|
|
|
|
Enhanced | 1200 | 2.00 | 3 |
28.99 | 37.75 |
Basic | 1175 | 1.00 | 1 |
12.50 | 15.83 |
*P < .001
Visits by time course for those 2375 participants who viewed smokeless tobacco cessation content are depicted in
Visits following enrollment
We observed 3783 visits for participants in the Enhanced condition and 2054 visits in the Basic condition. Consistent with usage patterns reported in other research of Web-based interventions [
We also examined the number of days following enrollment that participants continued to access their assigned website for program content (excluding visits to take online assessment only). For purposes of this survival analysis, the last content-accessing visit for each participant was designated as the final date of program usage. For example, 36.1% of participants (433/1200) in the Enhanced condition and 60.7% (713/1175) in the Basic condition stopped using the program on the day they enrolled in the program. Because
We assumed that each participant, regardless of condition, would eventually stop using the Web-based program. Thus we examined the differential pattern of program use atrophy. As depicted in
Website activity following enrollment (survival curve)
In addition to metrics of overall website exposure, we were interested in the extent to which participants accessed content that contained specific information most relevant to smokeless tobacco cessation and tobacco abstinence.
Web page viewing by participants who accessed at least one Web page containing smokeless tobacco cessation content
|
|
|
|
|
|
|
|
|
Enhanced | 1200 | 18.2 | 78.3 | 12.2 | 68.2 | 24.7 | 63.3 | 32.8 |
Basic | 1175 | 32.1 | 96.3 | 87.5 |
We found that 38.2% participants in the Enhanced condition (481/1260) posted content to the Web forum for peers, with 5.2% (65/1260) posting at least one message in the expert forum (
Web forum activity in the Enhanced condition (n = 1260 users)
|
|
|||
|
|
|
|
|
Peer | 481 | 38.2 | 2 | 11 |
Expert | 65 | 5.2 | 1 | 1 |
It is important to acknowledge several limitations to the present study. First, we did not design the Enhanced intervention website to track passive Web forum viewing. This limitation prevented us from analyzing the duration of Web forum visits by participants who observed postings but did not post their own comment on the forum posts of others. In addition, although study inclusion criteria required all study participants to be able to access their personal email at least once per week, we did not collect data on participants' previous experience using the Internet or on their computer self-efficacy [
It is thought that a key ingredient in determining the impact of any Web-based behavior change program is the extent to which participants are exposed to the program. This assumption is consistent with the finding that the efficacy and intensity of treatment programs tend to be positively related. For example, research on smoking cessation interventions—including self-help approaches—has illustrated the relationship between abstinence rates and program intensity, typically defined as contact time and number of sessions [
Some reviewers of this burgeoning field have recommended that fuller participation in Web-based interventions might be encouraged through the use of a "warm-up period" during which users can demonstrate their commitment by complying with precursor tasks while they become more familiar with what will be asked of them during the course of the program [
We found that the estimated median lifetime website usage (date when 50% of participants stopped using the program) was 11 days for the Enhanced condition and 0 days (ie, the enrollment day) for the Basic condition. We anticipate that some measures of exposure and outcome will likely share a curvilinear (inverted U-shaped) relationship such that those individuals who are least ready to make a meaningful change may be more likely to visit the Web-based program for a short time, while participants who are most prepared to change their behavior may similarly choose to visit the Web-based program for a relatively short time. Those participants who are interested in quitting and decide to learn more about how to do so will spend relatively more time visiting the program. It remains for future research to differentiate characteristics that illuminate the pattern of this relationship among motivation, readiness to quit, and program usage.
Measures of participant exposure can help researchers and program developers determine the extent to which content is viewed. These data can point to needed changes in the information architecture and design features of the website. It is reasonable to assume that program content cannot be helpful if it is never viewed. Exposure measures may have utility in that they inform us about whether certain content—or clusters of content—is related to outcome and thus might be considered to be active ingredients in accomplishing the desired behavioral goals. They enable us to better understand idiosyncratic patterns of program use, highlighting ways we can adapt program structure and content to better accommodate (be tailored to) individual differences in participant interests, needs, and learning styles.
In this regard, we intend to examine a variety of relationships between and among measures of exposure and the smokeless tobacco and tobacco cessation outcome measures in the ChewFree.com research project. For example, we will test whether participants who set a quit date are more successful in quitting, as well as whether, after quitting, there is a relationship between accessing content from the Staying Quit module (number and duration of visits) and lasting abstinence. Similarly, we will examine whether those participants who spend more time reviewing program content after they have lapsed are better able to regain control over their behavior and regain abstinence. We also plan to perform content and text analyses of Web forum postings [eg, 48] to explore whether smokeless tobacco cessation might be related to message types, whether cessation and maintenance strategies shared in postings are consistent with program recommendations, and the extent to which postings convey differing levels of confidence and self-efficacy across participants as well as within participants over time.
There is a significant risk of collecting so much detailed exposure and engagement data that the task of analyzing and interpreting results becomes difficult. We suggest that this task can become more manageable and, thus, more fruitful, by focusing its scope through the use of a rationale that incorporates both theory and pragmatism. Potentially relevant rationales are not difficult to identify. Consider, for example, a rationale that builds on the Web foraging model [
We view exposure as representing one of a set of complementary measures of the broader theme of program engagement. Other engagement measures include participant comprehension of program content, practice of that content (especially in the participant's everyday routines outside of interacting with the Web-based program), self-reported satisfaction with the function and content of the website, and measures of self-efficacy. While exposure is obviously important (indeed, it is best viewed as a prerequisite), it represents only one piece of the puzzle in seeking to understand program effectiveness.
Thanks to Edward Lichtenstein, John Noell, and John Seeley for their careful reviews of this paper. We also acknowledge the important contribution of Steven Christiansen and Tim Woolley of InterVision (Eugene, OR) in the development of the ChewFree.com Web-based programs. This project was funded by a grant from the National Cancer Institute (R01-CA84225).
None declared.
Table of detailed measures of program exposure
|
a) Percentage of participants receiving email prompts | Programmatic rules defined the timing of email prompts regarding completion of online assessments that were sent to participants in both conditions. Participants in the Enhanced condition also were sent "treatment-related" email prompts that contained tailored content related to quitting plan, quit date, and support of continued abstinence. |
|
b) Number of visits | Visits by each participant to access smokeless tobacco cessation content on their assigned website were counted. All visits in which only online assessments were accessed were excluded. |
c) Aggregate duration of visits | Duration of each visit was defined as the sum of Web page durations during that visit. With one exception (noted next), Web page durations were defined as their logged end time minus start time. Because our program logic did not include a session-expiration feature that automatically logged out after a period of inactivity, we conservatively approximated the end of the visit by using the date/time of the last Web page that had been accessed before the abrupt end of the session. In addition, we followed the operational definition for visit expiration recommended by Peterson; that is, any Web page viewed for 30 minutes or more was defined as having ended the visit using the ending date/time stamp of the Web page that immediately preceded the hiatus. Moreover, if, after the hiatus, the participant resumed activity, it was considered to be a new visit for measurement purposes. | |
d) Number of daily visits post-enrollment | The number of daily visits per participant was counted with "days" being defined in terms of their occurrence relative to the participant enrollment date. Participants could have more than one visit per day, and visits were defined using Peterson's recommendation (see above). Total visits per post-enrollment date aggregated these data across participants. | |
e) Number of days of program access post-enrollment | The number of days of post-enrollment access to smokeless tobacco cessation content was defined for each participant as the last visit date minus the enrollment date. | |
f) Plan to Quit | Participants in the Enhanced condition were given the opportunity to define a personal quitting plan and quit date. Although the program allowed participants to define their quitting plan and date multiple times, for purposes of the analyses in this report, we focused only on the first recorded date when a participant defined his/her quitting plan/date. | |
g) Smokeless tobacco quit status | Participants in the Enhanced condition were able to indicate that they had quit using smokeless tobacco. This report enabled them to access "Staying Quit" content. | |
|
h) Overall number of Web page views | The total number of Web pages viewed was logged. |
i) Specific page views (selected smokeless tobacco cessation content) | The total number of selected Web pages viewed related to smokeless tobacco cessation was logged, and a subset of these is described in this report. | |
j) Web forum postings | Individual forum postings—in each of the two forums—were logged for each participant (Enhanced condition only). |