This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The Internet provides us with tools (user metrics or paradata) to evaluate how users interact with online interventions. Analysis of these paradata can lead to design improvements.
The objective was to explore the qualities of online participant engagement in an online intervention. We analyzed the paradata in a randomized controlled trial of alternative versions of an online intervention designed to promote consumption of fruit and vegetables.
Volunteers were randomized to 1 of 3 study arms involving several online sessions. We created 2 indirect measures of breadth and depth to measure different dimensions and dynamics of program engagement based on factor analysis of paradata measures of Web pages visited and time spent online with the intervention materials. Multiple regression was used to assess influence of engagement on retention and change in dietary intake.
Baseline surveys were completed by 2513 enrolled participants. Of these, 86.3% (n = 2168) completed the follow-up surveys at 3 months, 79.6% (n = 2027) at 6 months, and 79.4% (n = 1995) at 12 months. The 2 tailored intervention arms exhibited significantly more engagement than the untailored arm (
By exploring participants’ exposures to online interventions, paradata are valuable in explaining the effects of tailoring in increasing participant engagement in the intervention. Controlling for intervention arm, greater engagement is also associated with retention of participants and positive change in a key outcome of the intervention, dietary change. This paper demonstrates the utility of paradata capture and analysis for evaluating online health interventions.
NCT00169312; http://clinicaltrials.gov/ct2/show/NCT00169312 (Archived by WebCite at http://www.webcitation.org/5u8sSr0Ty)
The major advantages of online interventions lie in their ability to reach large numbers of potential clients with very complex individually tailored designs and with relatively low cost [
This paper focuses on what is variously called dosage [
The goal of this paper was to use paradata to explore engagement in a randomized controlled trial (RCT) of an online intervention with several different arms. Specifically, we examined both breadth and depth of engagement defined in new measures built from paradata. We then explored how engagement was related to retention in the study, as measured by completion of the follow-up surveys. Finally, we addressed the relationship between engagement and key outcomes of the trial. Our expectation was that tailored interventions would result in greater engagement in the online material, leading to lower attrition in the intervention and improved outcomes. This paper provides a starting point to identify areas where online intervention design improvements may be required and, ultimately, may give us clues as to why a particular intervention may be more or less successful.
Data for this study came from the Making Effective Nutritional Choices for Cancer Prevention (MENU) study (Trial Registration: ClinicalTrials.gov NCT00169312), a randomized trial conducted in conjunction with the Cancer Research Network (CRN). The CRN is a consortium of 14 research organizations affiliated with nonprofit integrated health care delivery systems and the National Cancer Institute (NCI) [
Study subjects, aged 21 to 65 years, were randomly selected and recruited from the administrative databases of the 5 participating health care systems. Selection was limited to those members who had at least one-year enrollment in the respective health plan and had no record (according to diagnostic codes) of existing health conditions that might be negatively affected by increasing dietary fruit and vegetables. Equal numbers of men and women were selected, and 3 sites over-sampled minority racial/ethnic groups (African American or Hispanic) to enhance diversity in enrollment. Access to the Internet for personal use and use of a working email account, assessed during the study’s online eligibility survey, was also required for enrollment.
Of the 28,460 members mailed invitations to participate in the study, 4270 (15%) visited the website and 2540 (8.9% of those invited or 59.5% of those visiting the website) enrolled. Analysis following the 12-month survey identified 27 participants who reported inconsistencies in birth date and gender, suggesting different people may have completed the follow-up surveys. These cases were dropped from all analyses, leaving a final count of 2513 participants for analysis (
Participants were recruited with a single mailed invitation letter using health system stationery. The letter described eligibility criteria and included the Web address and a unique sign-on code which could be used to access more information about the study online. Also included were a US $2 bill preenrollment incentive and the promise of US $20 for completing each of the 3 follow-up surveys during the 12-month follow-up period [
Enrollees were randomized to 3 experimental arms receiving Web sessions that were (1) untailored, (2) tailored, or (3) tailored with email support which utilized a human online behavioral intervention (HOBI) consisting of behavior change counseling. Randomization was assigned by study site, gender, and stage of change with eating fruit and vegetables. Tailored Web sessions were based on health risk information and motivations for change obtained from baseline or 3-month post surveys. All materials were provided in English only.
An initial online welcome letter showed the participant’s current status of reported fruit and vegetable servings compared with recommended intakes [
The MENU tailored Web program included content and suggestions matched to each person’s gender, needs, characteristics, dietary preferences, and interests. Behavioral sessions were tailored to each person’s stage of change and were designed to increase participants’ motivation and self-efficacy for buying, preparing, and eating fruits and vegetables. Tailored web sessions also contained tailored video and audio files designed to reinforce behavioral advice featuring videos of food preparation by Graham Kerr, a well-known, health-conscious chef. Additionally, persons in the tailored arms were able to access an expert-tailored menu, which was generated based on their fruit and vegetable preferences, dietary restrictions, and other preferences.
In addition to the tailored program, participants in Arm 3 were offered corresponding email counseling support sessions. Each counseling session was initiated by a study counselor within a week after each Web session was first visited. Counselors provided additional support for dietary change, following the therapeutic principles outlined in motivational interviewing [
In addition to the sessions, participants could access “special features,” which were short, optional, and individually accessed clusters of Web pages that appeared periodically on the intervention website and which presented tips and other additional information in a pop-up window. Like sessions, notice of each feature’s availability was automatically delivered a fixed number of days after enrollment. Examples of special features included recipes developed by Graham Kerr, a dietary intake goal-setting tool, tips for eating out, food safety and storage, fun with fruit and vegetables, and nutritional similarities of fresh, frozen, and canned foods (for details, see [
The Web protocol for all data collection surveys was similar. Participants were asked to report fruit and vegetable intake at baseline, 3, 6 and 12 months, using one or both of two fruit and vegetables screeners. The first, used at baseline and 12 months, is based on a 16-item measure of fruit and vegetable servings, adapted from the NCI 19-item fruit and vegetable food frequency questionnaire [
Baseline description of the enrolled subjects by study arm
Study Arm | |||||
Variable | Total |
Arm 1 Control |
Arm 2 Tailored |
Arm 3 Tailored + |
|
Age (years), mean (SD) median | 46.3 (10.8) 48.0 | 46.1 (10.6) 47.0 | 46.5 (10.8) 48.0 | 46.4 (10.9) 47.0 | |
Female, n, % | 1729 (69) | 576 (69) | 577 (69) | 576 (69) | |
African American, n, % | 585 (24) | 192 (23) | 196 (24) | 197 (24) | |
Hispanic, n, % | 192 (8) | 69 (8) | 66 (8) | 57 (7) | |
Married/with partner, n, % | 1805 (72) | 595 (72) | 602 (72) | 609 (73) | |
High school education or less, n, % | 217 (9) | 76 (9) | 70 (8) | 71 (9) | |
Associate or some college, n, % | 1023 (41) | 334 (40) | 352 (42) | 337 (40) | |
College degree, n, % | 659 (26) | 219 (26) | 232 (28) | 208 (25) | |
Post bachelor’s education, n, % | 607 (24) | 205 (25) | 183 (22) | 219 (26) | |
|
|||||
Precontemplator stage, n, % | 49 (2) | 17 (2) | 14 (2) | 18 (2) | |
Contemplator stage, n, % | 1247 (50) | 412 (49) | 421 (50) | 414 (49) | |
Preparation stage, n, % | 511 (20) | 164 (20) | 175 (21) | 172 (21) | |
Action stage, n, % | 170 (7) | 54 (6) | 61 (7) | 55 (7) | |
Maintenance stage, n, % | 533 (21) | 189 (23) | 166 (20) | 178 (21) | |
|
|||||
Precontemplator stage, n, % | 40 (2) | 11 (1) | 17 (2) | 12 (1) | |
Contemplator stage, n, % | 1547 (62) | 519 (62) | 523 (62) | 505 (60) | |
Preparation stage, n, % | 389 (15) | 128 (15) | 124 (15) | 137 (16) | |
Action stage, n, % | 104 (4) | 35 (4) | 35 (4) | 34 (4) | |
Maintenance stage (%) | 430 (17) | 143 (17) | 138 (16) | 149 (18) | |
Fruits and vegetables/day, 16-item measure of servings: meana (SD) median | 4.4 (2.8) 3.8 | 4.6 (3.0) 3.9 | 4.2 (2.7) 3.6 | 4.5 (2.7) 4.0 | |
Fruits and vegetables/day, 2-item measure of servings: mean (SD) median | 3.3 (1.58) 3.0 | 3.3 (1.57) 3.0 | 3.2 (1.57) 3.0 | 3.4 (1.59) 3.0 |
a Using the Kruskal-Wallis test, the means by arms were statistically significantly different at
We examined the role of engagement in minimizing attrition or maximizing retention in the study. We defined retention as completion of the follow-up surveys at 3-, 6-, and 12-months after baseline.
We also examined two key substantive outcomes measured as change in mean fruit and vegetable consumption from baseline to 12-month follow-up. In both cases, a positive score indicated an increase in consumption. The 2 measures were correlated (
The baseline survey included 70 questions and took an average of 25 minutes to complete. The 3-month follow-up survey included 32 questions, taking an average of 13 minutes to complete; the 6-month survey included 30 questions, taking an average of 13 minutes to complete; and the 12-month survey included 80 questions, taking an average of 29 minutes to complete. A reminder letter was mailed to all enrollees a week prior to each survey due date, and an email reminder was sent to all enrollees on each survey due date. A series of 5 automated reminder emails were sent to anyone who had not completed the survey every 3 or 4 days after the due date. For the 3-month survey, phone call reminders were initiated in the final 5 days of the online completion “window” during which callers offered enrollees reminders to do the survey and the opportunity to complete the survey by phone. Nearly all of the assessments (> 96%) were completed online. Overall, 86.3% of baseline participants completed assessments at 3 months, 79.6% at 6 months, and 79.4% at 12 months with no significant differences by intervention arm.
The engagement measures were obtained using server-side paradata. For confidentiality reasons we did not embed JavaScript code in the Web pages to capture client-side paradata [
The MENU program consisted of 4 sessions, each made available at different time points: 3 days after baseline, 21 days after baseline, 3 days after the 3-month survey, and 21 days after the 3-month survey. Once new content was available, the user was automatically presented with the current new session at log-in. A bank of nearly 300 recipes and a goal-setting feature were available as optional elements throughout the study. All previous sessions remained available in a navigation bar at the top of the Web page. Participants could thus view up to 4 unique informational sessions by the end of the intervention program; however, the total count of sessions accessed could be higher if a session was viewed more than once.
The measure “unique sessions” was simply a count of the number of offered informational Web sessions visited at least once, with the maximum being 4.
To approximate the total time spent interacting with the website over the course of the study, we attributed the elapsed time between 2 time-stamped events to the action that generated the first of the events. These elapsed times were then accumulated across the various actions to give total elapsed times for each type of action done on the website. These accumulated times may have been slightly lower than the time actually spent on the site since we did not capture how long the participant spent reading the previously accessed Web session or special feature.
We focused on the 4 measures of engagement captured through the website paradata and described above: total session accesses, unique session accesses, total special feature accesses, and total time on the website (excluding time spent completing the surveys) (see
BREADTH is a summary measure of access to all activity on the website. It is composed of the sum of the 4 measures in
DEPTH is a summary measure of how deeply individuals engaged in the online material, for a given level of overall Web activity. NON_SURVEY_MINS (total minutes spent excluding survey completion) and SF_TOT (total number of special feature accesses) loaded positively on the second principal component, while SESS_UNIQ (number of unique session accesses) loaded negatively, with the loading of SESS_TOT (total number of session accesses) close to 0. The measure of DEPTH is thus obtained as the sum of the average (standardized) total of accessed special feature sessions (SF_TOT) and standardized nonsurvey minutes spent online (NON_SURVEY_MINS), minus twice the total number (standardized) of unique sessions accessed (SESS_UNIQ). The more special features a person accessed, and the longer they spent on the website relative to the number of different sessions they saw, the higher the value of DEPTH. DEPTH approximates the second component from the PCA.
Using the factor loadings from the PCA yielded similar results to those using the methods described above. The measures of BREADTH and DEPTH were again standardized (mean 0, SD 1) for further analyses. The two measures were slightly positively correlated,
In the multivariate models, we controlled for a number of additional variables measured at baseline. Fruit and vegetable consumption was based on the sum of 2 single measures and collapsed into low (less than 2 servings per day), medium (2 to 4 servings per day), and high (5 or more servings per day) consumption.
We focused on several outcomes of interest. First, utilizing our 2 newly derived indicators of the depth and breadth of engagement based on PCA, we explored the correlates of these engagement indicators from the baseline survey, using ordinary least squares (OLS) regression. Next, we examined completion of the follow-up surveys using both the baseline measures and the 2 new engagement indicators as predictors. These analyses used generalized estimating equations (GEE), reflecting the within-subject correlation across outcomes. A likelihood ratio chi-square was used to test whether the addition of the 2 engagement indicators improved the model fit. Finally, we examined 2 key outcome measures (fruit and vegetable consumption at 12 months) to explore how engagement may mediate the effect of the intervention on outcomes. The models again used OLS regression. Statistical analyses were done using SAS 9.1.3(SAS Institute Inc, Cary, NC, USA).
The 4 component indicators of engagement are presented in
Descriptive statistics on component engagement measures (n=2513)
Variable | Mean | SD | Median |
Total session accesses (SESS_TOT) | 10.64 | 7.14 | 9 |
Unique session accesses (SESS_UNIQ) | 3.14 | 1.20 | 4 |
Total special feature accesses (SF_TOT) | 11.13 | 10.79 | 8 |
Total time excluding survey completion (NON_SURVEY_MINS) | 42.16 | 42.93 | 29.55 |
The mean number of special feature accesses (8.3 for arm 1, 10.2 for arm 2, 10.3 for arm 3) and mean total minutes devoted to the intervention website (32.3 for arm 1, 44.1 for arm 2, 46.7 for arm 3) differed significantly by arm (F2,2512 = 9.57,
We regressed the standardized measures of depth and breadth, in turn, on a series of sociodemographic and related behavioral variables at baseline, using OLS regression (SAS 9.1.3 PROC GLM, SAS Institute Inc, Cary, NC, USA). These models are presented in
Models of standardized breadth and depth regressed on common demographic/baseline variables
Predictors | Breadth | Depth | |||
Coefficient | (SE) | Coefficient | (SE) | ||
|
|||||
Arm 1: Untailored | --- | --- | --- | --- | |
Arm 2: Tailored | 0.114 b | (0.047) | 0.234 b | (0.049) | |
Arm 3: Tailored with HOBI | 0.141 b | (0.047) | 0.305 b | (0.049) | |
|
0.407 b | (0.044) | 0.087 | (0.046) | |
|
|||||
< 29 | -0.475 b | (0.101) | -0.428 b | (0.104) | |
29-38 | -0.437 b | (0.081) | -0.315 b | (0.084) | |
39-48 | -0.230 b | (0.074) | -0.262 b | (0.077) | |
49-58 | -0.047 | (0.068) | -0.182 b | (0.070) | |
59+ | --- | --- | --- | --- | |
|
|||||
White | --- | --- | --- | --- | |
Black | -0.045 | (0.050) | 0.098 | (0.052) | |
Other | -0.034 | (0.071) | -0.110 | (0.073) | |
Hispanic versus non Hispanic | -0.127 | (0.086) | 0.094 | (0.088) | |
|
|||||
High school or lessc | --- | --- | --- | --- | |
Some college | 0.110 | (0.059) | -0.150 a | (0.061) | |
College graduate | 0.106 | (0.063) | -0.137 a | (0.065) | |
Postgraduate | -0.026 | (0.064) | -0.265 b | (0.067) | |
|
-0.167 b | (0.046) | -0.062 | (0.047) | |
|
|||||
Never married | --- | --- | --- | --- | |
Formerly married | -0.048 | (0.079) | 0.007 | (0.082) | |
Married/living with partner | -0.015 | (0.066) | 0.071 | (0.068) | |
|
|||||
Poor to good | -0.050 | (0.042) | 0.063 | (0.044) | |
Very good to excellent | --- | --- | --- | --- | |
|
|||||
Low | -0.047 | (0.063) | 0.101 | (0.065) | |
Medium | --- | --- | --- | --- | |
High | 0.031 | (0.052) | -0.054 | (0.054) | |
|
|||||
Low | -0.126 a | (0.057) | -0.071 | (0.059) | |
Medium | --- | --- | --- | --- | |
High | -0.078 | (0.048) | -0.069 | (0.050) | |
|
|||||
Low | -0.126 a | (0.057) | 0.018 | (0.056) | |
Medium | --- | --- | --- | --- | |
High | -0.078 | (0.063) | 0.015 | (0.060) | |
|
|||||
Low | 0.021 | (0.054) | -0.027 | (0.056) | |
Medium | --- | --- | --- | --- | |
High | -0.074 | (0.058) | -0.023 | (0.060) | |
|
|||||
Inactive | 0.197 a | (0.094) | 0.182 | (0.098) | |
Low activity | 0.163 a | (0.066) | 0.047 | (0.068) | |
Somewhat active | 0.100 | (0.059) | 0.065 | (0.061) | |
Very active | --- | --- | --- | --- | |
|
|||||
Intrinsic motivationd | 0.074 b | (0.021) | 0.026 | (0.026) | |
Extrinsic motivatione | -0.047 b | (0.014) | -0.017 | (0.014) | |
|
|||||
Constant | -0.287 | (0.195) | -0.259 | 0.202 | |
Observations | 2461 | 2461 | |||
R2 | .108 | .053 |
a
b
c Category includes those with vocational or technical training.
d Intrinsic motivation measures personal importance or internal drive to do a behavior. Examples are: “I have a strong value for eating healthy” and “I want to take responsibility for my own health.”
e Extrinsic motivation measures perceived outside influences on behavior. Examples are: “Others would be upset with me if I didn’t (eat more fruits and vegetables)” and “It is easier to do what I am told.”
Together these baseline measures explained a modest proportion of variation in the breadth (
Few of the baseline measures showed significant associations with the measures of engagement in the program. Low comfort using the Internet was significantly related to lower breadth, or amount of the website seen. Those with low motivation to eat fruit upon enrollment exhibited slightly lower breadth of engagement, but those who were less physically active showed higher levels. Intrinsic motivation was positively associated with depth, while extrinsic motivation was negatively associated with depth.
In the second step, we used the standardized breadth and depth measures of engagement along with all of the baseline measures included in
We used a generalized estimating equation (GEE) in SAS 9.1.3 PROC GENMOD to model survey completion, reflecting the within-subject correlation across outcomes [
Model of survey completion at 3-, 6-, and 12-months
Predictors | Odds Ratio | 95% Confidence Interval | |
|
|||
3-month | 1.0 | ||
6-month | 0.48 b | (0.42-0.56) | |
12-month | 0.48 b | (0.41-0.55) | |
|
|||
Arm 1: Untailored | 1.0 | ||
Arm 2: Tailored | 0.82 | (0.64-1.06) | |
Arm 3: Tailored with HOBI | 0.79 | (0.61-1.02) | |
|
1.07 | (0.85-1.35) | |
|
|||
< 29 | 1.06 | (0.63-1.81) | |
29-38 | 1.1 | (0.71-1.69) | |
39-48 | 1.03 | (0.70-1.54) | |
49-58 | 0.92 | (0.63-1.34) | |
59+ | 1.0 | ||
|
|||
White | 1.0 | ||
Black | 0.85 | (0.66-1.09) | |
Other | 0.82 | (0.57-1.17) | |
Hispanic | 0.63 a | (0.41-0.97) | |
|
|||
High school or less | 1.0 | ||
Some college | 0.86 | (0.64-1.15) | |
College graduate | 1.02 | (0.74-1.41) | |
Postgraduate | 1.32 | (0.94-1.85) | |
|
1.00 | (0.79-1.27) | |
|
|||
Married | 1.0 | ||
Formerly married | 0.99 | (0.66-1.50) | |
Never married | 1.24 | (0.87-1.77) | |
|
|||
Poor to good | 0.77 a | (0.62-0.96) | |
Very good to excellent | 1.0 | ||
|
|||
Low | 0.77 | (0.58-1.04) | |
Medium | 1.00 | ||
High | 1.00 | (0.74-1.34) | |
|
|||
Low | 0.97 | (0.72-1.30) | |
Medium | 1.0 | ||
High | 1.06 | (0.82-1.36) | |
|
|||
Low | 0.97 | (0.73-1.29) | |
Medium | 1.0 | ||
High | 1.17 | (0.86-1.59) | |
|
|||
Low | 1.29 | (0.96-1.73) | |
Medium | 1.0 | ||
High | 0.73 a | (0.54-0.99) | |
|
|||
Inactive | 0.68 | (0.43-1.09) | |
Low activity | 0.89 | (0.62-1.29) | |
Somewhat active | 0.91 | (0.65-1.26) | |
Very active | 1.0 | ||
|
|||
Intrinsic motivation (see |
0.98 | (0.86-1.12) | |
Extrinsic motivation (see |
0.96 | (0.90-1.03) | |
Breadth | 4.11 b | (3.61-4.69) | |
Depth | 2.12 b | (1.89-2.38) | |
Constant | 22.55 b | (7.95-63.92) | |
|
|||
Observations | 7383 | ||
Max-rescaled |
.32 |
a
b
From the model, we can see a significant drop-off in completion from the 3-month follow-up to the 6-month follow-up, but not from the 6-month to the 12-month. What is striking from
However, our main focus was on the role of the 2 engagement measures. Both were significantly and strongly associated with survey completion. The likelihood ratio (LR) chi-square test of the addition of these two variables to the model was significant (LR χ2
2 = 1005.8,
Finally we added BREADTH and DEPTH to a model regressing 2 key fruit and vegetable intake outcome variables on the baseline measures included in the models in
This paper focused on the use of paradata to measure the process of engagement in an online intervention aimed at increasing fruit and vegetable consumption. These data, collected throughout delivery of these online materials, reveal what pages of informational sessions are visited and the frequency and duration of the visit, but they do not reveal
First, those in the 2 tailored intervention arms showed higher levels of engagement—as indicated by the 2 composite measures, BREADTH and DEPTH of engagement—than those in the untailored arm. This suggests that the tailoring is responsible for participants’ increased use of the program materials. Variation in engagement by demographic characteristics may indicate groups’ differing levels of interest in the program or online materials. Whether this is a reaction to the intervention content or a reflection of preexisting differences in interest that were not captured by our baseline measures is unclear.
Second, the engagement indicators were significant correlates of attrition from the intervention. This suggests that the more participants are engaged with the online materials, the more likely they are to complete the follow-up surveys. This is a key finding, as discovering mechanisms that promote collection of more complete outcome measures is essential to research studies.
Finally, engagement was also significantly associated with the key behavioral outcomes of the study: changes in fruit and vegetable consumption. Those who spent more time on the website, who visited a greater number of pages, and who visited the site more often, as captured by the composite measure of breadth of engagement, showed significantly greater gains in fruit and vegetable consumption from baseline to 12-month follow-up than did those who exhibited less engagement. This finding provides further empirical evidence that “dose matters” in Web-based interventions. [
Key strengths include the large number of participants and the racial/ethnically diverse sample of relatively healthy adults from 5 geographic regions, providing a large number for analyses by subgroup. The relatively high response rates for the follow-up surveys permitted analysis of baseline and process variables to understand change in eating behaviors. Paradata measures were collected with date and time stamps over the 12-month study duration, which permitted the creation of duration and frequency variables and quantified the time lapse between website visits.
Limitations include the requirement that participation eligibility include both access to the Internet and an active email account, so findings may not generalize to all Internet users. We also were limited in the detail of the paradata we collected, as we were limited in measuring interruptions or distraction time during a Web encounter. This may have influenced our ability to distinguish between “sessions” and “visits” and did not provide details on what participants did within website sessions. Further, the incentives paid for participation, which were equivalent across intervention arms, and the effort taken to retain participants, relying mainly on automated email and single mailed reminders, may limit generalizability to other online interventions regarding the levels of engagement.
This paper demonstrates the usefulness of paradata in providing insight into the process by which an online intervention may affect outcomes. Such data are useful in identifying the “active ingredients” in a tailored intervention, that is, what works and what does not. Paradata could also be used to improve the design of online health interventions and websites, whether tailored or not, by identifying such components as which features visitors use, what pages they visit and revisit, and how long they spend on various parts of the site. This information could be used, in combination with other methods such as debriefing questionnaires or usability tests, to identify areas for program improvement, either in content or in navigation. We used a limited set of paradata captured in this online intervention. It is relatively easy to embed richer measures in health websites to provide more insight into what users are doing when they visit such sites. As online interventions increase in utilization and extend accessibility to various populations, we urge the collection and reporting of analysis of expanded paradata measures to improve the design and effectiveness of online health interventions.
This study was supported by National Cancer Institute awards U19-CA079689 (Cancer Research Network) and P50-CA101451 (Centers of Excellence in Cancer Communication). We are indebted to the participants in MENU Choices and to the research team for their contributions to the MENU trial.
None declared
Cancer Research Network
generalized estimating equations
human online behavioral intervention
likelihood ratio
Making Effective Nutritional Choices for Cancer Prevention study
National Cancer Institute
total time excluding survey completion
ordinary least squares regression
principal components analysis
randomized controlled trial
total session accesses
unique session accesses
total special features accesses
Treatment Self-Regulation Questionnaire