Background: In the Netherlands, human papillomavirus (HPV) vaccination uptake remains low. To improve informed decision making (IDM) and HPV vaccination acceptability, we systematically developed an interactive, web-based tailored intervention to which mothers of Dutch girls were invited to participate.
Objective: The aim of this study was to provide insight into the intervention’s working mechanisms by evaluating (1) program use, (2) program acceptability, and (3) the relationship of program use with program acceptability and intervention effects (ie, dose-response relationship).
Methods: Only mothers from the intervention arm of a randomized controlled trial that assessed the effectiveness of the web-based, tailored intervention were included in this study. They were invited to visit the website of the web-based intervention between baseline (January 2015, just before access to the intervention) and follow up (March 2015, prior to the first HPV vaccination). Indicators for program use were time of website use (ie, duration of intervention exposure) and completeness (ie, proportion of all available web pages visited). HPV vaccination uptake registered by Praeventis was used as the primary outcome. Secondary outcomes were IDM, decisional conflict, and social-psychological determinants of HPV vaccination uptake.
Results: Among the 3995 invited mothers, 2509 (62.80%) logged on to the website, 2239 of whom (89.24%) visited at least one page of the intervention components. On average, mothers spent 21.39 minutes (SD 12.41) on the website and completed 50.04% (SD 26.18%) of the website components. Participants rated the website 7.64 (SD 1.39) on a 10-point scale. Program acceptability was significantly associated with completeness (β=4.36, P<.001), but not with time of website use (β=–.07, P=.77). Intention-to-treat analysis (N=3995) showed a significant positive effect of completeness on all outcome measures (all P<.003; Bonferroni-corrected α=.05/15 factors), including on HPV vaccination uptake. Time of website use had a significant positive effect on all outcomes (all P<.003), except for uptake (P=.20), risk perception when not vaccinated (P=.14), subjective norms (P=.03), and habit (P=.01).
Conclusions: Program use and acceptability of the intervention were adequate. Completeness was positively associated with acceptability. Furthermore, positive effects (ie, dose-response effects) were found for completeness and time of website use on the mothers’ IDM, decisional conflict, and almost all of the social-psychological determinants of HPV vaccination acceptability. In addition, the extent to which mothers completed the intervention had a positive impact on their daughters’ vaccination uptake. This indicates that the web-based, tailored intervention fits well with the mothers’ needs, and that completeness of use is essential for improving HPV vaccination uptake, acceptability, and IDM. Program use should therefore be promoted.
Trial Registration: Netherlands Trial Register NTR4795; https://www.trialregister.nl/trial/4795
Cervical cancer is the fourth most common cancer among women worldwide. Globally, in 2018, there were 569,847 new cases and 311,365 deaths caused by cervical cancer . Persistent infection by human papillomavirus (HPV) appears to be the major cause of cervical cancer; nearly all cases (over 99%) are attributable to an HPV infection [ , ]. There is also an increasing body of evidence that strongly links HPV infection with cancers of the anus, vulva, vagina, penis, and head and neck (eg, [ , ]). In the Netherlands, on an annual basis since 2010, girls are invited by the Dutch National Immunization Program to receive the vaccine against HPV upon reaching 13 years of age. However, HPV vaccination uptake remains low (45.5% in 2017) [ ]. The uptake needs to be higher to reduce the incidence of cervical cancer in women. Therefore, we developed an interactive, web-based, tailored intervention promoting HPV vaccination acceptability among Dutch mothers of invited girls for HPV vaccination [ ]. Further, to improve vaccine acceptability, important intervention objectives were to improve informed decision making (IDM) and to decrease decisional conflict [ ].
To date, only four studies have examined the effectiveness of tailored interventions in promoting HPV vaccination acceptance [- ]; three of these studies showed positive effects [ - ]. To our knowledge, the only tailored intervention that was web-based turned out to be ineffective [ ]. The advantage of web-based interventions is that they have the potential to enable large-scale application at relatively low cost [ ] and therefore can have a substantial impact at the population level [ ].
Furthermore, only two of the existing studies using tailored interventions to promote HPV vaccination acceptance incorporated a process evaluation [, ]. Process evaluation provides insight into the extent to which an intervention was used, which is relevant for determining factors that influence the effectiveness of an intervention [ - ]. Specifically, this analysis can provide insight into whether an intervention is ineffective due to implementation failure or to the inferior quality of the intervention method(s), thereby avoiding the so-called type III error [ ]. It also provides recommendations for intervention improvement.
For the use of online interventions, three aspects are relevant: a first visit, staying, and revisiting [, ]. In the context of decision making about HPV vaccination, we consider a first visit and staying as the most relevant aspects. Revisiting seems less relevant, since getting the HPV vaccine is hardly a repetitive behavior (ie, it consists of two injections). Within the electronic health (eHealth) arena, limited intervention use in terms of a first visit and staying is a broadly recognized problem [ ]. The extent to which one uses an intervention may positively impact the intervention effects (ie, the so-called “dose-response relation”). Positive dose-response relationships have been found for eHealth interventions (eg, [ - ]). In the context of promoting HPV vaccination acceptability, none of the studies that incorporated a process evaluation addressed the dose-response relationship (for a review, see [ ]). Moreover, data on the use of web-based interventions have been poorly reported [ , ] or not reported at all [ ].
Furthermore, assessing program acceptability may shed light on the causes of nonuse or incomplete use . An intervention needs to be considered appropriate by the target group and to suit their needs in order to reach the intended outcomes [ ]. Acceptability will improve interaction and the users’ feelings of engagement, and, as a consequence, the likelihood of health-related behavior changes [ ].
Results from the effect evaluation showed positive intervention effects on IDM, decisional conflict, and nearly all determinants of HPV vaccination uptake (for more details on the results of the effect evaluation, see ). This paper describes the process evaluation of the web-based tailored intervention promoting HPV vaccination acceptance [ ]. The aim of this process evaluation was to provide insights into the intervention’s working mechanisms by evaluating (1) program use, (2) program acceptability, and (3) the relationship of program use with program acceptability and intervention effects (ie, dose-response relationships).
Participants and Study Design
The study was approved by the Ethical Committee of the VU Medical Center in Amsterdam (Netherlands Trial Register: NTR4795). This process evaluation was part of a randomized controlled trial on the effectiveness of the intervention . For the purpose of this study, only participants from the experimental condition (ie, those invited to use the web-based intervention; N=3995) were included in the analyses. Details of the trial design, such as power calculations, participant eligibility, and recruitment procedures, have been published elsewhere [ ]. This study was conducted between January 2015 (baseline, just before access to the intervention) and March 2015 (follow up, prior to the first HPV vaccination), in line with the national HPV vaccination round of 2015. Mothers were invited to use the web-based intervention between baseline and follow up (mid-January 2015).
We developed the web-based tailored intervention with virtual assistants using the 6-step Intervention Mapping protocol for developing theory- and evidence-based health promotion interventions . In short, the web-based intervention provided mothers with tailored feedback about topics on HPV vaccination, delivered by two virtual assistants. presents a selection of screenshots of the website. The website consisted of 4 menu options: (1) 2-sided, tailored information about HPV vaccination, (2) a decisional balance in which mothers could weigh their perceived pros and cons, (3) practical information, and (4) frequently asked questions. Within menus 1-3, mothers could visit several components. In addition, mothers were given the opportunity to visit an “in-depth” information page within some of the components (eg, educational movies). provides a brief overview of the intervention components. More detailed descriptions of the intervention, including its theoretical basis, are published elsewhere [ ].
|Menu and component||Description|
|Information about HPVa vaccination|
|General information||Mothers are provided with general information about HPV, cervical cancer, and HPV vaccination.|
|Importance of vaccinating at a young age||Mothers are challenged to consider whether the age of their daughter is appropriate to get vaccinated against HPV. The relationship between sexual activity in relation to the HPV vaccine is also discussed.|
|HPV-related risks||Mothers are asked to estimate both the risk of their daughter getting infected with HPV and the risk of their daughter developing cervical cancer later in life, and are then provided with tailored feedback accordingly.|
|Methods to protect against cervical cancer||Mothers are asked to rate the effectiveness of alternative methods for protecting against cervical cancer and are then provided with tailored feedback according to their answers.|
|From HPV to cervical cancer||Mothers are provided with an explanation of how infection with HPV can lead to cervical cancer (eg, by viewing an educational video).|
|Facts and stories||Mothers are provided with several statements regarding HPV, cervical cancer, and HPV vaccination, and are asked to indicate whether these are true (a fact) or false (a story). They then receive tailored feedback accordingly.|
|Side effects||Mothers are presented with a variety of potential side effects of the HPV vaccine and are asked to indicate whether or not they are scientifically proven. They are then provided with tailored feedback, stating the correct responses.|
|Effectiveness||Mothers are asked about the effectiveness of the HPV vaccine in protecting both against their daughter getting infected with HPV and developing cervical cancer. They are then provided with tailored feedback about the effectiveness of HPV vaccination.|
|Other mothers||Mothers are asked to indicate what they think most mothers in their direct environment will decide regarding their daughters’ HPV vaccination. They are then provided with tailored feedback and are shown the actual HPV vaccination uptake in different regions of the Netherlands in 2014.|
|Vaccine working mechanisms||Mothers are explained in a generic way how the HPV vaccine works in protecting against HPV and cervical cancer with an educational video.|
|Weighing the pros and cons|
|Decisional balance||Mothers are presented with a list of pros and cons of HPV vaccination. Based on pros and cons mothers marked as most salient, a decisional balance reveals their current position on a scale ranging between wanting and not wanting to get their daughter vaccinated.|
|Value clarification||Mothers are invited to list their central values for life and link these to HPV vaccination.|
|Practical information||Practical information provides mothers with information such as how and where to receive the HPV vaccine and provides them with advice on how they discuss HPV vaccination with personal important others (eg, their daughter and partner).|
|Frequently asked questions|
|About HPV vaccination||Frequently asked questions provides answers to known questions about the HPV vaccine (eg, “does my daughter know if she’s infected with HPV?”) or getting the HPV vaccine (eg, “where do I get the HPV vaccine?”).|
|Problems with the website||Mothers are provided with possible solutions to problems with the website such as not being able to hear or see the virtual assistants.|
aHPV: human papillomavirus.
All data were derived from web-based self-report questionnaires and computer logs.
Program use was measured according to time of website use and completeness. Both factors were constructed based on computer logs that registered the pages and components the mothers had visited and the amount of time they spent on the website during their visit(s).
Time of website use represents the total amount of time spent on the website. Each time mothers logged on to the website, a record was created, in which both the date and time of the first and last page the mothers entered were registered. Time per session was calculated by subtracting the time of the last entered page from the time of the earliest entered page. The total time of website use was then calculated by adding up the amount of time spent in each session.
Completeness represents the proportion of all available web pages visited by the mother while she was logged on to the website, ranging from 0% (visited no pages) to 100% (visited all pages).
Program acceptability was assessed at the 6-week follow up by asking mothers to rate the intervention and the virtual assistants on a 10-point scale, ranging from 0 (very bad) to 10 (excellent). The grade for the website was used as an indicator for overall program acceptability to examine the association with program use. In addition, mothers evaluated the information provided by the website (eg, credibility, relevance), perceived user control (eg, perceived degree of autonomy), and how they perceived the virtual assistants (eg, reliability) (Seeand for an overview). These acceptability measures were assessed on a 7-point scale, and some were averaged into one scale. All composite scores showed sufficient internal consistency (Cronbach α ≥.71 for scales with >2 items; Pearson r≥0.84 for scales with 2 items).
|Measures||Items||Score and scale||Cronbach α or Pearson r||Reference|
|Rate||On average, how would you rate the website on a scale from 0 to 10?||0=very bad to|
|Interest||In general, what did you think of the website?||1=very uninteresting to|
7=very interesting; 1=boring to 7=engaging
|Informative||In general, what did you think of the website?||1=very uninformative to 7=very informative; 1=very noneducational to 7 =very educational||0.86 (r)|||
|Perceived user control||I felt that I had a lot of control over my visiting experiences at this website.|
While I was on the website, I could choose freely what I wanted to see.
While surfing the website, I felt in control.
While surfing the website, my actions decided the kind of experiences I got.
|1=strongly disagree to 7=strongly agree||.71 (α)|||
|Elaboration||How well did you read the information?||1=not carefully at all to 7=very carefully||N/A|||
|Support||The website has helped me decide about my daughter’s HPVb vaccination.||1=strongly disagree to 7=strongly agree||N/A|||
|Recall||I can recall the information from the website.||1=strongly disagree to 7=strongly agree||N/A|||
|Personal relevance||I considered the website to be personally relevant.||1=strongly disagree to 7=strongly agree||N/A||[, ]|
|Tailoring||I considered the information on the website to be…||1=not tailored to me at all to 7=very tailored to me||N/A|||
|Comprehensibility||I considered the information on the website to be…||1=not at all understandable to 7=very understandable||N/A|||
|Reliability||I considered the information on the website to be…||1=very unreliable to 7=very reliable||N/A|||
|Credibility||I considered the information on the website to be…||1=very incredible to 7=very credible||N/A|||
|Usefulness||I considered the information on the website to be…||1=very useless to 7=very useful||N/A|||
|Readability||I considered the information on the website to be…||1=very unreadable to 7=very readable||N/A|||
|Sidedness||I considered the information on the website to be…||1=focused on the cons to 7=focused on the pros||N/A|||
|Enjoyment||I considered the information on the website to be…||1=very unenjoyable to 7=very enjoyable||N/A|||
|Novelty||The website contained new information for me.||1=strongly disagree to 7=strongly agree||N/A|||
|Attitude toward website||How good or bad did you find…|
- the possibility to first answer a question and then receive information?
- the possibility to weigh the pros and cons?
- the speed of the website?
- the layout of the website?
|1=very bad to 7=very good||.81 (α)|||
aN/A: not applicable.
bHPV: human papillomavirus.
|Measures||Items||Score and scale||Pearson r||Reference|
|Overall rating||On a scale from 0 to 10, how would you rate:|
- the mother-like assistant
- the doctor-like assistant
|0=very bad to 10=excellent||0.94|||
|Enjoyment||I considered the virtual assistants to be…||1=very unenjoyable to 7=very enjoyable||N/Aa|||
|Reliability||I considered the virtual assistants to be…||1=very unreliable to 7=very reliable||N/A|||
|Credibility||I considered the virtual assistants to be…||1=very incredible to 7=very credible||N/A|||
|Usefulness||I considered the virtual assistants to be…||1=very useless to 7=very useful||N/A|||
aN/A: not applicable.
Primary Outcome: HPV Vaccination Uptake
An objective measure for HPV vaccination uptake was derived from Praeventis, which is the national electronic vaccination register that monitors the vaccination status for all children up to 18 years of age living in the Netherlands . Uptake was dichotomized into having received no HPV injection (0=not vaccinated) and having received 1 or 2 HPV injections (1=vaccinated), as data analyses showed that determinants of HPV vaccination uptake contrasted the most between these groups.
Secondary Outcomes: IDM, Decisional Conflict, and Social-Psychological Determinants of HPV Vaccination Uptake
provides the measurement details of the constructs IDM, decisional conflict, and the social-psychological determinants of HPV vaccination uptake [ , - ]. Determinants accounted for were intention, attitude, risk perception of having received the HPV vaccine, risk perception without having received the HPV vaccine, anticipated regret about rejecting the HPV vaccine, beliefs, subjective norms, habit strength toward HPV vaccination, self-efficacy, knowledge, and perceived relative effectiveness of HPV vaccination compared to alternative methods.
Sociodemographics were included as background variables (ie, age, educational level, country of birth, and religion).
Level of education referred to the mothers’ highest completed level of education. It was classified as low (less than secondary or vocational education), intermediate (secondary through preuniversity education), or high (professional or university education) [, , ].
Country of birth was dichotomized into “Netherlands” vs “other,” because the percentage of mothers born in the latter category (276/3995, 6.91%) was too small for further subdivision in our sample.
Religion was measured by asking mothers about their religious convictions (Protestant, Roman Catholic, Muslim, Jewish, Buddhist, Hindu, other, or no religion). This was dichotomized into “Protestant” vs “not Protestant” as earlier research and data analyses showed that Protestants more frequently refrain from vaccination compared to other religious groups [, , ].
The sample and intervention use data are summarized using descriptive statistics. For determining the dose-response relation, intention-to-treat (ITT) analyses were used, which increases power, while decreasing the risk of possible bias caused by selective dropout . Missing data were imputed for uptake, determinants, and sociodemographics (not for use): we applied multiple imputation by chained equations [ , ]. In total, 15 imputed datasets were generated using the predictive mean matching algorithm in Statistical Package for the Social Sciences (SPSS, IBM Corp). Results from these datasets were pooled together using Rubin’s rules [ ]. Iteration plots were inspected to check convergence of the imputations.
The relationship between acceptability and use was examined using univariate regression analyses. The dose-response relation between program use and intervention effects was examined by logistic and linear regression analyses (for dichotomous and continuous variables, respectively) using the outcome score at follow up as the dependent variable and the outcome scores at baseline and program use (both completeness and time of website use) as the independent variables . Effects were considered significant when the P value was lower than .003 (Bonferroni-corrected α=.05/15 factors). For the logistic regressions, the odds ratio (OR) was used as an index of effect sizes. These were interpreted as small (OR=1.5), medium (OR=3.5), or large (OR=9.0) [ ]. For the linear regressions, effect sizes were calculated in R software (R Development Core Team, Vienna, Austria) using Cohen ƒ2 statistic, (R2AB – R2A)/(1 – R2AB), in which B is the variable of interest (ie, use), A is the set of all other variables (ie, the outcome at baseline), R2AB is the proportion of variance accounting for A and B together, and R2A is the proportion of variance accounted for by A [ ]. These were classified as small (ƒ2=0.02), medium (ƒ2=0.15), and large (ƒ2=0.35) [ ]. We performed complete case analyses as a sensitivity check for substantial differences with the results from ITT.
Response Rates and Attrition
A flow diagram of the recruitment and response of participants in the experimental condition, as part of the process evaluation, is shown in. From the 4555 participants who were randomized into the experimental condition at baseline (T0), 4277 (93.90%) completed the baseline questionnaire, 2511 (55.13%) visited the intervention, and 2197 (48.23%) completed the follow-up questionnaire 8 weeks later (T1). Dropout analysis showed that there was significantly more dropout with regard to participants not born in the Netherlands, those with a lower educational level, those having a daughter not being vaccinated, and those with low levels of risk perception. In total, 564 participants were excluded as they (a) did not meet the inclusion criteria (ie, being a mother of a daughter born in 2002 and aged 24-62 years), (b) were found to be duplicates, or (c) encountered a language barrier. The final sample for ITT analysis consisted of 3995 mothers.
shows the sample description of participants in the experimental condition. No data were available on sociodemographics of the population from which the sample was derived (ie, Dutch mothers of 12-year-old girls); hence, we were unable to assess whether the study sample was representative. The mothers’ mean age was 43.70 years (range 27.00-62.00 years). On average, mothers had a positive intention toward their daughters’ HPV vaccination at baseline (mean score 5.35, SD 1.69). Compared with the national HPV vaccination uptake at the time (ie, 2015), uptake was higher in the study sample (59,866/93,173; 60.98% vs 2923/3995; 73.17%) [ ].
|Variables||N (% missing)||Valuea|
|Age in years, mean (SD)||3995 (0)||43.70 (4.27)|
|Country of birth, n (%)||3991 (0.10)|
|The Netherlands||3715 (92.99)|
|Religion, n (%)||3998 (0.18)|
|Not Protestant||3235 (80.98)|
|Educational level, n (%)||3991 (0.10)|
|HPVb vaccination uptake, n (%)||3986 (0.23)|
aBy reporting 2 decimal points for the percentages, summing the percentages for each category may differ from 100%.
bHPV: human papillomavirus.
From the 3995 mothers in the intervention group, a total of 2509 mothers (62.80%) logged in (ie, time of website use >0). Of these, 73.06% (1833/2509) logged in once, 19.89% (499/2509) logged in twice, 5.00% (125/2509) logged in three times, and 2.07% (52/2509) logged in four times or more. On average, mothers spent 21.39 minutes (SD 12.41) on the website.
Of the 2509 mothers that had initially logged in, 2239 (89.24%) visited at least one page of a component of the intervention (ie, completeness >0). Only these 2239 mothers were included for describing use of specific intervention components and program acceptability (see below). On average, the mothers completed 50.04% (SD 26.18%) of the website components (ie, 51/101 pages).
Use of Intervention Components
shows the breakdown for the use of intervention components (N=2239). When mothers clicked on one or more links to receive in-depth information, it was marked as “visited.” Mothers visited the page “Ways to Protect Against Cervical Cancer” the most frequently (1971/2239, 88.03%), followed by “Chance” (1945/2239, 86.87%), whereas they visited “Value Clarification” the least (293/2239, 12.53%). “General Information” had the highest completion rate (1622/2239, 72.44%), whereas “Side Effects” and “Effectivity” were completed the least (both 19/2239, 0.85%). Only a small percentage of the mothers (4.20%-13.22%; n=94-298) visited in-depth information (comprising educational movies or extra information).
|Intervention component||Completeda, n (%)b||Partly completedc, n (%)||Not visitedd, n (%)||Visited in-depth informatione, n (%)|
|General Information||1622 (72.44)||71 (3.17)||546 (24.39)||N/Af|
|Ways to Protect against Cervical Cancer||130 (5.81)||1841 (82.22)||268 (11.97)||N/A|
|Chance||135 (6.03)||1810 (80.84)||294 (13.13)||142 (6.34)|
|From HPVg to Cervical Cancer||96 (4.29)||1110 (49.58)||1033 (46.14)||298 (13.22)|
|Age||154 (6.88)||1568 (70.03)||517 (23.09)||160 (7.15)|
|Side Effects||19 (0.85)||1686 (75.30)||534 (23.85)||299 (13.35)|
|Effectivity||19 (0.85)||1542 (68.87)||678 (30.28)||339 (15.14)|
|Other Mothers||1099 (49.08)||416 (18.58)||724 (32.34)||N/A|
|Working Mechanisms Vaccination||94 (4.20)||1015 (45.33)||1130 (50.47)||94 (4.20)|
|Facts and Stories||95 (4.24)||1220 (54.49)||924 (41.27)||99 (4.42)|
|Weighing Pros and Cons||615 (27.47)||911 (40.69)||713 (31.84)||N/A|
|Value Clarification||269 (12.01)||24 (1.07)||1946 (86.91)||N/A|
|Practical Information||556 (24.83)||1147 (51.23)||536 (23.94)||N/A|
aA component was considered “completed” when mothers visited every page of the component.
bBy reporting 2 decimal points for the percentages, summing the percentages for each category may differ from 100%.
c“Partly completed” indicates that the mothers had seen at least one, but not all pages.
d“Not visited” means that the mothers had not visited any of the component’s pages.
e“Visited in-depth information” was also part of the total completeness, but is depicted separately here to provide a better overall view of the mothers’ interest in this information. When one or more of the links to more in-depth information had been clicked on, the in-depth information was marked as “visited.”
fN/A: not applicable as the component did not contain in-depth information.
gHPV: human papillomavirus.
The overall acceptability of the intervention was rated 7.64 (SD 1.39) on a 10-point scale (1513/2239, 67.57%). Acceptability was significantly and positively associated with completeness (β=4.36, P<.001); this did not account for time of website use (β=–.07, P=.77). The virtual assistants were rated 7.41 (SD 1.57) on a 10-point scale (1513/2239, 67.57%). The mean scores on all other program acceptability measures were moderate to high (range 4.34-5.71 on a 7-point Likert scale; see). As the data were not normally distributed but were positively skewed, the median scores are also included in .
|Variable||Website (nmissing=726-734)||Virtual assistants (nmissing=960)|
|Mean (SD)||Median||Mean (SD)||Median|
|Rate (1-10)||7.64 (1.39)||8.00||7.41 (1.57)||8.00|
|Interest (1-7)||5.41 (1.05)||5.50||N/Ab||N/A|
|Informative (1-7)||5.67 (1.04)||6.00||N/A||N/A|
|Attitude toward website (1-7)||5.53 (.89)||6.00||N/A||N/A|
|Perceived user control (1-7)||5.26 (.97)||5.50||N/A||N/A|
|Elaboration (1-7)||5.70 (1.14)||6.00||N/A||N/A|
|Tailoring (1-7)||5.22 (1.21)||5.22||N/A||N/A|
|Comprehensibility (1-7)||5.40 (1.70)||6.00||N/A||N/A|
|Reliability (1-7)||5.27 (1.23)||6.00||5.04 (1.09)||5.00|
|Credibility (1-7)||5.35 (1.21)||6.00||5.01 (1.22)||5.00|
|Usefulness (1-7)||5.52 (1.10)||6.00||4.83 (1.33)||5.00|
|Readability (1-7)||5.71 (1.04)||6.00||N/A||N/A|
|Sidedness (1-7)||4.74 (1.09)||4.00||N/A||N/A|
|Enjoyment (1-7)||4.70 (1.08)||5.00||4.72 (1.37)||5.00|
|Novelty (1-7)||5.11 (1.39)||5.00||N/A||N/A|
|Support (1-7)||4.34 (1.80)||5.00||N/A||N/A|
|Recall (1-7)||5.16 (1.22)||5.00||N/A||N/A|
|Personal relevance (1-7)||4.87 (1.41)||5.00||N/A||N/A|
aA higher score represents a higher score on the program acceptability measure.
bN/A: not applicable.
Dose-Response Relation Between Program Use and Intervention Outcomes
provides an overview of the effects of program use (completeness and time of website use) on the primary and secondary outcomes according to the ITT analyses. Significant positive effects were found for completeness on all outcome measures (all P<.003). The more mothers had completed the intervention, the more likely they were to have their daughter vaccinated against HPV and make an informed decision, and were less likely to experience decisional conflict at follow up. In addition, positive effects of completeness were found with respect to all social-psychological determinants of HPV vaccination acceptability (eg, HPV vaccination intention, beliefs; see ).
Time of website use had a positive effect on all outcomes (all P<.003), except for HPV vaccination uptake, risk perception when not vaccinated, subjective norms, and habit (). The more time mothers spent on the intervention, the more likely they were to make an informed decision, experience less decisional conflict, have a higher intention to vaccinate their daughter, have a more positive attitude, have more positive beliefs, have a lower risk perception of HPV vaccination, anticipate more feelings of regret about rejecting the HPV vaccine, report a higher relative effectiveness of the HPV vaccine, have higher self-efficacy expectations, and have more knowledge at follow up.
Effect sizes were small overall (see). Results from the complete case analyses were similar for both completeness and time of website use, except for a lack of effect of time of website use on intention.
|Outcomea||Pretest, mean (SD)||Posttest, mean (SD)||Completeness||Time of website use|
|β (SEM)||Cohen ƒ2 or ORb||β (SEM)||Cohen ƒ2 or OR|
|Primary outcome: HPVc vaccination uptaked|
|Has received no HPV injection (reference)||N/Ae||26.67% (1066)||N/A||N/A||N/A||N/A|
|Has received one or two HPV injections||N/A||73.32% (2929)||.004 (.001)f||1.004||.003 (.003)g||1.003|
|IDMh: Not informed (reference)||67.31% (2689)||42.53% (1699)||N/A||N/A||N/A||N/A|
|IDM: Informed||32.69% (1306)||57.47% (2296)||.014 (.001)f||1.014||.021 (.003)f||1.021|
|IDM: continuous (0-48)||18.69 (11.21)||25.85 (12.30)||.087 (.006)f||0.075||.122 (.013)f||.027|
|Decisional conflict (1-7)||4.33 (1.75)||5.38 (1.36)||.007 (.001)f||0.043||.011 (.002)f||.018|
|Intention (1-7)||5.35 (1.69)||5.59 (1.87)||.004 (.001)f||0.014||.005 (.002)f||.004|
|Attitude (1- 7)||5.18 (1.45)||5.37 (1.51)||.005 (.001)f||0.027||.007 (.001)f||.010|
|Beliefs (1-7)||4.19 (.73)||4.47 (.81)||.004 (.000)f||0.051||.005 (.001)f||.016|
|Risk perception; not vaccinated (1-7)||3.74 (0.98)||3.77 (1.08)||.002 (.001)f||0.003||.002 (.001)i||.001|
|Risk perception; vaccinated (1-7)||2.77 (1.07)||2.64 (1.10)||-.003 (.001)f||0.008||–.005 (.001)f||.005|
|Anticipated regret (1-5)||3.71 (1.25)||3.59 (1.31)||.002 (.001)f||0.007||.004 (.001)f||.004|
|Subjective Norm (–20-20)||5.88 (7.81)||7.25 (9.20)||.020 (.004)f||0.009||.018 (.008)j||.002|
|Habit (1-7)||4.28 (1.78)||4.51 (1.83)||.004 (.001)f||0.011||.004 (.002)k||.003|
|Relative effectiveness (1-10)||-1.97 (2.22)||-1.35 (2.27)||.013 (.001)f||0.051||.016 (.003)f||.015|
|Self-efficacy (1-7)||6.27 (.73)||6.29 (.75)||.003 (.000)f||0.022||.004 (.001)f||.007|
|Knowledge (–8-8)||4.40 (2.14)||5.75 (2.09)||.017 (.001)f||0.095||.024 (.002)f||.034|
aA higher score means a higher outcome (eg, more positive attitude) except for decisional conflict in which a higher score means less decisional conflict.
bOR: odds ratio; OR>1 means the higher the score on a factor, the higher the outcome of IDM and higher chance of the daughter being vaccinated; OR<1 means the higher the score on a factor, the lower outcome of IDM and lower chance of the daughter being vaccinated.
cHPV: human papillomavirus.
dHPV vaccination uptake was not assessed at baseline.
eN/A: not applicable.
fP<.003 (significant; Bonferroni: 0.05/15 factors).
hIDM: informed decision making.
The aim of this process evaluation was to examine (1) program use, (2) program acceptability, and (3) the relationship between program use and acceptability and intervention outcomes for the web-based tailored intervention about HPV vaccination.
Almost two thirds (62.80%) of the mothers who were invited to visit the intervention logged in to the intervention. This reach is adequate, and is comparable to that reported for other eHealth interventions (eg, ). However, it still leaves room for improvement to increase the reach by employing strategies such as arousing interest in this eHealth intervention. Support for this notion comes from Crutzen et al [ ], who showed that arousing interest successfully increased the intention to visit a website about hepatitis A, B, and C virus and the likelihood of clicking on the link to visit the website. Interest was aroused in the invitation by challenging the potential visitor regarding their knowledge about hepatitis and it was emphasized that the website provided this information in a comprehensible manner. Not only could interest be aroused in the invitation for HPV vaccination for girls and their parents but also, for instance, via other channels by which the target group is reached (eg, the internet, social media). In addition, research has shown a positive recommendation by word of mouth to be an important trigger for visiting a web-based intervention for the first time [ ]. Word-of-mouth recommendations could be encouraged by providing “tell-a-friend” services at the web-based intervention [ ].
Of the mothers that logged in to the intervention, nearly all visited at least one component. However, a small portion of the mothers (10.76%) did not view any of the intervention’s content after having logged in. This is likely due to technical difficulties because at follow up, some mothers (320, 12.75%) indicated that they were not able to see or hear the virtual assistants.
On average, time of website use, indicating the time mothers spent with the intervention, was adequate (21.39 minutes). This is longer than that reported by Brouwer and colleagues  in their review of web-based, interactive healthy lifestyle interventions, in which intervention exposure time varied from less than 10 minutes to 10-20 minutes. However, the present exposure time was lower than found by Sanders and colleagues [ ] for their interactive, web-based tailored intervention promoting colorectal cancer screening (33 minutes). This difference could be explained by the controlled setting of their study [ ], in which participants arrived 60 minutes prior to a medical appointment and viewed the intervention in the waiting room in the presence of a research assistant.
In addition, the mean completeness of the intervention was 50.04%. This indicates that the intervention fits well with the mothers’ needs, which is also supported by the results regarding acceptability (see below). The mean completeness in this study is comparable to that found by Watts and colleagues , who reported a completeness of 49.09% for a web-based prostate cancer screening decision aid.
Use of Intervention Components
When looking at exposure to intervention components, we found that mothers were most interested in the effectiveness of (alternative) methods to protect against cervical cancer (88.03% visited) and the risks of their daughter getting infected with HPV and developing cervical cancer (86.87% visited). These appear to be essential components of communication about HPV vaccination.
The “Value Clarification” component was visited the least overall (13.09%). This may be attributed to the fact that it was not easily found by participants: this page was only accessible once mothers completed the component “Weighing Pros and Cons.” Voncken-Brewster and colleagues  also found that participants overlooked a certain part of their intervention, and they successfully improved its visibility by repositioning it. In order to promote exposure to values clarification, we could enhance its visibility.
The “General Information” component was completed the most overall (72.24%). This is likely due to the fact that the mother-like virtual assistant recommended starting with this component when mothers first entered the main menu, and that it was brief (2 pages). “Side Effects” and “Effectivity” were completed the least (both 0.85%). This could be explained by the fact that these two components contained the most links to in-depth information. Only a small proportion of the mothers (≤13.22%) visited in-depth information.
The mothers evaluated the website as positive, as shown by the overall acceptability (7.64/10) and the scores on all acceptability measures (mean scores ranging from 4.34 to 5.71 on a 7-point Likert scale). The virtual assistants were also well appreciated (7.41/10). This is likely because it matched with the mothers’ preferences for more interactive personalized feedback . This adequate acceptability is similar to the findings of Paiva and colleagues [ ] for a web-based tailored intervention to increase HPV vaccination among young adult women (3.27/4.0). We believe the high acceptability in the present study can be attributed to the systematic and user-centered development process of the intervention [ ]. The target group was extensively involved throughout the development. Not only did we fine-tune the content of the intervention to the mothers’ preferences and requirements but this also was considered in the design of the website, which was chosen by the mothers.
Furthermore, we found a positive relationship between acceptability and completeness (P<.001). This association was also found by others in a web-based intervention for breast cancer [- ]. This finding underlines the importance of an intervention to be considered appropriate by the target group.
Effects of Program Use on Intervention Outcomes
In line with our expectations, completeness had a significant positive effect on all outcome measures (IDM, decisional conflict, social-psychological determinants of HPV vaccination uptake), including actual HPV vaccination uptake itself. Mothers who had completed more of the intervention were more likely to have their daughter vaccinated against HPV, and had higher levels of IDM and more positive scores on determinants of HPV vaccination acceptance. In particular, the effect on HPV vaccination uptake is an important finding, which was not found when simply contrasting the experimental to the control condition . This stresses the importance of conducting a process analysis alongside such an effect evaluation. These results are very promising, considering the currently low HPV vaccination uptake rates in the Netherlands [ ].
We believe that these positive effects can be attributed to the extensive tailoring throughout the intervention. Support for this notion comes from the study results indicating that mothers perceived the intervention to be well-tailored (5.22 on a 7-point scale). Not only did we tailor the content of the intervention to the mothers’ personal interest, it was also used to guide the mothers’ personal route through the intervention. The latter is likely to have improved the usability of the intervention. Moreover, the intervention accounted for tailoring on a variety of themes. For example, it considered perceived barriers similar to the approach taken by Gerend and colleagues  as well as other beliefs, attitudes, subjective norms, habits, relative effectiveness, anticipated regret, risk perception, self-efficacy, and knowledge.
In contrast to our expectations, time of website use had a positive effect on all outcomes (P<.003), except for risk perception when not vaccinated (P=.14), subjective norms (P=.03), habit (P=.01), and vaccination uptake (P=.20). The latter is also in contrast to findings of a previous study . An explanation for this lack of effects may be the measurement we used for time of website use, which may be less reliable for measuring exposure. We measured the total time spent on the website, but this does not indicate the specific pages the mothers had visited and for how long. In addition, we were unable to determine if there were any timeouts during a session and what the duration of a timeout was, because the intervention was web-based. For instance, they could have been distracted in the home environment during website use, which may have influenced the measured time of website use and may also have caused the lack of effect of acceptability on time of website use. An alternative, more accurate, measure can be found within the domain of education, namely time on task [ ]. Future studies are needed to examine whether time on task (instead of total time of website use) has a positive influence on uptake.
There are three methodological considerations. First, as mentioned above, the measurement of total time spent on the website seemed to be not entirely adequate to measure program use. In this study, completeness seemed to be a better indicator of use as it demonstrates the mother’s navigation through the website, whereas time of website use did not. In addition, within (almost) every component, mothers were first asked a question about an HPV-related topic. The page was marked as “completed” if they provided an answer, which was necessary to obtain tailored feedback. Within the decisional balance, completeness was calculated based on the mother’s answer to each statement. These are more complete indicators of actual use. Nevertheless, it is not an entirely complete indicator for use, since we could not measure whether they read generic information, listened to the tailored feedback from the virtual assistant, or saw a video. This may have influenced the dose-response effects found.
Second, considering the aim of initiating active processing of information, the positive effect found of completeness on habit may seem unwanted. However, at follow up, the mothers had already been exposed to the intervention, and were therefore likely to have actively processed the information. These results may indicate that the more mothers completed the intervention, the less they had to think about getting the HPV vaccine. The intervention may have helped them make a decision rather than inducing perceptions about HPV vaccination as something you take for granted, without thinking. In retrospect, the label “habit” appears to be misleading.
Third, the effect sizes found were small, which is in line with other web-based interventions targeting health behavior outcomes . Despite this, we believe that even small effects are of relevance in public health as they become substantial at the population level. After all, in the Netherlands, approximately 100,000 girls are invited to receive the HPV vaccine on a yearly basis.
Strengths and Limitations
The most important strength of this study is that we, to our knowledge, are the first within the field of HPV vaccination to conduct an extensive process evaluation. According to a review , the process evaluation of an intervention promoting HPV vaccination acceptance by tailoring information to participants’ perceived barriers, such as that performed by Gerend and colleagues [ ], was limited to the evaluation of intervention information (eg, the extent to which the information was informative and convincing). Other notable strengths are the objective measurement of HPV vaccination uptake, the dose-response effect of completeness on actual HPV vaccination uptake, the broad focus on outcomes (ie, determinants of HPV vaccination acceptance, uptake, IDM, and decisional conflict), and the adequate level of both (objectively assessed) program use and acceptability.
However, some limitations need to be addressed. First, there appeared to be technical difficulties despite our best effort to minimize such issues (eg, by extensively pilot testing the website using various devices). For instance, it turned out that a portion of the mothers were unable to see or hear the virtual assistants and that certain components did not work adequately among some of the mothers. Such technical difficulties are likely to have influenced website usability and therefore may have had a negative impact on program use. Second, this eHealth intervention requires mothers to have eHealth literacy . People with lower eHealth literacy have been shown to have a lower educational level and to spend less time online [ , ]. It could be interesting to examine the extent to which eHealth literacy might have influenced use, acceptability, and effects. Considering the latter, we do not expect substantial differences since we did not find any differences in intervention effects among mothers with a low vs high educational level [ ]. Third, it was hard to compare the study results to other studies because data on program use are often very poorly reported or even completely lacking (for a review, see [ ]). Furthermore, the study was subject to a considerable amount of attrition. This is a common finding in studies on eHealth interventions [ ]. Dropout appeared higher, for example, among mothers with a lower educational level, which has also been reported for other web-based tailored interventions [ ]. In this study, missing data and selective dropout were handled by using multiple imputation [ , ]. Fortunately, there were nearly no differences between the results from the complete case analyses and the ITT analyses. Therefore, it seems unlikely that the effects found are spurious or due to selective dropout. Finally, caution is needed when generalizing the results of this study to the general population (ie, Dutch mothers of 12-year-old girls) because we were unable to check the representativeness of the sample. Besides, it should be noted that a very homogenous sample was reached. An undiversified reach is a known phenomenon within web-based interventions aimed at health promotion [ ]. In our study, the mothers were predominantly born in the Netherlands, highly educated, and likely to have a daughter that was vaccinated against HPV. However, results from the effect evaluation showed no differences in intervention effects in specific subgroups of participants, as indicated by moderation analyses with sociodemographic variables [ ].
Conclusions and Recommendations
This process evaluation has demonstrated that program use and acceptability of the web-based tailored HPV vaccination intervention were adequate. We found a positive association between completeness and acceptability. Furthermore, dose-response effects were found for completeness and time of website use on (nearly) all social-psychological determinants of the mothers’ decision making about vaccination, levels of IDM, and decisional conflict. Importantly, the extent to which mothers completed the intervention positively influenced their daughters’ actual HPV vaccination uptake. These results indicate that this web-based, tailored intervention fits well with the mothers’ needs and has the potential to increase HPV vaccination uptake. Because of the intervention’s adequate (dose-response) effects and acceptability, the intervention has recently been incorporated into the national HPV vaccination communication alongside the existing communication materials.
We recommend future interventions promoting HPV vaccination acceptance to (1) incorporate a process evaluation alongside the effect evaluation, (2), include strategies to arouse interest so as to expand reach, and to (3) include time on task. In addition, we recommend investigating the influence of different media types (eg, graphic vs nongraphic presentations) on decision making about HPV vaccination. For instance, Cox and colleagues  found that parents who viewed a graphic presentation of HPV-related risk information had a higher HPV vaccination intention compared to parents who viewed a nongraphic presentation. Furthermore, we recommend future research to examine differences in use and acceptability in specific subgroups of participants, for example by conducting moderation analyses with sociodemographics (eg, educational level). Finally, we recommend developing similar intervention models for (a) target groups other than mothers (eg, the daughter or sons in case they will become a next target group for HPV vaccination) and (b) other vaccinations (eg, maternal pertussis vaccination).
We would like to thank Petra Oomen (RIVM) and the panels (Veldkamp BV, Intromart GFK, and NGO Flycatcher) for their help with data collection. The Netherlands Organization for Health Research and Development (ZonMw) and Netherlands Organization for Scientific Research (NWO) support this study (grant-number: 50-51515-98-259).
Conflicts of Interest
Screenshots of the website.DOCX File , 2665 KB
Measurement details (secondary outcomes).DOC File , 121 KB
- Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 2018 Nov;68(6):394-424. [CrossRef] [Medline]
- Schiffman M, Castle PE. Human papillomavirus: epidemiology and public health. Arch Pathol Lab Med 2003 Aug;127(8):930-934. [CrossRef] [Medline]
- Bosch FX, Lorincz A, Muñoz N, Meijer CJLM, Shah KV. The causal relation between human papillomavirus and cervical cancer. J Clin Pathol 2002 Apr;55(4):244-265 [FREE Full text] [CrossRef] [Medline]
- Chaturvedi AK, Engels EA, Pfeiffer RM, Hernandez BY, Xiao W, Kim E, et al. Human papillomavirus and rising oropharyngeal cancer incidence in the United States. J Clin Oncol 2011 Nov 10;29(32):4294-4301 [FREE Full text] [CrossRef] [Medline]
- Gillison ML, Chaturvedi AK, Lowy DR. HPV prophylactic vaccines and the potential prevention of noncervical cancers in both men and women. Cancer 2008 Nov 15;113(Suppl 10):3036-3046. [CrossRef] [Medline]
- van Lier E, Geraedts J, Oomen P, Giesbers H, van Vliet J, Drijfhout I. Vaccinatiegraad en jaarverslag Rijksvaccinatieprogramma Nederland. 2017. URL: https://www.rivm.nl/bibliotheek/rapporten/2018-0008.pdf [accessed 2019-05-19] [WebCite Cache]
- Pot M, Ruiter RAC, Paulussen TWGM, Heuvelink A, de Melker HE, van Vliet HJA, et al. Systematically Developing a Web-Based Tailored Intervention Promoting HPV-Vaccination Acceptability Among Mothers of Invited Girls Using Intervention Mapping. Front Public Health 2018;6:226. [CrossRef] [Medline]
- Bennett AT, Patel DA, Carlos RC, Zochowski MK, Pennewell SM, Chi AM, et al. Human Papillomavirus Vaccine Uptake After a Tailored, Online Educational Intervention for Female University Students: A Randomized Controlled Trial. J Womens Health (Larchmt) 2015 Nov;24(11):950-957 [FREE Full text] [CrossRef] [Medline]
- Gerend MA, Shepherd MA, Lustria MLA. Increasing human papillomavirus vaccine acceptability by tailoring messages to young adult women's perceived barriers. Sex Transm Dis 2013 May;40(5):401-405. [CrossRef] [Medline]
- Grandahl M, Rosenblad A, Stenhammar C, Tydén T, Westerling R, Larsson M, et al. School-based intervention for the prevention of HPV among adolescents: a cluster randomised controlled study. BMJ Open 2016 Jan 27;6(1):e009875 [FREE Full text] [CrossRef] [Medline]
- Hopfer S. Effects of a narrative HPV vaccination intervention aimed at reaching college women: a randomized controlled trial. Prev Sci 2012 Apr;13(2):173-182. [CrossRef] [Medline]
- Neville LM, O'Hara B, Milat AJ. Computer-tailored dietary behaviour change interventions: a systematic review. Health Educ Res 2009 Aug;24(4):699-720 [FREE Full text] [CrossRef] [Medline]
- Noar SM, Benac CN, Harris MS. Does tailoring matter? Meta-analytic review of tailored print health behavior change interventions. Psychol Bull 2007 Jul;133(4):673-693. [CrossRef] [Medline]
- Moore G, Audrey S, Barker M, Bond L, Bonell C, Cooper C, et al. Process evaluation in complex public health intervention studies: the need for guidance. J Epidemiol Community Health 2014 Feb;68(2):101-102 [FREE Full text] [CrossRef] [Medline]
- Grimshaw JM, Zwarenstein M, Tetroe JM, Godin G, Graham ID, Lemyre L, et al. Looking inside the black box: a theory-based process evaluation alongside a randomised controlled trial of printed educational materials (the Ontario printed educational message, OPEM) to improve referral and prescribing practices in primary care in Ontario, Canada. Implement Sci 2007 Nov 26;2:38 [FREE Full text] [CrossRef] [Medline]
- Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract 2005 Apr;6(2):134-147. [CrossRef] [Medline]
- Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci 2007 Nov 30;2:40 [FREE Full text] [CrossRef] [Medline]
- Van 't Riet J, Crutzen R, De Vries H. Investigating predictors of visiting, using, and revisiting an online health-communication program: a longitudinal study. J Med Internet Res 2010 Sep 02;12(3):e37 [FREE Full text] [CrossRef] [Medline]
- Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Internet-delivered interventions aimed at adolescents: a Delphi study on dissemination and exposure. Health Educ Res 2008 Jun;23(3):427-439. [CrossRef] [Medline]
- Kohl LFM, Crutzen R, de Vries NK. Online prevention aimed at lifestyle behaviors: a systematic review of reviews. J Med Internet Res 2013 Jul 16;15(7):e146 [FREE Full text] [CrossRef] [Medline]
- Elfeddali I, Bolman C, Candel MJJM, Wiers RW, de Vries H. Preventing smoking relapse via Web-based computer-tailored feedback: a randomized controlled trial. J Med Internet Res 2012 Aug 20;14(4):e109 [FREE Full text] [CrossRef] [Medline]
- Rabius V, Pike KJ, Wiatrek D, McAlister AL. Comparing internet assistance for smoking cessation: 13-month follow-up of a six-arm randomized controlled trial. J Med Internet Res 2008 Nov 21;10(5):e45 [FREE Full text] [CrossRef] [Medline]
- Richardson A, Graham AL, Cobb N, Xiao H, Mushro A, Abrams D, et al. Engagement promotes abstinence in a web-based cessation intervention: cohort study. J Med Internet Res 2013 Jan 28;15(1):e14 [FREE Full text] [CrossRef] [Medline]
- Priest HM, Knowlden AP. Systematic review of primary prevention human papillomavirus interventions targeting college students. Int J Sex Health 2014 Aug 14;27(2):125-144. [CrossRef]
- Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JEWC. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012 Nov 14;14(6):e152 [FREE Full text] [CrossRef] [Medline]
- Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries NK, Brug J, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res 2011 Jan 06;13(1):e2 [FREE Full text] [CrossRef] [Medline]
- Ludden GD, van Rompay TJL, Kelders SM, van Gemert-Pijnen JEWC. How to Increase Reach and Adherence of Web-Based Interventions: A Design Research Viewpoint. J Med Internet Res 2015 Jul 10;17(7):e172 [FREE Full text] [CrossRef] [Medline]
- Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res 2017 Jan 26;17(1):88 [FREE Full text] [CrossRef] [Medline]
- Miller L, Appleby P, Christensen J, Godoy C, Si M, Corsbie-Massay C. Virtual interactive interventions for reducing risky sex: adaptations, integrations, innovations. In: Noar SM, Harrington NG, editors. eHealth applications: promising strategies for behavior change. New York, NY: Routledge; 2012.
- Pot M, Paulussen TG, Ruiter RA, Eekhout I, de Melker HE, Spoelstra ME, et al. Effectiveness of a Web-Based Tailored Intervention With Virtual Assistants Promoting the Acceptability of HPV Vaccination Among Mothers of Invited Girls: Randomized Controlled Trial. J Med Internet Res 2017 Sep 06;19(9):e312 [FREE Full text] [CrossRef] [Medline]
- Bartholomew Eldredge LK, Markham CM, Ruiter RAC, Fernandez ME, Kok G, Parcel GS. Planning health promotion programs: An Intervention Mapping approach. 4th ed. Hoboken, NJ: Wiley; 2016.
- van Keulen HM, Mesters I, Brug J, Ausems M, Campbell M, Resnicow K, et al. Vitalum study design: RCT evaluating the efficacy of tailored print communication and telephone motivational interviewing on multiple health behaviors. BMC Public Health 2008 Jun 19;8:216 [FREE Full text] [CrossRef] [Medline]
- Turner SA, Silvia PJ. Must interesting things be pleasant? A test of competing appraisal structures. Emotion 2006 Nov;6(4):670-674. [CrossRef] [Medline]
- Pauwels E, Van Hoof E, Charlier C, Lechner L, De Bourdeaudhuij I. Design and process evaluation of an informative website tailored to breast cancer survivors' and intimate partners' post-treatment care needs. BMC Res Notes 2012 Oct 03;5:548 [FREE Full text] [CrossRef] [Medline]
- Liu Y. Developing a scale to measure the interactivity of websites. J Advert Res 2003 Jun 01;43(2):207-216. [CrossRef]
- Chaiken S. Heuristic versus systematic information processing and the use of source versus message cues in persuasion. J Person Soc Psychol 1980;39(5):752-766. [CrossRef]
- Friederichs SAH, Oenema A, Bolman C, Guyaux J, Van Keulen HM, Lechner L. Motivational interviewing in a web-based physical activity intervention: questions and reflections. Health Promot Int 2015 Sep;30(3):803-815. [CrossRef] [Medline]
- Samoocha D, Snels IAK, Bruinvels DJ, Anema JR, van der Beek AJ. Effectiveness of an interactive website aimed at empowerment of disability benefit claimants: results of a pragmatic randomized controlled trial. J Occup Rehabil 2011 Sep;21(3):410-420 [FREE Full text] [CrossRef] [Medline]
- Mevissen FEF, Meertens RM, Ruiter RAC, Feenstra H, Schaalma HP. HIV/STI risk communication: the effects of scenario-based risk information and frequency-based risk information on perceived susceptibility to chlamydia and HIV. J Health Psychol 2009 Jan;14(1):78-87. [CrossRef] [Medline]
- Van Keulen H, Fekkes M, Otten W, Van DPS, Kocken P, Ruiter R. Onderzoek naar de HPV-vaccinatiebereidheid bij moeders en dochters naar aanleiding van de inhaalcampagne in Nederland. Leiden: TNO; 2010.
- Crutzen R, Cyr D, Larios H, Ruiter RAC, de Vries NK. Social presence and use of internet-delivered interventions: a multi-method approach. PLoS One 2013;8(2):e57067 [FREE Full text] [CrossRef] [Medline]
- Spook J, Paulussen T, Kok G, van Empelen P. Evaluation of a Serious Self-Regulation Game Intervention for Overweight-Related Behaviors ("Balance It"): A Pilot Study. J Med Internet Res 2016 Sep 26;18(9):e225 [FREE Full text] [CrossRef] [Medline]
- Friederichs S, Bolman C, Oenema A, Guyaux J, Lechner L. Motivational interviewing in a Web-based physical activity intervention with an avatar: randomized controlled trial. J Med Internet Res 2014 Feb 13;16(2):e48 [FREE Full text] [CrossRef] [Medline]
- van Lier A, Oomen P, de Hoogh P, Drijfhout I, Elsinghorst B, Kemmeren J, et al. Præventis, the immunisation register of the Netherlands: a tool to evaluate the National Immunisation Programme. Euro Surveill 2012 Apr 26;17(17):20153 [FREE Full text] [CrossRef] [Medline]
- van der Pal S, Otten W, Detmar S. Evaluatie van de voorlichting aan ouders over de hielprik. Tijds Gezondheids Wetenschappen 2012 Jun 19;88(8):449-453. [CrossRef]
- Michie S, Dormandy E, Marteau TM. The multi-dimensional measure of informed choice: a validation study. Patient Educ Couns 2002 Sep;48(1):87-91. [CrossRef] [Medline]
- Marteau TM, Dormandy E, Michie S. A measure of informed choice. Health Expect 2001 Jun;4(2):99-108 [FREE Full text] [CrossRef] [Medline]
- O'Connor AM, Jacobsen MJ, Stacey D. An evidence-based approach to managing women's decisional conflict. J Obstet Gynecol Neonatal Nurs 2002;31(5):570-581. [CrossRef] [Medline]
- van Keulen HM, Otten W, Ruiter RAC, Fekkes M, van Steenbergen J, Dusseldorp E, et al. Determinants of HPV vaccination intentions among Dutch girls and their mothers: a cross-sectional study. BMC Public Health 2013 Feb 06;13:111 [FREE Full text] [CrossRef] [Medline]
- Pot M, van Keulen HM, Ruiter RAC, Eekhout I, Mollema L, Paulussen TWGM. Motivational and contextual determinants of HPV-vaccination uptake: A longitudinal study among mothers of girls invited for the HPV-vaccination. Prev Med 2017 Jul;100:41-49 [FREE Full text] [CrossRef] [Medline]
- Paulussen T, Lanting C, Buijs G, Hirasing R. Ouders over het Rijksvaccinatieprogramma: Tevredenheid en vaccinatiebereidheid van ouders van jonge kinderen in Nederland Parents about the governments immunization program: Satisfaction and vaccine intention of Dutch parents of toddlers. Leiden: TNO; 2000.
- Reiter PL, Brewer NT, Gottlieb SL, McRee A, Smith JS. Parents' health beliefs and HPV vaccination of their adolescent daughters. Soc Sci Med 2009 Aug;69(3):475-480. [CrossRef] [Medline]
- Gerend MA, Weibley E, Bland H. Parental response to human papillomavirus vaccine availability: uptake and intentions. J Adolesc Health 2009 Nov;45(5):528-531. [CrossRef] [Medline]
- Verplanken B, Orbell S. Reflections on Past Behavior: A Self-Report Index of Habit Strength1. J Appl Social Pyschol 2003 Jun;33(6):1313-1330. [CrossRef]
- Van Buuren S. Flexible Imputation of Missing Data. Boca Raton: Chapman Hall/CRC; 2012.
- White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med 2011 Feb 20;30(4):377-399. [CrossRef] [Medline]
- Rubin D. Multiple Imputation for Nonresponse in Surveys. New York: Wiley; 1987.
- Van Breukelen GJP. ANCOVA versus change from baseline: more power in randomized studies, more bias in nonrandomized studies [corrected]. J Clin Epidemiol 2006 Sep;59(9):920-925. [CrossRef] [Medline]
- Cohen J. Statistical power analysis for the behavioral science. Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers; 1988.
- Selya AS, Rose JS, Dierker LC, Hedeker D, Mermelstein RJ. A Practical Guide to Calculating Cohen's f(2), a Measure of Local Effect Size, from PROC MIXED. Front Psychol 2012;3:111. [CrossRef] [Medline]
- van Lier E, Oomen P, Giesbers H, van Vliet J, Drijfhout I, Zonnenberg-Hoff I. Vaccinatiegraad Rijksvaccinatieprogramma Nederland: verslagjaar. 2016. URL: https://www.rivm.nl/bibliotheek/rapporten/2016-0064.pdf [accessed 2019-04-29] [WebCite Cache]
- Kelders SM, Van Gemert-Pijnen JEWC, Werkman A, Nijland N, Seydel ER. Effectiveness of a Web-based intervention aimed at healthy dietary and physical activity behavior: a randomized controlled trial about users and usage. J Med Internet Res 2011 Apr 14;13(2):e32 [FREE Full text] [CrossRef] [Medline]
- Crutzen R, Ruiter RA, de Vries NK. Can interest and enjoyment help to increase use of Internet-delivered interventions? Psychol Health 2014 May 28;29(11):1227-1244. [CrossRef] [Medline]
- Sanders M, Fiscella K, Veazie P, Dolan JG, Jerant A. Does patient time spent viewing computer-tailored colorectal cancer screening materials predict patient-reported discussion of screening with providers? Health Educ Res 2016 Aug;31(4):555-562 [FREE Full text] [CrossRef] [Medline]
- Watts KJ, Meiser B, Wakefield CE, Barratt AL, Howard K, Cheah BC, et al. Online prostate cancer screening decision aid for at-risk men: a randomized trial. Health Psychol 2014 Sep;33(9):986-997. [CrossRef] [Medline]
- Voncken-Brewster V, Moser A, van der Weijden T, Nagykaldi Z, de Vries H, Tange H. Usability evaluation of an online, tailored self-management intervention for chronic obstructive pulmonary disease patients incorporating behavior change techniques. JMIR Res Protoc 2013 Jan 16;2(1):e3 [FREE Full text] [CrossRef] [Medline]
- Paiva AL, Lipschitz JM, Fernandez AC, Redding CA, Prochaska JO. Evaluation of the acceptability and feasibility of a computer-tailored intervention to increase human papillomavirus vaccination among young adult women. J Am Coll Health 2014;62(1):32-38. [CrossRef] [Medline]
- van den Berg SW, Peters EJ, Kraaijeveld JF, Gielissen MFM, Prins JB. Usage of a generic web-based self-management intervention for breast cancer survivors: substudy analysis of the BREATH trial. J Med Internet Res 2013 Aug 19;15(8):e170 [FREE Full text] [CrossRef] [Medline]
- Ritterband LM, Bailey ET, Thorndike FP, Lord HR, Farrell-Carnahan L, Baum LD. Initial evaluation of an Internet intervention to improve the sleep of cancer survivors with insomnia. Psychooncology 2012 Jul;21(7):695-705 [FREE Full text] [CrossRef] [Medline]
- De Cocker K, Charlier C, Van Hoof E, Pauwels E, Lechner L, Bourgois J, et al. Development and usability of a computer-tailored pedometer-based physical activity advice for breast cancer survivors. Eur J Cancer Care (Engl) 2015 Sep;24(5):673-682. [CrossRef] [Medline]
- Kovanović V, Gašević D, Dawson S, Joksimović S, Baker RS, Hatala M. Does time-on-task estimation matter? Implications for the validity of learning analytics findings. J Learn Analyt 2015;2(3):81-110. [CrossRef]
- Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010 Feb 17;12(1):e4 [FREE Full text] [CrossRef] [Medline]
- Sørensen K, Van den Broucke S, Fullam J, Doyle G, Pelikan J, Slonska Z, (HLS-EU) Consortium Health Literacy Project European. Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health 2012 Jan 25;12:80 [FREE Full text] [CrossRef] [Medline]
- Richtering SS, Hyun K, Neubeck L, Coorey G, Chalmers J, Usherwood T, et al. eHealth Literacy: Predictors in a Population With Moderate-to-High Cardiovascular Risk. JMIR Hum Factors 2017 Jan 27;4(1):e4 [FREE Full text] [CrossRef] [Medline]
- Xesfingi S, Vozikis A. eHealth Literacy: In the Quest of the Contributing Factors. Interact J Med Res 2016 May 25;5(2):e16 [FREE Full text] [CrossRef] [Medline]
- Reinwand DA, Crutzen R, Elfeddali I, Schneider F, Schulz DN, Smit ES, et al. Impact of Educational Level on Study Attrition and Evaluation of Web-Based Computer-Tailored Interventions: Results From Seven Randomized Controlled Trials. J Med Internet Res 2015 Oct 07;17(10):e228 [FREE Full text] [CrossRef] [Medline]
- Cox DS, Cox AD, Sturm L, Zimet G. Behavioral interventions to increase HPV vaccination acceptability among mothers of young girls. Health Psychol 2010 Jan;29(1):29-39. [CrossRef] [Medline]
|eHealth: electronic health|
|HPV: human papilloma virus|
|IDM: informed decision making|
|OR: odds ratio|
Edited by G Eysenbach; submitted 26.05.19; peer-reviewed by A Peine, R Mpofu; comments to author 19.07.19; revised version received 04.11.19; accepted 14.05.20; published 17.07.20Copyright
©Mirjam Pot, Theo G W M Paulussen, Robert A C Ruiter, Liesbeth Mollema, Miranda Hofstra, Hilde M Van Keulen. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 17.07.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.