Published on in Vol 22, No 7 (2020): July

Preprints (earlier versions) of this paper are available at, first published .
Dose-Response Relationship of a Web-Based Tailored Intervention Promoting Human Papillomavirus Vaccination: Process Evaluation of a Randomized Controlled Trial

Dose-Response Relationship of a Web-Based Tailored Intervention Promoting Human Papillomavirus Vaccination: Process Evaluation of a Randomized Controlled Trial

Dose-Response Relationship of a Web-Based Tailored Intervention Promoting Human Papillomavirus Vaccination: Process Evaluation of a Randomized Controlled Trial

Original Paper

1Child Health, Netherlands Organization for Applied Scientific Research (TNO), Leiden, Netherlands

2Department of Work and Social Psychology, Maastricht University, Maastricht, Netherlands

3National Institute for Public Health and the Environment (RIVM), Centre for Infectious Disease Control, Bilthoven, Netherlands

Corresponding Author:

Mirjam Pot, PhD

Child Health

Netherlands Organization for Applied Scientific Research (TNO)

Schipholweg 77

Leiden, 2316ZL


Phone: 31 643234293


Background: In the Netherlands, human papillomavirus (HPV) vaccination uptake remains low. To improve informed decision making (IDM) and HPV vaccination acceptability, we systematically developed an interactive, web-based tailored intervention to which mothers of Dutch girls were invited to participate.

Objective: The aim of this study was to provide insight into the intervention’s working mechanisms by evaluating (1) program use, (2) program acceptability, and (3) the relationship of program use with program acceptability and intervention effects (ie, dose-response relationship).

Methods: Only mothers from the intervention arm of a randomized controlled trial that assessed the effectiveness of the web-based, tailored intervention were included in this study. They were invited to visit the website of the web-based intervention between baseline (January 2015, just before access to the intervention) and follow up (March 2015, prior to the first HPV vaccination). Indicators for program use were time of website use (ie, duration of intervention exposure) and completeness (ie, proportion of all available web pages visited). HPV vaccination uptake registered by Praeventis was used as the primary outcome. Secondary outcomes were IDM, decisional conflict, and social-psychological determinants of HPV vaccination uptake.

Results: Among the 3995 invited mothers, 2509 (62.80%) logged on to the website, 2239 of whom (89.24%) visited at least one page of the intervention components. On average, mothers spent 21.39 minutes (SD 12.41) on the website and completed 50.04% (SD 26.18%) of the website components. Participants rated the website 7.64 (SD 1.39) on a 10-point scale. Program acceptability was significantly associated with completeness (β=4.36, P<.001), but not with time of website use (β=–.07, P=.77). Intention-to-treat analysis (N=3995) showed a significant positive effect of completeness on all outcome measures (all P<.003; Bonferroni-corrected α=.05/15 factors), including on HPV vaccination uptake. Time of website use had a significant positive effect on all outcomes (all P<.003), except for uptake (P=.20), risk perception when not vaccinated (P=.14), subjective norms (P=.03), and habit (P=.01).

Conclusions: Program use and acceptability of the intervention were adequate. Completeness was positively associated with acceptability. Furthermore, positive effects (ie, dose-response effects) were found for completeness and time of website use on the mothers’ IDM, decisional conflict, and almost all of the social-psychological determinants of HPV vaccination acceptability. In addition, the extent to which mothers completed the intervention had a positive impact on their daughters’ vaccination uptake. This indicates that the web-based, tailored intervention fits well with the mothers’ needs, and that completeness of use is essential for improving HPV vaccination uptake, acceptability, and IDM. Program use should therefore be promoted.

Trial Registration: Netherlands Trial Register NTR4795;

J Med Internet Res 2020;22(7):e14822



Cervical cancer is the fourth most common cancer among women worldwide. Globally, in 2018, there were 569,847 new cases and 311,365 deaths caused by cervical cancer [1]. Persistent infection by human papillomavirus (HPV) appears to be the major cause of cervical cancer; nearly all cases (over 99%) are attributable to an HPV infection [2,3]. There is also an increasing body of evidence that strongly links HPV infection with cancers of the anus, vulva, vagina, penis, and head and neck (eg, [4,5]). In the Netherlands, on an annual basis since 2010, girls are invited by the Dutch National Immunization Program to receive the vaccine against HPV upon reaching 13 years of age. However, HPV vaccination uptake remains low (45.5% in 2017) [6]. The uptake needs to be higher to reduce the incidence of cervical cancer in women. Therefore, we developed an interactive, web-based, tailored intervention promoting HPV vaccination acceptability among Dutch mothers of invited girls for HPV vaccination [7]. Further, to improve vaccine acceptability, important intervention objectives were to improve informed decision making (IDM) and to decrease decisional conflict [7].

To date, only four studies have examined the effectiveness of tailored interventions in promoting HPV vaccination acceptance [8-11]; three of these studies showed positive effects [9-11]. To our knowledge, the only tailored intervention that was web-based turned out to be ineffective [8]. The advantage of web-based interventions is that they have the potential to enable large-scale application at relatively low cost [12] and therefore can have a substantial impact at the population level [13].

Furthermore, only two of the existing studies using tailored interventions to promote HPV vaccination acceptance incorporated a process evaluation [9,11]. Process evaluation provides insight into the extent to which an intervention was used, which is relevant for determining factors that influence the effectiveness of an intervention [14-16]. Specifically, this analysis can provide insight into whether an intervention is ineffective due to implementation failure or to the inferior quality of the intervention method(s), thereby avoiding the so-called type III error [17]. It also provides recommendations for intervention improvement.

For the use of online interventions, three aspects are relevant: a first visit, staying, and revisiting [18,19]. In the context of decision making about HPV vaccination, we consider a first visit and staying as the most relevant aspects. Revisiting seems less relevant, since getting the HPV vaccine is hardly a repetitive behavior (ie, it consists of two injections). Within the electronic health (eHealth) arena, limited intervention use in terms of a first visit and staying is a broadly recognized problem [20]. The extent to which one uses an intervention may positively impact the intervention effects (ie, the so-called “dose-response relation”). Positive dose-response relationships have been found for eHealth interventions (eg, [21-23]). In the context of promoting HPV vaccination acceptability, none of the studies that incorporated a process evaluation addressed the dose-response relationship (for a review, see [24]). Moreover, data on the use of web-based interventions have been poorly reported [20,25] or not reported at all [26].

Furthermore, assessing program acceptability may shed light on the causes of nonuse or incomplete use [27]. An intervention needs to be considered appropriate by the target group and to suit their needs in order to reach the intended outcomes [28]. Acceptability will improve interaction and the users’ feelings of engagement, and, as a consequence, the likelihood of health-related behavior changes [29].

Results from the effect evaluation showed positive intervention effects on IDM, decisional conflict, and nearly all determinants of HPV vaccination uptake (for more details on the results of the effect evaluation, see [30]). This paper describes the process evaluation of the web-based tailored intervention promoting HPV vaccination acceptance [7]. The aim of this process evaluation was to provide insights into the intervention’s working mechanisms by evaluating (1) program use, (2) program acceptability, and (3) the relationship of program use with program acceptability and intervention effects (ie, dose-response relationships).

Participants and Study Design

The study was approved by the Ethical Committee of the VU Medical Center in Amsterdam (Netherlands Trial Register: NTR4795). This process evaluation was part of a randomized controlled trial on the effectiveness of the intervention [30]. For the purpose of this study, only participants from the experimental condition (ie, those invited to use the web-based intervention; N=3995) were included in the analyses. Details of the trial design, such as power calculations, participant eligibility, and recruitment procedures, have been published elsewhere [30]. This study was conducted between January 2015 (baseline, just before access to the intervention) and March 2015 (follow up, prior to the first HPV vaccination), in line with the national HPV vaccination round of 2015. Mothers were invited to use the web-based intervention between baseline and follow up (mid-January 2015).

Intervention Components

We developed the web-based tailored intervention with virtual assistants using the 6-step Intervention Mapping protocol for developing theory- and evidence-based health promotion interventions [31]. In short, the web-based intervention provided mothers with tailored feedback about topics on HPV vaccination, delivered by two virtual assistants. Multimedia Appendix 1 presents a selection of screenshots of the website. The website consisted of 4 menu options: (1) 2-sided, tailored information about HPV vaccination, (2) a decisional balance in which mothers could weigh their perceived pros and cons, (3) practical information, and (4) frequently asked questions. Within menus 1-3, mothers could visit several components. In addition, mothers were given the opportunity to visit an “in-depth” information page within some of the components (eg, educational movies). Table 1 provides a brief overview of the intervention components. More detailed descriptions of the intervention, including its theoretical basis, are published elsewhere [7].

Table 1. Brief description of intervention components.
Menu and componentDescription
Information about HPVa vaccination

General informationMothers are provided with general information about HPV, cervical cancer, and HPV vaccination.

Importance of vaccinating at a young ageMothers are challenged to consider whether the age of their daughter is appropriate to get vaccinated against HPV. The relationship between sexual activity in relation to the HPV vaccine is also discussed.

HPV-related risksMothers are asked to estimate both the risk of their daughter getting infected with HPV and the risk of their daughter developing cervical cancer later in life, and are then provided with tailored feedback accordingly.

Methods to protect against cervical cancerMothers are asked to rate the effectiveness of alternative methods for protecting against cervical cancer and are then provided with tailored feedback according to their answers.

From HPV to cervical cancerMothers are provided with an explanation of how infection with HPV can lead to cervical cancer (eg, by viewing an educational video).

Facts and storiesMothers are provided with several statements regarding HPV, cervical cancer, and HPV vaccination, and are asked to indicate whether these are true (a fact) or false (a story). They then receive tailored feedback accordingly.

Side effectsMothers are presented with a variety of potential side effects of the HPV vaccine and are asked to indicate whether or not they are scientifically proven. They are then provided with tailored feedback, stating the correct responses.

EffectivenessMothers are asked about the effectiveness of the HPV vaccine in protecting both against their daughter getting infected with HPV and developing cervical cancer. They are then provided with tailored feedback about the effectiveness of HPV vaccination.

Other mothersMothers are asked to indicate what they think most mothers in their direct environment will decide regarding their daughters’ HPV vaccination. They are then provided with tailored feedback and are shown the actual HPV vaccination uptake in different regions of the Netherlands in 2014.

Vaccine working mechanismsMothers are explained in a generic way how the HPV vaccine works in protecting against HPV and cervical cancer with an educational video.
Weighing the pros and cons

Decisional balanceMothers are presented with a list of pros and cons of HPV vaccination. Based on pros and cons mothers marked as most salient, a decisional balance reveals their current position on a scale ranging between wanting and not wanting to get their daughter vaccinated.

Value clarificationMothers are invited to list their central values for life and link these to HPV vaccination.
Practical informationPractical information provides mothers with information such as how and where to receive the HPV vaccine and provides them with advice on how they discuss HPV vaccination with personal important others (eg, their daughter and partner).
Frequently asked questions

About HPV vaccinationFrequently asked questions provides answers to known questions about the HPV vaccine (eg, “does my daughter know if she’s infected with HPV?”) or getting the HPV vaccine (eg, “where do I get the HPV vaccine?”).

Problems with the websiteMothers are provided with possible solutions to problems with the website such as not being able to hear or see the virtual assistants.

aHPV: human papillomavirus.


All data were derived from web-based self-report questionnaires and computer logs.

Program Use

Program use was measured according to time of website use and completeness. Both factors were constructed based on computer logs that registered the pages and components the mothers had visited and the amount of time they spent on the website during their visit(s).

Time of website use represents the total amount of time spent on the website. Each time mothers logged on to the website, a record was created, in which both the date and time of the first and last page the mothers entered were registered. Time per session was calculated by subtracting the time of the last entered page from the time of the earliest entered page. The total time of website use was then calculated by adding up the amount of time spent in each session.

Completeness represents the proportion of all available web pages visited by the mother while she was logged on to the website, ranging from 0% (visited no pages) to 100% (visited all pages).

Program Acceptability

Program acceptability was assessed at the 6-week follow up by asking mothers to rate the intervention and the virtual assistants on a 10-point scale, ranging from 0 (very bad) to 10 (excellent). The grade for the website was used as an indicator for overall program acceptability to examine the association with program use. In addition, mothers evaluated the information provided by the website (eg, credibility, relevance), perceived user control (eg, perceived degree of autonomy), and how they perceived the virtual assistants (eg, reliability) (See Table 2 and Table 3 for an overview). These acceptability measures were assessed on a 7-point scale, and some were averaged into one scale. All composite scores showed sufficient internal consistency (Cronbach α ≥.71 for scales with >2 items; Pearson r≥0.84 for scales with 2 items).

Table 2. Overview of the program acceptability measures for the website.
MeasuresItemsScore and scaleCronbach α or Pearson rReference
RateOn average, how would you rate the website on a scale from 0 to 10?0=very bad to
InterestIn general, what did you think of the website?1=very uninteresting to
7=very interesting; 1=boring to 7=engaging
0.84 (r)[33]
InformativeIn general, what did you think of the website?1=very uninformative to 7=very informative; 1=very noneducational to 7 =very educational0.86 (r)[34]
Perceived user controlI felt that I had a lot of control over my visiting experiences at this website.
While I was on the website, I could choose freely what I wanted to see.
While surfing the website, I felt in control.
While surfing the website, my actions decided the kind of experiences I got.
1=strongly disagree to 7=strongly agree.71 (α)[35]
ElaborationHow well did you read the information?1=not carefully at all to 7=very carefullyN/A[36]
SupportThe website has helped me decide about my daughter’s HPVb vaccination.1=strongly disagree to 7=strongly agreeN/A[37]
RecallI can recall the information from the website.1=strongly disagree to 7=strongly agreeN/A[32]
Personal relevanceI considered the website to be personally relevant.1=strongly disagree to 7=strongly agreeN/A[32,37]
TailoringI considered the information on the website to be…1=not tailored to me at all to 7=very tailored to meN/A[37]
ComprehensibilityI considered the information on the website to be…1=not at all understandable to 7=very understandableN/A[37]
ReliabilityI considered the information on the website to be…1=very unreliable to 7=very reliableN/A[37]
CredibilityI considered the information on the website to be…1=very incredible to 7=very credibleN/A[37]
UsefulnessI considered the information on the website to be…1=very useless to 7=very usefulN/A[38]
ReadabilityI considered the information on the website to be…1=very unreadable to 7=very readableN/A[39]
SidednessI considered the information on the website to be…1=focused on the cons to 7=focused on the prosN/A[40]
EnjoymentI considered the information on the website to be…1=very unenjoyable to 7=very enjoyableN/A[41]
NoveltyThe website contained new information for me.1=strongly disagree to 7=strongly agreeN/A[39]
Attitude toward websiteHow good or bad did you find…
- the possibility to first answer a question and then receive information?
- the possibility to weigh the pros and cons?
- the speed of the website?
- the layout of the website?
1=very bad to 7=very good.81 (α)[42]

aN/A: not applicable.

bHPV: human papillomavirus.

Table 3. Overview of the program acceptability measures for the virtual assistants.
MeasuresItemsScore and scalePearson rReference
Overall ratingOn a scale from 0 to 10, how would you rate:
- the mother-like assistant
- the doctor-like assistant
0=very bad to 10=excellent0.94[43]
EnjoymentI considered the virtual assistants to be…1=very unenjoyable to 7=very enjoyableN/Aa[41]
ReliabilityI considered the virtual assistants to be…1=very unreliable to 7=very reliableN/A[43]
CredibilityI considered the virtual assistants to be…1=very incredible to 7=very credibleN/A[37]
UsefulnessI considered the virtual assistants to be…1=very useless to 7=very usefulN/A[37]

aN/A: not applicable.


Primary Outcome: HPV Vaccination Uptake

An objective measure for HPV vaccination uptake was derived from Praeventis, which is the national electronic vaccination register that monitors the vaccination status for all children up to 18 years of age living in the Netherlands [44]. Uptake was dichotomized into having received no HPV injection (0=not vaccinated) and having received 1 or 2 HPV injections (1=vaccinated), as data analyses showed that determinants of HPV vaccination uptake contrasted the most between these groups.

Secondary Outcomes: IDM, Decisional Conflict, and Social-Psychological Determinants of HPV Vaccination Uptake

Multimedia Appendix 2 provides the measurement details of the constructs IDM, decisional conflict, and the social-psychological determinants of HPV vaccination uptake [30,45-54]. Determinants accounted for were intention, attitude, risk perception of having received the HPV vaccine, risk perception without having received the HPV vaccine, anticipated regret about rejecting the HPV vaccine, beliefs, subjective norms, habit strength toward HPV vaccination, self-efficacy, knowledge, and perceived relative effectiveness of HPV vaccination compared to alternative methods.


Sociodemographics were included as background variables (ie, age, educational level, country of birth, and religion).

Level of education referred to the mothers’ highest completed level of education. It was classified as low (less than secondary or vocational education), intermediate (secondary through preuniversity education), or high (professional or university education) [30,49,50].

Country of birth was dichotomized into “Netherlands” vs “other,” because the percentage of mothers born in the latter category (276/3995, 6.91%) was too small for further subdivision in our sample.

Religion was measured by asking mothers about their religious convictions (Protestant, Roman Catholic, Muslim, Jewish, Buddhist, Hindu, other, or no religion). This was dichotomized into “Protestant” vs “not Protestant” as earlier research and data analyses showed that Protestants more frequently refrain from vaccination compared to other religious groups [30,49,50].

Statistical Analyses

The sample and intervention use data are summarized using descriptive statistics. For determining the dose-response relation, intention-to-treat (ITT) analyses were used, which increases power, while decreasing the risk of possible bias caused by selective dropout [55]. Missing data were imputed for uptake, determinants, and sociodemographics (not for use): we applied multiple imputation by chained equations [55,56]. In total, 15 imputed datasets were generated using the predictive mean matching algorithm in Statistical Package for the Social Sciences (SPSS, IBM Corp). Results from these datasets were pooled together using Rubin’s rules [57]. Iteration plots were inspected to check convergence of the imputations.

The relationship between acceptability and use was examined using univariate regression analyses. The dose-response relation between program use and intervention effects was examined by logistic and linear regression analyses (for dichotomous and continuous variables, respectively) using the outcome score at follow up as the dependent variable and the outcome scores at baseline and program use (both completeness and time of website use) as the independent variables [58]. Effects were considered significant when the P value was lower than .003 (Bonferroni-corrected α=.05/15 factors). For the logistic regressions, the odds ratio (OR) was used as an index of effect sizes. These were interpreted as small (OR=1.5), medium (OR=3.5), or large (OR=9.0) [59]. For the linear regressions, effect sizes were calculated in R software (R Development Core Team, Vienna, Austria) using Cohen ƒ2 statistic, (R2AB – R2A)/(1 – R2AB), in which B is the variable of interest (ie, use), A is the set of all other variables (ie, the outcome at baseline), R2AB is the proportion of variance accounting for A and B together, and R2A is the proportion of variance accounted for by A [60]. These were classified as small (ƒ2=0.02), medium (ƒ2=0.15), and large (ƒ2=0.35) [59]. We performed complete case analyses as a sensitivity check for substantial differences with the results from ITT.

Response Rates and Attrition

A flow diagram of the recruitment and response of participants in the experimental condition, as part of the process evaluation, is shown in Figure 1. From the 4555 participants who were randomized into the experimental condition at baseline (T0), 4277 (93.90%) completed the baseline questionnaire, 2511 (55.13%) visited the intervention, and 2197 (48.23%) completed the follow-up questionnaire 8 weeks later (T1). Dropout analysis showed that there was significantly more dropout with regard to participants not born in the Netherlands, those with a lower educational level, those having a daughter not being vaccinated, and those with low levels of risk perception. In total, 564 participants were excluded as they (a) did not meet the inclusion criteria (ie, being a mother of a daughter born in 2002 and aged 24-62 years), (b) were found to be duplicates, or (c) encountered a language barrier. The final sample for ITT analysis consisted of 3995 mothers.

Figure 1. Flow diagram of the recruitment and response of study participants from the experimental condition. Participants could be excluded based on multiple criteria (eg, male with an invalid age). Therefore, the total number of participants excluded differed from the sum of separate criteria for exclusion.
View this figure

Sample Description

Table 4 shows the sample description of participants in the experimental condition. No data were available on sociodemographics of the population from which the sample was derived (ie, Dutch mothers of 12-year-old girls); hence, we were unable to assess whether the study sample was representative. The mothers’ mean age was 43.70 years (range 27.00-62.00 years). On average, mothers had a positive intention toward their daughters’ HPV vaccination at baseline (mean score 5.35, SD 1.69). Compared with the national HPV vaccination uptake at the time (ie, 2015), uptake was higher in the study sample (59,866/93,173; 60.98% vs 2923/3995; 73.17%) [61].

Table 4. Sample description of mothers in the intervention group (N=3995).
VariablesN (% missing)Valuea
Age in years, mean (SD)3995 (0)43.70 (4.27)
Country of birth, n (%)3991 (0.10)

The Netherlands
3715 (92.99)

276 (6.91)
Religion, n (%)3998 (0.18)

753 (18.85)

Not Protestant
3235 (80.98)
Educational level, n (%)3991 (0.10)

588 (14.72)

1736 (43.45)

1660 (41.55)
HPVb vaccination uptake, n (%)3986 (0.23)

2923 (73.17)

1063 (26.60)

aBy reporting 2 decimal points for the percentages, summing the percentages for each category may differ from 100%.

bHPV: human papillomavirus.

Program Use

From the 3995 mothers in the intervention group, a total of 2509 mothers (62.80%) logged in (ie, time of website use >0). Of these, 73.06% (1833/2509) logged in once, 19.89% (499/2509) logged in twice, 5.00% (125/2509) logged in three times, and 2.07% (52/2509) logged in four times or more. On average, mothers spent 21.39 minutes (SD 12.41) on the website.

Of the 2509 mothers that had initially logged in, 2239 (89.24%) visited at least one page of a component of the intervention (ie, completeness >0). Only these 2239 mothers were included for describing use of specific intervention components and program acceptability (see below). On average, the mothers completed 50.04% (SD 26.18%) of the website components (ie, 51/101 pages).

Use of Intervention Components

Table 5 shows the breakdown for the use of intervention components (N=2239). When mothers clicked on one or more links to receive in-depth information, it was marked as “visited.” Mothers visited the page “Ways to Protect Against Cervical Cancer” the most frequently (1971/2239, 88.03%), followed by “Chance” (1945/2239, 86.87%), whereas they visited “Value Clarification” the least (293/2239, 12.53%). “General Information” had the highest completion rate (1622/2239, 72.44%), whereas “Side Effects” and “Effectivity” were completed the least (both 19/2239, 0.85%). Only a small percentage of the mothers (4.20%-13.22%; n=94-298) visited in-depth information (comprising educational movies or extra information).

Table 5. Program use of the various intervention components among those who visited at least one page of the intervention (N=2239).
Intervention componentCompleteda, n (%)bPartly completedc, n (%)Not visitedd, n (%)Visited in-depth informatione, n (%)
General Information1622 (72.44)71 (3.17)546 (24.39)N/Af
Ways to Protect against Cervical Cancer130 (5.81)1841 (82.22)268 (11.97)N/A
Chance135 (6.03)1810 (80.84)294 (13.13)142 (6.34)
From HPVg to Cervical Cancer96 (4.29)1110 (49.58)1033 (46.14)298 (13.22)
Age154 (6.88)1568 (70.03)517 (23.09)160 (7.15)
Side Effects19 (0.85)1686 (75.30)534 (23.85)299 (13.35)
Effectivity19 (0.85)1542 (68.87)678 (30.28)339 (15.14)
Other Mothers1099 (49.08)416 (18.58)724 (32.34)N/A
Working Mechanisms Vaccination94 (4.20)1015 (45.33)1130 (50.47)94 (4.20)
Facts and Stories95 (4.24)1220 (54.49)924 (41.27)99 (4.42)
Weighing Pros and Cons615 (27.47)911 (40.69)713 (31.84)N/A
Value Clarification269 (12.01)24 (1.07)1946 (86.91)N/A
Practical Information556 (24.83)1147 (51.23)536 (23.94)N/A

aA component was considered “completed” when mothers visited every page of the component.

bBy reporting 2 decimal points for the percentages, summing the percentages for each category may differ from 100%.

c“Partly completed” indicates that the mothers had seen at least one, but not all pages.

d“Not visited” means that the mothers had not visited any of the component’s pages.

e“Visited in-depth information” was also part of the total completeness, but is depicted separately here to provide a better overall view of the mothers’ interest in this information. When one or more of the links to more in-depth information had been clicked on, the in-depth information was marked as “visited.”

fN/A: not applicable as the component did not contain in-depth information.

gHPV: human papillomavirus.

Program Acceptability

The overall acceptability of the intervention was rated 7.64 (SD 1.39) on a 10-point scale (1513/2239, 67.57%). Acceptability was significantly and positively associated with completeness (β=4.36, P<.001); this did not account for time of website use (β=–.07, P=.77). The virtual assistants were rated 7.41 (SD 1.57) on a 10-point scale (1513/2239, 67.57%). The mean scores on all other program acceptability measures were moderate to high (range 4.34-5.71 on a 7-point Likert scale; see Table 6). As the data were not normally distributed but were positively skewed, the median scores are also included in Table 6.

Table 6. Mean (SD) and median scores on the program acceptability measures among those who visited at least one page of the intervention (N=2239)a.
VariableWebsite (nmissing=726-734)Virtual assistants (nmissing=960)

Mean (SD)MedianMean (SD)Median
Rate (1-10)7.64 (1.39)8.007.41 (1.57)8.00
Interest (1-7)5.41 (1.05)5.50N/AbN/A
Informative (1-7)5.67 (1.04)6.00N/AN/A
Attitude toward website (1-7)5.53 (.89)6.00N/AN/A
Perceived user control (1-7)5.26 (.97)5.50N/AN/A
Elaboration (1-7)5.70 (1.14)6.00N/AN/A
Tailoring (1-7)5.22 (1.21)5.22N/AN/A
Comprehensibility (1-7)5.40 (1.70)6.00N/AN/A
Reliability (1-7)5.27 (1.23)6.005.04 (1.09)5.00
Credibility (1-7)5.35 (1.21)6.005.01 (1.22)5.00
Usefulness (1-7)5.52 (1.10)6.004.83 (1.33)5.00
Readability (1-7)5.71 (1.04)6.00N/AN/A
Sidedness (1-7)4.74 (1.09)4.00N/AN/A
Enjoyment (1-7)4.70 (1.08)5.004.72 (1.37)5.00
Novelty (1-7)5.11 (1.39)5.00N/AN/A
Support (1-7)4.34 (1.80)5.00N/AN/A
Recall (1-7)5.16 (1.22)5.00N/AN/A
Personal relevance (1-7)4.87 (1.41)5.00N/AN/A

aA higher score represents a higher score on the program acceptability measure.

bN/A: not applicable.

Dose-Response Relation Between Program Use and Intervention Outcomes

Table 7 provides an overview of the effects of program use (completeness and time of website use) on the primary and secondary outcomes according to the ITT analyses. Significant positive effects were found for completeness on all outcome measures (all P<.003). The more mothers had completed the intervention, the more likely they were to have their daughter vaccinated against HPV and make an informed decision, and were less likely to experience decisional conflict at follow up. In addition, positive effects of completeness were found with respect to all social-psychological determinants of HPV vaccination acceptability (eg, HPV vaccination intention, beliefs; see Table 7).

Time of website use had a positive effect on all outcomes (all P<.003), except for HPV vaccination uptake, risk perception when not vaccinated, subjective norms, and habit (Table 7). The more time mothers spent on the intervention, the more likely they were to make an informed decision, experience less decisional conflict, have a higher intention to vaccinate their daughter, have a more positive attitude, have more positive beliefs, have a lower risk perception of HPV vaccination, anticipate more feelings of regret about rejecting the HPV vaccine, report a higher relative effectiveness of the HPV vaccine, have higher self-efficacy expectations, and have more knowledge at follow up.

Effect sizes were small overall (see Table 7). Results from the complete case analyses were similar for both completeness and time of website use, except for a lack of effect of time of website use on intention.

Table 7. Effects of use on the outcome measures according to intention-to-treat analyses (N=3995).
OutcomeaPretest, mean (SD)Posttest, mean (SD)CompletenessTime of website use

β (SEM)Cohen ƒ2 or ORbβ (SEM)Cohen ƒ2 or OR
Primary outcome: HPVc vaccination uptaked

Has received no HPV injection (reference)N/Ae26.67% (1066)N/AN/AN/AN/A

Has received one or two HPV injectionsN/A73.32% (2929).004 (.001)f1.004.003 (.003)g1.003
Secondary outcomes

IDMh: Not informed (reference)67.31% (2689)42.53% (1699)N/AN/AN/AN/A

IDM: Informed32.69% (1306)57.47% (2296).014 (.001)f1.014.021 (.003)f1.021

IDM: continuous (0-48)18.69 (11.21)25.85 (12.30).087 (.006)f0.075.122 (.013)f.027

Decisional conflict (1-7)4.33 (1.75)5.38 (1.36).007 (.001)f0.043.011 (.002)f.018

Intention (1-7)5.35 (1.69)5.59 (1.87).004 (.001)f0.014.005 (.002)f.004

Attitude (1- 7)5.18 (1.45)5.37 (1.51).005 (.001)f0.027.007 (.001)f.010

Beliefs (1-7)4.19 (.73)4.47 (.81).004 (.000)f0.051.005 (.001)f.016

Risk perception; not vaccinated (1-7)3.74 (0.98)3.77 (1.08).002 (.001)f0.003.002 (.001)i.001

Risk perception; vaccinated (1-7)2.77 (1.07)2.64 (1.10)-.003 (.001)f0.008–.005 (.001)f.005

Anticipated regret (1-5)3.71 (1.25)3.59 (1.31).002 (.001)f0.007.004 (.001)f.004

Subjective Norm (–20-20)5.88 (7.81)7.25 (9.20).020 (.004)f0.009.018 (.008)j.002

Habit (1-7)4.28 (1.78)4.51 (1.83).004 (.001)f0.011.004 (.002)k.003

Relative effectiveness (1-10)-1.97 (2.22)-1.35 (2.27).013 (.001)f0.051.016 (.003)f.015

Self-efficacy (1-7)6.27 (.73)6.29 (.75).003 (.000)f0.022.004 (.001)f.007

Knowledge (–8-8)4.40 (2.14)5.75 (2.09).017 (.001)f0.095.024 (.002)f.034

aA higher score means a higher outcome (eg, more positive attitude) except for decisional conflict in which a higher score means less decisional conflict.

bOR: odds ratio; OR>1 means the higher the score on a factor, the higher the outcome of IDM and higher chance of the daughter being vaccinated; OR<1 means the higher the score on a factor, the lower outcome of IDM and lower chance of the daughter being vaccinated.

cHPV: human papillomavirus.

dHPV vaccination uptake was not assessed at baseline.

eN/A: not applicable.

fP<.003 (significant; Bonferroni: 0.05/15 factors).


hIDM: informed decision making.




Principal Findings

The aim of this process evaluation was to examine (1) program use, (2) program acceptability, and (3) the relationship between program use and acceptability and intervention outcomes for the web-based tailored intervention about HPV vaccination.

Program Use

Almost two thirds (62.80%) of the mothers who were invited to visit the intervention logged in to the intervention. This reach is adequate, and is comparable to that reported for other eHealth interventions (eg, [62]). However, it still leaves room for improvement to increase the reach by employing strategies such as arousing interest in this eHealth intervention. Support for this notion comes from Crutzen et al [63], who showed that arousing interest successfully increased the intention to visit a website about hepatitis A, B, and C virus and the likelihood of clicking on the link to visit the website. Interest was aroused in the invitation by challenging the potential visitor regarding their knowledge about hepatitis and it was emphasized that the website provided this information in a comprehensible manner. Not only could interest be aroused in the invitation for HPV vaccination for girls and their parents but also, for instance, via other channels by which the target group is reached (eg, the internet, social media). In addition, research has shown a positive recommendation by word of mouth to be an important trigger for visiting a web-based intervention for the first time [19]. Word-of-mouth recommendations could be encouraged by providing “tell-a-friend” services at the web-based intervention [19].

Of the mothers that logged in to the intervention, nearly all visited at least one component. However, a small portion of the mothers (10.76%) did not view any of the intervention’s content after having logged in. This is likely due to technical difficulties because at follow up, some mothers (320, 12.75%) indicated that they were not able to see or hear the virtual assistants.

On average, time of website use, indicating the time mothers spent with the intervention, was adequate (21.39 minutes). This is longer than that reported by Brouwer and colleagues [26] in their review of web-based, interactive healthy lifestyle interventions, in which intervention exposure time varied from less than 10 minutes to 10-20 minutes. However, the present exposure time was lower than found by Sanders and colleagues [64] for their interactive, web-based tailored intervention promoting colorectal cancer screening (33 minutes). This difference could be explained by the controlled setting of their study [64], in which participants arrived 60 minutes prior to a medical appointment and viewed the intervention in the waiting room in the presence of a research assistant.

In addition, the mean completeness of the intervention was 50.04%. This indicates that the intervention fits well with the mothers’ needs, which is also supported by the results regarding acceptability (see below). The mean completeness in this study is comparable to that found by Watts and colleagues [65], who reported a completeness of 49.09% for a web-based prostate cancer screening decision aid.

Use of Intervention Components

When looking at exposure to intervention components, we found that mothers were most interested in the effectiveness of (alternative) methods to protect against cervical cancer (88.03% visited) and the risks of their daughter getting infected with HPV and developing cervical cancer (86.87% visited). These appear to be essential components of communication about HPV vaccination.

The “Value Clarification” component was visited the least overall (13.09%). This may be attributed to the fact that it was not easily found by participants: this page was only accessible once mothers completed the component “Weighing Pros and Cons.” Voncken-Brewster and colleagues [66] also found that participants overlooked a certain part of their intervention, and they successfully improved its visibility by repositioning it. In order to promote exposure to values clarification, we could enhance its visibility.

The “General Information” component was completed the most overall (72.24%). This is likely due to the fact that the mother-like virtual assistant recommended starting with this component when mothers first entered the main menu, and that it was brief (2 pages). “Side Effects” and “Effectivity” were completed the least (both 0.85%). This could be explained by the fact that these two components contained the most links to in-depth information. Only a small proportion of the mothers (≤13.22%) visited in-depth information.

Program Acceptability

The mothers evaluated the website as positive, as shown by the overall acceptability (7.64/10) and the scores on all acceptability measures (mean scores ranging from 4.34 to 5.71 on a 7-point Likert scale). The virtual assistants were also well appreciated (7.41/10). This is likely because it matched with the mothers’ preferences for more interactive personalized feedback [40]. This adequate acceptability is similar to the findings of Paiva and colleagues [67] for a web-based tailored intervention to increase HPV vaccination among young adult women (3.27/4.0). We believe the high acceptability in the present study can be attributed to the systematic and user-centered development process of the intervention [7]. The target group was extensively involved throughout the development. Not only did we fine-tune the content of the intervention to the mothers’ preferences and requirements but this also was considered in the design of the website, which was chosen by the mothers.

Furthermore, we found a positive relationship between acceptability and completeness (P<.001). This association was also found by others in a web-based intervention for breast cancer [68-70]. This finding underlines the importance of an intervention to be considered appropriate by the target group.

Effects of Program Use on Intervention Outcomes

In line with our expectations, completeness had a significant positive effect on all outcome measures (IDM, decisional conflict, social-psychological determinants of HPV vaccination uptake), including actual HPV vaccination uptake itself. Mothers who had completed more of the intervention were more likely to have their daughter vaccinated against HPV, and had higher levels of IDM and more positive scores on determinants of HPV vaccination acceptance. In particular, the effect on HPV vaccination uptake is an important finding, which was not found when simply contrasting the experimental to the control condition [30]. This stresses the importance of conducting a process analysis alongside such an effect evaluation. These results are very promising, considering the currently low HPV vaccination uptake rates in the Netherlands [6].

We believe that these positive effects can be attributed to the extensive tailoring throughout the intervention. Support for this notion comes from the study results indicating that mothers perceived the intervention to be well-tailored (5.22 on a 7-point scale). Not only did we tailor the content of the intervention to the mothers’ personal interest, it was also used to guide the mothers’ personal route through the intervention. The latter is likely to have improved the usability of the intervention. Moreover, the intervention accounted for tailoring on a variety of themes. For example, it considered perceived barriers similar to the approach taken by Gerend and colleagues [9] as well as other beliefs, attitudes, subjective norms, habits, relative effectiveness, anticipated regret, risk perception, self-efficacy, and knowledge.

In contrast to our expectations, time of website use had a positive effect on all outcomes (P<.003), except for risk perception when not vaccinated (P=.14), subjective norms (P=.03), habit (P=.01), and vaccination uptake (P=.20). The latter is also in contrast to findings of a previous study [64]. An explanation for this lack of effects may be the measurement we used for time of website use, which may be less reliable for measuring exposure. We measured the total time spent on the website, but this does not indicate the specific pages the mothers had visited and for how long. In addition, we were unable to determine if there were any timeouts during a session and what the duration of a timeout was, because the intervention was web-based. For instance, they could have been distracted in the home environment during website use, which may have influenced the measured time of website use and may also have caused the lack of effect of acceptability on time of website use. An alternative, more accurate, measure can be found within the domain of education, namely time on task [71]. Future studies are needed to examine whether time on task (instead of total time of website use) has a positive influence on uptake.

Methodological Considerations

There are three methodological considerations. First, as mentioned above, the measurement of total time spent on the website seemed to be not entirely adequate to measure program use. In this study, completeness seemed to be a better indicator of use as it demonstrates the mother’s navigation through the website, whereas time of website use did not. In addition, within (almost) every component, mothers were first asked a question about an HPV-related topic. The page was marked as “completed” if they provided an answer, which was necessary to obtain tailored feedback. Within the decisional balance, completeness was calculated based on the mother’s answer to each statement. These are more complete indicators of actual use. Nevertheless, it is not an entirely complete indicator for use, since we could not measure whether they read generic information, listened to the tailored feedback from the virtual assistant, or saw a video. This may have influenced the dose-response effects found.

Second, considering the aim of initiating active processing of information, the positive effect found of completeness on habit may seem unwanted. However, at follow up, the mothers had already been exposed to the intervention, and were therefore likely to have actively processed the information. These results may indicate that the more mothers completed the intervention, the less they had to think about getting the HPV vaccine. The intervention may have helped them make a decision rather than inducing perceptions about HPV vaccination as something you take for granted, without thinking. In retrospect, the label “habit” appears to be misleading.

Third, the effect sizes found were small, which is in line with other web-based interventions targeting health behavior outcomes [72]. Despite this, we believe that even small effects are of relevance in public health as they become substantial at the population level. After all, in the Netherlands, approximately 100,000 girls are invited to receive the HPV vaccine on a yearly basis.

Strengths and Limitations

The most important strength of this study is that we, to our knowledge, are the first within the field of HPV vaccination to conduct an extensive process evaluation. According to a review [24], the process evaluation of an intervention promoting HPV vaccination acceptance by tailoring information to participants’ perceived barriers, such as that performed by Gerend and colleagues [9], was limited to the evaluation of intervention information (eg, the extent to which the information was informative and convincing). Other notable strengths are the objective measurement of HPV vaccination uptake, the dose-response effect of completeness on actual HPV vaccination uptake, the broad focus on outcomes (ie, determinants of HPV vaccination acceptance, uptake, IDM, and decisional conflict), and the adequate level of both (objectively assessed) program use and acceptability.

However, some limitations need to be addressed. First, there appeared to be technical difficulties despite our best effort to minimize such issues (eg, by extensively pilot testing the website using various devices). For instance, it turned out that a portion of the mothers were unable to see or hear the virtual assistants and that certain components did not work adequately among some of the mothers. Such technical difficulties are likely to have influenced website usability and therefore may have had a negative impact on program use. Second, this eHealth intervention requires mothers to have eHealth literacy [73]. People with lower eHealth literacy have been shown to have a lower educational level and to spend less time online [74,75]. It could be interesting to examine the extent to which eHealth literacy might have influenced use, acceptability, and effects. Considering the latter, we do not expect substantial differences since we did not find any differences in intervention effects among mothers with a low vs high educational level [30]. Third, it was hard to compare the study results to other studies because data on program use are often very poorly reported or even completely lacking (for a review, see [20]). Furthermore, the study was subject to a considerable amount of attrition. This is a common finding in studies on eHealth interventions [20]. Dropout appeared higher, for example, among mothers with a lower educational level, which has also been reported for other web-based tailored interventions [76]. In this study, missing data and selective dropout were handled by using multiple imputation [55,56]. Fortunately, there were nearly no differences between the results from the complete case analyses and the ITT analyses. Therefore, it seems unlikely that the effects found are spurious or due to selective dropout. Finally, caution is needed when generalizing the results of this study to the general population (ie, Dutch mothers of 12-year-old girls) because we were unable to check the representativeness of the sample. Besides, it should be noted that a very homogenous sample was reached. An undiversified reach is a known phenomenon within web-based interventions aimed at health promotion [20]. In our study, the mothers were predominantly born in the Netherlands, highly educated, and likely to have a daughter that was vaccinated against HPV. However, results from the effect evaluation showed no differences in intervention effects in specific subgroups of participants, as indicated by moderation analyses with sociodemographic variables [30].

Conclusions and Recommendations

This process evaluation has demonstrated that program use and acceptability of the web-based tailored HPV vaccination intervention were adequate. We found a positive association between completeness and acceptability. Furthermore, dose-response effects were found for completeness and time of website use on (nearly) all social-psychological determinants of the mothers’ decision making about vaccination, levels of IDM, and decisional conflict. Importantly, the extent to which mothers completed the intervention positively influenced their daughters’ actual HPV vaccination uptake. These results indicate that this web-based, tailored intervention fits well with the mothers’ needs and has the potential to increase HPV vaccination uptake. Because of the intervention’s adequate (dose-response) effects and acceptability, the intervention has recently been incorporated into the national HPV vaccination communication alongside the existing communication materials.

We recommend future interventions promoting HPV vaccination acceptance to (1) incorporate a process evaluation alongside the effect evaluation, (2), include strategies to arouse interest so as to expand reach, and to (3) include time on task. In addition, we recommend investigating the influence of different media types (eg, graphic vs nongraphic presentations) on decision making about HPV vaccination. For instance, Cox and colleagues [77] found that parents who viewed a graphic presentation of HPV-related risk information had a higher HPV vaccination intention compared to parents who viewed a nongraphic presentation. Furthermore, we recommend future research to examine differences in use and acceptability in specific subgroups of participants, for example by conducting moderation analyses with sociodemographics (eg, educational level). Finally, we recommend developing similar intervention models for (a) target groups other than mothers (eg, the daughter or sons in case they will become a next target group for HPV vaccination) and (b) other vaccinations (eg, maternal pertussis vaccination).


We would like to thank Petra Oomen (RIVM) and the panels (Veldkamp BV, Intromart GFK, and NGO Flycatcher) for their help with data collection. The Netherlands Organization for Health Research and Development (ZonMw) and Netherlands Organization for Scientific Research (NWO) support this study (grant-number: 50-51515-98-259).

Conflicts of Interest

None declared.

Multimedia Appendix 1

Screenshots of the website.

DOCX File , 2665 KB

Multimedia Appendix 2

Measurement details (secondary outcomes).

DOC File , 121 KB

  1. Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin 2018 Nov;68(6):394-424. [CrossRef] [Medline]
  2. Schiffman M, Castle PE. Human papillomavirus: epidemiology and public health. Arch Pathol Lab Med 2003 Aug;127(8):930-934. [CrossRef] [Medline]
  3. Bosch FX, Lorincz A, Muñoz N, Meijer CJLM, Shah KV. The causal relation between human papillomavirus and cervical cancer. J Clin Pathol 2002 Apr;55(4):244-265 [FREE Full text] [CrossRef] [Medline]
  4. Chaturvedi AK, Engels EA, Pfeiffer RM, Hernandez BY, Xiao W, Kim E, et al. Human papillomavirus and rising oropharyngeal cancer incidence in the United States. J Clin Oncol 2011 Nov 10;29(32):4294-4301 [FREE Full text] [CrossRef] [Medline]
  5. Gillison ML, Chaturvedi AK, Lowy DR. HPV prophylactic vaccines and the potential prevention of noncervical cancers in both men and women. Cancer 2008 Nov 15;113(Suppl 10):3036-3046. [CrossRef] [Medline]
  6. van Lier E, Geraedts J, Oomen P, Giesbers H, van Vliet J, Drijfhout I. Vaccinatiegraad en jaarverslag Rijksvaccinatieprogramma Nederland. 2017.   URL: [accessed 2019-05-19] [WebCite Cache]
  7. Pot M, Ruiter RAC, Paulussen TWGM, Heuvelink A, de Melker HE, van Vliet HJA, et al. Systematically Developing a Web-Based Tailored Intervention Promoting HPV-Vaccination Acceptability Among Mothers of Invited Girls Using Intervention Mapping. Front Public Health 2018;6:226. [CrossRef] [Medline]
  8. Bennett AT, Patel DA, Carlos RC, Zochowski MK, Pennewell SM, Chi AM, et al. Human Papillomavirus Vaccine Uptake After a Tailored, Online Educational Intervention for Female University Students: A Randomized Controlled Trial. J Womens Health (Larchmt) 2015 Nov;24(11):950-957 [FREE Full text] [CrossRef] [Medline]
  9. Gerend MA, Shepherd MA, Lustria MLA. Increasing human papillomavirus vaccine acceptability by tailoring messages to young adult women's perceived barriers. Sex Transm Dis 2013 May;40(5):401-405. [CrossRef] [Medline]
  10. Grandahl M, Rosenblad A, Stenhammar C, Tydén T, Westerling R, Larsson M, et al. School-based intervention for the prevention of HPV among adolescents: a cluster randomised controlled study. BMJ Open 2016 Jan 27;6(1):e009875 [FREE Full text] [CrossRef] [Medline]
  11. Hopfer S. Effects of a narrative HPV vaccination intervention aimed at reaching college women: a randomized controlled trial. Prev Sci 2012 Apr;13(2):173-182. [CrossRef] [Medline]
  12. Neville LM, O'Hara B, Milat AJ. Computer-tailored dietary behaviour change interventions: a systematic review. Health Educ Res 2009 Aug;24(4):699-720 [FREE Full text] [CrossRef] [Medline]
  13. Noar SM, Benac CN, Harris MS. Does tailoring matter? Meta-analytic review of tailored print health behavior change interventions. Psychol Bull 2007 Jul;133(4):673-693. [CrossRef] [Medline]
  14. Moore G, Audrey S, Barker M, Bond L, Bonell C, Cooper C, et al. Process evaluation in complex public health intervention studies: the need for guidance. J Epidemiol Community Health 2014 Feb;68(2):101-102 [FREE Full text] [CrossRef] [Medline]
  15. Grimshaw JM, Zwarenstein M, Tetroe JM, Godin G, Graham ID, Lemyre L, et al. Looking inside the black box: a theory-based process evaluation alongside a randomised controlled trial of printed educational materials (the Ontario printed educational message, OPEM) to improve referral and prescribing practices in primary care in Ontario, Canada. Implement Sci 2007 Nov 26;2:38 [FREE Full text] [CrossRef] [Medline]
  16. Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract 2005 Apr;6(2):134-147. [CrossRef] [Medline]
  17. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci 2007 Nov 30;2:40 [FREE Full text] [CrossRef] [Medline]
  18. Van 't Riet J, Crutzen R, De Vries H. Investigating predictors of visiting, using, and revisiting an online health-communication program: a longitudinal study. J Med Internet Res 2010 Sep 02;12(3):e37 [FREE Full text] [CrossRef] [Medline]
  19. Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Internet-delivered interventions aimed at adolescents: a Delphi study on dissemination and exposure. Health Educ Res 2008 Jun;23(3):427-439. [CrossRef] [Medline]
  20. Kohl LFM, Crutzen R, de Vries NK. Online prevention aimed at lifestyle behaviors: a systematic review of reviews. J Med Internet Res 2013 Jul 16;15(7):e146 [FREE Full text] [CrossRef] [Medline]
  21. Elfeddali I, Bolman C, Candel MJJM, Wiers RW, de Vries H. Preventing smoking relapse via Web-based computer-tailored feedback: a randomized controlled trial. J Med Internet Res 2012 Aug 20;14(4):e109 [FREE Full text] [CrossRef] [Medline]
  22. Rabius V, Pike KJ, Wiatrek D, McAlister AL. Comparing internet assistance for smoking cessation: 13-month follow-up of a six-arm randomized controlled trial. J Med Internet Res 2008 Nov 21;10(5):e45 [FREE Full text] [CrossRef] [Medline]
  23. Richardson A, Graham AL, Cobb N, Xiao H, Mushro A, Abrams D, et al. Engagement promotes abstinence in a web-based cessation intervention: cohort study. J Med Internet Res 2013 Jan 28;15(1):e14 [FREE Full text] [CrossRef] [Medline]
  24. Priest HM, Knowlden AP. Systematic review of primary prevention human papillomavirus interventions targeting college students. Int J Sex Health 2014 Aug 14;27(2):125-144. [CrossRef]
  25. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JEWC. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012 Nov 14;14(6):e152 [FREE Full text] [CrossRef] [Medline]
  26. Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries NK, Brug J, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res 2011 Jan 06;13(1):e2 [FREE Full text] [CrossRef] [Medline]
  27. Ludden GD, van Rompay TJL, Kelders SM, van Gemert-Pijnen JEWC. How to Increase Reach and Adherence of Web-Based Interventions: A Design Research Viewpoint. J Med Internet Res 2015 Jul 10;17(7):e172 [FREE Full text] [CrossRef] [Medline]
  28. Sekhon M, Cartwright M, Francis JJ. Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res 2017 Jan 26;17(1):88 [FREE Full text] [CrossRef] [Medline]
  29. Miller L, Appleby P, Christensen J, Godoy C, Si M, Corsbie-Massay C. Virtual interactive interventions for reducing risky sex: adaptations, integrations, innovations. In: Noar SM, Harrington NG, editors. eHealth applications: promising strategies for behavior change. New York, NY: Routledge; 2012.
  30. Pot M, Paulussen TG, Ruiter RA, Eekhout I, de Melker HE, Spoelstra ME, et al. Effectiveness of a Web-Based Tailored Intervention With Virtual Assistants Promoting the Acceptability of HPV Vaccination Among Mothers of Invited Girls: Randomized Controlled Trial. J Med Internet Res 2017 Sep 06;19(9):e312 [FREE Full text] [CrossRef] [Medline]
  31. Bartholomew Eldredge LK, Markham CM, Ruiter RAC, Fernandez ME, Kok G, Parcel GS. Planning health promotion programs: An Intervention Mapping approach. 4th ed. Hoboken, NJ: Wiley; 2016.
  32. van Keulen HM, Mesters I, Brug J, Ausems M, Campbell M, Resnicow K, et al. Vitalum study design: RCT evaluating the efficacy of tailored print communication and telephone motivational interviewing on multiple health behaviors. BMC Public Health 2008 Jun 19;8:216 [FREE Full text] [CrossRef] [Medline]
  33. Turner SA, Silvia PJ. Must interesting things be pleasant? A test of competing appraisal structures. Emotion 2006 Nov;6(4):670-674. [CrossRef] [Medline]
  34. Pauwels E, Van Hoof E, Charlier C, Lechner L, De Bourdeaudhuij I. Design and process evaluation of an informative website tailored to breast cancer survivors' and intimate partners' post-treatment care needs. BMC Res Notes 2012 Oct 03;5:548 [FREE Full text] [CrossRef] [Medline]
  35. Liu Y. Developing a scale to measure the interactivity of websites. J Advert Res 2003 Jun 01;43(2):207-216. [CrossRef]
  36. Chaiken S. Heuristic versus systematic information processing and the use of source versus message cues in persuasion. J Person Soc Psychol 1980;39(5):752-766. [CrossRef]
  37. Friederichs SAH, Oenema A, Bolman C, Guyaux J, Van Keulen HM, Lechner L. Motivational interviewing in a web-based physical activity intervention: questions and reflections. Health Promot Int 2015 Sep;30(3):803-815. [CrossRef] [Medline]
  38. Samoocha D, Snels IAK, Bruinvels DJ, Anema JR, van der Beek AJ. Effectiveness of an interactive website aimed at empowerment of disability benefit claimants: results of a pragmatic randomized controlled trial. J Occup Rehabil 2011 Sep;21(3):410-420 [FREE Full text] [CrossRef] [Medline]
  39. Mevissen FEF, Meertens RM, Ruiter RAC, Feenstra H, Schaalma HP. HIV/STI risk communication: the effects of scenario-based risk information and frequency-based risk information on perceived susceptibility to chlamydia and HIV. J Health Psychol 2009 Jan;14(1):78-87. [CrossRef] [Medline]
  40. Van Keulen H, Fekkes M, Otten W, Van DPS, Kocken P, Ruiter R. Onderzoek naar de HPV-vaccinatiebereidheid bij moeders en dochters naar aanleiding van de inhaalcampagne in Nederland. Leiden: TNO; 2010.
  41. Crutzen R, Cyr D, Larios H, Ruiter RAC, de Vries NK. Social presence and use of internet-delivered interventions: a multi-method approach. PLoS One 2013;8(2):e57067 [FREE Full text] [CrossRef] [Medline]
  42. Spook J, Paulussen T, Kok G, van Empelen P. Evaluation of a Serious Self-Regulation Game Intervention for Overweight-Related Behaviors ("Balance It"): A Pilot Study. J Med Internet Res 2016 Sep 26;18(9):e225 [FREE Full text] [CrossRef] [Medline]
  43. Friederichs S, Bolman C, Oenema A, Guyaux J, Lechner L. Motivational interviewing in a Web-based physical activity intervention with an avatar: randomized controlled trial. J Med Internet Res 2014 Feb 13;16(2):e48 [FREE Full text] [CrossRef] [Medline]
  44. van Lier A, Oomen P, de Hoogh P, Drijfhout I, Elsinghorst B, Kemmeren J, et al. Præventis, the immunisation register of the Netherlands: a tool to evaluate the National Immunisation Programme. Euro Surveill 2012 Apr 26;17(17):20153 [FREE Full text] [CrossRef] [Medline]
  45. van der Pal S, Otten W, Detmar S. Evaluatie van de voorlichting aan ouders over de hielprik. Tijds Gezondheids Wetenschappen 2012 Jun 19;88(8):449-453. [CrossRef]
  46. Michie S, Dormandy E, Marteau TM. The multi-dimensional measure of informed choice: a validation study. Patient Educ Couns 2002 Sep;48(1):87-91. [CrossRef] [Medline]
  47. Marteau TM, Dormandy E, Michie S. A measure of informed choice. Health Expect 2001 Jun;4(2):99-108 [FREE Full text] [CrossRef] [Medline]
  48. O'Connor AM, Jacobsen MJ, Stacey D. An evidence-based approach to managing women's decisional conflict. J Obstet Gynecol Neonatal Nurs 2002;31(5):570-581. [CrossRef] [Medline]
  49. van Keulen HM, Otten W, Ruiter RAC, Fekkes M, van Steenbergen J, Dusseldorp E, et al. Determinants of HPV vaccination intentions among Dutch girls and their mothers: a cross-sectional study. BMC Public Health 2013 Feb 06;13:111 [FREE Full text] [CrossRef] [Medline]
  50. Pot M, van Keulen HM, Ruiter RAC, Eekhout I, Mollema L, Paulussen TWGM. Motivational and contextual determinants of HPV-vaccination uptake: A longitudinal study among mothers of girls invited for the HPV-vaccination. Prev Med 2017 Jul;100:41-49 [FREE Full text] [CrossRef] [Medline]
  51. Paulussen T, Lanting C, Buijs G, Hirasing R. Ouders over het Rijksvaccinatieprogramma: Tevredenheid en vaccinatiebereidheid van ouders van jonge kinderen in Nederland Parents about the governments immunization program: Satisfaction and vaccine intention of Dutch parents of toddlers. Leiden: TNO; 2000.
  52. Reiter PL, Brewer NT, Gottlieb SL, McRee A, Smith JS. Parents' health beliefs and HPV vaccination of their adolescent daughters. Soc Sci Med 2009 Aug;69(3):475-480. [CrossRef] [Medline]
  53. Gerend MA, Weibley E, Bland H. Parental response to human papillomavirus vaccine availability: uptake and intentions. J Adolesc Health 2009 Nov;45(5):528-531. [CrossRef] [Medline]
  54. Verplanken B, Orbell S. Reflections on Past Behavior: A Self-Report Index of Habit Strength1. J Appl Social Pyschol 2003 Jun;33(6):1313-1330. [CrossRef]
  55. Van Buuren S. Flexible Imputation of Missing Data. Boca Raton: Chapman Hall/CRC; 2012.
  56. White IR, Royston P, Wood AM. Multiple imputation using chained equations: Issues and guidance for practice. Stat Med 2011 Feb 20;30(4):377-399. [CrossRef] [Medline]
  57. Rubin D. Multiple Imputation for Nonresponse in Surveys. New York: Wiley; 1987.
  58. Van Breukelen GJP. ANCOVA versus change from baseline: more power in randomized studies, more bias in nonrandomized studies [corrected]. J Clin Epidemiol 2006 Sep;59(9):920-925. [CrossRef] [Medline]
  59. Cohen J. Statistical power analysis for the behavioral science. Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers; 1988.
  60. Selya AS, Rose JS, Dierker LC, Hedeker D, Mermelstein RJ. A Practical Guide to Calculating Cohen's f(2), a Measure of Local Effect Size, from PROC MIXED. Front Psychol 2012;3:111. [CrossRef] [Medline]
  61. van Lier E, Oomen P, Giesbers H, van Vliet J, Drijfhout I, Zonnenberg-Hoff I. Vaccinatiegraad Rijksvaccinatieprogramma Nederland: verslagjaar. 2016.   URL: [accessed 2019-04-29] [WebCite Cache]
  62. Kelders SM, Van Gemert-Pijnen JEWC, Werkman A, Nijland N, Seydel ER. Effectiveness of a Web-based intervention aimed at healthy dietary and physical activity behavior: a randomized controlled trial about users and usage. J Med Internet Res 2011 Apr 14;13(2):e32 [FREE Full text] [CrossRef] [Medline]
  63. Crutzen R, Ruiter RA, de Vries NK. Can interest and enjoyment help to increase use of Internet-delivered interventions? Psychol Health 2014 May 28;29(11):1227-1244. [CrossRef] [Medline]
  64. Sanders M, Fiscella K, Veazie P, Dolan JG, Jerant A. Does patient time spent viewing computer-tailored colorectal cancer screening materials predict patient-reported discussion of screening with providers? Health Educ Res 2016 Aug;31(4):555-562 [FREE Full text] [CrossRef] [Medline]
  65. Watts KJ, Meiser B, Wakefield CE, Barratt AL, Howard K, Cheah BC, et al. Online prostate cancer screening decision aid for at-risk men: a randomized trial. Health Psychol 2014 Sep;33(9):986-997. [CrossRef] [Medline]
  66. Voncken-Brewster V, Moser A, van der Weijden T, Nagykaldi Z, de Vries H, Tange H. Usability evaluation of an online, tailored self-management intervention for chronic obstructive pulmonary disease patients incorporating behavior change techniques. JMIR Res Protoc 2013 Jan 16;2(1):e3 [FREE Full text] [CrossRef] [Medline]
  67. Paiva AL, Lipschitz JM, Fernandez AC, Redding CA, Prochaska JO. Evaluation of the acceptability and feasibility of a computer-tailored intervention to increase human papillomavirus vaccination among young adult women. J Am Coll Health 2014;62(1):32-38. [CrossRef] [Medline]
  68. van den Berg SW, Peters EJ, Kraaijeveld JF, Gielissen MFM, Prins JB. Usage of a generic web-based self-management intervention for breast cancer survivors: substudy analysis of the BREATH trial. J Med Internet Res 2013 Aug 19;15(8):e170 [FREE Full text] [CrossRef] [Medline]
  69. Ritterband LM, Bailey ET, Thorndike FP, Lord HR, Farrell-Carnahan L, Baum LD. Initial evaluation of an Internet intervention to improve the sleep of cancer survivors with insomnia. Psychooncology 2012 Jul;21(7):695-705 [FREE Full text] [CrossRef] [Medline]
  70. De Cocker K, Charlier C, Van Hoof E, Pauwels E, Lechner L, Bourgois J, et al. Development and usability of a computer-tailored pedometer-based physical activity advice for breast cancer survivors. Eur J Cancer Care (Engl) 2015 Sep;24(5):673-682. [CrossRef] [Medline]
  71. Kovanović V, Gašević D, Dawson S, Joksimović S, Baker RS, Hatala M. Does time-on-task estimation matter? Implications for the validity of learning analytics findings. J Learn Analyt 2015;2(3):81-110. [CrossRef]
  72. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010 Feb 17;12(1):e4 [FREE Full text] [CrossRef] [Medline]
  73. Sørensen K, Van den Broucke S, Fullam J, Doyle G, Pelikan J, Slonska Z, (HLS-EU) Consortium Health Literacy Project European. Health literacy and public health: a systematic review and integration of definitions and models. BMC Public Health 2012 Jan 25;12:80 [FREE Full text] [CrossRef] [Medline]
  74. Richtering SS, Hyun K, Neubeck L, Coorey G, Chalmers J, Usherwood T, et al. eHealth Literacy: Predictors in a Population With Moderate-to-High Cardiovascular Risk. JMIR Hum Factors 2017 Jan 27;4(1):e4 [FREE Full text] [CrossRef] [Medline]
  75. Xesfingi S, Vozikis A. eHealth Literacy: In the Quest of the Contributing Factors. Interact J Med Res 2016 May 25;5(2):e16 [FREE Full text] [CrossRef] [Medline]
  76. Reinwand DA, Crutzen R, Elfeddali I, Schneider F, Schulz DN, Smit ES, et al. Impact of Educational Level on Study Attrition and Evaluation of Web-Based Computer-Tailored Interventions: Results From Seven Randomized Controlled Trials. J Med Internet Res 2015 Oct 07;17(10):e228 [FREE Full text] [CrossRef] [Medline]
  77. Cox DS, Cox AD, Sturm L, Zimet G. Behavioral interventions to increase HPV vaccination acceptability among mothers of young girls. Health Psychol 2010 Jan;29(1):29-39. [CrossRef] [Medline]

eHealth: electronic health
HPV: human papilloma virus
IDM: informed decision making
ITT: intention-to-treat
OR: odds ratio

Edited by G Eysenbach; submitted 26.05.19; peer-reviewed by A Peine, R Mpofu; comments to author 19.07.19; revised version received 04.11.19; accepted 14.05.20; published 17.07.20


©Mirjam Pot, Theo G W M Paulussen, Robert A C Ruiter, Liesbeth Mollema, Miranda Hofstra, Hilde M Van Keulen. Originally published in the Journal of Medical Internet Research (, 17.07.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.