The study was approved by the Ethical Committee of the VU Medical Center in Amsterdam (Netherlands Trial Register: NTR4795). This process evaluation was part of a randomized controlled trial on the effectiveness of the intervention [30]. For the purpose of this study, only participants from the experimental condition (ie, those invited to use the web-based intervention; N=3995) were included in the analyses. Details of the trial design, such as power calculations, participant eligibility, and recruitment procedures, have been published elsewhere [30]. This study was conducted between January 2015 (baseline, just before access to the intervention) and March 2015 (follow up, prior to the first HPV vaccination), in line with the national HPV vaccination round of 2015. Mothers were invited to use the web-based intervention between baseline and follow up (mid-January 2015).

We developed the web-based tailored intervention with virtual assistants using the 6-step Intervention Mapping protocol for developing theory- and evidence-based health promotion interventions [31]. In short, the web-based intervention provided mothers with tailored feedback about topics on HPV vaccination, delivered by two virtual assistants. Multimedia Appendix 1 presents a selection of screenshots of the website. The website consisted of 4 menu options: (1) 2-sided, tailored information about HPV vaccination, (2) a decisional balance in which mothers could weigh their perceived pros and cons, (3) practical information, and (4) frequently asked questions. Within menus 1-3, mothers could visit several components. In addition, mothers were given the opportunity to visit an “in-depth” information page within some of the components (eg, educational movies). Table 1 provides a brief overview of the intervention components. More detailed descriptions of the intervention, including its theoretical basis, are published elsewhere [7].

All data were derived from web-based self-report questionnaires and computer logs.

Program use was measured according to time of website use and completeness. Both factors were constructed based on computer logs that registered the pages and components the mothers had visited and the amount of time they spent on the website during their visit(s).

Time of website use represents the total amount of time spent on the website. Each time mothers logged on to the website, a record was created, in which both the date and time of the first and last page the mothers entered were registered. Time per session was calculated by subtracting the time of the last entered page from the time of the earliest entered page. The total time of website use was then calculated by adding up the amount of time spent in each session.

Completeness represents the proportion of all available web pages visited by the mother while she was logged on to the website, ranging from 0% (visited no pages) to 100% (visited all pages).

Program acceptability was assessed at the 6-week follow up by asking mothers to rate the intervention and the virtual assistants on a 10-point scale, ranging from 0 (very bad) to 10 (excellent). The grade for the website was used as an indicator for overall program acceptability to examine the association with program use. In addition, mothers evaluated the information provided by the website (eg, credibility, relevance), perceived user control (eg, perceived degree of autonomy), and how they perceived the virtual assistants (eg, reliability) (See Table 2 and Table 3 for an overview). These acceptability measures were assessed on a 7-point scale, and some were averaged into one scale. All composite scores showed sufficient internal consistency (Cronbach α ≥.71 for scales with >2 items; Pearson r≥0.84 for scales with 2 items).

Sociodemographics were included as background variables (ie, age, educational level, country of birth, and religion).

Level of education referred to the mothers’ highest completed level of education. It was classified as low (less than secondary or vocational education), intermediate (secondary through preuniversity education), or high (professional or university education) [30,49,50].

Country of birth was dichotomized into “Netherlands” vs “other,” because the percentage of mothers born in the latter category (276/3995, 6.91%) was too small for further subdivision in our sample.

Religion was measured by asking mothers about their religious convictions (Protestant, Roman Catholic, Muslim, Jewish, Buddhist, Hindu, other, or no religion). This was dichotomized into “Protestant” vs “not Protestant” as earlier research and data analyses showed that Protestants more frequently refrain from vaccination compared to other religious groups [30,49,50].

The sample and intervention use data are summarized using descriptive statistics. For determining the dose-response relation, intention-to-treat (ITT) analyses were used, which increases power, while decreasing the risk of possible bias caused by selective dropout [55]. Missing data were imputed for uptake, determinants, and sociodemographics (not for use): we applied multiple imputation by chained equations [55,56]. In total, 15 imputed datasets were generated using the predictive mean matching algorithm in Statistical Package for the Social Sciences (SPSS, IBM Corp). Results from these datasets were pooled together using Rubin’s rules [57]. Iteration plots were inspected to check convergence of the imputations.

The relationship between acceptability and use was examined using univariate regression analyses. The dose-response relation between program use and intervention effects was examined by logistic and linear regression analyses (for dichotomous and continuous variables, respectively) using the outcome score at follow up as the dependent variable and the outcome scores at baseline and program use (both completeness and time of website use) as the independent variables [58]. Effects were considered significant when the P value was lower than .003 (Bonferroni-corrected α=.05/15 factors). For the logistic regressions, the odds ratio (OR) was used as an index of effect sizes. These were interpreted as small (OR=1.5), medium (OR=3.5), or large (OR=9.0) [59]. For the linear regressions, effect sizes were calculated in R software (R Development Core Team, Vienna, Austria) using Cohen ƒ² statistic, (R²AB – R²A)/(1 – R²AB), in which B is the variable of interest (ie, use), A is the set of all other variables (ie, the outcome at baseline), R²AB is the proportion of variance accounting for A and B together, and R²A is the proportion of variance accounted for by A [60]. These were classified as small (ƒ²=0.02), medium (ƒ²=0.15), and large (ƒ²=0.35) [59]. We performed complete case analyses as a sensitivity check for substantial differences with the results from ITT.

A flow diagram of the recruitment and response of participants in the experimental condition, as part of the process evaluation, is shown in Figure 1. From the 4555 participants who were randomized into the experimental condition at baseline (T0), 4277 (93.90%) completed the baseline questionnaire, 2511 (55.13%) visited the intervention, and 2197 (48.23%) completed the follow-up questionnaire 8 weeks later (T1). Dropout analysis showed that there was significantly more dropout with regard to participants not born in the Netherlands, those with a lower educational level, those having a daughter not being vaccinated, and those with low levels of risk perception. In total, 564 participants were excluded as they (a) did not meet the inclusion criteria (ie, being a mother of a daughter born in 2002 and aged 24-62 years), (b) were found to be duplicates, or (c) encountered a language barrier. The final sample for ITT analysis consisted of 3995 mothers.

Table 4 shows the sample description of participants in the experimental condition. No data were available on sociodemographics of the population from which the sample was derived (ie, Dutch mothers of 12-year-old girls); hence, we were unable to assess whether the study sample was representative. The mothers’ mean age was 43.70 years (range 27.00-62.00 years). On average, mothers had a positive intention toward their daughters’ HPV vaccination at baseline (mean score 5.35, SD 1.69). Compared with the national HPV vaccination uptake at the time (ie, 2015), uptake was higher in the study sample (59,866/93,173; 60.98% vs 2923/3995; 73.17%) [61].

From the 3995 mothers in the intervention group, a total of 2509 mothers (62.80%) logged in (ie, time of website use >0). Of these, 73.06% (1833/2509) logged in once, 19.89% (499/2509) logged in twice, 5.00% (125/2509) logged in three times, and 2.07% (52/2509) logged in four times or more. On average, mothers spent 21.39 minutes (SD 12.41) on the website.

Of the 2509 mothers that had initially logged in, 2239 (89.24%) visited at least one page of a component of the intervention (ie, completeness >0). Only these 2239 mothers were included for describing use of specific intervention components and program acceptability (see below). On average, the mothers completed 50.04% (SD 26.18%) of the website components (ie, 51/101 pages).

Table 5 shows the breakdown for the use of intervention components (N=2239). When mothers clicked on one or more links to receive in-depth information, it was marked as “visited.” Mothers visited the page “Ways to Protect Against Cervical Cancer” the most frequently (1971/2239, 88.03%), followed by “Chance” (1945/2239, 86.87%), whereas they visited “Value Clarification” the least (293/2239, 12.53%). “General Information” had the highest completion rate (1622/2239, 72.44%), whereas “Side Effects” and “Effectivity” were completed the least (both 19/2239, 0.85%). Only a small percentage of the mothers (4.20%-13.22%; n=94-298) visited in-depth information (comprising educational movies or extra information).

The overall acceptability of the intervention was rated 7.64 (SD 1.39) on a 10-point scale (1513/2239, 67.57%). Acceptability was significantly and positively associated with completeness (β=4.36, P<.001); this did not account for time of website use (β=–.07, P=.77). The virtual assistants were rated 7.41 (SD 1.57) on a 10-point scale (1513/2239, 67.57%). The mean scores on all other program acceptability measures were moderate to high (range 4.34-5.71 on a 7-point Likert scale; see Table 6). As the data were not normally distributed but were positively skewed, the median scores are also included in Table 6.

Table 7 provides an overview of the effects of program use (completeness and time of website use) on the primary and secondary outcomes according to the ITT analyses. Significant positive effects were found for completeness on all outcome measures (all P<.003). The more mothers had completed the intervention, the more likely they were to have their daughter vaccinated against HPV and make an informed decision, and were less likely to experience decisional conflict at follow up. In addition, positive effects of completeness were found with respect to all social-psychological determinants of HPV vaccination acceptability (eg, HPV vaccination intention, beliefs; see Table 7).

Time of website use had a positive effect on all outcomes (all P<.003), except for HPV vaccination uptake, risk perception when not vaccinated, subjective norms, and habit (Table 7). The more time mothers spent on the intervention, the more likely they were to make an informed decision, experience less decisional conflict, have a higher intention to vaccinate their daughter, have a more positive attitude, have more positive beliefs, have a lower risk perception of HPV vaccination, anticipate more feelings of regret about rejecting the HPV vaccine, report a higher relative effectiveness of the HPV vaccine, have higher self-efficacy expectations, and have more knowledge at follow up.

Effect sizes were small overall (see Table 7). Results from the complete case analyses were similar for both completeness and time of website use, except for a lack of effect of time of website use on intention.

The aim of this process evaluation was to examine (1) program use, (2) program acceptability, and (3) the relationship between program use and acceptability and intervention outcomes for the web-based tailored intervention about HPV vaccination.

Almost two thirds (62.80%) of the mothers who were invited to visit the intervention logged in to the intervention. This reach is adequate, and is comparable to that reported for other eHealth interventions (eg, [62]). However, it still leaves room for improvement to increase the reach by employing strategies such as arousing interest in this eHealth intervention. Support for this notion comes from Crutzen et al [63], who showed that arousing interest successfully increased the intention to visit a website about hepatitis A, B, and C virus and the likelihood of clicking on the link to visit the website. Interest was aroused in the invitation by challenging the potential visitor regarding their knowledge about hepatitis and it was emphasized that the website provided this information in a comprehensible manner. Not only could interest be aroused in the invitation for HPV vaccination for girls and their parents but also, for instance, via other channels by which the target group is reached (eg, the internet, social media). In addition, research has shown a positive recommendation by word of mouth to be an important trigger for visiting a web-based intervention for the first time [19]. Word-of-mouth recommendations could be encouraged by providing “tell-a-friend” services at the web-based intervention [19].

Of the mothers that logged in to the intervention, nearly all visited at least one component. However, a small portion of the mothers (10.76%) did not view any of the intervention’s content after having logged in. This is likely due to technical difficulties because at follow up, some mothers (320, 12.75%) indicated that they were not able to see or hear the virtual assistants.

On average, time of website use, indicating the time mothers spent with the intervention, was adequate (21.39 minutes). This is longer than that reported by Brouwer and colleagues [26] in their review of web-based, interactive healthy lifestyle interventions, in which intervention exposure time varied from less than 10 minutes to 10-20 minutes. However, the present exposure time was lower than found by Sanders and colleagues [64] for their interactive, web-based tailored intervention promoting colorectal cancer screening (33 minutes). This difference could be explained by the controlled setting of their study [64], in which participants arrived 60 minutes prior to a medical appointment and viewed the intervention in the waiting room in the presence of a research assistant.

In addition, the mean completeness of the intervention was 50.04%. This indicates that the intervention fits well with the mothers’ needs, which is also supported by the results regarding acceptability (see below). The mean completeness in this study is comparable to that found by Watts and colleagues [65], who reported a completeness of 49.09% for a web-based prostate cancer screening decision aid.

When looking at exposure to intervention components, we found that mothers were most interested in the effectiveness of (alternative) methods to protect against cervical cancer (88.03% visited) and the risks of their daughter getting infected with HPV and developing cervical cancer (86.87% visited). These appear to be essential components of communication about HPV vaccination.

The “Value Clarification” component was visited the least overall (13.09%). This may be attributed to the fact that it was not easily found by participants: this page was only accessible once mothers completed the component “Weighing Pros and Cons.” Voncken-Brewster and colleagues [66] also found that participants overlooked a certain part of their intervention, and they successfully improved its visibility by repositioning it. In order to promote exposure to values clarification, we could enhance its visibility.

The “General Information” component was completed the most overall (72.24%). This is likely due to the fact that the mother-like virtual assistant recommended starting with this component when mothers first entered the main menu, and that it was brief (2 pages). “Side Effects” and “Effectivity” were completed the least (both 0.85%). This could be explained by the fact that these two components contained the most links to in-depth information. Only a small proportion of the mothers (≤13.22%) visited in-depth information.

The mothers evaluated the website as positive, as shown by the overall acceptability (7.64/10) and the scores on all acceptability measures (mean scores ranging from 4.34 to 5.71 on a 7-point Likert scale). The virtual assistants were also well appreciated (7.41/10). This is likely because it matched with the mothers’ preferences for more interactive personalized feedback [40]. This adequate acceptability is similar to the findings of Paiva and colleagues [67] for a web-based tailored intervention to increase HPV vaccination among young adult women (3.27/4.0). We believe the high acceptability in the present study can be attributed to the systematic and user-centered development process of the intervention [7]. The target group was extensively involved throughout the development. Not only did we fine-tune the content of the intervention to the mothers’ preferences and requirements but this also was considered in the design of the website, which was chosen by the mothers.

Furthermore, we found a positive relationship between acceptability and completeness (P<.001). This association was also found by others in a web-based intervention for breast cancer [68-70]. This finding underlines the importance of an intervention to be considered appropriate by the target group.

In line with our expectations, completeness had a significant positive effect on all outcome measures (IDM, decisional conflict, social-psychological determinants of HPV vaccination uptake), including actual HPV vaccination uptake itself. Mothers who had completed more of the intervention were more likely to have their daughter vaccinated against HPV, and had higher levels of IDM and more positive scores on determinants of HPV vaccination acceptance. In particular, the effect on HPV vaccination uptake is an important finding, which was not found when simply contrasting the experimental to the control condition [30]. This stresses the importance of conducting a process analysis alongside such an effect evaluation. These results are very promising, considering the currently low HPV vaccination uptake rates in the Netherlands [6].

We believe that these positive effects can be attributed to the extensive tailoring throughout the intervention. Support for this notion comes from the study results indicating that mothers perceived the intervention to be well-tailored (5.22 on a 7-point scale). Not only did we tailor the content of the intervention to the mothers’ personal interest, it was also used to guide the mothers’ personal route through the intervention. The latter is likely to have improved the usability of the intervention. Moreover, the intervention accounted for tailoring on a variety of themes. For example, it considered perceived barriers similar to the approach taken by Gerend and colleagues [9] as well as other beliefs, attitudes, subjective norms, habits, relative effectiveness, anticipated regret, risk perception, self-efficacy, and knowledge.

In contrast to our expectations, time of website use had a positive effect on all outcomes (P<.003), except for risk perception when not vaccinated (P=.14), subjective norms (P=.03), habit (P=.01), and vaccination uptake (P=.20). The latter is also in contrast to findings of a previous study [64]. An explanation for this lack of effects may be the measurement we used for time of website use, which may be less reliable for measuring exposure. We measured the total time spent on the website, but this does not indicate the specific pages the mothers had visited and for how long. In addition, we were unable to determine if there were any timeouts during a session and what the duration of a timeout was, because the intervention was web-based. For instance, they could have been distracted in the home environment during website use, which may have influenced the measured time of website use and may also have caused the lack of effect of acceptability on time of website use. An alternative, more accurate, measure can be found within the domain of education, namely time on task [71]. Future studies are needed to examine whether time on task (instead of total time of website use) has a positive influence on uptake.

There are three methodological considerations. First, as mentioned above, the measurement of total time spent on the website seemed to be not entirely adequate to measure program use. In this study, completeness seemed to be a better indicator of use as it demonstrates the mother’s navigation through the website, whereas time of website use did not. In addition, within (almost) every component, mothers were first asked a question about an HPV-related topic. The page was marked as “completed” if they provided an answer, which was necessary to obtain tailored feedback. Within the decisional balance, completeness was calculated based on the mother’s answer to each statement. These are more complete indicators of actual use. Nevertheless, it is not an entirely complete indicator for use, since we could not measure whether they read generic information, listened to the tailored feedback from the virtual assistant, or saw a video. This may have influenced the dose-response effects found.

Second, considering the aim of initiating active processing of information, the positive effect found of completeness on habit may seem unwanted. However, at follow up, the mothers had already been exposed to the intervention, and were therefore likely to have actively processed the information. These results may indicate that the more mothers completed the intervention, the less they had to think about getting the HPV vaccine. The intervention may have helped them make a decision rather than inducing perceptions about HPV vaccination as something you take for granted, without thinking. In retrospect, the label “habit” appears to be misleading.

Third, the effect sizes found were small, which is in line with other web-based interventions targeting health behavior outcomes [72]. Despite this, we believe that even small effects are of relevance in public health as they become substantial at the population level. After all, in the Netherlands, approximately 100,000 girls are invited to receive the HPV vaccine on a yearly basis.

The most important strength of this study is that we, to our knowledge, are the first within the field of HPV vaccination to conduct an extensive process evaluation. According to a review [24], the process evaluation of an intervention promoting HPV vaccination acceptance by tailoring information to participants’ perceived barriers, such as that performed by Gerend and colleagues [9], was limited to the evaluation of intervention information (eg, the extent to which the information was informative and convincing). Other notable strengths are the objective measurement of HPV vaccination uptake, the dose-response effect of completeness on actual HPV vaccination uptake, the broad focus on outcomes (ie, determinants of HPV vaccination acceptance, uptake, IDM, and decisional conflict), and the adequate level of both (objectively assessed) program use and acceptability.

However, some limitations need to be addressed. First, there appeared to be technical difficulties despite our best effort to minimize such issues (eg, by extensively pilot testing the website using various devices). For instance, it turned out that a portion of the mothers were unable to see or hear the virtual assistants and that certain components did not work adequately among some of the mothers. Such technical difficulties are likely to have influenced website usability and therefore may have had a negative impact on program use. Second, this eHealth intervention requires mothers to have eHealth literacy [73]. People with lower eHealth literacy have been shown to have a lower educational level and to spend less time online [74,75]. It could be interesting to examine the extent to which eHealth literacy might have influenced use, acceptability, and effects. Considering the latter, we do not expect substantial differences since we did not find any differences in intervention effects among mothers with a low vs high educational level [30]. Third, it was hard to compare the study results to other studies because data on program use are often very poorly reported or even completely lacking (for a review, see [20]). Furthermore, the study was subject to a considerable amount of attrition. This is a common finding in studies on eHealth interventions [20]. Dropout appeared higher, for example, among mothers with a lower educational level, which has also been reported for other web-based tailored interventions [76]. In this study, missing data and selective dropout were handled by using multiple imputation [55,56]. Fortunately, there were nearly no differences between the results from the complete case analyses and the ITT analyses. Therefore, it seems unlikely that the effects found are spurious or due to selective dropout. Finally, caution is needed when generalizing the results of this study to the general population (ie, Dutch mothers of 12-year-old girls) because we were unable to check the representativeness of the sample. Besides, it should be noted that a very homogenous sample was reached. An undiversified reach is a known phenomenon within web-based interventions aimed at health promotion [20]. In our study, the mothers were predominantly born in the Netherlands, highly educated, and likely to have a daughter that was vaccinated against HPV. However, results from the effect evaluation showed no differences in intervention effects in specific subgroups of participants, as indicated by moderation analyses with sociodemographic variables [30].

This process evaluation has demonstrated that program use and acceptability of the web-based tailored HPV vaccination intervention were adequate. We found a positive association between completeness and acceptability. Furthermore, dose-response effects were found for completeness and time of website use on (nearly) all social-psychological determinants of the mothers’ decision making about vaccination, levels of IDM, and decisional conflict. Importantly, the extent to which mothers completed the intervention positively influenced their daughters’ actual HPV vaccination uptake. These results indicate that this web-based, tailored intervention fits well with the mothers’ needs and has the potential to increase HPV vaccination uptake. Because of the intervention’s adequate (dose-response) effects and acceptability, the intervention has recently been incorporated into the national HPV vaccination communication alongside the existing communication materials.

We recommend future interventions promoting HPV vaccination acceptance to (1) incorporate a process evaluation alongside the effect evaluation, (2), include strategies to arouse interest so as to expand reach, and to (3) include time on task. In addition, we recommend investigating the influence of different media types (eg, graphic vs nongraphic presentations) on decision making about HPV vaccination. For instance, Cox and colleagues [77] found that parents who viewed a graphic presentation of HPV-related risk information had a higher HPV vaccination intention compared to parents who viewed a nongraphic presentation. Furthermore, we recommend future research to examine differences in use and acceptability in specific subgroups of participants, for example by conducting moderation analyses with sociodemographics (eg, educational level). Finally, we recommend developing similar intervention models for (a) target groups other than mothers (eg, the daughter or sons in case they will become a next target group for HPV vaccination) and (b) other vaccinations (eg, maternal pertussis vaccination).