The questionnaire used in the study is an adaptation of a well-tested questionnaire used by the Department of Epidemiology of the Netherlands Cancer Institute in a large-scale Dutch cohort study on long-term effects of ovarian stimulation for in vitro fertilization [24]. It addresses the following issues: sociodemographic characteristics, medical history, menstrual and reproductive history, pregnancy outcomes, menopausal symptoms and menopause, and family history of cancer and family history of subfertility or infertility.

The paper- and Web-based version of the questionnaire were identical in terms of the questions asked, their wording, and their order of presentation. In the Web-based version, however, questions not relevant to the participant were automatically skipped. The Web-based version of the questionnaire was accessible for participants through a website which was specially designed for the nationwide study.

The study population for this study consisted of 277 female childhood cancer survivors from three participating centers of the nationwide study (Emma Children’s Hospital/ Academic Medical Center Amsterdam, Leiden University Medical Center, University Medical Center Utrecht/Wilhelmina Children's Hospital). These women were randomly allocated to two groups: the mixed invitation group and the Web-only invitations group.

If an envelope appeared undeliverable because of an incorrect or nonexistent postal address, the online telephone directory was used to find the correct address. If this proved unhelpful, vital status and current address were checked by means of the Gemeentelijke Basis Administratie (Dutch Municipal Population Register).

If the questionnaire was not returned within 3 weeks, a reminder was sent by postal mail. For participants in the mixed invitation group, this reminder consisted of a letter in which the relevance of the study was again stressed and in which the individual was asked to respond. For participants in the Web-only invitation group, a paper-based version of the questionnaire was added to this reminder letter. When, after 3 weeks, still no response was received, patients in both groups were contacted by telephone and were asked to respond.

For the purpose of the current study, response time is defined as the time (number of days) elapsed between the day the envelope with the study information package was sent and the day a response was received.

Data were analyzed using SPSS for Windows, version 15.0 (SPSS Inc, Chicago, IL, USA). Descriptive statistics were used to describe differences between (1) participants allocated to the mixed invitation group and the Web-only invitation group, (2) respondents filling out the paper-based questionnaire and the Web-based questionnaire, and (3) responders and nonresponders. Independent samples t tests and Pearson chi-square tests were used to test whether these differences were statistically significant. A P value of less than .05 was considered to be statistically significant.

Multivariable logistic regression analysis was used to predict the probability of filling out the Web-based version of the questionnaire by reporting odds ratios (ORs) and 95% confidence intervals (CIs). A prediction model was developed using a backward selection procedure with a P value of .10 as the criterion for exclusion of variables.

Included in this study were 277 women. Table 1 outlines the response characteristics of the participants allocated to the two randomization groups. Response rates were comparable in both groups (83% in mixed invitation group vs 89% in Web-only invitation group, P = .20). In addition, participation rates—defined as the number of women who were willing to fill out the questionnaire—did not differ significantly between the mixed invitation group and the Web-only invitation group (66% or 90/137 vs 59% or 83/140, respectively, P = .27). Moreover, median response time was comparable in both groups. In the mixed invitation group, significantly more respondents filled out the paper-based questionnaire compared with the Web-only invitation group (83% or 75/90 and 65% or 54/83, respectively, P = .01).

Figure 1 summarizes the participant flow. Overall there were no differences between the responders in the mixed invitation group and the Web-only invitation group with respect to the number of women responding after the initial invitation, after the first reminder, and after the second reminder. However, when the results regarding the timing of the response are related to the type of questionnaire filled out by the respondents, some differences between the two groups appear. Among the group of responders who sent back either type of the questionnaire before the first postal reminder was sent, that is, the “fast responders,” the proportion of responders filling out the paper-based version of the questionnaire was significantly larger in the mixed invitation compared with the Web-only invitation group (74% or 32/43 vs 51% or 18/35, respectively, P = .04). This difference in type of response remained after the first postal reminder (to which a paper-based version of the questionnaire was added) was sent, but it was no longer statistically significant: 91% (30/33) in the mixed invitation group compared with 75% (24/32) in the Web-only invitation group filled out the paper-based version of the questionnaire after being reminded by postal mail (P = .09).

When the 238 women who responded to the study invitation by sending back the questionnaire, the informed consent form, the refusal form, or the entire study information package were compared with the 39 women who did not respond at all, it appeared that these two groups did not significantly differ regarding age, age at diagnosis, or type of diagnosis. The nonresponder group included 20 survivors (12 of 137 or 9% in the mixed invitation group and 8 of 140 or 6% in the Web-only invitation group) whose postal address could not be verified and who could not be contacted by telephone either. It was decided to consider these survivors to be nonresponders. However, they might not be “true” nonresponders since it is not known whether they indeed received the study information package and the postal reminder.

Comparing the 173 women who participated in this study with the 104 women who did not participate (ie, women indicating they refused to participate and women who did not respond) did not reveal significant differences regarding current age or age at diagnosis. However, it appeared that the proportion of women with leukemia was significantly higher in the participant group compared with the nonparticipant group (52% or 90/173 vs 38% or 39/104, respectively, P = .02) (data not shown).

Table 2 shows the characteristics of the 173 women who returned a questionnaire. It appeared that the 44 women who filled out the Web-based version of the questionnaire were more likely to have had a higher educational level than the 129 women who filled out the paper-based version (P = .01). No differences were found regarding age, type of diagnosis, age at diagnosis, employment status, or marital status.

Among the group of respondents filling out the questionnaire, it was investigated which factors influenced the probability of filling out either the paper- or the Web-based version of the questionnaire. Table 3 shows the odds ratios and 95% confidence intervals (CI) for the variables in the final model of the logistic regression analysis. Age, educational level, employment status, and randomization group were significant factors influencing the probability of filling out the Web-based questionnaire. More specifically, the probability of filling out the Web-based questionnaire was higher for participants allocated to the Web-based invitation group for participants who were older, and for participants with a higher educational level. Finally, students appeared to have a higher probability of filling out the Web-based questionnaire compared with participants who were employed.

In the present study, we examined differences in response between female childhood cancer survivors who received either a mixed invitation (paper-based questionnaire together with log-in details for Web-based questionnaire) or a Web-only invitation (log-in details only). The results show that survivors receiving the mixed invitation preferred filling out the paper-based version instead of the Web-based questionnaire as compared with the survivors receiving the Web-only invitation. Thus, when a paper-based version of the questionnaire was added to an invitation in which also the possibility of filling out the Web-based version was mentioned, the survivors were more likely to choose the paper-based option. Moreover, when the results regarding the timing of the response are taken into account this finding is endorsed since a large proportion (75%, 24/32) of females who initially received the log-in details only responded by filling out the paper-based questionnaire after they received a postal reminder (3 weeks later) to which a paper-based version of the questionnaire was added. This proportion is comparable to the proportion of females filling out the paper-based questionnaire immediately after the invitation (ie, before the postal reminder) among those who initially received the log-in details together with the paper-based version of the questionnaire (74%, 24/32).

To our knowledge, no studies are available that have compared response rates to a Web- and paper-based version of a questionnaire on reproductive and fertility issues among young adult women. However, a few studies are available that have evaluated these issues by means of a Web-based questionnaire only. In a group of female survivors of breast cancer, the response rate to this type of questionnaire was 51% [27,28] whereas in a group of women aged 17 to 21 years, this rate was 72% [29]. However, no information was provided regarding characteristics of the nonresponder group.

In our study, the overall response rates in the mixed invitation group and the Web-only invitation group did not differ. This result is in line with the results found in the study of Quigley et al [25]. In the study by Quigley et al, military personnel were requested to participate in a survey on information services. In one study group, a paper-based questionnaire was used with an added option of completing the questionnaire via the Internet. In the other study group, an Internet-based questionnaire was used with an added option of completing a paper version of the questionnaire by mail. Although response rates in both study groups were lower (42% and 37% respectively) than the response rates found in our study, differences in response rates between the two groups were not found, as was the case in our study. Furthermore, of the participants receiving the paper-based questionnaire with the Internet option, 77% chose to complete paper-based questionnaire. In our study, a similar proportion of participants in the mixed invitation group filled out the paper-based questionnaire, that is, 83%. However, Quigley et al found that of the participants receiving the Web-based questionnaire with the option of the paper-based version, 73% chose to complete the Web-based questionnaire, while in our study the proportion of women in the Web-only invitation group who filled out the Web-based questionnaire was much lower (35%).

Furthermore, the participation rates (ie, the proportion of women who filled out the questionnaire) measured in our study can be considered as being rather high (66% in the mixed invitation group and 59% in the Web-only invitation group). In other studies using Web-based questionnaires in combination with paper-based versions these rates are, in general, lower [17,25,30]. A possible explanation for the high response rates found in our study might be the salience of the study topic. It is known that potential participants are more likely to respond to both paper-based and Web-based surveys when the salience of the topic, defined as the degree to which the topic is of interest or is relevant for participants, is high [5,12,31]. Moreover, the questionnaire used in the current study was one of the three study components used in a nationwide study on fertility issues in female childhood cancer survivors, with the other two study components being the provision of a blood sample and a transvaginal ultrasound measurement of the reproductive organs. It is known that female survivors of childhood cancer are in need of information regarding their reproductive function [32,33]. Therefore, participation in this study might be appealing for a large group of the invited females, resulting in higher response rates compared with studies in which a questionnaire is the only measurement instrument used.

Our results show that the use of reminders improved the response rates substantially. After the first reminder (a letter sent by postal mail), the response almost doubled in both randomization groups. Other studies support our finding that both postal and telephone reminders are effective in increasing response rates for both Web-based surveys as well as traditional paper-based surveys [16,34,35].

In our study, the majority of the respondents preferred filling out the paper-based version of the questionnaire over filling out the Web-based version. Moreover, age, educational level, and employment status appeared to be important factors influencing the decision to fill out either version of the questionnaire. Our finding that women with a high education level as well as students tended to choose Web-based questionnaires over paper-based questionnaires is in line with previously published results [5,30,36]. However, results of the present study could not endorse other study results stating that Web-based questionnaires are more likely to attract younger respondents than paper-based questionnaires [6,36].

Another factor that may have played a role in the decision of the respondents to fill out either the paper-based or the Web-based version of the questionnaire is the length of the questionnaire used in the current study, which was rather long. The paper-based version consisted of 122 questions covering 32 pages. The Web-based questionnaire required several computer screens, with the number of questions on one screen depending on the type and length of the questions. For women filling out the Web-based version of the questionnaire, the median (IQR) time spent on filling out the questionnaire, which was automatically registered by the Web-based questionnaire tool, was 42.7 minutes (28.7 minutes to 67.8 minutes). Unfortunately, in the current study these data were not collected for the group of women filling out the paper-based version of the questionnaire. However, in the larger nationwide study, of which this study is part, a question was added to the paper-based questionnaire at a later point in time asking how much time was spent filling out the questionnaire. Median (IQR) time spent in this group (n = 145) was 30.0 minutes (30.0 minutes to 60.0 minutes) minutes. Thus, although this information was recorded among a different group of participants, it seems that filling out the paper-based version of the questionnaire took less time compared with the Web-based version. Various studies have shown the length of both Web-based and paper-based questionnaires to be negatively related to response rates [31,37-39]. However, no literature is available on differences in response rates to paper-based and Web-based questionnaires related to the length of a questionnaire. The results of our study seem to indicate that people tend to choose the paper-based version of a questionnaire when it concerns a long questionnaire. However, whether shorter surveys result in higher response rates when they are offered through the Web and longer surveys result in higher response rates when offered on paper needs further investigation.

In addition, the topic of our questionnaire can be considered to be rather personal. It is known that questionnaires containing questions of a sensitive nature result in lower response rates [31,38]. Although to our knowledge, studies investigating differences in response rates to paper-based and Web-based questionnaires taking into account the degree of sensitivity of the questions are lacking, one could assume that the sensitivity of our topic may have resulted in more respondents filling out the paper-based questionnaire especially since it is known that respondents filling out questionnaires through the Internet have doubts about their privacy and the confidentiality of their responses [10,40]. Despite the fact that data security and confidentiality were stressed in the letter accompanying our Web-based questionnaire, this might have led to more women filling out the paper-based questionnaire.

An important limitation of our study is the generalizability of the results found. Our study population mainly consisted of relatively young women, and thus the results may be less representative of older age groups or mixed groups including males. Moreover, our study population represents a rather unique clinic population, that is, long-term survivors of childhood cancer. In addition, the topic of the questionnaire used cannot be considered a conventional subject. Therefore, caution should be exercised when translating the results found in the current study to other study groups or other study topics. Furthermore, the available data on the nonresponders in the present study were limited to age, age at diagnosis, and type of diagnosis. As a consequence, potential bias introduced due to nonresponse, also influencing the generalizability, could not be investigated extensively. However, in many of the studies using both paper-based as well as Web-based questionnaires, data on nonresponders are not available at all. As nonresponse to surveys seems to be increasing in recent years [41,42], future studies investigating the degree of bias as well as its consequences for the interpretation of data collected by paper-based and Web-based questionnaires are of great importance.

Survivors of childhood cancer from this era represent a highly mobile group, and they may not be as available or as responsive to contact by traditional mail methods [43]. Successful recruitment of this population will require new methods of contact such as email and Web-based methods. Therefore, although our findings indicate that most survivors preferred the paper-based version over the Web-based version when offered both, we conclude that Web-based questionnaires are promising data collection tools for childhood cancer survivors. However, researchers should carefully weigh the methodological benefits and barriers of using either a paper-based or a Web-based questionnaire for this group of subjects, taking into account possible response bias.