In the Netherlands, health care policy stresses regulated competition between health care providers [1]. Efforts are made to enhance the transparency of health care quality, to stimulate informed decision making among consumers, and to improve the performance of health care providers. Comparative information about the performance of health care providers is needed for consumers to make informed decisions. This comparative information can be gathered in different ways. One possibility is to ask a sample of patients about their actual experiences concerning quality of care provided by health care providers.

Measuring the quality of care from the patients’ perspective has been standardized in the Netherlands since 2006, using a new instrument called the Consumer Quality Index (CQ-index or CQI) [2]. CQI questionnaires are usually self-administered paper questionnaires (eg, CQI Rheumatoid Arthritis [3], CQI Breast Care [4]). Individual structured interviews are conducted in cases where a self-administered paper questionnaire is not feasible because of respondents’ visual, physical, or cognitive limitations (CQI Care for the Disabled [5], CQI Long-Term Care [6]). Postal surveys (with multiple reminders) and interviews are relatively expensive and time consuming. It would therefore be interesting to know whether other data collection methods can be applied in this field.

The Internet is increasingly considered to be an efficient medium for assessing quality of care from a patient’s perspective. In populations that already use the Internet, Internet surveys have been found to be a useful means of conducting research [7-9]. Efficiency gains are found in shorter response times and field costs reductions (50%–80%) [10-12]. In contrast to paper questionnaires, Internet questionnaires can contain various interactive features that allow complex skip patterns that are invisible to respondents, and the Internet allows validation of responses by using an instant feedback function while respondents are still online [12,13]. Consequently, the quality of data collected with an Internet survey is higher. Some Internet surveys have shown promising response rates (up to 94% in Web forums) [10,12,14]. The extreme response in Web forums can be explained by a probable selection bias in these studies. Those who participate in Web forums are most likely people who are familiar with the Internet and frequently use the Internet, leading to a higher response rate to Internet questionnaires. This high response rate has not been realized in other studies; response rates ranged from 17% to 70% [15]. In CQI research, the response rate to paper questionnaires varied from 20% to 79% with an average response rate of 55% [16]. One CQI study compared an Internet questionnaire with a paper questionnaire. The response rate to the Internet questionnaire (8%) was considerably lower than to the paper questionnaire (35%) [17,18]. To increase the response rate one can send a prenotification or reminders, give an incentive, or use short questionnaires. A salient subject of a questionnaire also increases the response rate [19].

The potential of Internet surveys should, however, be balanced against an equally large weakness. The Netherlands has the largest percentage of households with Internet access in the European Union, but there are still 1.2 million Dutch people (7.3% of the population) with no Internet access at home and 0.5 million Dutch people (3.1% of the population) who do not use the Internet [20]. People who use the Internet are more affluent, better educated, more often male, and younger than people who do not use Internet. Only a part of the population can thus be reached through the Internet [10,11,21]. To compensate for the selection of people in an Internet survey, a combination of data collection methods can be used such as combining an Internet questionnaire with a more traditional postal follow-up [19].

It is known that the way questionnaire are administered has an effect on answers of respondents (so-called mode effects). For example, telephone respondents were found to be more likely to rate health care positively and their own health status negatively than postal respondents [22,23]. This finding is similar to a study where telephone respondents provide more positive ratings than Web respondents [24]. Another example is that students who completed a Web-based questionnaire responded more favorably on different scales (such as college challenge and learning, education, and personal and social gains) than students who filled out a paper questionnaire [25]. It is suggested that computer anxiety affects participants’ responses. Moreover, biases could occur in the way people perceive and process questions presented on screen versus on paper. A study that tested the difference in test–retest reliability and internal consistency between Internet and paper versions of the SF-36, however, found little or no evidence for mode effects [26]. Knowing that these mode effects exist, it is important to investigate whether the answers of respondents in a postal and mixed-mode survey differ.

To examine whether a mixed-mode survey can be an alternative to postal survey, our research question is “What are the differences between a mixed-mode survey (Internet with paper follow-up) and a more traditional postal survey in terms of respondent characteristics, response rates and time, quality of data, costs, and mode effects?” The differences were examined within a sample of breast care patients who reported their experiences with health care using the CQI Breast Care questionnaire.

This study examined whether a mixed-mode survey (Internet questionnaire with paper follow-up) is an alternative to the more traditional postal survey. The results showed that combining an Internet questionnaire with a paper follow-up improved the quality of data and was less expensive than a postal survey. However, the time before questionnaires were received was longer in the mixed-mode survey. No differences between the mixed-mode survey and postal survey were found concerning respondent characteristics, response rates, and global ratings of different health care providers.

The findings showed that the characteristics of the respondents were the same for the two surveys. This means that mixed-mode surveys attract the same population as postal surveys. In total, 53% of respondents in the mixed-mode survey (128/242) filled out the questionnaire online. It appeared that in the mixed-mode survey Internet respondents were younger and more often highly educated and that they reported better psychological health than paper respondents. The younger people probably were more familiar with the Internet and were more likely to have access to the Internet than older people [11,21]. To overcome the problem of possible exclusion of the elderly and less highly educated, a mixed-mode survey should be chosen rather than an Internet survey [11,29].

The response rate was relatively high for both surveys (over 60%). In other CQI surveys, the response rates varied between 20% and 79% [16]. Perhaps the relatively high response rate is due the subject under study, namely abnormality of the breast. The response rate among women referred for mammography in another study was comparably high, both for the Internet (64%) and for the paper questionnaire (77%) [26]. Breast abnormality is a disease that has a huge impact on the emotional and physical quality of life of patients [30]. A review showed that saliency of the subject of questionnaires yields higher response rates [19]. Our results confirm the result of that review. In the mixed-mode survey, the response rate for patients with breast cancer was higher than the response rate for patients with benign abnormalities, even though the questionnaire for breast cancer was longer.

The response time for the questionnaires to be returned was longer in the mixed-mode survey than in the postal survey. This effect was unexpected because using the Internet can reduce the time taken to return a questionnaire [10-12]. Both groups in the mixed-mode survey (paper and Internet respondents) responded relatively quickly (median number of days 7 and 9 days, respectively), but respondents with no access to or interest in the Internet questionnaire only responded after 4 weeks when the paper questionnaire was sent. The relatively quick response by postal respondents in the mixed-mode survey could be explained by the fact that respondents had already been informed about the study. Use of prenotification has been shown to shorten response times [19,31]. Another method to reduce the return time is sending the paper questionnaire out earlier.

Research has shown that an Internet survey results in more complete data compared with a postal survey [32]. This conclusion is confirmed in our study; the quality of data was higher in the mixed-mode survey than in the postal survey. One of the advantages of using the Internet for survey research is the technique of designing questionnaires so that complex skip patterns are invisible to respondents. As a consequence, the online questionnaire resulted in zero missing items and zero invalid answers (eg, answers to questions that had to be skipped). However, given the fact that some groups of people are underrepresented on the Internet (for instance, the elderly), conducting surveys through the Internet alone is not (yet) possible [11,21].

One of the key potential advantages of using the Internet over paper questionnaires is reducing costs. This study showed that the costs per returned questionnaire was €2 lower in the mixed-mode survey than in the postal survey. In the present study, the information technology costs were, however, relatively high for the mixed-mode survey. This was due to the need to program two applications, one for scanning the paper questionnaires and one for the Internet questionnaires. In the future, more costs can possibly be saved by using one and the same program for the different data collection methods within a mixed-mode survey. In addition, the variable costs per questionnaire were lower and the fixed costs per questionnaire were higher in the mixed-method survey than in the postal survey. Fixed costs per questionnaire can be reduced if a larger sample is taken, because the fixed activities are divided over the number of returned questionnaires. In other words, the larger the sample, the more money can be saved by using a mixed-mode survey.

Our study was the first to examine so-called mode effects between a mixed-method survey (Internet with paper follow-up) and postal survey. We found no differences between the two surveys concerning global ratings respondents gave to different health care providers. This is beneficial, because it implies that there is no bias in the scores that is a function of the manner of data collection. Other studies did find mode effects between the answers of telephone respondents and postal respondents [23], Internet respondents and telephone respondents [24], and Internet respondents and postal respondents [25,28,33]. One study investigated the differences between a postal and an Internet questionnaire, where a subset of the participants filled out also the alternative version (Internet and paper questionnaire, respectively). They found little or no evidence for a difference in test–retest reliability and internal consistency when they compared the Internet and paper versions of the questionnaire [26].

We did not ask why respondents in the mixed-mode survey did not fill out the questionnaire online. In one study among nonrespondents of an Internet questionnaire, the nonrespondents indicated that they did not have a computer or access to the Internet. Other reasons were having no experience with the Internet or not trusting the Internet [31]. This corresponds with findings by other researchers, who showed that factors influencing response times are privacy concerns and computer anxiety [19,28,33