The methodological pilot study used a sample of children and adolescents registered in the local resident registries of 20 municipalities in 5 federal states of Germany, covering urban and rural areas as well as the eastern and western regions of the country.

Data were collected using SAQ-Web or SAQ-paper methods. All selected individuals were invited by mail to participate in the study. They were sent a cover letter with the invitation to participate, information about the study and data privacy, and an informed consent form. Depending on the allocated mode, the invitation comprised a username and password for participation through the Web option along with a paper questionnaire for those allocated to the concurrent mixed-mode design, only a paper questionnaire in the single-mode design, or only the access data for the online questionnaire in the sequential mixed-mode design. The SAQ-Web questionnaire was only optimized for desktop computers. A reminder was sent by mail to respondents who had not replied within 3 weeks of the initial invitation. Participants who did not respond to the reminder were telephoned up to 5 times 4 weeks after the initial invitation. As an additional motivation for prospective participants, each parent and adolescent who had completed a questionnaire received a shopping voucher to the value of €10. The methodological pilot study strictly adhered to the data protection regulations set out in the German Federal Data Protection Act. Participation in the study was voluntary. All parents and participating adolescents were informed about the study’s aims and content, as well as data protection, and they provided informed consent. Following the strict data privacy protocol, prospective participants between the ages of 11 and 17 years received their questionnaires only after their parents provided consent.

Different questionnaires were used for different age groups. Main health indicators were included on the health questionnaires for parents of all age groups (0-17 years), and self-report data for main health indicators were obtained from adolescents aged 11-17 years. To reduce the risk of mode effects, the 2 questionnaires were designed to be as similar as possible and contained the same wording for the questions and response categories. On the basis of the unified-mode design [27], the wording and formatting of questions and response categories were standardized. To help participants visually distinguish single-choice questions from multiple-choice questions, all survey modes used the same checkbox design. Single-choice checkboxes were round, whereas multiple-choice checkboxes were rectangular. Additionally, multiple-choice questions included the instruction “Multiple entries are possible.” For filter questions, Web-based questionnaires were optimized with filter skips whenever the perceivability of the questions was not impaired. Plausibility checks and ranges were defined for the Web-based questionnaire. Additionally, soft prompting was programmed into the Web-based questionnaire to reduce item nonresponse. These differences were used to capitalize on the advantages of the Web mode for better data quality, and they were the only mode-specific design differences. Detailed information of the survey design and other technical aspects of the Web-based part of the survey are described in a “Checklist for Reporting Results of Internet E-Surveys” [28] (Multimedia Appendix 1).

As shown in Figure 1, a gross sample of 11,140 children and adolescents was randomly allocated to four survey designs:

There were no statically significant differences in the (gross) sample composition across the 4 design groups in terms of known sample characteristics, such as age, sex, municipality size, region, or respondent citizenship, which were obtained from local registries.

The combined response rate for all survey designs was 38.43% (n=4032), following the internationally used Standard Definitions of Outcome Rates for Surveys of the American Association for Public Opinion Research (AAPOR Response Rate 2) [29]. There were no significant differences in response rates among the concurrent mixed-mode design, the sequential mixed-mode design, and the single-mode control group design. However, there was a significantly lower response rate in the preselect mixed-mode design. Detailed comparisons of response rates, sample compositions, data quality, and efforts among the different survey designs have been published previously [26].

For this study, only survey design groups using the same version of the questionnaire were included, with 3468 completed parent-reported health questionnaires for children and adolescents aged 0-17 years and 1194 questionnaires completed by adolescents aged 11-17 years. A response was defined as one completed health questionnaire from either parents or children. Hence, a valid response did not require both parents and children to complete all requested questionnaires. To answer the first research question regarding mode equivalence across the different survey designs, we compared the single-mode control group with each of the 3 mixed-mode groups. To answer the second research question regarding mode equivalence between the 2 data collection modes, data from all survey designs were pooled (Table 1).

A wide range of health status indicators and health behaviors for children and adolescents with high public health relevance were analyzed to identify differences between the mixed-mode designs and the single-mode control group, as well as mode differences between SAQ-paper and SAQ-Web.

The main aim of this study was to examine the risk of mode effects in a mixed-mode health interview survey for children and adolescents that combined paper-and-pencil questionnaires and Web-based questionnaires. Therefore, we compared prevalence rates and mean values of a broad range of health indicators from 3 alternative mixed-mode designs (all combining paper-and-pencil and Web-based questionnaires) with a single-mode control group (paper-and-pencil only). We also compared results between online respondents and paper-and-pencil respondents regardless of the survey design. First, we examined differences in sociodemographic characteristics by survey design and data collection mode, as it is well documented that sociodemographic characteristics are associated with health status and health behavior [43,44]. Regarding survey design, there were no statistically significant differences in sample composition, prevalence rates, or mean values of the examined health indicators. There were differences in sociodemographic characteristics across the data collection mode groups. After adjusting for these differences, only one of the analyzed health indicators (lifetime alcohol consumption) showed between-group differences. These results indicate that there is limited mode bias in health interview surveys for children and adolescents using a mixed-mode design with Web-based and paper questionnaires.

Consistent with previous findings, the sample composition of responding parents and of responding adolescents differed by data collection mode. We confirmed the so-called “digital divide” [47-50]—male adolescents and younger fathers preferred the online mode, a well-known systematic difference [5] between these modes[10,49,51-54]. Additionally, SAQ-Web respondents had higher household incomes [15,49,55] and higher household education levels [10,49,54-57]. Despite these differences, and differences in online response rates between the mixed-mode survey designs, there were no statistically significant differences in sample composition between the paper-and-pencil single-mode control group and the 3 mixed-mode groups. To control for the influence of sociodemographic on health indicators, we adjusted for sociodemographic characteristics by first calculating crude prevalence rates. Then, the analysis was complemented with adjusted prevalence rates or adjusted mean values using predictive margins to identify possible mode effects. Comparisons between the mixed-mode survey designs and the single-mode control group were made using only the crude prevalence rates. Using this approach, hardly any statistically significant differences by data collection mode or by survey design were found for the analyzed health indicators.

Prevalence rates of health complaints, such as diagnosed allergies, diagnosed ADHD, obesity, and chronic pain, were equivalent between the modes, as previous studies of adults [9,11,53,58] and adolescents [12] have shown. A population-based Norwegian study found higher asthma prevalence rates among online respondents; this was interpreted as possible nonresponse bias and not as a mode effect because there were no differences in the prevalence rates for any other condition [59]. A literature review by Hox et al showed that after controlling for selection, small mode effects do appear, most often distinguishing between modes that involve interviewers (face-to-face, telephone) and modes that do not (mail, Web) [42].

We found similar prevalence rates for SRH, chronic diseases, and impairment owing to health problems between SAQ-paper and SAQ-Web respondents. The 2 previous studies examining these health indicators among adults in general [11] and among older adults [10] also found no differences between these 2 data collection modes. Another study of adults interpreted the higher SRH found among online respondents compared with paper-based respondents as an expression of different sample characteristics linked to the digital divide era [49], or a case of better-situated people with better health using Web-based questionnaires, and not as a mode effect. We cannot say whether this holds true for the KiGGS methodological pilot study, because we controlled for most characteristics linked to the preference for online participation, such as region of residence and education or income.

For mental and psychosocial problems, we calculated risk groups for emotional and behavioral problems and for impairment owing to psychosocial problems based on SDQ scores [34]. Both parent- and adolescent-reported scores were equivalent across the examined modes. Several other studies have postulated the comparability of measurement results between these 2 self-administered modes for other standardized mental health questionnaires (eg, depression or anxiety) [12,58,60,61].

In their review of 55 studies investigating 79 instruments, Campbell et al [9] found measurement equivalence for electronic- and paper-based patient outcomes and concluded that standardized instruments can generally be used electronically without measurement effects. In our study, we also found comparable results for standardized instruments (the SDQ and AUDIT-C), as well as for self-reported HRQoL (KIDSCREEN-27). No existing studies have compared these particular instruments, but previous studies have compared the Short Form Health Survey-36, a frequently used standardized HRQoL instrument for adults, and found measurement equivalence [9,58,62-64].

All reports of health care utilization were equivalent between the self-administered modes; this is consistent with prior empirical results, including studies of adult vaccination use [11], adolescent health care use [12], and multiple health care quality indicators [56]. The greater use of pediatric services (and of any doctor) before adjustment for sociodemographic characteristics may be explained by the younger age of children in the online group—in Germany, all children are invited to undergo regular health screening examinations (U3-U9 examinations) from early childhood until the age of 5 years, with a well-established system of reminders and reporting.

Most of the analyzed adolescent health behaviors (current smoking, 12-month water pipe consumption, second-hand smoke exposure, physical activity, and screen-based media use) showed comparable results and no differences between the 2 modes. These results are consistent with the results of other studies on adolescents [12,65].

Considering alcohol consumption, the crude and adjusted prevalence rates for lifetime consumption were significantly higher among SAQ-Web-responding adolescents. After adjusting for sociodemographic characteristics, the difference decreased but could not be explained by the sociodemographic differences between the 2 groups of respondents. The prevalence of hazardous consumption and binge drinking were comparable between data collection modes after controlling for sample composition.

Most previous studies have reported no statistically significant differences in alcohol consumption among adolescents or young adults by these 2 data collection modes [12,66]. However, research comparing sensitive health behaviors is inconsistent. Some studies have found higher adult binge drinking [53] and higher adolescent alcohol consumption [17] in online reports, whereas others have found no difference in sensitive health behaviors in general for college students [67,68] and young adults [69].

The higher rate of reported lifetime alcohol consumption among SAQ-Web-responding adolescents, in the absence of frequently reported hazardous consumption or binge drinking, may be interpreted in multiple ways. For example, this may be a result of different sample properties, such as SAQ-Web-preferring adolescents being more likely to experiment with alcohol consumption. However, it is also possible that this result is a mode effect based on the assumption of identical alcohol consumption in both groups. Web-based questionnaires afford greater privacy because there is no risk of parents checking the responses. Another possible explanation is the lower social orientation in the Internet mode [13]. Both these explanations assume that Web-based questionnaires are more likely to elicit honest reports, but the similar results between the 2 mode groups for reported harmful alcohol consumption after adjustment contradict this assumption. Taken together, the results for alcohol consumption suggest that lifetime consumption should be used with caution as a health indicator in a mixed-mode design. Hazardous consumption and binge drinking are better indicators because they exhibit mode equivalence and have greater public health relevance than lifetime consumption, which is measured by a single question asking whether the respondent has ever consumed alcohol.

Other empirical comparisons of measurement results between different mixed-mode survey designs are rare. In accord with one other result for the adult population [70], all of the analyzed health indicators for children and adolescents showed comparable results, with no statistically significant differences between the single-mode control group and the 3 mixed-mode groups. Additionally, sociodemographic characteristics did not differ by survey design for parents or adolescents. Regarding measurement comparability, any of the tested mixed-mode health interview survey designs, which offer both Web-based and paper questionnaires, could be used for children and adolescents.

The strengths of the methodological pilot study are the randomized study design, the population-based sample, and the inclusion of a single-mode control group as a reference to interpret the results. However, there are also some limitations, predominantly the relatively small size of the net samples of the analyzed groups. Each survey design had a relatively low number of cases, so interpretations of the results based on the net samples must be made with caution. Possible differences across the 4 survey designs or between the 2 data collection modes could have been overlooked because of a lack of statistical power, particularly regarding the need for correction for multiple testing. Other limitations concern the external validity of the results; the study was conducted in a German setting using register-based samples of children and adolescents, so the results are difficult to generalize to other countries, settings, or populations.

Our results are consistent with those of most previous studies. We found comparable results between the 2 self-administered modes (SAQ-Web and SAQ-paper) for almost all analyzed health indicators, except for lifetime consumption of alcohol among adolescents aged 11-17 years. Thus, no differences were found between the single-mode control group design and 3 mixed-mode survey designs that combined the 2 data collection modes.

These results suggest that it is possible to measure health indicators for children and adolescents using a mixed-mode design combining SAQ-Web and SAQ-paper methods, with a low risk of mode effects and high comparability across different mixed-mode survey designs combining these 2 data collection modes [4]. The implementation of a Web-based option in the existing paper-based interview surveys of children and adolescents has a low risk of changed measurement values caused by the mixed-mode survey design.