Background: Electronic applications are increasingly being used in hospitals for numerous purposes.
Objective: Our aim was to assess differences in the characteristics of patients who choose paper versus electronic questionnaires and to evaluate the data quality of both approaches.
Methods: Between October 2012 and June 2013, 136 patients participated in a study on diagnosis-induced stress and anxiety. Patients were asked to fill out questionnaires at six different moments during the diagnostic phase. They were given the opportunity to fill out the questionnaires on paper or electronically (a combination of tablet and Web-based questionnaires). Demographic characteristics and completeness of returned data were compared between groups.
Results: Nearly two-thirds of patients (88/136, 64.7%) chose to fill out the questionnaires on paper, and just over a third (48/136, 35.3%) preferred the electronic option. Patients choosing electronic questionnaires were significantly younger (mean 47.3 years vs mean 53.5 in the paper group, P=.01) and higher educated (P=.004). There was significantly more missing information (ie, at least one question not answered) in the paper group during the diagnostic day compared to the electronic group (using a tablet) (28/88 vs 1/48, P<.001). However, in the week after the diagnostic day, missing information was significantly higher in the electronic group (Web-based questionnaires) compared to the paper group (41/48 vs 38/88, P<.001).
Conclusions: Younger patients and patients with a higher level of education have a preference towards filling out questionnaires electronically. In the hospital, a tablet is an excellent medium for patients to fill out questionnaires with very little missing information. However, for filling out questionnaires at home, paper questionnaires resulted in a better response than Web-based questionnaires.
With the evolution of modern technology, electronic applications are increasingly being used in hospitals. Web-based applications and touchscreen devices are finding their way into hospitals for numerous purposes. These electronic applications can be useful for research purposes, for collecting patient-reported outcomes, and questionnaires [- ]. Some of the most important advantages of electronic over paper questionnaires include easy usage and immediate electronic storage of results. The use of electronic applications has been evaluated for informed consenting procedures, assessing quality of life, medical education, interventions, diagnostics, and filling out questionnaires [ , - ].
Obtaining high response rates without missing information is important for research purposes, as non-responders can bias study results . Response rates have been found to be lower using electronic questionnaires compared to paper questionnaires [ - ]. In order to potentially improve response rates, specific patient subgroups with a preference for electronic questionnaires could be identified. For example, elderly patients may not be as experienced with electronic applications. Aiello et al compared the use of a tablet to paper questionnaires in a mammography clinic. They found that older women (>60 years) had a slightly harder time learning to use the tablet compared to younger patients, but preference towards the tablet was similar in both groups [ ].
The aim of our study was to assess differences in demographic characteristics of patients choosing paper versus electronic questionnaires and to evaluate data quality and completeness of data of both approaches.
This study was performed in the University Medical Center Utrecht, the Netherlands (approximate caseload of 180 newly diagnosed breast cancer patients per year). In 2011, same-day diagnosis for breast cancer was introduced with the aim to provide a definitive diagnosis within one day in over 80% of patients. Reducing the time of uncertainty about a diagnosis could potentially reduce anxiety and stress. All patients suspected of breast cancer visited the outpatient breast clinic and underwent physical examination, diagnostic imaging (mammography and ultrasound) with a histological biopsy if indicated, and received a final diagnosis at the end of the day after a multidisciplinary meeting.
Between mid-October 2012 and June 2013, all patients referred to the same-day diagnosis out-patient breast clinic were eligible to participate in the study. Approval for this study was obtained from the local ethics committee, and all patients signed written informed consent. All patients were asked to fill out the 6-item State Trait Anxiety Inventory (STAI) [, ] questionnaire at six different time points (measuring moments) during the diagnostic phase to evaluate levels of stress and anxiety ( ). Patients were given the opportunity to fill out the questionnaires electronically ( ) or on paper ( ). Preference towards paper or electronic questionnaires was measured at baseline. The paper questionnaires were returned by mail. In the electronic scenario, the first three questionnaires (administered on the day of diagnosis in the hospital) were offered by means of tablets (iPad). For the last three electronic questionnaires that were to be filled out at home, we used Web-based (hypertext markup language [HTML]) questionnaires. An email with login information to the questionnaires was sent to participants by email on the diagnostic day. The STAI questionnaire was displayed on one page, and all six questions needed to be answered before the form could be submitted. If a question was left blank, an automated message appeared saying that all questions needed to be answered. Patients were not able to look back at previously completed questionnaires. The tablets were also used for providing information and entertainment. An information app was built to provide information on the diagnostic process, diagnostic procedures, treatment team, and routing during the diagnostic day. Several forms of entertainment were available on the tablet, including digital newspapers, magazines, games, and music. The paper questionnaires were returned by mail in a pre-stamped return envelope.
Outcome measures included differences in demographic characteristics between patients choosing paper or electronic questionnaires and data quality, focusing on age, reason for referral, breast cancer history, level of education, and baseline anxiety. Data quality was assessed by focusing on missing information, defined as a questionnaire containing at least one unanswered question. To assess if a breast cancer diagnosis affected the quality of data, subgroup analysis including only patients with a benign diagnosis was performed.
Methods for Data Analysis
Demographics, history of breast disease, and diagnostic findings were described as proportions and means with standard deviation. Differences in demographic characteristics, reported anxiety score, and completeness of reported data between the electronic and the paper group were compared by means of chi-square test and independent samples t test, where appropriate. Significant differences were defined as P values of .05 or less. All statistical analyses were performed using SPSS version 20.0.
Of 321 patients referred to our out-patient breast clinic, 136 patients (42.4%) agreed to participate in the study. All patients were offered the choice of paper or electronic questionnaires.
The mean age was 51.3 years (range 18-85 years) and 35.3% (48/136) patients chose to fill out the questionnaires electronically (). Reason for referral, family history of breast cancer, and breast-related medical history were similar in both groups. Baseline anxiety scores (as measured by the STAI) did not differ between the groups (46.4 in the paper group versus 43.8 in the electronic group, P=.30). Diagnostic imaging findings and proportion of patients undergoing biopsy were similar in both groups. Patients choosing to fill out questionnaires electronically were significantly younger compared to those opting for paper questionnaires (47.3 years vs 53.5, respectively; P=.01) and had a higher level of education (P=.004).
Outcome Data: Missing Information
There was significantly more missing information (ie, questionnaires containing at least one unanswered question) in the paper group during the diagnostic day (measuring moments 1-3) compared to the electronic group (28/88 vs 1/48, P<.001) (). In the paper group, this included two patients who did not fill out one or two questions (instead of complete questionnaires not filled out).
In the week after the diagnostic day (measuring moments 4-6), missing information was significantly more prevalent in the electronic group (41/48, 85%) compared to the paper group (38/88, 43%) (P<.001). This included 7 patients in the paper group who left one or two questions unanswered. These differences persisted in subgroup analysis including only patients with a benign diagnosis.
|Characteristics||Paper, n (%)||Electronic, n (%)||P valuea|
|Female||86 (98)||47 (98)||.94b|
|Age in years, mean (SD), range||53.5 (14.1), 22-85||47.3 (11.4), 18-66||.01c|
|Reason for referral|
|Screening||27 (31)||16 (37)||.75b|
|Palpable lesion/symptoms||61 (69)||32 (67)|
|Positive family history of breast cancer – yes||34 (39)||13 (27)||.18b|
|Previous breast disease/complaints – yes||28 (32)||19 (40)||.36b|
|Previous breast cancer diagnosis – yes||4 (5)||3 (6)||.67 b|
|Breast Imaging Reporting and Data System (BI-RADS) classification suspected lesion|
|I||18 (21)||9 (19)||.54b|
|II||34 (39)||22 (46)|
|III||9 (10)||9 (19)|
|IV||15 (17)||5 (10)|
|V||10 (11)||3 (6)|
|VI||1 (1)||0 (0)|
|No imaging performed||1 (1)||0 (0)|
|No||51 (58)||29 (60)||.62b|
|Yes, histology||32 (36)||18 (38)|
|Yes, cytology||5 (6)||1 (2)|
|Cancer – yes||16 (18)||7 (15)||.59b|
|Level of educationd|
|Low-moderatee||21 (26)||1 (3)||.004b|
|Moderate-highf||27 (33)||10 (29)|
|Highg||33 (41)||24 (69)|
|Mean baseline anxiety score, STAI (SD)||46.4 (13.3)||43.8 (13.5)||.30c|
aP values are based on valid proportions.
bCalculated by chi-square test.
cCalculated by independent samples t test.
dLevel of education is based on the Dutch educational system
eLow-moderate education includes primary education/low pre-vocational/secondary general education.
fModerate-high education includes secondary vocational/higher general and pre-university education.
gHigh education includes higher vocational education/university.
|Paper, n (%)||Electronic, n (%)||P value|
|Baseline – moment 1 – STAI 1||25 (28)||0 (0)|
|Measuring moment 2 – STAI 2||24 (27)||1 (2)|
|Measuring moment 3 – STAI 3||25 (28)||1 (2)|
|Measuring moment 4 – STAI 4||28 (32)||27 (56)|
|Measuring moment 5 – STAI 5||31 (35)||34 (71)|
|Measuring moment 6 – STAI 6||35 (40)||33 (69)|
|Measuring moment 1-3 – in hospitala||28 (32)||1 (2)||<.001|
|Measuring moment 4-6 – at homeb||38 (43)||41 (85)||<.001|
aIncludes all patients with at least 1 incomplete questionnaire in measuring moment 1, 2, or 3.
bIncludes all patients with at least 1 incomplete questionnaire in measuring moment 4, 5, or 6.
The use of tablets and Web-based questionnaires for collection of patient-reported data has many potential advantages over the use of paper questionnaires. Still, the present study shows that a majority of patients preferred paper over electronic questionnaires. Younger patients and those with a higher level of education were more likely to opt for electronic questionnaires. When using tablets (during the diagnostic day in the hospital), more complete information was collected compared to using paper questionnaires. These data suggests that tablets are superior to paper questionnaires. However, the use of Web-based questionnaires resulted in less complete data collection than paper questionnaires. This might be due to the study design where patients could fill out the electronic questionnaires only on a specific day.
A major advantage of filling out electronic questionnaires is that information is immediately saved. Other advantages of the use of tablets include the possibility of automatically reminding the patient to fill out the questionnaires, and providing information and entertainment. We did not electronically remind patients by email to fill out the questionnaires. Considering the high percentage of incompletely filled out Web-based questionnaires (85%), we would definitively incorporate this in a future study. We did use an automated message when not all questions were answered. This resulted in completely filled out questionnaires in the electronic group, which could possibly lead to more complete data.
Possible drawbacks of using Web-based questionnaires are high non-response rates, impaired reliability and validity, and safety or confidentiality issues . Drawbacks of tablets are the need for upgrades, wireless network unreliability, hardware theft [ ], and costs. Fritz et al performed a cost-effectiveness analysis comparing the costs of electronic questionnaires offered on a tablet with paper questionnaires. They found the break-even point to be at 1737 paper sheets per year [ ].
Completeness of data collection was very high in the tablet group, with only 1 of the 48 patients not filling out all questions at the first three measuring moments. Missing information was highest in the Web-based group, where many patients (41/48) did not fill out all questions at the last three measuring moments. One likely reason for the high rate of missing information in this group was that patients could fill out the questionnaires only on the correct day (ie, exactly 1, 3, or 7 days after the patient’s visit). Our aim was to measure patients’ anxiety at specific moments in time, and we limited the possibility of filling out the questionnaires to the correct day only. Patients in the paper group, however, were able to fill out the questionnaires at any given time. This led to more missing information in the Web-based group, and results on missing information need to be interpreted with this information in mind. However, limiting patients to one specific moment to fill out the questionnaires might lead to more accurate measurements of patients’ anxiety at that specific moment. For higher response in the Web-based group, automated email reminders could be useful.
A possible limitation of this study was that we included only breast cancer patients and consequently, 98% were female. These results are therefore not generalizable to other populations. There could be reduced missing data in the electronic group when other groups are included in a similar study (eg, men, young adults).
Younger patients and patients with a higher level of education have a preference towards filling out questionnaires electronically. In the hospital, a tablet is an excellent medium for patients to fill out questionnaires with very little missing information. However, for filling out questionnaires at home, paper questionnaires result in a better response compared to Web-based questionnaires.
The authors thank Peter Sommer and Pieter Ruijssenaars from BeagleBoxx for their cooperation and providing the software and hardware for measuring patient impact of same-day diagnosis.
Conflicts of Interest
- Fritz F, Balhorn S, Riek M, Breil B, Dugas M. Qualitative and quantitative evaluation of EHR-integrated mobile patient questionnaires regarding usability and cost-efficiency. Int J Med Inform 2012 May;81(5):303-313. [CrossRef] [Medline]
- Aiello EJ, Taplin S, Reid R, Hobbs M, Seger D, Kamel H, et al. In a randomized controlled trial, patients preferred electronic data collection of breast cancer risk-factor information in a mammography setting. J Clin Epidemiol 2006 Jan;59(1):77-81. [CrossRef] [Medline]
- Duracinsky M, Lalanne C, Goujard C, Herrmann S, Cheung-Lung C, Brosseau JP, et al. Electronic versus paper-based assessment of health-related quality of life specific to HIV disease: reliability study of the PROQOL-HIV questionnaire. J Med Internet Res 2014;16(4):e115 [FREE Full text] [CrossRef] [Medline]
- Chalil Madathil K, Koikkara R, Obeid J, Greenstein JS, Sanderson IC, Fryar K, et al. An investigation of the efficacy of electronic consenting interfaces of research permissions management system in a hospital setting. Int J Med Inform 2013 Sep;82(9):854-863 [FREE Full text] [CrossRef] [Medline]
- George P, Dumenco L, Doyle R, Dollase R. Incorporating iPads into a preclinical curriculum: a pilot study. Med Teach 2013;35(3):226-230. [CrossRef] [Medline]
- Dorr M, Lesmes LA, Lu ZL, Bex PJ. Rapid and reliable assessment of the contrast sensitivity function on an iPad. Invest Ophthalmol Vis Sci 2013 Nov;54(12):7266-7273. [CrossRef] [Medline]
- Glaser D, Jain S, Kortum P. Benefits of a physician-facing tablet presentation of patient symptom data: comparing paper and electronic formats. BMC Med Inform Decis Mak 2013;13:99 [FREE Full text] [CrossRef] [Medline]
- Mayfield CH, Ohara PT, O'Sullivan PS. Perceptions of a mobile technology on learning strategies in the anatomy laboratory. Anat Sci Educ 2013;6(2):81-89. [CrossRef] [Medline]
- Geraedts AS, Kleiboer AM, Twisk J, Wiezer NM, van Mechelen W, Cuijpers P. Long-term results of a web-based guided self-help intervention for employees with depressive symptoms: randomized controlled trial. J Med Internet Res 2014;16(7):e168 [FREE Full text] [CrossRef] [Medline]
- Lalloo C, Kumbhare D, Stinson JN, Henry JL. Pain-QuILT: clinical feasibility of a web-based visual pain assessment tool in adults with chronic pain. J Med Internet Res 2014;16(5):e127 [FREE Full text] [CrossRef] [Medline]
- Hanberger L, Ludvigsson J, Nordfeldt S. Use of a web 2.0 portal to improve education and communication in young patients with families: randomized controlled trial. J Med Internet Res 2013;15(8):e175 [FREE Full text] [CrossRef] [Medline]
- Partin MR, Malone M, Winnett M, Slater J, Bar-Cohen A, Caplan L. The impact of survey nonresponse bias on conclusions drawn from a mammography intervention trial. J Clin Epidemiol 2003 Sep;56(9):867-873. [Medline]
- Hohwü L, Lyshol H, Gissler M, Jonsson SH, Petzold M, Obel C. Web-based versus traditional paper questionnaires: a mixed-mode survey with a Nordic perspective. J Med Internet Res 2013;15(8):e173 [FREE Full text] [CrossRef] [Medline]
- Kongsved SM, Basnov M, Holm-Christensen K, Hjollund NH. Response rate and completeness of questionnaires: a randomized study of Internet versus paper-and-pencil versions. J Med Internet Res 2007;9(3):e25 [FREE Full text] [CrossRef] [Medline]
- Leece P, Bhandari M, Sprague S, Swiontkowski MF, Schemitsch EH, Tornetta P, et al. Internet versus mailed questionnaires: a controlled comparison. J Med Internet Res 2004 Oct 29;6(4):e39 [FREE Full text] [CrossRef] [Medline]
- Marteau TM, Bekker H. The development of a six-item short-form of the state scale of the Spielberger State-Trait Anxiety Inventory (STAI). Br J Clin Psychol 1992 Sep;31 ( Pt 3):301-306. [Medline]
- van der Bij AK, de Weerd S, Cikot RJ, Steegers EA, Braspenning JC. Validation of the Dutch short form of the state scale of the Spielberger State-Trait Anxiety Inventory: considerations for usage in screening outcomes. Community Genet 2003;6(2):84-87. [Medline]
- van Gelder MM, Bretveld RW, Roeleveld N. Web-based questionnaires: the future in epidemiology? Am J Epidemiol 2010 Dec 1;172(11):1292-1298 [FREE Full text] [CrossRef] [Medline]
- McDonnell CH, Seidenwurm DJ, McDonnell DE, Bobolis KA. Self administered screening for hereditary cancers in conjunction with mammography and ultrasound. Fam Cancer 2013 Dec;12(4):651-656. [CrossRef] [Medline]
|BI-RADS: Breast Imaging Reporting and Data System|
|HTML: hypertext markup language|
|STAI: State Trait Anxiety Inventory|
Edited by G Eysenbach; submitted 03.06.14; peer-reviewed by F Fritz, L Hohwü, C Schäfer; comments to author 17.07.14; revised version received 14.08.14; accepted 23.08.14; published 31.10.14Copyright
©Maarten W Barentsz, Hester Wessels, Paul J van Diest, Ruud M Pijnappel, Cees Haaring, Carmen C van der Pol, Arjen J Witkamp, Maurice A van den Bosch, Helena M Verkooijen. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 31.10.2014.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.