Surveys facilitate data collection on unobservable constructs such as symptoms, psychological disorders, and patient experiences. Surveys also enable the collection of important patient information such as health behaviors, and knowledge and understanding of conditions, which may be otherwise difficult to obtain. In the inpatient setting, surveys have been used to assess patient satisfaction [1], patient perceptions of communication [2], and patient willingness to engage in hand hygiene [3]. However, surveys often have low response rates (eg, 6%, 58%, and 32% [1-3]) and high rates of participant drop-off midsurvey [4]. This can lead to low statistical power, increased Type II error [5], and nonresponse bias, defined as “systematic and significant variation between responders and nonresponders” [6]. These biases can result in misleading conclusions [7] and affect the validity of survey findings [8]. High response rates and survey completion rates are therefore essential.

Response rate, defined as the number of individuals who are offered a survey who begin the survey, tends to be low in studies that survey patients. Some research has shown that response rate may be affected by survey length, with longer surveys having a lower response rate compared to shorter surveys [9-11]. Other studies, however, have failed to show this effect [12,13]. Regardless, a high response rate does not guarantee that those who begin the survey provide complete, usable responses.

Drop-off is defined as the number of participants who respond to a survey but do not finish it; that is, the participant abandons the survey before the end. Item missingness is defined as items that a participant does not answer, which may be related to drop-off, but can also include items the participant skipped intentionally or unintentionally. Reasons for drop-off and item missingness are multifaceted; similar to response rate, one possible contributor is survey length and associated response burden. For example, one study found that 10% of participants dropped off a survey almost immediately and completion rates continued to decline for a loss of 2% of participants after every 100 items [4]. Other studies have found similar results, showing that drop-off rates and item missingness are higher for longer surveys compared to shorter surveys [14,15]. Further, poor survey design can also lead to missing data and drop-off [16,17]. At the same time, some aspects of survey design that are common options for web-based surveys can reduce survey drop-off. For instance, prior work has shown that providing motivational reminder statements when participants fail to answer a survey question can reduce the rates of missing data [18]. Additional research suggests that incorporating page breaks leads to higher survey completion rates as compared to using a scrolling design [19].

Because most of the work on survey response and completion rates has been done outside the hospital setting, little is known about how these findings apply to surveys of hospitalized patients. Hospitalized patients differ from the general population in ways that could either decrease or increase the response rate, drop-off, and item missingness. For instance, a decreased response rate and higher drop-off or item missingness may be expected as many hospitalized patients have low physical or cognitive capacity due to their illness, and this could impact their ability to conduct tasks such as fully completing a survey [20,21]. On the other hand, many hospitalized patients find themselves with free time [22] and in an environment that has limited opportunities for alternative activities that could compete with survey-taking, which could contribute to an increased response rate and lower drop-off or item missingness. While satisfaction surveys of hospitalized patients are a common subject of research, they are generally sent to patients post discharge [23], and thus do not provide insight into surveying patients while in the hospital. Of the few studies that have reported results of surveys conducted in the inpatient environment, patients’ family members or caregivers, rather than the patients themselves, have often been the subject of the survey [24,25]. It is therefore unknown how surveys of the hospitalized patient population can be optimized to increase response rate and reduce drop-off and response burden.

In this study, we sought to close this gap by examining survey response patterns and drop-off using a sample of over 3500 hospitalized patients to answer the following research questions: (1) Do response rates of hospitalized patients differ as a function of survey length? (2) What is the average survey (i) drop-off rate and (ii) rate of item missingness for hospitalized patients? (3) How does the trajectory of participant drop-off change over the course of a long survey (ie, what is the ideal length of a survey for this population)? (4) What electronic survey design features (eg, page breaks and missing item reminders) are associated with reduced item missingness and drop-off for this population? (5) What is the response burden in this population, in terms of duration (ie, how much time does it take hospitalized patients to complete a survey item)?

Participants were patients in a large, midwestern US hospital system composed of 6 noncancer hospitals. Data were collected in the context of a randomized controlled trial investigating the impact of an inpatient portal on patient experience [26]. Patients were eligible if they were aged 18 years or older, able to speak or read English, and not involuntarily confined or detained (further details of the sampling strategy are in McAlearney et al [26]).

Patients were provisioned Samsung tablets that provided access to the MyChart Bedside patient portal (Epic Systems). MyChart Bedside is an inpatient portal allowing hospitalized patients to conduct activities such as ordering meals, receiving health education, and taking surveys. Tablets were provisioned no sooner than 6 hours from patients’ admissions, and up to 10 days after admission. Patients were recruited in one of two ways: (1) via an embedded URL on the tablet, or (2) in person by a study team member. Patients were provided with a study overview including the goals of the study, the expectations of participation, and risks and benefits associated with study participation. Patient informed consent was collected via the web with an electronic signature, or via written signature (if patient preference or due to internet connectivity issues).

Demographics, including gender, race, age, length of stay, and Charlson Comorbidity Index (CCI), were pulled from the institution’s Information Warehouse (IW) and linked to survey responses. Descriptive statistics for demographics and response rate (defined as the number of participants who were offered the survey and who completed at least 1 survey item), drop-off rate (defined as whether participants responded to the last survey item), and item missingness (defined as whether participants completed every item on the survey) were also computed. Missingness (ie, percentage of items not responded to) within each survey version was also examined via descriptive statistics. Chi-square tests were conducted to compare response rates, drop-off rates, and item missingness by survey version. Kaplan-Meyer Survival curves were computed for each survey version to assess the proportion of respondents who remained in the survey across the items. Distribution charts were developed to map response rates by item.

Duration of the survey response (in minutes) was computed for participants who responded to the last survey item, for each version. The duration was operationalized as time from opening the survey to ending the survey (hitting “submit” or timing out). This was inclusive of any breaks wherein the participant may have been interrupted or closed the survey and returned at a later date or time. This variable was highly negatively skewed, likely due to such breaks. Thus, the medians, IQRs, and the 25th percentiles are interpreted. Analyses were conducted in Stata (version 15; StataCorp) [27].

The study was conducted in accordance with the Declaration of Helsinki and approved by the institutional review board of The Ohio State University (#2015B0272). Participants provided informed consent and HIPAA (Health Insurance Portability and Accountability Act) authorization.

Overall, 3578 patients were offered the survey, and 2981 completed at least 1 item. Demographics are in Table 3.

The overall response rate was 83.3% (n=2981). Response rate by survey version is shown in Table 4. Response rate did not differ significantly between versions A (174 items) and B (111 items; χ²₁=0.39; P=.53), addressing research question (RQ) 1, and showing that survey length was not associated with response rate in this sample.

The overall percentage of participants who dropped off before the end of the survey was 29.3% (n=873), addressing RQ2(1). The overall percentage of participants who had any missing data (including drop-off but excluding items subject to display logic) was 50.7% (n=1512), addressing RQ2(2). Table 4 provides more details on item missingness by survey version, indicating that the mean percentage of items with missing data ranged from 11.7% (11.4 items; version C) to 40.7% (70.4 items; version A).

Drop-off was substantially reduced from version A (174 items; 65.7%) to version B (111 items; 20.2%). The percentage of participants who had missing data on at least 1 item was high for versions A (621/708, 87.7%) and B (291/377, 77.2%), and decreased substantially for version C (600/1896, 31.7%). Together, these findings suggest that shorter surveys yield less drop-off (χ²₁=203.9, P<.001 comparing version A to B for responding to the last item), addressing RQ3. In addition, missing item reminders were shown to reduce item missingness (χ²₁=273.7, P<.001 comparing version B to C for responding to all items), addressing RQ4.

To investigate RQ3 further, we computed Kaplan-Meier survival curves for each version (Figure 1). The chart indicates that the survival curves for each version trend down slowly, indicating participants slowly drop off as the survey progresses. For version A, the longest survey, which had 174 items, 25% (n=217) of participants had dropped off by approximately item number 120, and more than half of the participants (n=434) had discontinued the survey by item 158.

To address RQ4 more fully, we also examined the proportion of responses to each item in each version of the survey in the context of design features such as page breaks (Figure 2). For all versions, there is generally a downward trend of responses to items as the survey progresses. Another common trend is that missing data appears to increase as participants progress within a page, although this effect appears less pronounced in version C (the version where missing item reminders were added). Similarly, for versions A and B, the first items after a page break appear to have higher response rates compared to other items. Relatedly, there are early increases in missing data in versions A and B that recover after page breaks. Related to item type, we noted a higher rate of missing data in the free-response items. We also noted a dip in responses to the question that asks about income, which did not have a “prefer not to respond” option; this effect was present in all survey versions. These trends suggest that both page breaks and reminders about unanswered items are important, particularly in an electronic survey where display cues may not be optimal to prompt participants to scroll for more items.

The proportion of items that are complete versus missing when averaged across respondents differed by survey version. Version A had a large proportion of items that were missing, with a mean of 70.4 (40.7% of items), SD 52.8 (30.5%) items. The number and proportion of missing items decreased substantially in version B, with a mean of 19.9 (20.3% of items), SD 26.7 (27.2%) items, and there was an additional decrease in missing items in version C, with a mean of 11.4 (11.7% of items), SD 24.9 (26.4%) items.

Duration for version A was the longest, with a median time to completion of 120.9 (IQR 39.8-1333.0) minutes. This corresponds to a median of 41.7 seconds per item. Versions B and C had median durations of 46.2 (IQR 22.7-1162.9), and 42.4 (IQR 24.1-361.0) minutes, respectively. This corresponds to a median of 26.0 seconds per item for version B and 22.8 seconds per item for version C. Overall, across the 3 versions, the median participant took 24.6 seconds per item, indicating that the median participant was able to complete 2-3 items per minute, addressing RQ5.

The response rate was not related to survey length. However, survey length was associated with completion rate, such that a significantly higher proportion of participants dropped off before the end of the survey in the 174-item version, compared to the 111-item version. The survival curves indicate that an ideal survey length—one in which at least 75% of the participants are retained—may be around 120 items for hospitalized patients. A lower number of items should be used if researchers wish to retain a higher proportion of their sample. Our drop-off rate was higher than that of Hoerger [4], which found an initial drop-off of 10% plus an additional 2% per every 100 items, as we found a drop-off of about 15%-20% in our first 100 items. This difference may be due to the populations included in these studies as Hoerger [4] focused on undergraduate students, who may have been more motivated to complete the survey (eg, as a component of an educational course). Further, it is possible that drop-off was more prevalent in our study due to factors related to our population, such as hospitalized patients possibly feeling too sick to finish the survey. Follow-up work should be done to better investigate this.

Additionally, we found that independent of drop-off, item missingness was prevalent until missing item prompts were added. Before including the prompts, item missingness seemed to occur frequently after the first item on each page, indicating that there may have been a visual issue as participants were not cued to scroll to see additional questions. Regardless of the missing item prompt, the income item was skipped about 20% of the time. This finding is similar to results from prior work showing that about 24% of participants tend to skip income survey questions [28].

Participants took a median of 120.9 minutes to complete the 174-item version of the survey, and 46.2 and 42.4 minutes to complete the different 111-item versions. An important consideration, however, is the context of these findings. Specifically, the duration variable was inclusive of any breaks a participant may have taken while completing their survey—whether for a few moments (eg, due to clinical care), or for hours (eg, if the patient’s tablet needed to be charged at the nursing station). While these durations may be considered more as maximums, they may be realistic in practice for surveys deployed in inpatient environments where interruptions are frequent [29]. These durations are, however, well beyond the recommended optimal survey length of 10-13 minutes [30,31]. Given this, it is surprising that our drop-off was only modest. This suggests that hospitalized patients may have a higher tolerance for longer surveys due to factors such as the lack of other activities in the hospital environment that could compete for their attention. In addition, it is our finding that the median participant was able to answer 2-3 survey questions per minute. Further work should be done to better understand these factors and their implications.

Based on our findings, we present best practices for survey design in Table 5. While these suggestions are developed to guide surveys of hospitalized patient populations, we also indicate the extent to which we believe each best practice can be generalized to other settings. We supplement these suggestions with references that provide additional evidence for each best practice.

This study has several limitations. First, additional work is needed to examine its generalizability to outpatients and other settings. Second, this study used surveys that were completed on Samsung tablets. It is not clear how these findings generalize to paper surveys, or to surveys taken on a computer or mobile phone. Prior work has established equivalence in responses for tablet, mobile phone, and paper-based surveys [38], but others have found differing response rates [39] and durations [40] between web and mobile surveys. This should be tested in the current context. Further, this study used a survey that was completed by the participants themselves, yet particularly in a hospitalized patient context, family or other caregivers may assist patients with survey completion. Future work should examine differences in response rate, drop-off, missing data, and duration of response in these situations, as it is possible there may be differences. Last, several other factors (eg, demographic and clinical characteristics of patients, patient’s attitude toward surveys, sequence of items in the survey, and participants’ interest in the survey topic) may also impact response rates, survey drop-off, and missing data in the survey, but we were unable to examine these in this study.

We found that hospitalized patients had a higher tolerance for longer surveys than the general population, with most participants completing at least 120 items. Participants tolerated a median survey duration of 121 minutes for the longest version. In addition, the inclusion of missing item prompts substantially reduced the amount of missing data. Overall, the median participant was able to complete 2-3 items per minute. These findings can be informative for future research when designing surveys for use in the inpatient setting.