Alongside measures of clinical effectiveness and safety outcomes, patient experience is increasingly recognized as an important indicator of the quality of health care provision and is frequently cited in health policies [1-3]. Yet, in practice and research, the concept of patient experience has had varied uses and is often discussed with little more explanation than the term itself [4]. In 2011, the English National Health Service (NHS) [5] outlined 8 domains that help define patient experience and are critical to patients’ experience of health care services. This has been used as an agreed working definition of patient experience to guide the measurement of patient experience across the NHS.

One of the essential elements of a strategic approach to improving patients’ experience is to measure and report on patients’ experiences to assess progress, strengthen accountability, and identify new opportunities for improving performance [6]. Evidence suggests that this can be achieved using real-time feedback (RTF) [7]. RTF involves the systematic collection, analysis, and reporting of information from individuals at the point of care [8]. Previous studies have found that RTF has the potential to enable health care organizations to respond promptly to patients’ concerns and make timely improvements to services [7,9-11].

With the ability to maximize the scalability and speed of data collection while reducing cost [12], digital tools (tablets, kiosks, emailed survey, and websites) are increasingly being utilized to gather patient experience feedback [13,14], allowing summary results to be reported on an ongoing [15] real-time basis. In a recent Cochrane review [16], self-administered survey questionnaire responses collected using mobile apps were compared with those collected using other methods, and it was concluded that the delivery of survey questionnaires through apps does not affect data equivalence and can improve data completeness. However, none of the questionnaires evaluated patient experience.

There is a growing pressure on health care services to embrace digital technologies tosignificantly improve the patient experience [17]. With an increase in adoption of digital systems pertaining to RTF, health care services must recognize and overcome the barriers that may hinder successful integration and uptake of such technologies. One of the ways of achieving this is through systematic evaluation and monitoring of digital systems to ensure they operate in the way they are intended and cultivate a better patient experience [17]. Digital maturity—the extent to which digital technologies are used as enablers to deliver a high-quality health service—is an emerging concept across developed health care systems [17]. Evidence suggests that digital maturity is linked to better outcomes and is indicative of a well-performing organization [18]. Evaluating digital systems for digital maturity highlights where gaps exist in maturity, which presents an opportunity to address the specific shortcomings [17]. The digital maturity of the systems that are being used to collect RTF has not been previously evaluated. A deeper understanding of the limitations of individual digital systems may help health care services pinpoint areas in need of improvement and organizational change. Without this, digital systems used for RTF may not be successful, and patients’ voices may go unheard.

Therefore, this systematic review aims to evaluate the digital maturity of digital RTF systems. Specific objectives were to (1) describe the digital modes utilized; (2) provide insights into patients’ and staff views of these digital systems; and (3) demonstrate digital maturity by systematically scoring individual digital systems.

The following databases were searched: Medical Literature Analysis and Retrieval System Online, EMBASE, PsycINFO, The Cochrane Library (Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials, Cochrane Methodology Register), Global Health, Health Management Information Consortium, CINAHL, and Web of Science. In addition, gray literature and Google Scholar were utilized to extract papers that were not retrieved in the databases searched. Publications from January 2000 to February 2017 were included. We limited our electronic searches to studies on or after 2000 because it was around this time when the digital technology revolution in health care began to emerge [19]. Owing to the diversity of terms used inferring patient experience, combinations of search terms were used. Multimedia Appendix 1 provides the complete list of subheadings (Medical Subject Headings) and keywords.

Studies were deemed eligible for inclusion if (1) digital modes of administration were employed; (2) collection was in real time (at the point of care) or near real time (immediately after discharge); and (3) they were conducted in a primary or secondary care setting. There was no restriction on the age of the patient population of interest. Both quantitative and qualitative studies were included. Multimedia Appendix 1 provides further details of the inclusion criteria.

The research adhered to the guideline presented in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2009 checklist [20]. Data analysis involved the comparison of included studies and extracted data. Due to the heterogeneous nature of the studies, a narrative synthesis was deemed most appropriate. This was followed by scoring the digital systems reported in the studies to determine the digital maturity.

An existing framework [17] was utilized to determine the digital maturity of individual digital systems. This framework is embedded in the literature and has the benefit of systematically assessing the effectiveness of any digital system in health care. Each included study was scored across four key domains—capacity or resource, usage, interoperability, and impact. The framework highlights key questions for each domain and is based on “yes” or “no” response to each question. We assigned a maximum of 1 point if the digital system in each study demonstrated appropriate evidence for that particular domain. The maximum overall points a study could achieve was 4. This indicates the overall success of the digital platform [17]. Two reviewers (MK and KF) independently scored each study; disagreements in scoring were resolved by discussion between the two reviewers. Interrater agreement kappa was calculated.

The initial search returned 3456 papers; after removing duplicates, 3438 papers were retained. The titles and abstracts were screened, and 112 papers were identified as potentially eligible for inclusion. Full-text papers were retrieved and assessed for inclusion, of which 13 were retained for final inclusion. RTF was defined as feedback collected while patients are in a hospital, at the point of care, while receiving care, or immediately after discharge. The main reason for exclusion was the papers did not report patient experiences or reported user experience of other digital technology. Figure 1 illustrates the PRISMA flowchart representing the study selection process and reasons for exclusion.

The reasons for exclusion of records (n=3326) and full-text papers (n=99) were the following: nondigital mode of administration (n=241), not real time or near real time (n=130), assessment or evaluation of other parameters (n=1489), feasibility or usability testing of digital systems not related to patient experience data collection (n=655), experience of other digital technology (n=848), and reviews (n=62).

This review highlights that digital modes of administration of RTF are well accepted by patients and staff, with response rates equivalent to, and in some studies better than, non-RTF. From a patient’s perspective, it has been reported that its influx alongside demographic shifts such as increasing aging population and ethnicity [32,33] can lead to patient disengagement and poor uptake of these technologies [34]. On the contrary, we have shown that patients are in fact willing to engage with this technology for experience reporting, thereby disrupting previous nonreal-time, paper-based feedback. However, digital technology may not be preferred by all patients; therefore, health care organizations need to be mindful and consider other means of inclusivity when developing digital health care systems. From a staff perspective, although RTF is well received, problems arising from the lack of robust digital infrastructure [25] can thwart improvement efforts. From a health care organizational perspective, most digital systems were unable to demonstrate interoperability and very few demonstrated impact and, therefore, were not deemed digitally mature, compromising success within the organization.

By nature, maturity frameworks not only identify components of a successful system but also capture the evolution of a digital system from conception to implementation to impact. Using evidence where possible from the included studies, we highlight how each of the 4 domains contribute to ensuring digital maturity.

The success of digital health is contingent on establishing the necessary capacity and resources to build, use, and support access to high-quality health services and harvest useful information in the health system. There was a general lack of analytical support in most digital systems to extract valuable information such as the ability to examine user-specific interaction [10] or adjust survey responses [23]. A Kings’ Fund report explains that gleaning information from experience data requires the same analytical capability as interpreting clinical data; however, this is often unavailable [35]. The resource capacity requirements stretch beyond health professionals and technology specialists across the continuum of care to include health information managers to information security professionals [36]. Staff time was an important barrier in data collection, and this is in keeping with other studies that reported a lack of time or resources to collect, analyze, or act on data and need for staff training in data analysis and statistics to facilitate full understanding and use of results [37-39]. To circumvent this, some of the studies used volunteers [25,31] and incentives [30,31] to gather data. This generates concerns such as response bias and competition among clinicians, and there may be an element of the Hawthorne effect, whereby clinicians modify their behavior as they are being observed, which may explain the positive outcomes in those studies. Going forward, a strategic approach should take into consideration building human and institutional capacity, nurturing clinical and community champions, and developing the base of knowledgeable users to drive appropriate adoption of digital systems [36].

As the needs and experiences differ greatly across these groups, a flexible or responsive data collection mode is needed, which can aid patients during the data collection process. While most patients in the included studies were comfortable using the digital system and required little prompting or help once they engaged with the survey, some differences were noted. Concerns exist that older patients are less comfortable with technology [40]; however, there was an overrepresentation in responses from patients aged >65 years in the included studies [22,25,27,29,31]. Moreover, some of these studies were conducted in the elderly population [25,27]. Despite this, the response rate in this patient group was adequate, suggesting that their movement into digital life is evolving. Furthermore, when certain conditions are met such as providing a supportive, nonhurried environment [41]; bold, plain, and large font with fewer graphics; and avoidance of certain colors [42], older adults can be successful users. The use of videos in the software can enhance accuracy and acceptability [10]. Furthermore, trained volunteers can provide a responsive approach to real-time data collection from lesser heard groups [14], increasing patient engagement and, subsequently, improving response rates. In addition, they can help reduce the data collection burden, which may otherwise fall on clinical staff and may even account for the false positives as seen in some surveys [43] due to the presence of staff during survey completion.

If data collection is obtrusive, unrealistic, or inaccurate, it can undermine enthusiasm for assessing and improving practice quality [44,45]. However, the digital systems in this review demonstrated quick turnaround of data collection, collation, and dissemination of results. This was key to successful implementation of digital systems as it promoted “buy-in” from staff. Furthermore, patients are fatigued of requests for feedback in health care and in daily life; hence, data collection should be quick, focused, and part of routine care to encourage participation from patients and clinicians and to be sustainable in a busy setting [10].

From a health care perspective, interoperability is needed to reform the chaotic, and, at times, dysfunctional nature of how information is shared within and among health care services. There was a ubiquitous shortfall in achieving interoperability among the digital systems in this review. Data that are not interoperable cannot be analyzed alongside other data indicative of care quality. This perpetuates the siloed approach to data interpretation and creates a chasm between data and information for improvement. Some of the challenges in achieving interoperability include resistance from some vendors, prohibitively high data exchange fees, and lack of incentives to develop interoperability and technical variations. Without tackling the interoperability of RTF systems, it is likely that health care organizations will fail on delivering impactful quality improvement. This must now be a major focus for health care services, but it should be done in an organized way that prioritizes interoperability so that patient feedback data can communicate seamlessly to generate information that will benefit health care services and patients alike.

To genuinely demonstrate impact, the digital system should not only be able to generate quality improvement activities but also demonstrate sustainability and cost-effectiveness. Three studies [10,25,30] achieved a score for impact as they demonstrated a change in practice following the implementation of RTF. However, the majority of the digital systems were not able to demonstrate any impact. The duration of studies in this review was short (between 1 and 12 months), and the sustainability of the digital system to continue to deliver quality improvement could not be evaluated. Without considerations of sustainability, digital programs are of limited value.

Due to publication bias, there may be unpublished evidence of nonsignificant or negative findings or findings held locally, which are not published or otherwise publicly available. We ascribed a level of digital maturity of individual systems based on the evidence provided in the studies. If this was not discernible while reviewing the studies, we assumed it to be lacking. As the remit of the included papers differed greatly, their digital systems may, in fact, have existing procedures for ensuring capacity, usage, interoperability, and impact, but the authors did not specify this information in the published paper. Of note, the studies were not evaluated for research quality before being included in the synthesis owing to the limited number of studies that were identified after exclusion.

There was not a large body of published literature on digital modes in relation to RTF. However, the evidence provided by the studies in this review demonstrates that there is a potential in using digital modes of administration of RTF as an agent for improving service delivery. Patients and staff alike are willing to engage in RTF, demonstrated by acceptable response rates. However, for RTF to be impactful, health care organizations must ensure that they have strategies in place to deliver across all levels of digital maturity. Health care services have the capacity to introduce digital solutions for RTF; however, lack of interoperability is slowing down progress. In addition, it is possible that some health care services may be wasting effort and resources when they invest in digital technologies for RTF. On balance, the direction of the health care ecosystem toward embracing digital technology looks promising, and as health care shifts toward a patient-centered model, digital technology will be an important partner in this transformation.