The Donabedian SPO framework was used to identify the indicators of structure, process, and outcome that potentially affect the integration of eHealth into health care organizations. The Donabedian framework covers all relevant aspects of an organization’s structure, process, and outcome and their interrelations, and it combines these aspects with health and social factors. Therefore, it is a suitable model to evaluate the organization of eHealth within health care organizations. The SPO categories are thematically explained in Figure 2 [24,25].

The Donabedian SPO framework was designed in the 20th century before the introduction of eHealth. For this review, the SPO framework was adjusted to be compatible with the current time and incorporated the application of eHealth. The adjustments are described in the themes presented in Textbox 1.

The adjustments to the SPO framework are shown in Figure 3.

This systematic review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) guidelines. The research question was as follows: “How are structure indicators and/or process indicators related to eHealth or blended health care outcome indicators?”

Two authors (RT and MK) searched PubMed, EMBASE, Web of Science, Cochrane, and Emcare databases for relevant studies published up to December 12, 2019. They searched for the following terms in the titles, abstracts, and keywords of the published papers: structure* indicators* or process* indicators* or outcomes* indicators* and [blended care or eHealth* or telehealth*]. Multimedia Appendix 1 contains the full search details. After the search, two authors (RT and AV) screened the titles and abstracts of the relevant articles. Studies were included if they mentioned (1) the use of eHealth or blended care for diagnostics or treatment and (2) structure, process, or outcome indicators. Quantitative, qualitative, and mixed method study designs were included. A study was excluded if (1) it was a protocol, review, meta-analysis, grey literature, book chapter, oral presentation, or poster presentation; (2) it was published in a language other than English or Dutch; (3) full-text of the article was not available; (4) the intervention was not implemented (eg, conducted research regarding the users’ expectations towards a prototype); or (5) the intervention used an analog application via plain-old-telephone lines. Of the remaining articles, RT reviewed the full texts. To ensure reliability, AV randomly selected about 10% of the fully reviewed articles for a blind review. Discrepancies were resolved by discussion. In case of uncertainty, a third author (MK) was consulted.

Data extraction sheets were designed to provide an overview of the (1) study characteristics (eg, title, author, study aim, setting, disease, and quality appraisal); (2) characteristics of the eHealth intervention (eg, technology and function) and description of the intervention; (3) distribution of indicators into themes and categories related to the integration of eHealth into health care; and (4) interaction among the indicators, presented as themes.

RT designed the first concepts of the data extraction sheets. Authors RT, MK, NC, and ET discussed the design of the data extraction sheets to ensure their usability. Improved sheets were developed accordingly. The blind reviewer (AV) did not discuss the data extraction sheets. The included articles were reread by RT to check whether data clustering was complete and logical and for purposes of data pooling itself. AV selected a sample of 10% of the included articles for data extraction. Discrepancies were resolved by discussion.

The Mixed Methods Appraisal Tool (MMAT) was used to appraise the quality of eligible studies in mixed methods systematic reviews—that is, reviews that included qualitative research, randomized controlled trials, nonrandomized studies, quantitative descriptive studies, and mixed methods studies [30]. The MMAT allows determination of the quality of different empirical study designs by using the same measure of five criteria in the chosen category. MMAT scores range on a scale of 1 to 5, with 1 indicating the lowest quality and 5 indicating the highest quality.

eHealth interventions were ordered by type of technology and functionality. For technology, the classification proposed by Nictiz was used, distinguishing websites, apps, video communication, sensors, and wearables, domotics, robotics, and big data (ie, artificial intelligence) [31]. This classification is based on international studies [10,32]. For the present study, eHealth only concerns digital interventions and not analog ones such as analog applications via plain-old-telephone lines; this is in line with the classification proposed by Nictiz. For labelling the functionality, the second and third tiers of the National Institute for Health and Care Excellence (NICE) [33] were used, because these functionalities measure patient outcomes (Tier one consists of system services with no measurable patient outcomes). The functions were classified as communication, self-management, clinical calculation, active monitoring, diagnosis, and treatment [33].

Indicators that had a potential impact on the integration of eHealth in health care were extracted and organized by the relevant theme according to the adjusted SPO framework (Textbox 1). In addition, the reported interactions between the indicators were extracted and organized by the relevant categories and themes. For a clear overview, the indicators within each theme were further divided into two subthemes by RT and ET (Table 1). The creation of subthemes was an iterative process. When reading the full texts, we found some definitions that sharpened some of the subthemes. The full definitions of the themes and subthemes are provided in Multimedia Appendix 2.

For each of the extracted indicators, the relevant impact on the integration of eHealth was noted. As there is no general standard for when eHealth is successful or effective [3,19], nor did the included studies specify such standards, these indicators were labeled as advantage, disadvantage, or neutral. An advantage in the structure and process categories indicates a positive effect on the integration and/or a positive effect on the outcome. A disadvantage in the structure and process categories indicates a negative effect on the integration and/or a negative effect on the outcome. An indicator that did not turn out to be an advantage nor a disadvantage was labeled neutral. The extracted indicators were noted as advantage, disadvantage, or neutral, in line with the evaluation performed in the corresponding study.

The following results are presented in this paper: (1) distribution of the indicators into (sub)themes and categories, and the impact on the integration of eHealth into health care; (2) most frequently reported indicators (ie, reported 10 times or more); and (3) interaction among indicators organized into themes and categories.

The systematic search led to the identification of 11 eligible articles, selected from a total of 739 articles shortlisted initially (Figure 4).

This literature review analyzed how eHealth can be organized optimally by using the Donabedian SPO framework. General organizational developments were identified, regardless of the type of illness, setting, or the eHealth application used. A review of the literature of selected cases highlighted three important findings. First, the role of the care recipient needs to be incorporated into the organizational structure and daily care process. Second, the technology must be well attuned to the structure of the organization and daily care process. Third, the deployment of the human resources to the daily processes needs to be aligned with the desired end results. Not adhering to these points could negatively affect the organization, daily process, or the end results. Findings from this research using the Donabedian framework corresponds to the conclusions of other studies using different research methodologies, which is explained below.

First, the SPO analysis showed that the care recipient plays a crucial role in the successful integration of eHealth. Patient-centered interaction and communication are important, to activate patients in managing their health care and to improve health outcomes in the application of eHealth [5,31,44-46]. Kuipers et al [44] and Rathert [45] demonstrated with systematic literature reviews that patient-centered care and co-creation are positively associated with the physical and social well-being of patients and with satisfaction of patients and health care professionals. These findings are in line with the review of Wildevuur and colleagues [46], demonstrating that organizations that are more patient centered with eHealth interventions achieve better outcomes with regard to patient health and quality of life. Although most health care professionals embrace more patient involvement and engagement, delegating power and responsibilities could be a challenge for health care professionals’ authority [47,48]. Another important issue is knowing who the customers are, what they want, and how the customer’s demand is answered [49]. A previous study reported that eHealth is not suitable for all care receivers [18,50]. Therefore, identifying who benefits most from which kind of therapy is an essential addition to the screening process, and it could lead to more effective targeting and resourcing [51]. Furthermore, insufficiently incorporating the patients’ family, work, and life goals into care plans will likely result in dropouts or nonadherence to care plans [50].

The second noteworthy finding is the essential role of excellent technology in the integration of eHealth. The way technology is set up has an influence on the working environment of health care professionals [52]. Inflexibility and complexity of the technology comes at the expense of effective daily processes and their quality [53,54]. Several studies demonstrated that the adaptability of eHealth technologies to fit to the local context, its ease of use, and its integration into clinical workflow benefit the users’ acceptance and meaningful use [22,55,56]. This was also reflected in the early phase of the COVID-19 pandemic, where rapid scalable technologies were the easiest to use and quickly implementable [53]. However, the health care system continued to face challenges in adopting digital technology after the first emergency phase of the COVID-19 pandemic, due to inadequate information and communication technology infrastructure and a bad fit of the technology into the clinical workflow that is primarily designed for face-to-face care [53]. Granja et al [54] demonstrated that the application of eHealth is often not fitted to the existing workflow due to time and space constraints and breaking of traditions. Although eHealth is seen as an innovative solution for alleviating the increasing burden for health care professionals [2], it could have a counterproductive effect on the working conditions for employees if the technology is not properly adapted to the structure and processes [57,58]. Third, integrating eHealth into a health care organization requires adjustments of the care processes and utilization of the human resources, with appropriate process monitoring. Working with eHealth also poses logistical challenges; for example, a clear understanding is needed of the expected achievements, processes, and staffing requirements in order to bring about changes and create new capabilities [59]. Vissers and De Vries [49] pointed out that it is necessary to know how the logistical capacities should be assigned to the process, how the processes are measured, and who is responsible for the management of the process. Changes in the workflow are inevitable and necessary for eHealth interventions to be successful [54]. However, integrating eHealth technology into daily care processes is complex, and it needs coordination and process communication [19]. For example, a living laboratory experiment conducted over 3 years with patients, health care professionals, enterprises, and researchers to accelerate the integration of eHealth in daily practice showed that workflow, responsibilities, and roles needed to change, but health care professionals did not know how to approach this and had difficulties in integrating eHealth into their daily care processes [18].

The strengths of this research are that international studies were included and represented a wide range of patient groups and settings. The findings were representative for the included studies, and they were not dependent of the study design, disease, target population, setting, or type or function of the eHealth application used. The wide range of settings of the included studies is supportive of a broader application of the present study’s findings. In the Methods, we stated that there is no clear consensus on what constitutes as good eHealth and how it is best organized [3,19]. Nevertheless, we believe that our findings make a significant contribution to improving the integration of eHealth in regular health care by identifying the most common indicators in the organization’s structure, processes, and outcomes. Thus, this research contributes to a new model for integrating organizational, health, and social factors.

A limitation of this study is that the health outcomes were rarely mentioned in this review. We hypothesized that this is because the main method used in the included studies was process evaluation. Therefore, although the health outcomes played a major role in earlier RCTs, this was not the case in process evaluation studies. The included studies did not define clear standards for the indicators to determine their quality. However, an indicator only becomes meaningful if a standard is specified [60,61]. There are also limitations in the selection procedure. The interrater reliability was not calculated. Due to this complex, broad topic, the predefined inclusion and exclusion criteria were sharpened at the time of selection. It was an iterative process, with a lot of consultation and coordination. In the process, full consensus was reached for all inclusion and exclusion criteria for selection at each step of the research. Another limitation is the classification of indicators into subthemes and themes at the discretion of the authors. It is conceivable that different classifications would reach different conclusions. Yet, the conclusions of each included study fit with the overall conclusion; therefore, the chance of this bias seems to be small. However, the findings of this literature review are dependent on the results of the included studies and may be subject to publication bias. Even though the included publications contain either positive or negative results (eg, a failed randomized trial [35] or interventions with no or less impact [40,43]), a chance of publication bias cannot be precluded automatically [62,63].

It is also noted that the Donabedian framework itself was designed before the introduction of eHealth and may not include the latest prevailing ideas on the organization of health care. For this reason, the model has been adapted in order to represent eHealth. By doing so, an attempt has been made to reduce the limitation as far as possible. Nevertheless, this literature review confirmed that it is still useful to analyze what contributes to the successful integration of eHealth into traditional health care. Additionally, there are other reputable models for evaluating eHealth interventions, such as the nonadoption, abandonment, scale-up, spread, sustainability (NASSS) framework [20]; Consolidated Framework for Implementation Research (CFIR) [64]; and the holistic framework to improve the uptake and impact of eHealth technologies [19]. These models describe the different phases from the design of the intervention to its adoption and implementation. This literature review focused on quality improvement of the way eHealth is organized, that has already passed the initial phase (of design and adoption). The Donabedian framework covers all relevant aspects for sustaining the integration of eHealth into health care and the interrelations of organization’s structure, processes, and outcomes, as well as integrating these aspects with human and social factors, after the adoption and uptake phase of eHealth.

For optimal integration of eHealth into health care, the following main principles should be considered and approached simultaneously. First, the role of the care recipient needs to be incorporated in the organizational structure and daily care process. Second, the technology must be well attuned to the structure of the organization and daily care process. Third, the deployment of human resources to the care process needs to be aligned with the desired end results.

Thus far, no study has presented a complete overview of the successful and effective organization of eHealth. Therefore, it is desirable to supplement this research with knowledge from other sources, such as in-depth research into the experiences from different perspectives, as this can help us to obtain a complete overview of how eHealth can be successfully integrated into health care organizations.