Emerging digital technologies promise to shape the future health care industry [1,2]. According to our previous review [3], most researchers had a positive impression of digital health interventions (DHIs). The number of DHIs is proliferating [4-6], which is affecting the way patients receive their health care services compared with face-to-face health care services and ultimately influencing the patient journey and overall patient experience (PEx) [7,8]. Good PEx is a key intent of patient-centered care [9] and a core measure of care quality in digital health (DH) [10,11]. Digital technologies have the potential to enhance or provide comparable PEx compared with some face-to-face health care services [8,12-14]. However, the uptake of digital technologies in health care is not as rapid as it has been in many other industries [15], and their potential in health care remains unfulfilled [16]. According to a report by the World Health Organization (WHO) on the classification of DHIs, the health system is not responding adequately to the need for improved PEx [17].

Despite the growing number of DHIs, evaluations that are timely, cost-effective, and robust have not kept pace with this growth [7,18,19]. PExs in the wide range of DHIs are mixed [20,21]. Few published DHIs have resulted in high download numbers and active users [22]; most are released with minimal or no evaluation and require patients to assess the quality for themselves and take responsibility for any consequences [23]. Low-quality DH may disrupt user experience (UX) [24], resulting in low acceptance, and some may even be harmful [25]. In addition, a DHI may be popular with patients but not valued by clinicians [26]. To generate evidence and promote the appropriate integration and use of digital technologies in health care, an overview of how to evaluate PEx or UX in varied DHIs is needed [3,27].

In this study, we used the definition of digital PEx from our previous review [3]: “the sum of all interactions affected by a patient’s behavioral determinants, framed by digital technologies, and shaped by organizational culture, that influence patient perceptions across the continuum of care channeling digital health.” This incorporates influencing factors of digital PEx [3] and the existing definitions of DHIs [28,29], PEx [30], and UX [31]. Compared with the general PEx and UX, it highlights patient perceptions that are affected by technical, behavioral, and organizational determinants when interacting with a DHI. DHI has become an umbrella term that often encompasses broad concepts and technologies [32], such as DH applications, ecosystems, and platforms [28]. In this study, we followed the WHO’s definition of DHIs [29], that is, the use of digital, mobile, and wireless technologies to support the achievement of health objectives. It refers to the use of information and communication technologies for health care, encompassing both mobile health and eHealth [29,33]. Compared with evaluating DHIs, PEx, and UX, little is known about evaluating digital PEx. However, combining the definition of digital PEx with the extensively explored measurement of PEx, UX, and DHIs can lead to an improved understanding of and enable the development of evaluation approaches for measuring digital PEx. Therefore, the evaluations of PEx, UX, and DHIs will be used as a starting point in this study to clarify when to measure, what to measure, and how to measure digital PEx.

First, the timing of measuring and evaluating digital PEx is an important consideration and must align with the contextual situation, such as evaluation objectives and stakeholders, to ensure practicality and purposefulness [34,35]. According to the European Union [36] and the Department of Health of The King’s Fund [37], an evaluation can be scheduled during the design phase or during or after the implementation phase. Similarly, the WHO [29] introduced 3 DHI evaluation stages: efficacy, effectiveness, and implementation. The evaluation of efficacy refers to where the intervention is under highly controlled conditions, the evaluation of effectiveness is carried out in a real world context, and the evaluation of implementation occurs after efficacy and effectiveness have been established. Furthermore, an evaluation can be performed before, during, or after the evaluated intervention in both research and nonresearch settings [36]. However, decision-making on when to collect PEx data can be more complicated. As argued in earlier studies [35,37], immediate feedback has the benefit of gaining real-time insights, but patients may be too unwell, stressed, or distracted to provide detailed opinions. In contrast, when the feedback is related to medical outcomes or quality of life, it often requires a lengthy period after the intervention to observe any changes. However, responses gathered long after a care episode may be inferior because of recall bias.

Second, there is a need for a decision on what is required to measure to assess digital PEx. The frequently mentioned UX evaluation concepts, such as usability, functionality, and reliability, from studies [38-40] investigating UX can be applied to evaluate the intervention outputs to anticipate digital PEx at a service level. Moreover, according to the existing constructs and frameworks of understanding or evaluating PEx [41-45], such as emotional support, relieving fear and anxiety, patients as active participants in care, and continuity of care and relationships, they can be adjusted to evaluate digital PEx by understanding patient outcomes at an individual level. In addition, the National Quality Forum [11] proposed a set of measurable concepts to be used to evaluate PEx in telehealth, for example, patients’ increased confidence in, understanding of, and compliance with their care plan; reduction in diagnostic errors and avoidance of adverse outcomes; and decrease in waiting times and eliminated travel. Some of these concepts can be used to understand digital PEx at an organizational level by assessing the impact of the health care system.

The third consideration is how to choose evaluation approaches appropriate for evaluating the digital PEx [35], starting from widely used theories, study designs, methods, and tools for evaluating DHIs and the related PEx or UX. There is rapidly evolving guidance for guiding DH innovators [18], such as the National Institute for Health and Care Excellence Evidence Standards Framework for Digital Health Technologies [46]. The strength of the evidence in the evaluation of DHIs often depends on the study design [18]. However, the high bar for evidence in health care usually requires a longer time for evidence generation, such as prospective randomized controlled trials (RCTs) and observational studies, which often conflicts with the fast-innovation reality of the technology industry [16,18]. In addition, many traditional approaches, such as qualitative and quantitative methods, can be used to collect experience-related data to evaluate the DHIs [18,29]. Qualitative methods such as focus groups, interviews, and observations are often used to obtain an in-depth understanding of PEx [37] in the early intervention development stages [29]. Surveys using structured questionnaires, such as patient satisfaction ratings [37,47], patient-reported experience measures (PREMs) [35,48], and patient-reported outcome measures (PROMs) [35,37,48], are often used to examine patterns and trends from a large sample. Hodgson [49] believed that strong evidence results from UX data that are valid and reliable, such as formative and summative usability tests, and stated that behavioral data are strong, but opinion data are weak.

This study aims to systematically identify (1) evaluation timing considerations (ie, when to measure), (2) evaluation indicators (ie, what to measure), and (3) evaluation approaches (ie, how to measure) with regard to digital PEx. The overall aim of this study is to generate an evaluation guide for further improving digital PEx evaluation research and practice.

This study consists of 2 phases. In phase 1, we followed the same study search and selection process as our previous research [3] but focused on a different data extraction and analysis process to achieve our objectives in this study. In the previous study [3], we identified the influencing factors and design considerations of digital PEx, provided a definition, constructed a design and evaluation framework, and generated 9 design guidelines to help DH designers and developers improve digital PEx. To highlight the connections between “design” and “evaluation” works in the development of DH and provide readers with a clear road map, we included some evaluation-related information in the previous paper as well. However, it was limited and described at a very abstract level. In this study, detailed information on the evaluation was provided, including evaluation timing considerations, evaluation indicators, and evaluation approaches, and we aimed to generate an evaluation guide for improving the measurement of digital PEx. Given that this is an evolving area, after we finished phase 1, we conducted an updated literature search as a subsequent investigation to determine whether an update of a review was needed in this study.

The decision to undertake an update of a review requires several considerations. Review authors should consider whether an update for a review is necessary and when it will be more appropriate [104]. In light of the “decision framework to assess systematic reviews for updating, with standard terms to report such decisions” [105], we consider that research on PEx in DH remains important and evolves rapidly. In case we missed some newly published articles that would bring significant changes to our initial findings, we conducted a rapid scoping search for articles published after our last search. We reran the search strategy as specified before with the addition of date (from December 16, 2020, to August 18, 2023) limits set to the period following the most recent search. After removing duplicates (73/367, 19.8%), we collected 294 articles in total. Following the same screening process and selection criteria, we finally identified 102 new eligible articles. The excluded articles were either not a literature review with systematic search (74/294, 25.2%), not about DH (87/294, 29.6%), not about PEx (26/294, 8.8%), our own parallel publications (2/294, 0.7%), or not accessible in full text (3/294, 1%). The eligible and ineligible articles in this phase are available in Multimedia Appendix 2. We found that the outcomes in the new studies were almost consistent with the existing data. For example, these articles either aimed to investigate what factors influence the feasibility, efficacy, effectiveness, design, and implementation of DH; examine how patients expect, perceive, and experience the DHIs; or intend to compare the DHIs with conventional face-to-face health care services. The research objectives of these new eligible articles are available in Multimedia Appendix 3. We considered that their findings were unlikely to meaningfully impact our findings on when to measure, what to measure, and how to measure digital PEx. As suggested by Cumpston and Chandler [104], review authors should decide whether and when to update the review based on their expertise and individual assessment of the subject matter. We decided to use these new articles as supplementary materials (Multimedia Appendices 2 and 3) but did not integrate them into the synthesis of this review.

This paper is a part of a larger study, and we have presented results related to study characteristics in a previous publication [3]. Multimedia Appendix 4 [53-97] provides detailed information regarding the characteristics of the included reviews, including research questions or aims, review types, analysis methods, number of included studies, target populations, health issues, and DHIs reported in each review. In this study, to achieve our research objectives, we identified reviews that reported different intervention maturity stages, timing of the evaluation, and timing of data collection. In addition, we identified a set of evaluation indicators of digital PEx and classified them into 3 predefined categories (ie, intervention outputs, patient outcomes, and health care system impact), which in turn included 9 themes and 22 subthemes. Furthermore, we highlighted evaluation approaches in terms of evaluation theories, study designs, data collection methods and instruments, and data analysis methods. We found that it was valuable to compare the evaluation objectives of the included studies. Therefore, we captured 5 typical evaluation objectives and the stakeholders involved, which clarified why and for whom DH evaluators carried out the evaluation tasks. The detailed findings are presented in the Evaluation Objectives section.

Our review findings highlighted 5 typical evaluation objectives.

The first objective was to broaden the general understanding of the digital PEx and guide evaluation research and practice (11/45, 24%) [53-63]. For instance, 1 review [61] aimed to identify implications for future evaluation research and practice on mental health smartphone interventions by investigating UX evaluation approaches.

The second was to improve the design, development, and implementation of the DHI in terms of a better digital PEx (15/45, 33%) [53-62,64-68]. As demonstrated in an included review [58], the evaluation of DHIs is critical to assess progress, identify problems, and facilitate changes to improve health service delivery and achieve the desired outcomes.

The third was to achieve evidence-based clinical use and increase DHIs’ adoption and uptake (14/45, 31%) [53,55,56,58-62,64,65,67,69-71].

The fourth was to drive ongoing investment (3/45, 7%) [53,70,71]; without compelling economic supporting evidence, the proliferation of DHIs will not occur. Therefore, ensuring the sustained clinical use, successful implementation, and adoption of and continued investment in DHIs require more evaluative information. This helps ensure that resources are not wasted on ineffective interventions [53].

The fifth was to inform health policy practice (3/45, 7%) [55,56,61]. As the 2 included articles stated [55,56], ongoing evaluation and monitoring of DHIs is critical to inform health policy and practice. In addition, in terms of the varied evaluation objectives, the evaluation activities serve different stakeholder groups, including program investigators, evaluators, and researchers; designers, developers, and implementers; end users, patients, and health care providers (HCPs); clients and investors; and governments and policymakers.

Among the included studies, evaluations were carried out at various stages of the intervention to fulfill the 5 evaluation objectives. Our findings showed that most reviews reported feasibility, efficacy, and pilot studies (32/45, 71%) [53,54,56,58-64,67,69-89] and then investigated effectiveness (20/45, 44%) [53,58,60,61,63,65,69,70,72,75,79,80,82-84,86,​90-93] and implementation studies (20/45, 44%) [54,56,58,61,62,64,68,70,73,75,78,81-83,85,87,88,90,91,94]. Notably, some reviews included >1 type of study. Our findings show that the timing of evaluation can be directly at pre- or postintervention [53,56,59,60,62-65,69-72,77,78,81,82,84,85,​87,88,90,91], at the baseline point or after a short- or long-term follow-up intervention [58,59,62,65,67,70-72,76,77,79,81,​82,84,85,88,91,95,96], during intervention use [76,85], continued monitoring [56,75], and even at dropout [53]. One study [84] suggested providing a period of technical training and conducting a baseline test to reduce the evaluation bias caused by individual technology familiarity and novelty. As demonstrated by another study [53], pre- and postintervention assessments using clinical trials can measure intervention effectiveness (eg, patients’ blood glucose levels). In terms of the timing of data collection, 1 included study [53] suggested that evaluations directly after the intervention are appropriate so that the users retain fresh memories of the experience. To sustain intervention outcomes over a longer period, longitudinal evaluations and long-term follow-up evaluations were recommended in 2 studies [55,84].

The goals of this umbrella review were to systematically review the evaluation timing considerations, indicators, and approaches of digital PEx. Furthermore, we identified 5 typical evaluation objectives and related audiences. The timing of a digital PEx evaluation should be a critical consideration when conducting an evaluation study; however, we found limited information about when to measure digital PEx. Moreover, the identified evaluation indicators are often heterogeneous and appear to be related to the different aspects of digital PEx. In terms of evaluation approaches, various theories were reported in the included papers. Furthermore, we noted that not only did the evaluation methods differ between the reviews but also the classification bases or perspectives used to describe these methods. Following our findings on when to measure, what to measure, and how to measure digital PEx, we generated a step-by-step evaluation guide and proposed 6 research directions for future studies.

DHIs change throughout the product life cycle, so to provide better-quality results and evidence-based health practice, evaluations need to be incorporated into the intervention maturity stages [55,56,58,80]. Our findings showed that many studies were not performed in a real-world setting for a long period, and most studies were either feasibility or pilot studies; these results are directly in line with previous findings [56,65,73,82,85,89,97]. Pilot or feasibility studies can help improve new intervention development but only provide limited evidence for increasing sustained clinical use and large-scale practice [58,80]. Two studies [55,77] reported a lack of information on the long-term experience. Others have shown that some solutions may be less sustainable outside the trial context [80,85]. In addition, it is possible that participants were more adherent during the study period and decreased their use of the apps over time [74]. Therefore, some authors call for further research on digital PEx when incorporating the DHIs into existing health care services and processes [76]; there is a need to move DHIs from promise into policy and practice [56].

One study [72] reported significantly different evaluation results before and after the treatment. It is likely that patients’ initial emotional state or understanding of DHIs may affect their final PEx evaluation outcomes. Therefore, a baseline test on individual differences would be a valuable step to limit evaluation bias, as noted in a previous study [84]. We found that the data gathered could occur at a specific moment or at different time points along the care pathway to reflect a rapid or delayed digital PEx. Thus, posttreatment evaluations should account for the recall bias caused by the time delay between treatment and recollection of experience, as has been noted in previous studies [53,76]. In line with other studies [29,64], we believe that real-world testing and direct feedback from actual users will help improve the usability of DHIs and directly benefit new users.

In comparison with intervention outputs and health care system impact, we discovered more evaluation indicators related to patient outcomes. We assume that this is owing to the consideration of the strength of the evidence and duration of the study. Patient outcomes enable the identification of patients’ actual experiences and reactions in uncontrolled settings, providing evidence for clinical use and further improvements. However, intervention outputs seem more suitable for exploring experts’ (eg, designers, health care professionals, and policymakers) or patients’ anticipated understandings of DHIs in the early stages of design and for addressing any potential system barriers. The health care system impact can be useful in predicting the sustainability of the DHIs on a large scale through a long-term study.

We used a set of themes and subthemes to describe each category. For instance, patient outcomes include emotional, perceptual, capability, behavioral, and clinical outcomes, as noted in 2 studies [60,75]: one study categorized the variables of patient engagement as behavioral, cognitive, and emotional outcomes, whereas the other study used biomarkers, perceptions, and behaviors to describe patient clinical outcomes with regard to DHIs. Furthermore, we noted that the evaluation outcome of one indicator is often unable to anticipate the outcome of another indicator. For instance, some patients reported high acceptance of a certain DHI, but they rarely used it [95]. Aligned with the arguments among the differences between patient satisfaction, PEx, PREMs, and PROMs [35,37,47,48], our findings indicate that digital PEx evaluations are not equivalent to the measurement of patient satisfaction, PEx, PREMs, or PROMs, but that these measures can be used to assess some of the digital PEx. We showed that the priorities of the evaluation indicators can differ between projects. In terms of what to measure first, as stated in a previous study [107], the goal of evaluations should be to focus on those processes that should be optimized by the digital catalyst. Furthermore, the evaluation indicators need to be continually updated as the DH landscape is rapidly evolving and the technology infrastructure is constantly shifting [54].

As demonstrated in an included review [58], an evidence-based theoretical evaluation framework is helpful in informing the evaluation process. Across the included reviews, we found that not only specifically designed evaluation theories were used to guide the evaluation activities but also theories from other fields were adopted to assess the evaluative data. We identified various traditional approaches across the included reviews. In addition, our results showed that more than half of the included reviews reported RCTs in their studies. RCTs were recommended in 2 reviews [108,109] to evaluate DHIs for stronger evidence. However, a recent systematic review [110] noted that only a handful of clinical decision support systems have been tested in this way. Others argued that there is a tension between the amount of time needed for evidence generation with traditional approaches and the speed of digital product development and iterative upgrading [16,18], which requires more innovative methods for fast evidence generation [18].

We identified a wide range of evaluation methods and instruments, although most were modified based on the evaluations for traditional face-to-face treatment or usability testing in human-computer interactions. This is also in line with the findings from previous studies [53,61,72,79]. Semistructured interviews and questionnaires were the most common evaluation methods for collecting evaluative data among the included reviews, which is in line with previous studies [53,76]. Semistructured interviews are the key methods used to understand the details of UX [59,61,62,65,66], whereas questionnaires are often modified from existing assessments to assess large-scale interventions [53,61]. It is likely that more in-depth, observational data collection methods are necessary to better capture experience data [53,66]. The use of a descriptive approach might be appropriate for a smaller sample size, collecting qualitative data through surveys, focus groups, and interviews [76]. Standard functional questionnaires may be preferred when DHIs are compared with other interventions [53]. However, we found that detailed interview outlines or questionnaires were generally not published, as mentioned in another study [53]. Comprehensive information on user evaluation methods and results is often lacking [65]. The determination of evaluation approaches depends on the specific context. In alignment with 2 studies [4,40], we state that the choice of evaluation approaches heavily depends on evaluation objectives, timing, indicators, and evaluation requirements and resources. An included review [58] recommended using multiple research methods, such as combining qualitative, quantitative, co-design principles, and process measures, for evaluation designs.

Thanks to the use of digital technologies [54,75], patients’ illness experience and what they feel when participating in a health care intervention can be monitored. However, we found that these may blur the boundaries between interventions, monitoring, and evaluations. For example, the diary function can be used as an intervention feature (eg, a self-management diary to track symptoms and identify exacerbations [78]), as a monitoring tool (eg, diary entries [97] or adherence [90]), or as an evaluation method (eg, to capture user feedback [53]). Furthermore, a study indicated that with the advancement of technology, the ability of DHIs to collect “passive data” for assessing digital PEx may gain more attention and eventually eclipse the utility of DH-aided self-report [74]. Finally, we believe that involving multiple stakeholders is not only essential in the design process but is also a requirement for the evaluation process. Both end users and experts can contribute to the evaluation activities [53]. This aligns with a recent study that suggests that digital solution evaluation requires collective efforts from multiple parties, such as health authorities, HCPs, and manufacturers [18].

Our analysis showed that the evaluation of a DHI follows the same evaluative process as that of traditional interventions, which supports a previous study [53]. To make the evaluation findings more comparable, more rigorous studies and standardized evaluations are suggested, including unified terminology [53,65,68], predefined measurable indicators [79,81], standardized methods [61,66], validated instruments [84,96], uniform time intervals [84], and adequate patient selection [81]. Intervention characteristics (eg, aims, expected outcomes, elements, length, frequency, and duration), study designs (eg, sample size, period, regulations, investigator, evaluators, recruitment, ethics, topic guides, or questions asked by the researchers), objectively measured patient health outcomes, and adverse events should be carefully considered when conducting and reporting an evaluation study [53,60,69,84].

Inspired by the challenges for the evaluation of DHIs [18]; shaped by the Performance of Routine Information System Management framework [100], the monitoring and evaluation DHIs guide [29], PEx measures [37], and our previous publications on influencing factors and design considerations of digital PEx [3,111]; and based on the findings of this study, we have developed a step-by-step evaluation guide for DH innovators, such as designers, developers, and evaluators (Figure 2): The first step is to clarify the evaluation objectives and determine the target audiences for the evaluation. We proposed 5 typical evaluation purposes and their related audiences. The selection of evaluation objectives can help determine the stages for evaluating the DHI. For example, we consider effectiveness and implementation studies more appropriate for achieving evidence-based clinical use and increasing adoption and uptake compared with efficacy studies. The second step is to determine the intervention contexts and foci in terms of the intervention maturity stages, including efficacy, effectiveness, and implementation. The determination of the evaluation stage is not only because of the evaluation objective but also because of the current condition of the DHI. The determination of the evaluation objectives and identification of the evaluation stage affect the consideration of influencing factors and evaluation indicators at the next step. For example, the evaluation of patient outcomes in an uncontrolled setting can provide evidence for clinical use and further improvement. The third step includes a set of influencing factors (ie, inputs and processes) and evaluation indicators (ie, outputs, outcomes, and impacts) that can be used for further formulating evaluation constructs. The former is more appropriate for formative evaluations, which often occur during the design and development process, whereas the latter is suitable for summative evaluations, which often occur during and after the implementation process. In the fourth step, we present 2 types of evaluations. On the basis of the frequency of evaluations, we can capture momentary experiences before, during, and following an intervention or monitor continuous feedback throughout the intervention. With regard to the time interval between the intervention and evaluation, assessments can reflect immediate experiences directly after the intervention or recalled experiences over an extended period. In the fifth step, we present various evaluation approaches that can be used to plan and carry out specific evaluation activities, such as study designs, data collection methods and instruments, and data analysis approaches. The consideration of study designs often affects the strength of the evidence and determines the data collection and analysis methods. Behavioral data may provide stronger evidence than opinion data. Qualitative methods, such as interviews, are more appropriate for collecting in-depth experience data for a smaller sample size in the early intervention development stages, and quantitative methods, such as questionnaires, are more suitable for investigating experience data at a large scale or comparing it with other interventions during or after the implementation stages. In the sixth step, we proposed 6 questions for the evaluation investigators to guide them in reporting the evaluation results and 5 questions to inspire them to generate theoretical or practical implications for responding to the related stakeholder groups. The answers to these 11 questions should reflect the evaluation processes and serve the evaluation objectives.

The guide can be used when setting up a digital PEx evaluation plan or guiding evaluation practice. Notably, the interrelationships between these 6 steps are not fixed; the entire evaluation plan is an iterative process; and the decisions made at the previous steps may influence the following steps, and vice versa. In addition, other considerations beyond this guide can also impact the evaluation process, such as human, time, and financial resources. Our guide presents an ideal way to conduct the evaluation of digital PEx; however, in the real world, the order of these steps may be changed or some steps may even be skipped depending on the specific project context. For instance, in certain assessment procedures, selecting an evaluation construct, such as usability, may come first, rather than taking evaluation objectives or target audiences into account. We developed this guide based on our literature analysis. It provides an overview of the most common evaluation timing considerations, indicators, and approaches used to collect digital PEx–related data. However, it may be incomplete and require updating in the future. For example, owing to the methodological limitations, we did not provide concrete recommendations on which evaluation approaches are superior for what types of DHIs. We believe that without providing a specific context and concrete project requirements, it is difficult to draw a conclusion.

This study had several limitations. First, we noted possible resource restrictions and the newness of the field, which may have led to missing articles. To overcome this, we searched 3 databases and used the snowballing method. In addition, we performed an updated literature search to check whether there were any meaningful new insights that would significantly change our conclusions. To our knowledge, although there were some newly published reviews in this area, we confirmed that our results were quite stable, and the newly identified studies were unlikely to significantly impact our results. Second, we could not perform a quality assessment because of the diversity in reviews and methodological limitations. As previous studies on investigating umbrella reviews have indicated, there are currently no official standards for determining the certainty of evidence when performing umbrella reviews [112,113]. In addition, among the included reviews, only 53% (24/45) of the studies assessed the risk of bias and used diverse quality assessment instruments. After a thorough attempt, we found that none of these instruments were suitable for assessing the various reviews included. These encompass systematic reviews, scoping reviews, comprehensive overviews, and general literature reviews, which incorporate various primary and secondary studies extending beyond RCTs and nonrandomized studies of interventions. This diversity makes it challenging to use a standardized method for assessing the quality of evidence across the extensive range of included reviews. However, we tried to reduce the risk of bias by only including reviews that were published in peer-reviewed journals. Third, reviewing secondary research may have led to the omission of crucial information and reporting bias. To minimize potential bias, we used the most common terms used across the included papers as themes and subthemes. Owing to the cross-disciplinary nature of the topic, there is a lack of consistency or clarity in the terminology used to describe the evaluation indicators and approaches. For instance, in one study [55], a user study was pitched at the same level as interviews or observations, whereas in another study [56], interviews and questionnaires are methods that form part of “user study” research. In addition, information related to the evaluation approaches was reported at different levels among the included studies. For example, one study provided information related to data collection methods, such as focus groups, design sessions, and questionnaires [55], whereas another study reported information related to study designs, including qualitative, quantitative, and mixed methods designs [71]. These inconsistencies complicated the comparison between different studies. To counter this, we analyzed the different classification bases behind these study designs. Finally, we could not draw firm conclusions regarding which evaluation approaches are better suited for which types of DHIs. Owing to the nature of this study being a review of reviews, details such as the characteristics of DHIs are not always adequately covered in each included review. In addition, the included reviews contained a large number of primary studies, which makes referring back to each primary study challenging. The lack of details about the characteristics of these primary studies limits the classification of DHIs in this study. Moreover, the included reviews represent a wide range of studies, making comparison across the included reviews challenging. In addition, we decided that this was out of the scope of this study. When planning this study, we deemed it more appropriate to initially offer an overview of diversities rather than begin with a best practice recommendation. Consequently, we aimed to map possible evaluation considerations and approaches for evaluating digital PEx, instead of discussing which approach is better. However, we encourage future research to address this issue.

Considering our research limitations, to further facilitate evaluations of digital PEx, we propose 6 future research directions. First, further research into how one indicator mediates another indicator’s impact on digital PEx is required. For example, is there a correlation between clinical outcomes and perceptual outcomes? To explore this, we performed an experimental study to investigate whether patients’ initial pain perception and technology acceptance (using virtual reality distraction) affected their experienced pain during wound care treatment. Our findings will be published in a future article. Second, the variables that influence the selection or prioritization of evaluation indicators and approaches should be further investigated. For example, it would be valuable to investigate whether some evaluation indicators and approaches are better suited for evaluating certain types of DHIs according to the strength of the evidence and the length of the evidence generation time. Third, agreement is needed on standardized measures to evaluate digital PEx, particularly innovative approaches for faster and high-quality evidence generation. In a follow-up interview study, we aim to summarize the often-used agile evaluation approaches based on designers’ experiences. Furthermore, in cases where an interview or questionnaire is used to collect evaluative information, we recommend reporting the detailed interview outlines or questionnaires together with the evaluation results. Fourth, research is needed on how the intervention maturity stages and timing of the evaluation of the evaluation affect the evaluation results. Fifth, future studies should not only investigate whether DHIs achieve the intended results in a research setting but also assess the long-term digital PEx regarding the uptake, institutionalization, and sustainability of evidence-based DHIs in a given context and a real-world setting, including policies and practices. Finally, research is required on how to analyze and respond to the evaluative data. We recommend that future evaluation research and practice provide theoretical and practical guidance on how to use the evaluative information.

To effectively improve the digital PEx, knowing how to evaluate the digital PEx is as important as knowing what factors influence the digital PEx and how to design the digital PEx. Evaluating digital PEx requires clarifying the evaluation objectives, identifying stakeholder groups, considering reasonable evaluation timings, choosing relevant evaluation indicators, and selecting appropriate evaluation approaches. Following our previous publication on the influencing factors and design considerations of digital PEx [3], we first identified 5 typical evaluation objectives and related stakeholder groups. We then described potential evaluation timing considerations in terms of 4 intervention maturity stages and 3 evaluation timings. We collected knowledge on evaluation indicators of digital PEx and grouped them into 3 categories: intervention outputs, patient outcomes, and health care system impact. These were then classified into 9 themes (intervention functionality, usability, care quality, patient emotional outcomes, perceptual outcomes, capability outcomes, behavioral outcomes, clinical outcomes, and system financial outcomes) and 22 subthemes. Furthermore, we noted a set of common study designs, data collection methods and instruments, as well as data analysis methods, which can be used or adapted to evaluate digital PEx. On the basis of our findings, we developed an evaluation guide to help DHI researchers, designers, and developers further evaluate digital PEx. Finally, we recommend 6 directions for further research on digital PEx evaluation. Multimedia Appendix 5 (the PRISMA checklist) provides more detail on the structure of this review.