The use of electronic health (eHealth) is still driven by assumptions about the benefits of eHealth rather than its evidence [1]. With time, the trustworthiness and robustness of eHealth to facilitate safe and cost-efficient care are being questioned because of a lack of evidence [2]. This may trigger reluctance in investing and developing policies related to eHealth in organizations as well as in countries [3].

The term eHealth was introduced in the 1990s [4]; however, it was hardly in use until 1999 [5]. According to Eysenbach [5]:

Eysenbach believes eHealth stands for more than internet and medicine [5]. In our study, eHealth was used as the broadest umbrella encompassing everything that comes within information and communication technology and health care, including telemedicine, mobile health, and health informatics.

Systematic evaluation can capture the evidence and criteria that evaluative judgment is based on and curtail the sources of biases [6]. The quality of an evaluation is assessed by the credibility of evidence assembled through it and using evidence in refining the policies and programs [7]. Evaluation of eHealth interventions is complex because of several reasons (eg, the need for multidisciplinary collaboration [8], context dependency [9], and differences in epistemological beliefs considering the interventions in clinical studies or including social aspects as well [10-14]). Therefore, variety exists concerning how the evaluation of eHealth interventions is performed and presented. Garnering robust evidence through evaluation becomes difficult because of these circumstances.

It is relevant to understand evidence-based medicine (EBM) while discussing the importance of evidence in eHealth interventions. A common query is how EBM can help generate evidence for eHealth interventions [15]:

As per this definition, evidence in EBM is conspicuously related to the clinical aspect. Although it is argued whether EBM is only about randomized controlled trials [16] or not [15], it is quite explicit that EBM usually does not contemplate anything outside clinical practices. However, eHealth interventions have more aspects to evaluate besides the clinical aspects. An extensive assessment of the aspects including sociotechnical aspects is needed through each phase of the technology’s life cycle while evaluating eHealth interventions [10,11,17]. Hence, to gather evidence from eHealth intervention, the evaluation process requires a distinct approach than what is usually put forward within EBM.

Our objective was to elucidate how the evidence of effectiveness and efficiency of eHealth can be generated through evaluation. Consequently, a literature review was conducted to understand the evaluation process regarding both theories of eHealth and the practices in case studies of eHealth interventions. We decided to employ a broader perspective at the beginning of the review process to achieve our research objective. As the literature review progressed, our research objective narrowed, and the research questions were redefined several times. However, the objective was always to understand the evaluation of eHealth from a comprehensive perspective. It was pertinent to recognize the phases of eHealth interventions where evaluation occurs and the aspects of effectiveness and efficiency that are evaluated during such interventions. Our 3 research questions were as follows:

Finally, we presented an approach to evaluate eHealth interventions by developing a model—Evidence in eHealth Evaluation. To our knowledge, this model is a novel way of looking into evaluation of eHealth interventions for comprehensive evidence. This conceptual model was based on the findings of the literature review.

On the basis of the summary of studies mentioned in the previous section, the papers (N=46) were classified into 2 categories: (1) conceptual exploration of eHealth interventions (n=21) and (2) case studies of eHealth interventions (n=25). Using the summary table (Multimedia Appendix 1), the papers in the first category were divided into 2 groups: (A) evaluation of distinct eHealth intervention phases (n=10) and (B) aspects of evaluation in eHealth interventions (n=11).

This subsection concentrates on how evaluation is conducted in distinct phases of an eHealth intervention. From the selected studies in group A, a spectrum of phases of an eHealth intervention was identified including design, pretesting, pilot study, pragmatic trial, evaluation, and postintervention. Table 1 provides a compilation of the phases along with the area of focus and key activities within each phase.

It can be ascertained from Table 1 that evaluation is not commonly performed in the design and pretesting phases. Although some researchers see evaluation as an ongoing process throughout the intervention [12,21,23], others believe there is value in evaluating the intervention at the end of the study period [12]. Concerning the latter, evaluation itself is one of the phases of the intervention.

We determined the aspects that researchers evaluated to gather the evidence of efficiency and effectiveness of an eHealth intervention. To understand the aspects of efficiency and effectiveness, we compiled the dimensions of eHealth interventions that are proposed to be measured by the studies categorized in group B (n=11). While excerpting the dimensions during the qualitative analysis, we found that they can be classified into 7 aspects: organizational aspect, technological aspect, human and social aspect, clinical aspect, cost and economic aspect, ethical and legal aspect, and transferability aspect. Table 2 exhibits these aspects along with their key area of measurements.

The papers categorized as case studies of eHealth intervention show substantial variation in the approaches taken to evaluate the interventions. The use of standardized frameworks and theories for evaluating the interventions was hardly noticed in these studies. Multimedia Appendix 2 provides the result of the analysis [20,42-65].

To summarize Multimedia Appendix 2, it can be said that out of 25 case studies, 16 (64%, 16/25) evaluate clinical aspects, 12 (48%, 12/25) evaluate human and social aspects, 5 (20%, 5/25) evaluate technological aspect, and 4 (16%, 4/25) evaluate organizational aspect. The other aspects discussed by the theory-based literature (Table 2) are not evaluated in any of the case studies.

From the papers reviewed in this study, it has been revealed that numerous approaches to conceptualize and conduct eHealth intervention coexist. Several attributes of evaluation of eHealth intervention have become known through this review. There are vivid differences between how evaluation is conducted in practice (case studies) and how it is discussed in the conceptual papers. Moreover, a wide range of variety prevails within each group. Evaluation has been depicted as both static action performed at the end of the intervention [20,24,26-28] and dynamic action dividing it further into summative and formative evaluation [12,21,23]. Depending on the evaluators, evaluation can also be classified into internal and external assessment [12]. However, all case studies conducted evaluation at the end of the intervention. Although several aspects of evaluation have been found in conceptual papers [32-39,41], the case studies mostly evaluated clinical [20,42-44,46-49,52,53,55,59-61,64,65] and human and social aspects [42,46-49,53-55,59,61,63,65].

Although analyzing standardization of eHealth evaluation was not an objective of this review, the variability found in the studies compelled us to think whether it hinders the sharing of evidence among eHealth interventions. Scarcity of evidence, in turn, could delay the growth of eHealth. It is noticeable that the need for evidence is not clearly stated in any of the papers. The evaluation of the empirical studies typically focused on the success or failure of the technology (eHealth) in that intervention. It seems that the numerous efforts taken in eHealth research are still quite disconnected, and they are thus unable to create a synergic effect on the growth of eHealth.

It was noticeable from the review that although some studies elaborate on the aspects of evaluation for eHealth intervention (studies from group B) and some organize the evaluation of intervention into certain phases (studies from group A), no visible interaction has been made so far between these 2 groups of works (ie, what to assess during what phase). There is a gap where a connection can be made between the distinct phases of intervention and aspects of evaluation. This led us to develop the Evidence in eHealth Evaluation model (Figure 2), which exhibits the accumulation of evidence by assessing certain aspects of evaluation in distinct intervention phases. The Evidence in eHealth Evaluation model is a novel approach to investigate the evaluation of eHealth interventions.

In this study, an eHealth intervention comprising all 6 phases (ie, design, pretesting, pilot study, pragmatic trial, evaluation, and postintervention) was conceived as a comprehensive intervention. We propose that the generation of robust evidence of effectiveness and efficiency would be plausible when the evaluation is conducted through all intervention phases. Moreover, the aspects of evaluation (ie, organizational aspect, technological aspect, human and social aspect, clinical aspect, cost aspect, ethical and legal aspect, and transferability aspect) would vary in each phase depending on activities of the phases. For example, when an eHealth intervention initiates with the design phase, the decisions are made based on the evaluation of the technological aspect and cost of technology development. The formal evaluation of the intervention begins in succeeding phases. The evaluation of technological, human and social, and cost aspects occurs in the pretesting phase. During the pilot study phase, the focus of evaluation shifts primarily to clinical aspect followed by human and social, technological, and ethical and legal. Depending on the evidence garnered in the pilot study, the intervention may proceed to the next phase or go back to the design phase. As the intervention is scaled up in the pragmatic trial, the evaluation is conducted to identify whether the technology-enabled care can be executed within the realistic layout of an organization. Hence, the key areas of evaluation in this phase are organizational and cost aspects along with other aspects such as clinical, human and social, technological, and ethical and legal. The last phase of gathering evidence is summative evaluation, where all the aspects are assessed including transferability. This comprehensive evaluation process gradually accumulates the evidence that reaches its peak in the summative evaluation phase and is used in the postintervention phase to make future decisions. The model also exhibits how the involvement of patients increases continuously from the design phase to the pragmatic trial, escalating the complexity of the evaluation process.

The inclusion of relevant information regarding other aspects besides the clinical aspect (eg, organizational aspect and cost aspect) allows creating reusable knowledge to facilitate the transfer of results to other settings [36] and to obtain useful insights for long-term implementations. It can be assumed that assessing all the aspects in a single study might conclude with a confounding result, as all the aspects are interrelated and inferior performance in an aspect can affect the performance in other aspects, which might create a misleading result. Therefore, our model proposes to extend the evaluation process throughout the 6 phases of eHealth intervention. The underlying idea is to assess specific aspects in each phase instead of evaluating all aspects in a single phase. This way of evaluating eHealth interventions can capture comprehensive evidence that is usually dynamic and complex in nature.

We acknowledge the fact that an eHealth intervention including all the phases presented in the model will become cumbersome because of high resource consumption. This conceptual model is not a prescription but just a way to show the progression of evidence in eHealth intervention in a reliable manner.

To date, the importance of evidence has not been discussed as rigorously as the diverse research approaches and evaluation frameworks have been discussed. In this study, the Evidence in eHealth Evaluation model was developed to exhibit how evidence can be generated by evaluating certain aspects in each intervention phase. Assessing distinct aspects during distinct phases is a novel concept discussed in this study and requires further analysis. Moreover, this study implies an inconsistency between the literary concepts and practices of eHealth intervention, which has not been noted until now.

As health interventions are context-specific, the transferability of results from eHealth studies may be difficult. Moreover, neither the conceptual nor the case studies suggested the long-term implementation of specific technology into the health care settings where it has been tested. We believe that this might be caused by a lack of or insufficiency of preliminary evidence of the effectiveness and efficiency after conducting the micro-trials or short-term tests on the effects of the technology. Consequently, it appears that lack of evidence hinders the growth of eHealth. Further research directed toward evidence-based evaluation can not only improve the quality of that intervention study but also facilitate long-term implementation of eHealth in general. We conclude that the development of more robust and comprehensive evaluation of eHealth studies or an improved validation of evaluation methods could ease the transferability of results among similar studies. Thus, the resources can be used for supplementary research in eHealth.

This study is not devoid of limitations. We tried to include and analyze as many papers as possible; however, unknowingly and unintentionally, some papers may have been omitted. Furthermore, regarding the model, its development is in the preliminary stages; therefore, it cannot be compared with other validated frameworks.