Cocreation and quality improvement methodologies reported previously [20] were used to consolidate a pragmatic assessment protocol that was applied in the 4 digital health prototypes supporting the interventions (studies 1-4) described below.

Study 1 involved home-based noninvasive ventilation (NIV) of patients with hypercapnic respiratory failure. The study addressed enhanced management of chronic patients requiring specialized respiratory care for home-based NIV by means of a mobile app for patient self-management [24].

Study 2 involved prehabilitation of high-risk patients undergoing major abdominal surgery. The study assessed the potential of the supporting digital health tool, PREHAB [25], to enhance collaborative work among health professionals and patients using a mobile app for self-management at the community level.

The third cluster involved community-based care of frail chronic patients. This cluster of digital health interventions included 2 studies addressing specific objectives: (1) Study 3 assessed an adaptive case management platform [26] for community-based care of chronic patients; and (2) Study 4 investigated the potential of a secure communication platform, prototyped during 2019, for enhanced management of frail chronic patients [27] (Table 1). The details of the digital health interventions are provided in Multimedia Appendix 1.

The premarket assessment of the 4 digital health interventions (Table 1) piloted during the period 2017-2019 primarily focused on assessment of technical robustness and usability. The former was assessed with a technical log book on the cloud that was updated daily with technical issues and suggestions for improvement from all study participants. The end users’ perceived usability and satisfaction were assessed by means of the System Usability Scale (SUS) [29] and Net Promoter Score (NPS) [30], alongside general questions about satisfaction.

Besides the technical and usability assessments mentioned above, compliance with other operational aspects, such as privacy and security, interoperability, transferability, and value generation of the accompanying integrated care services, was part of an overall evaluation framework [31], but the premarket analysis did not systematically analyze this. It is worth mentioning that digital support for the 4 digital health interventions (Table 1) was designed to operate on top of existing hospital information systems to minimize the need for ad-hoc integration via standard application programming interfaces.

When writing this manuscript, we adhered to the Consolidated Criteria for Reporting Qualitative Research (COREQ) [32]. All information retrieved from the technical and usability assessments of the 4 digital health interventions (including the end users’ interviews) was processed according to the protocol-specific ethics statement mentioned in the Ethics Approval and Consent section.

In recent years, digital health technology has developed rapidly in the market as software-only novel therapies or has been embedded into medical devices or clinical workflows as a companion MDSW device in the market. MDSW is defined, under EU Regulation (EU) 2017/745-MDR [16], as a software, used alone or in combination, that is intended by its manufacturer as a medical device for human beings for a specific medical purpose: diagnosis, prevention, investigation, monitoring, prediction, treatment, alleviation, prognosis, and prediction.

The analysis of the European MDSW regulatory frame was focused on the EU Regulation 2017/745-MDR [16] and the fast-track process generated by Germany’s Federal Institute for Drugs and Medical Devices, known as the “BfArM” guidelines for the evaluation of digital health applications (DiGA) [14]. Likewise, for the United States, we considered the FDA 21 CFR Part 820 [15]. Moreover, the EU General Data Protection Regulation (GDPR) [18] and its American counterpart, the US Health Insurance Portability and Accountability Act (HIPAA) [19], were also considered in the analysis.

Within the postmarket surveillance analysis at the EU level, we collected the numbers of devices that had been recalled, irrespective of their risk level and the product end user, over 5 years (January 1, 2017, to December 31, 2021) from the German BfArM website [33]. The results included malfunctioning software that may result in a severe adverse event, device deficiency, incident, or serious incident.

For the postmarket surveillance analysis in the United States, we collected data from the following 2 public databases: Manufacturer and User Facility Device Experience (MAUDE) database [34] and MEDSUN Reports [35]. Considered MDSW recalls included software failures in terms of security flaws, privacy risks, internal controls, technical controls, physical controls, and implementation.

As a result of the experience-based cocreation process [20], 4 domains of digital health system validation (Table 2) and 3 eHealth contextual strategies (Table 3) were considered essential for the assessment of digital health tools. Therefore, they were used to guide a thematic analysis on the assessment results of the 4 digital health interventions, by the author EB, as well as on the overview of the MDSW current regulations and the analysis of MDSW recalls of the past 5 years (2017-2021) in the European Union and the United States, by the author HWH.

Then, the author EB discussed the findings with the participants of each digital health intervention, the final users of the supporting digital health tools, and all other coauthors. For each of the 4 digital health interventions, the evaluation results were judged by the author EB according to the contextual eHealth strategy, defined by the role of the digital tools in the health care service.

Finally, we evaluated the thematic analysis results to generate recommendations for assessment of the sustained adoption of future digital health interventions.

Letters of medical ethics approval of the Ethics Committee for Medical Research of the HCB and signed informed consent forms were obtained for the 4 studies (study 1, HCB/2019/0510; study 2, HCB/2016/0883; study 3, HCB/2018/0803; and study 4, HCB/2018/0805).

A total of 99 chronic patients and 9 health care professionals were assessed during the interaction and cocreation process of the 4 different digital health interventions piloted at the HCB (studies 1-4). Assessment results of the technical and usability performances are summarized in Table 4. See Multimedia Appendix 2 for further details.

Most (20/27, 74%) reported incidences by end users had to do with the need to login with a username and a password that were easy to forget, which precluded the ease of use. However, on a Likert scale from 1 (very bad) to 10 (very good), the general impression of patients was scored 7.5, user friendliness was scored 8.2, and the ability to use the app without assistance was scored 8.5. This was in line with the mean patient usability score (78 out of 100).

Fifty percent (5/10) of incidences were due to comfortability and accessibility (the system forced the use of a random password when the password was reset) and 30% (3/10) were due to technology robustness. The remaining 20% (2/10) of incidences were due to various factors. As in study 1, the general impression and user friendliness was scored 8 (out of 10) and the ability to use the app without assistance was scored 7.5 (out of 10). In contrast, the patient usability score had a mean value of 67, which is considered an average usability grading. With respect to the experience of professionals, a neutral experience using the web backend for professionals was reported, with an overall satisfaction score of 5 (out of 10) and a mean SUS score of 52.

Most (4/7, 57%) observations during the pilot were due to lack of robustness of the Bluetooth connection with the pedometer. Reported observations regarding motivation, reliability, comfortability, and accessibility reached 14% (1/7) each. In general, patients had a slightly positive experience (6/10) using the system, but its usability was graded low (SUS score of 56). In case of professionals, perceived usability was graded lower (SUS score of 43); thus, the professionals involved would not recommend the CONNECARE system.

High proportions of observations were due to usability (11/18, 61%), and comfortability and accessibility (6/18, 33%) issues, mostly due to lack of robustness of the multimedia communication channel. In general, most patients had a positive experience using the system, which was reinforced by the fact that the median overall satisfaction score was 7.8 (out of 10) and the mean patient usability score was 76. However, the NPS reported by professionals (n=5) was negative (−80%), but since the median overall satisfaction score was 5 and the perceived usability score was 54, we could consider that professionals had a neutral experience using the prototype.

The review of the German regulatory framework (BfArM) retrieved a total of 556 postmarket events that fitted the research requirements focused on digital health tools (representing 13% of all events that included drugs, assays, and medical devices). Likewise, the review of the MAUDE and MEDSUN databases (United States) found a total of 114 software-related issues, representing 18% of all queried events. See Multimedia Appendix 3 for details.

The vast majority of reported issues were related with software problems, incorrect results, data mismatch, error codes, system unexpected shutdowns, incorrect procedures, and cybersecurity-related aspects, which folded into the robustness (620/665, 93.2%) and privacy and security (26/665, 3.9%) components of the technical domain mentioned in Table 2. Therefore, software technical defects were reported to have the highest potential risk of harming end users.

No issues were reported with respect to transferability, smartness, safety, or medical benefit, whereas very few issues were reported in relation to interoperability (2/665, 0.3%), ease of use (8/665, 1.2%), and feasibility (9/665, 1.3%).

The overarching analysis of the process of premarket assessment of the 4 digital health interventions piloted at the HCB, as well as the analysis of current MDSW regulations and postmarket surveillance in the European Union and United States provided a source of experience-based knowledge that is described below and summarized in Tables 5 and 6 in terms of recommendations for each of the 4 digital health evaluation domains (Table 2) and lessons learnt toward sustained adoption in the 3 eHealth contexts considered (Table 3).

The study aimed to update health professionals on the current landscape of MDSW for enhanced management of chronic patients. The research generated recommendations on target evaluation domains and eHealth categories through an overarching analysis of 3 sources of information: (1) premarket evaluation of 4 pilots carried out at the HCB, using ongoing technological developments; (2) assessment of the regulatory frames of MDSW in the United States and Europe; and (3) postmarket surveillance reporting from the same 2 areas of the world.

Evaluation results of the 4 digital health interventions piloted at the HCB showed that patients tended to score higher than professionals in terms of the experience with supporting digital health tools, and in general, they would recommend the use of supporting digital health tools. This can be partly explained by the fact that the technology readiness level of the assessed digital health interventions was rather low at the precommercial stage, which most likely had a negative impact on the perceived usability by health care professionals who had to lead with technical issues at the same time than with the inherent complexities of case management. Moreover, the lack of integration with existing hospital information systems influenced the poor results with respect to the experience of health professionals. The 4 premarket digital health interventions were not considered mature for integration with existing health information systems, and in general, hospital information technology departments tend to reject integration of noncommercial digital health tools, which precludes usability, especially among health professionals who are not strongly motivated.

As mentioned above, EU and US MDSW regulations include premarket MDSW assessment with respect to clinical data, product information, performance testing, labeling, benefit-risk assessments, residual risks, etc. However, MDSW manufacturers should also plan, establish, document, implement, maintain, and update a postmarket surveillance system in a manner that is proportionate to the risk class and appropriate for the type of device. This system should be an integral part of the manufacturer’s quality management system and should be notified to the corresponding regulatory body.

Overall, the postmarket surveillance review of both German and US regulatory frameworks confirmed the crucial role of software verification and validation, the voluntary testing of cybersecurity, and the need for testing user interfaces.

The inherent heterogeneity of the 4 digital health interventions considered in this study represents both a strength, because it reinforces transferability of the assessment approach, and a limitation, because it precludes comparability of the results among the 4 study protocols. However, the cocreation process and the application of the same structured evaluation protocol over the 4 digital health interventions contributed to the evolution of the mindset of health professionals toward the use of digital health tools. Specifically, the participation of all stakeholders in the overarching analysis concluded with the generation of a set of general recommendations for adoption in routine clinical practice.

The evaluation protocol focused on technical and usability performance because the primary objective of the 4 digital health interventions piloted at the HCB was to demonstrate the feasibility of the approach. If large-scale deployment is the primary aim, other functional, technical, and ethical aspects of the supporting digital health tools will need to be assessed.

The 4 digital health interventions were piloted within the context of research and innovation projects. This represents a clear advantage for stimulating cocreation of the supporting digital health tools, but establishes the threshold of required technological maturity at the prototype level and limits the transferability of the results beyond the boundaries of pilot settings.

Considered MDSW regulations in the European Union and United States neglect the tools and general requirements of other countries or regions of the world. Although similar premarket approval applications and postmarket surveillance tools are being put in place in recent years [11-14], they do not apply to the full range of technology readiness levels, that is, they are intended to be used to evaluate mature technology. In this respect, development and production phases of digital health tools (focus of the 4 digital health interventions assessed in this study) may help to generate more mature and robust digital health tools that are ready to be assessed by corresponding health technology assessment frameworks [11-14].

Technical performance and usability are arguably among the most important considerations with patient-oriented mobile and digital-based solutions [39,40].

The overarching analysis showed a clear link between premarket assessment and postmarket surveillance in terms of technical failures hindering stakeholder adoption, regardless of usability and acceptability success. Such technical failures can be overcome by generating not only robust and secure products, but also online open access databases (the likes of clinical registries for single diseases) where basic MDSW approval information, medical specialty, and algorithm details can be publicly shared, thus enhancing transparency and collaborative work. Two recent studies [41,42] explored this concept when applied to artificial intelligence MDSW.

Moreover, digital health apps must be easy to use for their intended purpose, require minimal effort to complete tasks, have minimal data entry burden, and allow the user to control preferences when appropriate (eg, notifications). Since systems can be designed for users with disabilities (eg, impaired vision, motor deficits, and cognitive dysfunction), design considerations must ensure that accessibility compliance reflects the target user audience and different potential users, including family members and caretakers. Moreover, to maximize acceptability, digital health solutions require input from clinicians.

Developing digital health tools not only implies technological robustness and usability, but also guarantees data privacy. It requires thinking about how the newly collected data will need to be shared with health care professionals and whether the intended use of the technology ethically makes sense. During the long process of creating and validating a digital health application (starting with an idea, followed by its implementation and dissemination in different application markets), many stages must be achieved, and each one has its own particularities and methodologies.

Accordingly, a redefinition of the digital health ambit is needed. While some evaluation models in terms of the maturity of digital health interventions have been proposed [43], they are either very “technology specific” or “hospital oriented.” Moreover, as acknowledged previously [43], none of the identified models can be used as an overarching tool to encompass the wide range of digital tools used in a complex context such as integrated care. Moreover, they highlight the lack of a holistic approach to identify influencing factors.

The maturity grading criteria for digital health explored in this study cover a wide scope of tools and policies that correspond to the abovementioned new model of the comprehensive understanding of digital medicine. Recently, a review was conducted on the use of digital technologies in health care [44], and not surprisingly, it proposed an approach similar to the one proposed in the 4 domains and 3 eHealth contexts (Tables 2 and 3) to assess the different elements of MDSW adoption.

Usability performance was consistently perceived higher by patients than by health care professionals. This can be partly explained by the fact that the technology readiness level of the supporting digital health tools was within the interval 5-6, and health care professionals had to lead with technical issues at the same time than with the inherent complexities of case management.

However, the active participation of health care professionals in the co-design and application of the evaluation protocol contributed to the evolution of the mindset of the health professionals toward the use of digital health tools in routine clinical practice.

The overarching analysis resulted in lessons learnt and recommendations that could contribute to the large-scale adoption of digital health tools.