A systematic literature review was undertaken on published studies reporting the design and development of digital tools to support populations with diabetes. The methodology adopted for the systematic review followed the University of York Centre for Reviews and Dissemination [23], the Cochrane Handbook on Systematic Reviews of Interventions [24], and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [25]. The review was registered on PROSPERO in April 2022 (CRD42022322040) and conducted according to our pre-established protocol. It was reported using PRISMA guidelines (Multimedia Appendix 1) [26].

We conducted a systematic review of studies on diabetes digital health apps to report their adherence to design and development guidelines [19,20]. We analyzed 6 years (January 2017-November 2023) of diabetes digital health app development papers published in English. We searched the MEDLINE and Embase databases using the strategy outlined in Table 1, with a combination of free text and Medical Subject Headings (MeSH) for the search terms. Embase and MEDLINE were used as they have the highest two-database recall rate (92.8%) compared to other databases [27]. The key terms we identified were combined using the Ovid platform’s operators “AND/OR.” Reference lists of the included papers were also searched, authors of key papers were identified, and their publication lists were checked for other relevant research. The searches were also rerun just before the final analyses to identify and retrieve any further studies for inclusion.

The citations and abstracts of the search results were imported into Covidence systematic review software [28], and duplicates were removed. The records identified in the search strategy were downloaded and uploaded to Covidence. When screening the studies, we included those on the design or development of digital health apps used exclusively by patients with diabetes that were available on a smartphone or tablet. There were no restrictions on the type of diabetes or the age of the participants. We excluded studies on diabetes health technologies not available on a smartphone or tablet, papers solely providing opinions, industry reports, studies on practitioner behaviors, and reports of individual users or cases. Table 2 lists the inclusion and exclusion criteria used. Eligibility was determined based on paper type, language, population, intervention, outcomes, and publication date. Studies were included if they reported on the use, design, or development of mobile apps.

Two reviewers (authors SLW and HM) independently screened the titles and abstracts of the selected studies for inclusion by applying eligibility criteria and selecting studies for inclusion in the systematic review. Any discrepancies in the findings were resolved through discussion with the third senior reviewer (author REH). One reviewer extracted relevant data from the included studies, and 10% were checked for accuracy by a second reviewer. Reviewer decisions and the reasons studies were excluded were recorded. A framework was created based specifically on the items included in DTAC [19] and the NICE DHG [20], where studies were scored on adherence (Table 3); each category was evaluated using a yes/no scoring system. The second reviewer checked 10% of the guideline scores against the full texts. The primary outcome of the systematic review was framework adherence scores for each diabetes digital health app included in the study using DTAC [19] and the NICE DHG [20]. These scores were presented as summary statistics.

The data extracted from each study included the following:

Ethical approval was not required as this study was a systematic review.

The initial search from January 2017 to November 2023 identified 580 studies from MEDLINE and Embase. At this early stage, 54 (9.3%) duplicates were removed, and of the remaining 526 (90.7%) papers, 405 (69.8%) were excluded as they did not meet the inclusion criteria. These studies were removed as they lacked user-related outcomes and app quality assessment (n=202, 49.9%), focused on web-based interventions (n=38, 9.6%), or were reviews (n=119, 29.4%). Subsequently, 121 (23%) papers were retrieved, of which 7 (5.8%) were removed as the full text was not available for download. This left 114 (94.2%) full-text papers to be reviewed for eligibility. Of these, 43 (37.7%) met the inclusion criteria [29-71], and 71 (62.3%) were excluded due to being conference abstracts (n=16, 22.5%), having inappropriate outcomes (n=29, 40.8%), having an incorrect study design (n=9, 12.7%), and having an inappropriate setting (n=10, 14.1%). Figure 1 depicts the flowchart of study selection.

Data extracted (Multimedia Appendix 2) showed that the 43 studies [29-71] spanned a range of populations with diabetes, including type 1 diabetes (n=10, 23.3%), type 2 diabetes (n=22, 51.2%), both type 1 and type 2 diabetes (n=5, 11.6%), gestational diabetes (n=5, 11.6%), and diabetes and hemoglobin A_1c (HbA_1c)>6.5% (n=1, 2.3%). The studies also explored a range of diabetes health-related functions (eg, self-management, medication adherence, lifestyle changes, and diabetes education) [29-71].

The methodology data extracted from the included studies identified the details of the app design process, study design, type of diabetes classification, length of the study, duration of user testing, and items from DTAC [19] and NICE [20] guidelines. The items used from DTAC [19] and NICE [20] guidelines are shown in Table 4 and Multimedia Appendix 3. Results of the review are presented using descriptive statistics.

Based on the recommended design and development criteria, 40 (93%) of the 43 studies received 6 “yes” scores or lower; only 3 (7%) studies scored between 7 and 9. No studies adhered to all the DTAC [19] and NICE [20] minimum development standards. A high percentage of the studies (n=36, 83.7%) did not adhere to guidelines on equality considerations, provide sufficient evidence of accurate and reliable measurements (n=38, 88.4%), assess app accuracy (n=37, 86%), or provide appropriate quality and safeguarding (n=34, 79.1%).

In critically examining the adherence of diabetes DHTs to DTAC [19] and the NICE DHG [20] frameworks, our review highlights a large gap between theoretical ideals and the practical realities of app design, development, and evaluation. Despite the well-intentioned criteria of these frameworks to establish standards for safety, usability, and efficacy, there are significant differences in how developers design and test diabetes health technology. Not a single study met the selected minimum development standards, as outlined, and 93% failed to meet even 40% of these standards, which suggests that researchers and developers focus on app development to the detriment of comprehensive evaluation and adherence to these quality standards. This oversight not only raises questions about how developers perceive quality apps but also queries whether this is a systemic issue within the diabetes DHT sector.

This lack of alignment may be attributed to the overwhelming range of frameworks assessing app quality, leading to confusion among developers about what constitutes a quality app. The criteria of frameworks that assess app quality also vary substantially, making it difficult for developers to know what a quality app needs. For example, a systematic review showed that in 2018, a total of 48 different international frameworks with different criteria were in circulation to rate the quality and usability of digital health apps [72]. A common theme between these frameworks is the inclusion of user feedback and usability in both general health apps [73] and diabetes-specific health apps [74]. This still seems to be the case, and it had a knock-on impact on developers’ focus in this review, with user feedback (88.4%) and usability and acceptability (72.1%) scoring the highest. It seems usability has often been foregrounded at the expense of comprehensive efficacy testing, as developers seem to interpret a quality app as one that gathers user feedback and usability over strict adherence to quality, accuracy, and safety guidelines.

Although usability is undeniably important, the emphasis on iterative user feedback—a practice not uniformly adopted across the 43 studies—underscores the need for a more systematic and inclusive approach to app development. People with diabetes tend to have other comorbid conditions or impairments, making the accessibility and usability of DHTs a priority [75]. Ensuring that the needs of these users are understood and met is central to the design process, and poorly designed apps create barriers that increase health inequalities and prevent the uptake and long-term usage of apps [76]. However, it simply is not enough to gain user feedback once. Prior to the rapid software deployment of the apps, user input should take the form of consultation, testing, and feedback throughout development [77,78]. Regular iterations and collaborations with users within populations with diabetes are needed to pinpoint functionalities that may be unsuitable [79].

The amount of iterative user and clinician feedback incorporated in the process highly varied in our included studies [29-71] and, within existing research, ranged from 0% to 87% [80]. Similar findings were also present in research by Fu et al [81], whose analysis of several usability studies reported poor (38%) to average (80%) rates of user feedback. This iterative user feedback goes hand in hand with data on adherence, where users help create apps they are willing to use. Fu et al [81] also found that apps with design flaws (eg, Manual blood sugar level entry) increase user error and result in limited adherence and more varied user input.

If research suggests that poorly developed apps result in less adherence, it seems clear why developers should adhere to these frameworks but not why they do not [81]. One explanation is that the role of health care professionals in endorsing apps for monitoring, diagnosing, treating, and calculating further complicates things. In the United Kingdom, for example, research suggests that 77% of National Health Service (NHS)–recommended high-risk digital health apps do not have any peer-reviewed evidence [82]. This was the case for many categories, including app effectiveness, accuracy, usability, and acceptability. This is particularly striking as health care professional approval could impact whether apps are recommended by clinicians and used by patients [83]. Developers are permitted to reference expert opinion without explicit critical evaluation, or based on physiology, bench research, or first principles [84]. Additionally, there is no requirement for the evidence to undergo peer review or to be accessible to the public. It seems that if developers do not need to provide evidence before their app can be used, they will not do it, which underscores the need for a more rigorous and transparent evaluation process. Another explanation for the lack of adherence may be the complexity and time commitment of evaluating apps. Funding for these apps tends to come alongside tight turnaround times and pressure to meet grant and investor deadlines [85]. Therefore, some start-ups and established companies may prioritize a working prototype rather than the accuracy of content provided in their apps [85]. This fast development time means that only a small number of apps show, involve, and receive feedback from medical professionals during app development [86].

In addition, user, content, platform, and interface components are continually evolving. They can be updated, modified, released, or removed by developers, meaning that the collected feedback may no longer be relevant. Additionally, to keep up with these continually evolving apps, frameworks are updated, leaving developers “chasing their tail” to keep up with the latest requirements.

Ultimately, this lack of adherence has led to a concerning oversight of equality considerations, accurate measurements, and user safety. Across our included studies, a high percentage did not adhere to guidelines on equality considerations (83.7%), provide sufficient evidence of accurate and reliable measurements (88.4%), assess app accuracy (86%), or provide appropriate quality and safeguarding (79.1%). While acknowledging the strengths of this review, particularly its exploration of adherence in diabetes apps to design and development guidelines, [73] it is also necessary to recognize its limitations. Moreover, privacy and safeguarding seem to be a reoccurring concern across the field, where poorly secured personal health information and inaccurate apps have the potential to impact patient safety [87-89]. This was also found by Sharma et al [74] in diabetes management, where equality considerations and privacy and safeguarding scored 50% and 40%, respectively. Such oversights not only compromise the safety and efficacy of diabetes digital health apps but also challenge health care providers and users in identifying trustworthy apps. Our review suggests a pressing need for a unified approach toward adherence to quality and safety guidelines, emphasizing the importance of integrating user concerns early in the development process to enhance reliability, security, and inclusivity.

A fundamental way to improve these scores would be to adhere to the key criteria from DTAC [19] and the NICE DHG [20] more stringently, while taking user concerns on board during development or before it entirely.

The exploration of adherence of diabetes digital health apps to design and development guidelines [19,20] was a strength of this review. While acknowledging the strengths of this review, it is also necessary to recognize its limitations.

None of the apps were downloaded, as some apps required a unique patient access code or were restricted to clinical trials. The review relied on app descriptions for the scoring of studies, which may not fully capture the nuanced and dynamic nature of app development and user interaction. This approach was used to assess apps based on the developers’ stated intentions in the peer-reviewed literature. Although downloading an app may offer insight into some app features (ie, surface-level accessibility and usability), it provides limited information about the underlying design rationale, validation evidence, and other criteria assessed in this review (eg, clinical effectiveness, preventative behavior outcomes, accuracy and reliability of measurements, safeguarding features, and integration of user feedback) [90]. Therefore, although there is some value to downloading the apps, it would not have meaningfully enhanced the quality or depth of our assessment, particularly given that many of the criteria we evaluated were not readily observable through the app interface alone. For this reason, the absence of app downloads should not compromise the rigor or relevance of our review.

Moreover, the use of an unvalidated measurement tool that takes individual items from separate frameworks to assess app quality and usability may have introduced a degree of subjectivity and potential bias into the analysis. However, it offered greater flexibility than the frameworks individually for this analysis. A formal checklist developed through consensus with relevant stakeholders and experts that encompasses all these items, akin to the CONSORT (Consolidated Standards of Reporting Trials) checklist for clinical trials [91], the Evidence DEFINED (Evidence in Digital Health for Effectiveness of Interventions With Evaluative Depth) framework for intervention studies [92], or the STARD (Standards for Reporting of Diagnostic Accuracy) statement for diagnostic test accuracy studies [93], would be an ambition that can be used by regulatory bodies and journals to assess a new app’s adherence to best-practice guidelines.

Currently, the multiplicity of international frameworks creates additional confusion for developers, particularly those attempting to adhere to guidelines across multiple countries. To address this point, we suggest a set of core essential quality criteria that appear consistently across international frameworks and are particularly critical for diabetes digital health app development: safety, accurate and reliable information, usability, data privacy, and clinical effectiveness. These domains appear across the United Kingdom’s Medicines and Healthcare Products Regulatory Agency (MHRA) guidelines [94], the European Union’s Medical Device Regulation (MDR) [95], and digital health guidelines from the US Food and Drug Administration (FDA) [96], Brazil’s Agência Nacional de Vigilância Sanitária (ANVISA) [97], Health Canada [98], China’s National Medical Products Administration (NMPA) [99], and Japan’s Pharmaceuticals and Medical Devices Agency (PMDA) [100]. By foregrounding these shared principles, developers may be able to design applications with a universal quality baseline that aligns with global expectations, even if the ambition for full compliance with every national framework remains challenging.

Nonetheless, certain country-specific requirements, particularly those related to device regulations, data security, and privacy, are unlikely to be harmonized in the future. For example, developers targeting the United Kingdom must comply with the UK Medical Device Regulations (UK MDR) of 2002 and the Medicines and Medical Devices Act of 2021, the United States mandates FDA compliance [96], and other jurisdictions (eg, Brazil’s ANVISA [97], Japan’s PMDA [99], or China’s NMPA [100]) impose unique regulatory expectations each. As such, a fully unified framework is currently impractical, particularly for developers seeking regulatory clearance in multiple regions.

To address this, we propose that developers adopt a tiered compliance strategy: designing digital health apps to align with shared international principles (eg, usability, safety, and efficacy), while preparing supplemental documentation and adjustments to meet specific regulatory requirements of each target country. Furthermore, ensuring transparency in reporting the regulatory status of digital health tools and publishing postmarket surveillance data, where available, may help bridge current gaps in defining and recognizing quality internationally.

In conclusion, our results demonstrate that most diabetes digital health apps do not abide by best-practice development standards. The findings call for a collaborative effort to align app development more closely with the criteria identified in DTAC [19] and NICE [20] guidelines, ensuring that DHTs not only meet the needs of users but also do so in a manner that is safe, effective, and equitable. This endeavor is not just about improving the technological aspects of health apps but also about rethinking the entire ecosystem in which they are created, evaluated, and used to prevent research wastage in the longer term.