Type 2 diabetes mellitus (T2DM) is one of the leading causes of death as well as disability-adjusted life years [1]. The rapidly rising global prevalence of T2DM in adults, especially in rural and high-income areas, was estimated to be 10.5% in 2021 [2]. Those levels were previously estimated to occur in 2030 [3]. The metabolic disease is characterized by elevated blood glucose levels, which are associated with an increased risk for vascular and cardiovascular complications, such as heart disease, chronic kidney disease, diabetic retinopathy, and diabetic foot ulcers [4,5]. In addition, T2DM is related to the occurrence of mental health disorders, such as depression [6]. This causes T2DM to be associated with an increased mortality rate in patients enduring the disease [4,7]. Consequently, the chronic condition poses a major health burden on the physical and mental health of those affected as well as on their families [3].

There are several known risk factors for developing T2DM. In particular, conditions such as obesity, high blood pressure, and hyperlipidemia, are associated with relevant lifestyle factors such as high-caloric diets, sedentary behavior, smoking, and alcohol consumption [8,9]. To counteract the multiple risk factors of the disease, guidelines describe lifestyle modification measures as the foundation of all therapeutic interventions in patients with T2DM [10]. Behavioral interventions toward a healthy lifestyle play an important role in preventing the onset of T2DM in individuals at high risk and decreasing the development and progression of diabetic complications for patients with manifested T2DM [11].

A multimodal therapeutic approach for people with obesity and T2DM combines nutritional medical intervention, exercise, and behavioral therapy support and addresses treatment adherence to enhance the long-term success of such interventions [12]. These components of interdisciplinary treatment can be communicated to patients digitally. For example, eating behavior can be tracked and then a reduction in calorie intake and optimization of food composition can be implemented using app features. In addition, training sessions or tasks to increase physical activity can also be integrated. To increase therapy engagement, interventions should not only be multidisciplinary but also patient-centered and address as many risk factors as possible [5,11].

In line with that, behavioral interventions can also reduce the pharmacotherapy escalation. In a trial applying a multimodal approach, including nutritional and behavioral counseling, educational content, digital coaching, and medication management, most participants, 42.7%, decreased their medication, while another 8% eliminated their medications. Hemoglobin A_1c (HbA_1c; long-term blood glucose) levels were also significantly reduced with 56.1% of participants achieving a HbA_1c value of <6.5% [13].

Blood glucose control is a key target for such interventions, to lower cardiovascular risk as well as mortality from T2DM [5]. In adults with T2DM, the HbA_1c target value is set individually, depending on various factors, such as diabetes duration, comorbidities, and patient preference, but it lies within the corridor of 6.5%-8.5% [10]. Accordingly, guidelines for the treatment of T2DM formulate lifestyle interventions to manage indicators such as weight, blood pressure, and blood lipids as the foundation of the prevention and treatment of T2DM and also promote self-management and patient autonomy as a therapeutic goal [10,14]. Due to the chronic nature of T2DM, patients must manage their disease in their daily lives independently of medical care [15].

In this context, digital technologies, for example, tracking and providing a feedback loop with health care professionals (HCPs) are already included in guidelines as having great potential to improve diabetes care [5,16].

With the Digital Healthcare Act (January 2020) Germany was the first country to use a legal health claim for insured individuals to receive evidence-based treatment in the form of digitale Gesundheitsanwendung (digital health apps [DiGA]), that is, apps prescribed by health care providers [17]. Generally, DiGAs are not primary prevention tools but support the monitoring, discovery, and treatment of diseases, injuries, and disabilities. Other countries, in particular France and Belgium, are following the example and pursuing easier market access for digital solutions in the health care sector [18].

The potential of digital interventions to reduce HbA_1c levels has been shown in previous systematic reviews [19-21]. For instance, one umbrella review reports mean reductions in patients with T2DM using telehealth interventions of between −0.01% and −1.13% [21]. The medical purpose of a DiGA is achieved through the interaction of the patient with digital technology [22]. This means that DiGAs are not purely digital communication channels between patients and HCPs, but treatment is largely carried out independently by the patient through an app [20,22]. This means that therapy with DiGAs is independent of the prescription site in which it is used, as the driver of the therapeutic effect is the patient’s interaction with the application. By focusing on apps that meet the definition of DiGA, we aim to analyze the evidence for these technologies in diabetes care. This also allows us to expand the current evidence base for the digitalization of usual care, for example, delivering care through phone calls or chats.

For a medical device to be approved as a DiGA in Germany, its effectiveness has to be shown with clinical evidence generated by randomized controlled trials (RCTs) [23]. Especially before approval or after a device is already approved and on the market, digital applications, including DiGAs, offer new opportunities to generate evidence based on vast amounts of actual user and patient data. In this context, real-world data (RWD) and real-world evidence are increasingly recognized as complementary to RCTs. For example, user data allows us to continuously investigate the efficacy after approval as part of postmarket research, offering additional insights into how medical devices perform [24]. The European Medicines Agency, defines RWD as “routinely collected data relating to a patient’s health status or the delivery of health care from a variety of sources other than traditional clinical trials” [25]. One type of such data is health records directly tracked in-app. Evidence generated with RWD is often considered pragmatic and externally valid, making it cost- and time-efficient, particularly for regulatory purposes, such as postmarket surveillance under the European Medical Device Regulation [23,24,26,27].

RCTs provide high internal validity by controlling for confounding variables and minimizing bias through more controlled circumstances and more rigid protocols. Non-RCTs contribute by offering evidence generated outside a study setting and, while less controlled, still provide valuable information on how interventions perform in the real world. Both approaches offer opportunities for implementation in pragmatic settings [28]. To better distinguish between the internal and external validity of trials and the potential balance between them, we use both the GRADE (Grading of Recommendations, Assessment, Development and Evaluation) and Pragmatic Explanatory Continuum Indicator Summary (PRECIS-2) tools, respectively.

Current evidence on digital therapies exists for a broad range of intervention types but not specifically for apps that meet the definition of a DiGA addressing T2DM. The objective of this systematic review was to analyze the efficacy of such and to elicit their potential, including explanatory as well as pragmatic studies.

A systematic literature search was conducted in January 2023 to identify evidence on the efficacy of multimodal, app-based lifestyle interventions that meet the definition of DiGAs in reducing HbA_1c in patients with T2DM. The search was performed in the electronic databases PubMed, LIVIVO, and Cochrane and the search strategy was individually tailored for each database. A combination of search terms, MeSH (Medical Subject Headings) terms, and appropriate Boolean operators was developed (Multimedia Appendix 1) based on the PICO (Population; Intervention; Control; Outcome) scheme (Table 1) to identify relevant studies. To generate a broader evidence base, not only RCTs but also other study designs were considered eligible (eg, observational studies or analyzing user data). Study selection was carried out per the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [29] and the title-abstract screening and full-text screening of the search results were performed using CADIMA [30]. The screening was independently conducted by EB and LR, and disagreements were resolved through discussion and consensus, involving a third party if necessary.

Both RCTs and non-RCTs were graded by EB, KW, and LR using the PRECIS-2 tool [31] to represent how explanatory or pragmatic the trials were. The PRECIS-2 tool covers nine domains: “eligibility,” “recruitment,” “organization,” “setting,” “flexibility (adherence),” “flexibility (delivery),” “follow-up,” “primary endpoint,” and “primary analysis.” All domains are rated on a 5-point Likert scale from 0 (very explanatory) to 5 (very pragmatic). Generally, a trial is more pragmatic the less strict its protocol is or the fewer additional resources are used, and is therefore closer to usual care. On the other hand, a very explanatory trial would differ vastly from the usual care setting, using strict protocols, a lot of extra resources, specific patients as well as study settings or collecting a vast amount of extra data.

To grade the pragmatism of the studies included in this systematic review, a mean over all domains for each study and across all studies within the respective study type (RCT vs non-RCT) was calculated and was used to compare pragmatism between domains, studies, and study types.

Risk of bias (RoB) assessments for the included studies were independently performed by 2 reviewers, and disagreements were resolved through discussion and consensus. The RCTs were assessed by CB and EB using the RoB 2 tool [32] and nonrandomized and observational studies were assessed by EB and LR, using the Risk Of Bias in Non-Randomized Studies of Interventions (ROBINS-I) tool [33]. The quality of evidence was evaluated by EB, KW, and LR using the GRADE tool [34]. This established tool was used to assess the overall certainty of evidence according to RoB, imprecision, inconsistency, indirectness, and publication bias (see Multimedia Appendix 2) [34]. This was carried out using the GRADEpro Guideline Development Tool (GRADEpro GDT) for the three outcomes: changes in HbA_1c between groups in RCTs, within groups in RCTs, and within groups in non-RCTs. In addition, publication bias was evaluated with the help of funnel plots using the R (R Foundation) function meta::forest.meta [35].

The data analysis, that is, pooling the results, was performed with R (version 4.3.0) in the RStudio environment.

Two approaches were used to pool the results of the studies. First, the unstandardized between group differences were used to assess the effect of the intervention group (IG) or unit compared with the control group (CG) or unit. Second, with the within-group mean differences (MDs) the absolute change in HbA_1c postintervention was quantified. Both effect sizes were not standardized because the included studies measured the relevant outcome on the same scale, that is, HbA_1c (in percentage) [36]. The results (mean [SD]) with the highest statistical quality were extracted: (1) for RCTs intention-to-treat (ITTs) analyses that analyze all participants as randomized independent of protocol adherence (with imputation or complete cases analysis) were preferred over per-protocol analyses including only patients adhering to the protocol and hence hold the potential of biased “best case” results and (2) generally adjusted values were preferred over the potentially confounded.

While meta::metacont can calculate the mean between group difference and the respective 95% CIs based on each within-group MD and corresponding SD, the within-group MD and corresponding SE needed to be precalculated before effects could be pooled with meta::metagen [37,38]. The function further allowed us to calculate prediction intervals for the expected intervention effect in a single new study [39]. Random-effects models were used to pool results because study heterogeneity cannot be excluded (ie, due to different observation periods and different digital apps). Missing within-group MDs and SDs for the IG (and CG) were calculated from the available information reported in the studies per the following suggested procedures. MDs between IG and CG were calculated based on the formula for within-group differences, that is, the mean of the follow-up value subtracted by the mean of the baseline value [36]. In case 95% CIs, t statistic values, or P values were available, the SD was calculated based on the formulas provided by Higgins et al [40]. If only baseline or follow-up SDs were available, one was substituted with the other; under the assumption that the intervention did not alter the variance [40]. Assuming that the correlation coefficient between the pre- and postvalue of the IGs was similar, missing SDs for the within-group differences were calculated by the mean of the baseline and follow-up SDs [40].

For heterogeneity assessment, the I² statistic was used. It quantifies the amount of variation in the results that is not random. Generally, an I² below 40% can be considered low, an I² between 30%-60% moderate, and an I² between 50%-70% or even 70%-90% high [41]. In case of high heterogeneity outlier analysis based on the study by Harrer et al [42] was conducted and reported.

Overall, 21 trials and 3 systematic reviews met the inclusion criteria and were found to be eligible. Systematic reviews were not included in this review and meta-analysis but were used to identify further eligible trials. Their content was screened and thereby, two further trials were identified and included as cross-references. In total, 23 studies were included in this systematic review and were eligible for meta-analysis (see PRISMA flow diagram in Figure 1 and checklist in Multimedia Appendix 3).

A total of 23 studies were eligible for the review, 12 RCTs and 11 non-RCTs. Of those final studies, 3 were performed as secondary analyses looking at specific subgroups from another study (Lee et al 2021 [43] and Lim et al 2022 [44]) or presented preliminary results of a trial (Torbjørnsen et al 2014 [45]). They were hence not included in the meta-analysis because the main results are covered by other included studies. Another study (Dixon et al 2020 [46]) only looked at HbA_1c changes based on different HbA_1c baseline categories and was also not included in the meta-analysis because a result for the whole sample could not be obtained and including several subgroups would overrepresent the single study.

Overall, nine of the studies included in the meta-analysis were RCTs and 10 were non-RCTs. Most of the non-RCTs were single-arm observational (pilot) studies and two had an intraindividual CG (Table 2). The average calculated pragmatism of each study is reported in Table 2, while detailed information on each domain is provided in Multimedia Appendix 4. Detailed information on the content and the features of the apps is provided in Multimedia Appendix 5. Most RCTs were performed in Asia, most non-RCTs were performed in the United States. The patient characteristics of study participants and app users are provided in Table 3.

To rate the level of pragmatism of the included studies, the PRECIS-2 tool was used (Multimedia Appendix 4). The mean score of pragmatism for RCTs was 4.2 and for non-RCTs 4.5, that is, both study types were on average (very) pragmatic, that is, close to usual care. As a routinely collected parameter in usual care, the HbA_1c, is a very pragmatic parameter, resulting in high ratings in the domain of “primary outcome” in all studies. The “eligibility,” that is, inclusion and exclusion criteria were also rated as rather pragmatic, as they were close to real-world conditions in almost all studies for the use of a health app; including mostly patients who would be eligible under real-world conditions as well. The organization was mostly rated (very) pragmatic as well, carrying out the study in the usual care environment.

Due to more extensive data collection, RCTs had a slightly lower average score in the domain of “follow-up,” mostly due to patient-reported outcomes. Non-RCTs, on the other hand, were rated as less pragmatic than RCTs in the domains of “primary analysis” and “recruitment” due to insufficient inclusion of all available data and sometimes extensive recruitment strategies. The highest rating of pragmatism was achieved when studies analyzed actual user data.

Only 3 RCTs were rated as having a low RoB (Figure 2 [47-55]). All of the remaining RCTs were rated as having some concerns, with domain 5 raising concerns in all of them, due to missing study protocols. Missing blinding of participants was present in all studies and some studies also lacked blinding of the outcome assessor (domain 4). We nevertheless decided to rate the RoB as low in this respect, as the outcome of interest, the HbA_1c, is a physiological parameter that is less likely to be affected by knowledge of the received intervention, when compared to patient-reported outcomes [26].

All of the non-RCTs had at least a serious (n=4), if not a critical (n=5) RoB according to ROBINS-I; the detailed ratings can be found in Multimedia Appendix 6. Due to missing information on clear time frames for assessment as well as patient flow, one study could not be ranked [57]. All of the ranked studies had a serious RoB due to confounding because no adjustment for possible confounders was performed or, as in the case of Berman et al [58], adjustment for postintervention variables was performed. The categories leading to critical risks were due to bias in the selection of participants. This particularly relates to observational studies that analyzed app users but excluded all participants who did not have a follow-up HbA_1c, leading to exclusion rates of up to almost 80% [60] (Table 3). As it has to be assumed that reporting and continuation of the intervention is influenced by the intervention itself, the RoB of selection was judged to be critical. Another problem of RWD that led to a critical selection bias, was that start and follow-up times did not coincide for all users. Again, the lack of statistical adjustments to correct for missing data resulted in a selection bias and also did not offer evidence that the results were robust to missing data. Two of the studies also had a serious RoB in the category of measurement outcome due to self-reported HbA_1c values.

The grading of evidence was based on the meta-analyses results for the between group differences using 9 RCTs and the within-group differences using 12 RCTs as well as 11 observational studies. Using the GRADE Framework, the levels of certainty for the HbA_1c outcomes were rated as low quality of evidence for the RCTs and very low quality of evidence for the observational studies (Table 4).

For the between group differences in RCTs, this rating is due to a serious RoB, illustrated by the low to moderate RoB 2 rating. It also reflects a serious indirectness of the results, which arises from the RCTs being conducted primarily in Asia with widely varying intervention durations of 3 to 12 months. However, imprecision was not detected and neither was inconsistency, based on a low heterogeneity score of I²=19%. The same applies to publication bias, which was also not detected. For the RCTs used for the within-group differences, the same grading applies.

For the non-RCTs, the very low quality of evidence arose from a very serious RoB rating, pictured by the results of the ROBINS-I-Tool, which showed a serious to critical RoB in the non-RCTs (Multimedia Appendix 6). A high heterogeneity score of I²=82%-94% showed a high, very serious inconsistency. Due to studies being primarily conducted in the United States with intervention durations widely varying or remaining unclear, as is the nature of RWD, the quality was further downgraded for very serious indirectness. It was further downgraded for publication bias because funnel plots showed some asymmetry in favor of positive results. Imprecision however was not detected.

Overall, this systematic review and meta-analysis aimed at giving an overview of the current evidence regarding the efficacy and effectiveness of app-based interventions for the treatment of T2DM. For this, RCTs as well as non-RCTs were included. To our knowledge, this review is the first to include RCTs and non-RCTs for a pooled pre-post effect of digital lifestyle therapy on the HbA_1c in patients living with diabetes. Evidence from 9 RCTs shows that app-based interventions for the treatment of T2DM support patients in significantly reducing their HbA_1c levels by, on average, –0.36% and had favorable results compared to CGs receiving usual diabetes care. Looking at the within-group differences, patients were able to reduce their HbA_1c values on average by –0.79% after 3-12 months. While non-RCTs showed slightly higher reductions compared to the RCTs, the differences between study types were not significant. However, the heterogeneity (I² statistic) for the within-group differences across study types, as well as within-study types was high, likely due to differences in the study duration and setting, methodology, and patients’ characteristics [66]. In contrast, the RCTs’ between-subject analysis resulted in a low heterogeneity. This shows the importance of controlling for baseline characteristics in statistical models to analyze outcomes. The pre-post effects only adjusted for confounding factors within their study population. In contrast, between group differences cancel out several effects unrelated to the intervention. First, the effect of the patient population characteristics itself. Second, the effect of the “usual care” that potentially differs between study sites (nationality, resources, study information provided to the participants, etc). Third, both groups are followed up over the same period, which allows controls for unexpected or spontaneous outcome-related events influencing the disease process of the whole population. As a result, only the between group differences show the isolated efficacy of the intervention, while within-group differences show the effectiveness of the intervention including external factors that might influence study outcomes, that is, the clinical effect under real-world conditions. This review shows that non-RCTs are likely to produce similar outcomes to RCTs per the effectiveness of interventions and as such can complement the evidence from RCTs. To address the limiting factor of a missing comparator in non-RCTs, appropriate artificial CGs could be used, such as matching methods or intraindividual cohorts. In fact, some of the included non-RCTs used the latter method.

The average additional mean HbA_1c reduction of –0.36% achieved with app-based interventions in comparison to usual care only, is comparable to previous systematic reviews on tele-medical treatment of diabetes that found mean group-differences of –0.44% [19] and –0.52% [21]. To reduce long-term diabetes-related complications, an HbA_1c reduction of 0.3% is considered clinically meaningful [67-69]. Stronger reductions in previous reviews could be due to broader inclusion criteria because the current review only included studies on apps that meet the DiGA definition and thus largely function through their technology independently of HCPs. Systematic reviews that analyze the efficacy of app-based interventions for type 1 diabetes mellitus and T2DM show that additional remote access to HCPs [19] as well as a higher frequency and intensity of feedback and interaction are associated with greater HbA_1c reductions [20]. Whether or not the feedback is automated or manual, on the other hand, might not be relevant [70].

RCTs are designed such that the result is highly controlled, providing strong internal validity. However, this does not mean an RCT cannot be implemented in a real-world setting. The reasons for choosing a non-RCT design can be very diverse, with pragmatic implementation being a major aspect. In our study, the concept of pragmatism was used to judge the external validity of studies with the help of the PRECIS-2 tool. The most pragmatic studies were those that analyzed actual users of the app-based interventions [46,60]. Both RCTs and non-RCTs had, on average, high grades for pragmatism indicating (very) pragmatic trials and good external validity. Due to a lack of transparency and suitable statistical methods to account for missing data, non-RCTs had slightly lower grades than RCTs in the domains “primary analysis” and “recruitment.”

The GRADE ratings in both RCTs and non-RCTs further showed potential for improvement in future trials aiming for analyzing in-app user data or studies without a CG. First of all, improved statistical methods should be used to control for baseline covariates. Especially in studies analyzing user data or data collected from health records, clear information on data selection and suitable statistical methods are lacking. Particularly, in the absence of a CG, the handling of missing data needs to be clearly justified (eg, patients who do not use the app or do not provide any data). One way to improve the quality might be the use of preregistered analysis that predefines statistical methods and inclusion or exclusion criteria of patients [23]. Additional sensitivity analyses under different assumptions or conditions may increase the interpretability of results, as might blinding that so far is a major limitation of most digital interventions.

While non-RCTs in the context of explanatory trials show several limitations, they do have strengths in depicting results in a real-world context as pragmatic trials. However, the results of the PRECIS-2 rating showed that non-RCTs and RCTs included in our study can be classified as similarly pragmatic, and as such hold the potential to analyze interventions under real-world conditions to complement RCTs conducted in a controlled setting. Especially, in the context of fast-changing digital interventions, non-RCTs offer a cost- and time-efficient addition to RCTs. It is important to note that, for both RCTs and non-RCTs, the results of the study depend on the specific recruitment strategy and patient characteristics. The findings offer insights into the shortcomings of both RCTs and non-RCTs concerning pragmatism, however, they should be read with care, and translation of the evidence to other care settings should be made with great caution.

This study has several limitations. First, it cannot be ruled out that relevant publications were overseen in the course of the conducted search. However, this was counteracted by building extensive search terms, based on the PICO scheme, and by using text words as well as MeSH terms. In addition, the search was also conducted in three different databases, which might have further contributed to limiting the risk of selection bias. Second, the present results are limited to studies without long-term, follow-up periods, generating short-term rather than long-term evidence of the interventions. Other reviews suggest that the initialization of a treatment and as such the immediate short-term outcomes of treatment duration might be associated with better treatment success [21] making long-term and follow-up effects of particular interest. Third, even though the comparator of the included RCTs was theoretically the same, that is, usual care, in practice diabetes care differs between countries and could not be considered thoroughly which might have influenced the results. Due to the limited number of studies that fit the inclusion criteria, the heterogeneity between studies could not be accounted for in further subgroup analyses, for example, based on the time of the interventions. In fact, the results of the GRADE rating further resulted in (very) low evidence for the results of meta-analysis. Therefore, the results should be evaluated with caution. Finally, it is important to note that non-RCTs including real-world evidence exist on a broad continuum of possible settings. Unlike RCTs, which are more clearly defined per their study conditions, they vary greatly in their design and context. The PRECIS-2 tool was used to obtain an estimate of how pragmatic the delivery of digital care was in the different studies, including RCTs and non-RCTs. The PRECIS-2 tool was originally developed for designing pragmatic RCTs. As the authors leave freedom and flexibility in the usage of the tool, we applied it for the retrospective grading for RCTs as well as for non-RCTs. Downgrading of specific dimensions within the PRECIS tool depends on the deviation from the usual care condition, which can also vary across different settings. This makes it difficult to apply coherent criteria to a set of different studies.

Overall, it can be concluded that app-based lifestyle interventions that meet the definition of DiGA can effectively reduce HbA_1c in patients with T2DM. This has been shown not only in RCTs but also in non-RCTs. While the latter still have several limitations per their design and statistical analyses, non-RCTs that implement suitable designs and methodologies have the potential to become an important source of complementary evidence, for example, in the context of postmarket analyses or piloting studies.

A DiGA approval requires high-quality evidence with minimal sources for biases, such as RCTs, before manufacturers can claim their medical product as a DiGA. Non-RCTs and in particular the analysis of in-app data can complement evidence from RCTs as a cost and time-efficient source of evidence to continuously monitor clinical outcomes of the medical device after being placed on the market.

Beyond the monitoring of the clinically relevant endpoints of DiGAs, in-app data can be a relevant addition to understanding patients’ needs and support postmarket analysis. As such the combination of evidence generated by both RCTs and non-RCTs is gaining relevance to develop the potential of DiGAs, for example, implemented in usual care as hybrid models. Moreover, introducing digital care solutions in the health care system may pave the way for artificial intelligence to further enhance the treatment of T2DM worldwide. Yet, future studies should aim for more methodological transparency and appropriate statistical evaluation procedures and methodologies to account for current limitations of non-RCTs, such as the missing comparator.