This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
There is growing evidence that apps and digital interventions have a positive impact on diabetes self-management. Standard self-management for patients with diabetes could therefore be supplemented by apps and digital interventions to increase patients’ skills. Several initiatives, models, and frameworks suggest how health apps and digital interventions could be evaluated, but there are few standards for this. And although there are many methods for evaluating apps and digital interventions, a more specific approach might be needed for assessing digital diabetes self-management interventions.
This review aims to identify which methods and criteria are used to evaluate apps and digital interventions for diabetes self-management, and to describe how patients were involved in these evaluations.
We searched CINAHL, EMBASE, MEDLINE, and Web of Science for articles published from 2015 that referred to the evaluation of apps and digital interventions for diabetes self-management and involved patients in the evaluation. We then conducted a narrative qualitative synthesis of the findings, structured around the included studies’ quality, methods of evaluation, and evaluation criteria.
Of 1681 articles identified, 31 fulfilled the inclusion criteria. A total of 7 articles were considered of high confidence in the evidence. Apps were the most commonly used platform for diabetes self-management (18/31, 58%), and type 2 diabetes (T2D) was the targeted health condition most studies focused on (12/31, 38%). Questionnaires, interviews, and user-group meetings were the most common methods of evaluation. Furthermore, the most evaluated criteria for apps and digital diabetes self-management interventions were cognitive impact, clinical impact, and usability. Feasibility and security and privacy were not evaluated by studies considered of high confidence in the evidence.
There were few studies with high confidence in the evidence that involved patients in the evaluation of apps and digital interventions for diabetes self-management. Additional evaluation criteria, such as sustainability and interoperability, should be focused on more in future studies to provide a better understanding of the effects and potential of apps and digital interventions for diabetes self-management.
As the number of people with diabetes continues to rise worldwide [
To improve diabetes self-management with apps and digital interventions, the World Health Organization and the European Commission [
Another relevant issue is who should be involved in these evaluations. As patients are often required to make critical decisions based on their own generated health information [
In this systematic review, we identify the specific methods and evaluation criteria that were used to assess apps and digital interventions for diabetes self-management. We also report how patients were involved in these assessments.
This review followed the PRISMA approach [
We performed a single data search in June 2018. The search strategy covered all studies that assessed diabetes self-management apps and digital interventions, involved patients, and were published in English after 2015. We chose a short search period to get a rapid overview of the most recent methods and evaluation criteria. The search strategy covered the following databases: CINAHL, EMBASE, MEDLINE, and Web of Science. The full search strategy is available in
We included articles for review if they were (1) primary studies referring to the evaluation of apps or digital interventions for diabetes self-management; and (2) involved patients in the evaluation.
Article were excluded if (1) the evaluation only measured medical values (ie, weight, glycated hemoglobin [HbA1c], blood glucose); (2) it was not a primary study; (3) it did not focus on apps or digital interventions for diabetes self-management; (4) the full-text was not available; (5) it was not a peer-reviewed publication; (6) it was not in English; or (7) it was published before 2015.
We uploaded all references captured by the search strategy to Rayyan and EndNote and removed duplicates. The eligibility of the articles was assessed in two stages. In the first stage, 2 independent reviewers (PR and EG) examined all titles and abstracts. Eligibility doubts were discussed and agreed with a third and fourth reviewer (KA and EÅ). In the second stage, the full texts of the selected articles were carefully examined by 2 independent reviewers (PR and EG) to confirm their eligibility.
Two reviewers (PR and MB) independently extracted and recorded the data from these articles on an Excel spreadsheet (Microsoft). We extracted the following information from each article: type of platform, targeted health condition, study population, methods of evaluation, and evaluation criteria. Incongruences with the data extraction were discussed among the research group.
Two reviewers (EG and KA) assessed the confidence in the evidence and risk of bias of the articles. We used an approach based on the CERQual guidelines [
We provide a narrative qualitative synthesis of the findings from the included articles, structured around confidence in the evidence and risk of bias; type of platform (apps, web, or multiplatform [ie, ≥2 types of platform delivering the same intervention in a study]); targeted health condition (type 1 diabetes [T1D], T2D, gestational diabetes mellitus, both T1D and T2D, and unspecified diabetes type); methods of evaluation (questionnaires, interviews, user-group meetings, health measures, system usage analysis, or other); and evaluation criteria (usability, clinical impact, cognitive impact, behavioral impact, feasibility, engagement, acceptability and acceptance, or security and privacy).
The search strategy resulted in 1681 articles. After removing duplicates, 967 articles remained. In the abstract screening, we excluded 910 articles in accordance with one or more of the exclusion criteria. A total of 57 articles were eligible for full-text screening, 26 of which were excluded (see
PRISMA flowchart of the selection procedure.
The 31 articles in this review included evaluations from 3689 participants. The number of participants in each study ranged from 7 [
Most of the 31 included studies evaluated interventions delivered via apps (18/31, 58%) [
Distribution of types of platform and targeted health conditions among included articles (n=31).
The methods of evaluation were grouped into 6 categories: questionnaires, interviews, user-group meetings, health-related measures, system usage analysis, and other measurements. We also identified 20 specific methods that were either used once or multiple times by the studies during the evaluation process.
The interrater agreement for the methods of evaluation was found to be κ=0.550, which represents a moderate agreement [
Questionnaires were the most common method used to evaluate diabetes self-management apps and digital interventions (21/31 studies, 68%) [
Among the 7 studies considered of high confidence in the evidence, the evaluations of the apps and digital diabetes self-management interventions were based mostly on standardized questionnaires [
Specific methods of evaluation and studies that used them.
Method of evaluation, specific type (n=times used), and details | Reference(s) | ||
|
|||
|
|
||
|
|
Block Food Frequency Assessment | [ |
Dietary Knowledge, Attitude, and Behavior Questionnaire | [ |
||
Health Care Climate Questionnaire | [ |
||
Paffenbarger Questionnaire | [ |
||
Patient Enablement Instrument | [ |
||
Patient Health Questionnaire-9 | [ |
||
Patient Reported Diabetes Symptoms Scale | [ |
||
Perceived Competence in Diabetes | [ |
||
Problem Areas in Diabetes-5 | [ |
||
Problem Areas in Diabetes | [ |
||
RAND 36-Item Health Survey 1.0 | [ |
||
Self-Efficacy for Diabetes Scale | [ |
||
System Usability Scale | [ |
||
The Health Education Impact Questionnaire | [ |
||
The Service User Technology Acceptability Questionnaire | [ |
||
36-Item Short Form Survey | [ |
||
|
|||
|
|
N/Aa | [ |
|
|||
|
|
||
|
|
N/A | [ |
|
|||
|
N/A | [ |
|
|
|||
|
|
||
|
|
N/A | [ |
|
|||
|
N/A | [ |
|
|
|||
|
|
||
|
|
N/A | [ |
|
|
||
|
|
N/A | [ |
|
|||
|
N/A | [ |
|
|
|||
|
|
||
|
|
HbA1c | [ |
Fasting blood glucose | [ |
||
Blood pressure and cholesterol | [ |
||
Gestational weight gain | [ |
||
|
|||
|
Self-reported blood glucose | [ |
|
Self-reported physical activity and nutritional habits | [ |
||
|
|||
|
|
||
|
|
N/A | [ |
|
|||
|
N/A | [ |
|
|
|||
|
N/A | [ |
|
|
|||
|
N/A | [ |
|
|
|||
|
N/A | [ |
|
|
|||
|
Star rating | [ |
|
|
|||
|
Bertini’s mobile tool | [ |
|
|
|||
|
Open text review | [ |
|
|
|||
|
N/A | [ |
aN/A: not applicable.
The evaluated criteria were grouped into 8 categories: usability, clinical impact, cognitive impact, behavioral impact, feasibility, engagement, acceptability and acceptance, and security and privacy. The included studies evaluated one or several of these identified criteria. The interrater agreement (κ) for the evaluation criteria was found to be 0.563, which represents a moderate agreement [
Among the 7 studies considered of high confidence in the evidence, the most commonly evaluated criteria were clinical impact [
Qualitative and mixed-method studies that used thematic analysis in their evaluation focused mostly on usability as an evaluation criterion. Three of the studies considered of high confidence in evidence were qualitative and mixed-method studies. Of these, 2 evaluated cognitive impact [
Number of studies using the various methods of evaluation and evaluation criteria. Blank boxes (NaN): No studies within this category.
This review aimed to identify the existing methods and criteria used to assess apps and digital diabetes self-management interventions that involved patients in their evaluations. A total of 31 articles were included in the review, 7 of which were considered of high confidence in the evidence [
In our review, we have found that studies dealing specifically with apps and digital interventions for diabetes self-management focus on the evaluation of more technology-related and users’ interaction aspects (ie, acceptability and acceptance, and engagement). In addition, these studies focus on the impact that these digital self-management interventions have on the individual. Behavioral impact, cognitive impact, and clinical impact were used as relevant criteria for assessing all types of digital interventions for diabetes self-management. It is vital to measure the interventions’ impact on their users because those that have shown benefits related to behavioral, cognitive, and clinical impact could reduce health-related costs [
Evidence shows that involving individuals in the assessment of different health interventions has a positive impact on health [
Involving patients with diabetes in assessing apps and digital self-management interventions, and obtaining their feedback regarding additional evaluation criteria could also increase our knowledge about the features that support engagement with these technologies. This could also help create better digital health interventions that encourage more continuous and effective use [
Apps and digital health interventions have evolved quickly. Yet, compared with other sectors, the health industry seems to be behind with regard to digitalization [
In 2016, the mobile health (mHealth) evidence reporting and assessment checklist was developed by the World Health Organization to help with reporting evidence of the effectiveness of mHealth interventions [
Evaluation reports for apps and digital interventions for diabetes self-management must be standardized, as recommended by the CONSORT-EHEALTH guidelines for reporting digital health interventions [
As electronic health (eHealth) research is a multidisciplinary field, we assume that the authors chose these terms based on the various educational or professional backgrounds and the corresponding target audiences. By following the World Health Organization classification of digital health interventions [
In our review we have found that there was an almost even split of studies that used standardized questionnaires, author-created questionnaires, and semistructured interviews to evaluate usability. Our results are to some extent in line with the findings of a previous review that found that usability was mainly assessed though polls and questionnaires [
Comparing the methods for the evaluation of usability with those for the evaluation of cognitive impact, we identified that it was more common to use author-created questionnaire for usability. A possible explanation might be the wide variety of intervention delivery platforms (eg, different types of apps and online resources) that might create different evaluation needs not captured in existing standardized usability questionnaires. Another explanation might be the different research traditions in different disciplines. Usability might be more often a concern of computer science researchers, whereas cognitive impact a concern of health researchers and professionals.
Finally, health outcomes were almost exclusively evaluated by medical tests, showing the preference of health researchers and professionals in using standardized tests to determine the impact of digital interventions. Several other methods can be used to evaluate multiple criteria; however, depending on the aim and the type of study, researchers must endeavor to exhaust all available methods to ensure consistency of results.
Although most apps and digital health interventions are intended for self-management, some of them also provide access to the health care system, such as communication with HCPs and electronic health journals. The reviewed studies consistently reported that this is in response to patients’ interest in being able to contact their HCPs or share results (eg, their blood glucose results with their health care team). This was the case not only within our review [
Several studies, including many in this review, have shown that involving HCPs in digital interventions is associated with improved self-management of diabetes and the success of these interventions [
The search for articles covered a short period (2015-2018) and focused on articles published in the English language. Therefore, we may have missed relevant studies that reported additional evaluation methods or evaluation criteria. Our interrater agreement of the data extraction was only moderate; however, all incongruences were discussed among the research group. Our findings have provided a useful overview of the recent evaluation methods and criteria that researchers are using to assess current apps and digital interventions for diabetes self-management. Furthermore, our review included both quantitative and qualitative studies which provided a better characterization of different evaluation methods and criteria that are being used to assess digital diabetes self-management interventions.
There are only few studies that involved patients in the evaluation of apps and digital interventions for diabetes self-management, and even fewer still considered of high confidence in the evidence. The most common evaluation methods were questionnaires, interviews, and user-group meetings, whereas evaluation criteria were cognitive impact, clinical impact, and usability. Studies with high confidence in the evidence did not evaluate feasibility or security and privacy, neither were patients involved in evaluating the latter criterion which was evaluated in only 2 [
It is important to the successful implementation and continuous use of apps and digital interventions for diabetes self-management that patients are involved in evaluating every criteria. In that way, they can contribute to the development and modification of these digital interventions to better meet their specific self-management needs. Furthermore, the methods and criteria evaluated in digital diabetes self-management interventions should be expanded to assess and ensure sustainability and interoperability. In addition, studies should evaluate the association between cognitive, clinical, and behavioral impact of these apps and digital interventions, and health-related costs for individuals with diabetes. This could help improve health care associated with the management of diabetes and promote the incorporation of apps and digital interventions for self-management in the services provided at health care facilities.
Search strategy (search date: June 21, 2018).
List of rejected articles after full-text review (n=26).
Articles included in qualitative synthesis (n=31).
PRISMA checklist.
glycated hemoglobin
health care professionals
type 1 diabetes
type 2 diabetes
This project is funded by Helse Nord (HNF1425-18). The coauthors acknowledge the advice of the project’s Advisory board: Professor Gunnar Hartvigsen, Anne Grethe Olsen MD, and Dr. Med. Anne Helen Hansen, and also the support and contributions of Per Erlend Hasvold in his role as an “internal reviewer.” Furthermore, we thank Dr. Steven Bradway for his assistance with the coarse data extraction and organization at the start of this review. The publication charges for this article have been funded by a grant from the publication fund of UiT The Arctic University of Norway.
KA was responsible for database searching; EG, EÅ, KA, and PR were responsible for title, abstract, and full-text screening; MB and PR performed independent data extraction; EG and KA evaluated risk of bias; and DL, EG, EÅ, MB, KA, and PR performed data analysis and interpretation. All the coauthors contributed to drafting and revising the review. All coauthors approved the final version of the manuscript.
None declared.