Advertisement: Preregister now for the Medicine 2.0 Congress
Mobile Applications for Diabetics: A Systematic Review and Expert-Based Usability Evaluation Considering the Special Requirements of Diabetes Patients Age 50 Years or Older
Madlen Arnhold, Dipl-Soz; Mandy Quade, MSc; Wilhelm Kirch, DMD, MD
Research Association Public Health Saxony and Saxony-Anhalt, Medizinische Fakultät Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany
Research Association Public Health Saxony and Saxony-Anhalt
Medizinische Fakultät Carl Gustav Carus
Technische Universität Dresden
Phone: 49 351 458 13457
Fax: 49 351 458 5338
Background: A multitude of mhealth (mobile health) apps have been developed in recent years to support effective self-management of patients with diabetes mellitus type 1 or 2.
Objective: We carried out a systematic review of all currently available diabetes apps for the operating systems iOS and Android. We considered the number of newly released diabetes apps, range of functions, target user groups, languages, acquisition costs, user ratings, available interfaces, and the connection between acquisition costs and user ratings. Additionally, we examined whether the available applications serve the special needs of diabetes patients aged 50 or older by performing an expert-based usability evaluation.
Methods: We identified relevant keywords, comparative categories, and their specifications. Subsequently, we performed the app review based on the information given in the Google Play Store, the Apple App Store, and the apps themselves. In addition, we carried out an expert-based usability evaluation based on a representative 10% sample of diabetes apps.
Results: In total, we analyzed 656 apps finding that 355 (54.1%) offered just one function and 348 (53.0%) provided a documentation function. The dominating app language was English (85.4%, 560/656), patients represented the main user group (96.0%, 630/656), and the analysis of the costs revealed a trend toward free apps (53.7%, 352/656). The median price of paid apps was €1.90. The average user rating was 3.6 stars (maximum 5). Our analyses indicated no clear differences in the user rating between free and paid apps. Only 30 (4.6%) of the 656 available diabetes apps offered an interface to a measurement device. We evaluated 66 apps within the usability evaluation. On average, apps were rated best regarding the criterion “comprehensibility” (4.0 out of 5.0), while showing a lack of “fault tolerance” (2.8 out of 5.0). Of the 66 apps, 48 (72.7%) offered the ability to read the screen content aloud. The number of functions was significantly negative correlated with usability. The presence of documentation and analysis functions reduced the usability score significantly by 0.36 and 0.21 points.
Conclusions: A vast number of diabetes apps already exist, but the majority offer similar functionalities and combine only one to two functions in one app. Patients and physicians alike should be involved in the app development process to a greater extent. We expect that the data transmission of health parameters to physicians will gain more importance in future applications. The usability of diabetes apps for patients aged 50 or older was moderate to good. But this result applied mainly to apps offering a small range of functions. Multifunctional apps performed considerably worse in terms of usability. Moreover, the presence of a documentation or analysis function resulted in significantly lower usability scores. The operability of accessibility features for diabetes apps was quite limited, except for the feature “screen reader”.
(J Med Internet Res 2014;16(4):e104)
mobile applications; apps; mobile health; mHealth; diabetes mellitus; market analysis; systematic review; elderly; usability test; expert review
Compared to early mobile phones, today’s smartphones and tablet PCs offer a considerably wider range of functionalities. Mobile applications (apps) are increasingly used in managing various tasks in daily life. Currently, more than 900,000 apps are available in the Apple App Store (operating system: iOS, developer: Apple) and more than 700,000 apps in the Google Play Store (operating system: Android, developer: Google) . The number of health-related apps increased to 31,000 in 2013 .
Within the health care sector, apps are supporting the management of illnesses, thereby promoting health awareness and well-being [3-5]. Specifically, a multitude of apps have been developed to assist patients in the management of diabetes mellitus type 1 or 2 [3,6]. For example, a topic-specific search in the Google Play Store resulted in more than 1000 hits. The ability, however, to sort the results according to individual needs is lacking. One reason for the large number of diabetes apps is the high and steadily increasing diabetes prevalence, especially among people older than 50 years [7-9]. In 2012, 371 million people between the ages of 20 and 79 suffered from diabetes worldwide and this number is estimated to increase to 552 million people by 2030 . The high self-therapy potential certainly has a major influence on the high number of currently available apps.
We carried out a systematic review of all currently available diabetes apps for the operating systems iOS and Android, between February 2013 and April 2013. Our review aimed to provide an overview of the number of newly released apps, range of functions, target user groups, languages, acquisition costs, popularity/user ratings, the ability to connect to measurement devices, and the connection between acquisition costs and user ratings.
Diabetes prevalence increases with age. Thus, the elderly are a large target group that could benefit from diabetes apps. However, several studies have shown a lack of acceptance and a subpar use of innovative mobile technologies among this age group [11-15]. As one possible reason, Holzinger et al  and Mallenius et al  have pointed out the insufficient consideration of usability requirements of the elderly. Their experiences in handling mobile devices and apps are frequently limited. Inhibition thresholds and entry barriers are therefore particularly pronounced among this age group. In addition, cognitive and physical skills are declining with age  and result in needs that are considerably different from those of young users. Hence, this age group would benefit from apps that consider their specific usability requirements.
In order to better assess and quantify usability for the elderly, we carried out an expert-based usability evaluation based on a representative 10% sample of diabetes apps available as of April 2013. Therewith, we examined to what extent existing diabetes applications serve the usability requirements of diabetes patients aged 50 or older.
Until now, just a few reviews of diabetes apps had been conducted [3,5,6,16]. They differ from the review presented here in several ways: they considered a broader range of health care applications, they were restricted to one operating system, they reviewed solely the offered functionalities, or they were done more than one year ago. There is an absence of usability evaluation for diabetes apps [16,17]. Especially, formative usability evaluations of health apps are rare. To our knowledge, just one article has been published that links a diabetes app review with a formative usability evaluation , but their evaluation is limited to Android apps and gives no special consideration to the requirements of elderly diabetes patients as we do.
Search and Screening Strategy
Our review focused on the leading operating systems for mobile devices, iOS and Android. The analysis was carried out using the Apple App Store for iOS apps and the Google Play Store for Android apps. We focused exclusively on diabetes apps available in English and German.
As a first step, we identified keywords to ensure that every relevant diabetes app was detected. Therefore, we chose the following German and English keywords, directly related to diabetes mellitus: Diabetes, Blood Sugar/Blutzucker, Glucose/Glukose. Every hit was reviewed in terms of its relevance and explicit link to diabetes mellitus. This pre-selection was necessary due to the growing number of misleading descriptions (spam techniques) for apps, caused partly by non-existent or low admission requirements for novel apps. In the Google Play Store, no admission requirements currently exist for newly developed apps, whereas iOS apps are first internally reviewed by an app review board. All apps with an explicit link to diabetes mellitus were included in the analysis. The basis for the systematic and comparative market analysis was defined by categories and respective subcategories/specifications outlined in Table 1.
We considered all the available information given by both the stores and the apps and collected the information for all categories and subcategories/specifications. In some cases, the structure of the app stores and the provided information differed strongly from one another, so we applied different approaches for the analysis of iOS and Android apps.
[view this table]
|Table 1. Categories and respective subcategories/specifications extracted from diabetes apps.|
Search and Screening Strategy for iOS Apps
The analysis of iOS apps was conducted using the information available in the Apple App Store. In contrast to the Google Play Store, the Apple App Store offers several options for filtering the search results by choosing thematic subcategories. The results can additionally be sorted by relevance, popularity, user rating, and date of release. During the survey period, a sorting function was only available for the iPad, so the whole iOS app survey was performed via the iPad.
For the analysis, we chose the subcategories “Health and Fitness” and “Medicine”. Subsequently, the displayed apps were sorted by their date of release. The date of release served as an objective characteristic, which was necessary for a reliable and reproducible acquisition of all diabetes apps. The number of hits given by the Apple App Store corresponded exactly to the number of relevant apps.
We checked every app hit with regard to its availability for iPad and iPhone. Additionally, we verified whether the app was offered exclusively for the operating system iOS or also for Android. The market analysis of diabetes apps for iOS resulted in 390 hits.
Search and Screening Strategy for Android Apps
By using the information available in the Google Play Store, the analysis of Android apps was conducted. To date, this app store offers no option to filter the search results for apps according to individual needs. Furthermore, the given “numbers of hits” is not only the number of apps but also the number of detected search terms in the app title and the app description. Thus, the search term “diabetes” led to more than 1000 hits in the Google Play Store. Keeping the limitations in mind, the number of available apps was a considerable overestimation.
In order to ensure a representative analysis despite missing selection criteria, we defined one day (03/06/2013) to record all found apps with title and developer. This definition will enable future app review processes. Additionally, every app was crosschecked for availability of an iOS version. Altogether, we found 380 diabetes apps available for the operating system Android.
Expert-Based Usability Evaluation
To examine the usability of currently available diabetes applications for the elderly, we performed an expert-based usability evaluation. With this method, usability experts put themselves in the role of potential or current users to examine products in terms of usability. We performed a summative evaluation as we exclusively included apps whose development was already finished .
Due to the high number of apps available for review, the usability evaluation was based on a representative 10% sample of existing diabetes apps as of April 2013. The sample was chosen on a random basis. The evaluation was performed by three independent experts, as suggested by Nielsen  and Barnum . They were chosen due to their comprehensive experience in handling and testing mobile devices and applications with regard to usability for the elderly and operability of accessibility features. In addition, they had specific expertise in the field of diabetes and diabetes management. They were already involved in the accompanying systematic review and a survey of diabetes patients aged 50 or older and physicians investigating the acceptance factors of diabetes applications. The authors of this article were not involved in the usability evaluation for reasons of independence.
The basis for the usability evaluation was defined by a specially created set of usability criteria considering interaction processes, interface design, and comprehensibility of content (Table 2). Therefore, we reviewed usability guidelines (ISO, DIN) with explicit regard to the requirements of the elderly concerning mobile applications. Additionally, we considered usability requirements that have been proven as relevant in previous studies with this age group (Table 2). This guideline-based approach of usability testing is in accordance with the proposals of Nielsen , and Sarodnick and Brau . We divided the selected criteria into main and subcriteria, added a clear description of their specific characteristics and defined respective assessment criteria. The experts rated each subcriterion and the expression of its characteristics by means of a 5-point Likert scale to grade the evaluation or by means of a dichotomous scale [16,19-21]. The main criteria were not evaluated themselves; their scores were calculated from the mean of the respective subcriteria.
To lower barriers for persons with reduced or limited cognitive and physical skills, iOS and Android offer different accessibility features. We tested the operability of three features for each tested app in a separate test run. We have chosen features that are relevant to the elderly and were offered by both operating systems:
- Screen reader—Voice over (iOS)/Talk back (Android): dichotomous scale
- “Larger Type” as an additional measurement for “possibility to flexibly adapt the size of operating elements and displayed images”: dichotomous scale
- “Invert colors” as an additional measurement for “sufficient color contrast”: 5-point Likert scale
According to the methodical approach of Barnum, the evaluators run through typical scenarios of use to conduct their evaluation . They were asked to take the perspective of a diabetes patient aged 50 or older. Each expert tested the main functionalities of the app, listed in the app description (eg, record of blood glucose data and/or medication, plotting graphs, search for information on diabetes mellitus, etc). All Android apps were tested on a Samsung Galaxy Note 10.1. All iOS apps were tested on an iPad 4thgeneration. If apps were offered for multiple platforms by the manufacturer, they were tested on an iPad 4thgeneration.
The chosen method offers a high level of validity and comparability due to its guideline-based approach and closed response categories . Additionally, a user-based test would not have been able to represent the pronounced heterogeneity among the age group 50 or older regarding health status, skills and preferences, experience in technology use, sociodemographics, etc, which is much more pronounced than for younger age groups [13,15,23-25]. At the same time, it would have been rather difficult to find test persons whose characteristics corresponded exactly to the distribution within the basic population.
[view this table]
|Table 2. Evaluated usability and assessment criteria for diabetes apps for the elderly.|
Search and Screening
In total, we examined 656 apps during the review process. As a result, we created three data sets (Multimedia Appendix 1), which separated the currently available diabetes apps into apps available exclusively for the operating system iOS (276 apps), apps available exclusively for the operating system Android (266 apps), and apps available for both operating systems (114 apps).
Annual Development of App Releases
The first diabetes app for iOS (according to Apple App Store as of April 2013) was developed and released on July 17, 2008 (name: Glucose-Charter, developer: e-agent). The first Android diabetes app (according to Google Play Store as of April 2013) followed on November 8, 2009 (name: Body Sugar, developer: Adibu). The number of diabetes apps released annually increased during the last five years, from 6 in 2008 to 267 in 2012. In the first four months of 2013, 149 new diabetes apps were released. The number of apps for Android more than doubled each year (Figure 1); however, this was not by publication date (unavailable in Google Play Store) but rather the date of the last update. More than half of the iOS diabetes apps (50.7%, 140/276) were specially designed for use on the iPhone. Only 87/276 (31.5%) were designed for both iPhone and iPad. Due to a lack of information in the Google Play Store, this subdivision into smartphone and tablet PC apps could not be made for Android apps.
[view this figure]
|Figure 1. Annual release figures for diabetes apps.|
The majority (85.4%, 560/656) of the examined apps were in English, especially the apps running exclusively on an Android operating system, (90.2%, 240/266). Apps with German as operating language were of relatively low number (14.6%, 96/656) (Table 3). Some apps offered the opportunity to choose between several languages after download.
[view this table]
|Table 3. Language of available diabetes apps as of April 2013.|
The acquisition costs and the ratio of free to paid apps differed strongly between the two operating systems (Table 4). While most of the iOS apps required payment (62.3%, 172/276), the vast majority of Android and Android/iOS apps were free (63.5%, 169/266 and 69.3%, 79/114). Nonetheless, some of the free apps and the inexpensive apps worked with specially designed test strips or were able to be linked to measurement devices. In these cases, the apps could not have been used without compatible devices.
The analysis of app price distribution revealed that a greater number of free apps were available across all apps (53.7%, 352/656). This appeared to be driven by Android apps where 63.5% (169/266) were free compared with 36.5% (97/266) paid. The reverse trend was observed for iOS where only 37.7% (104/276) were free compared with 62.3% (172/276) paid (Table 4).
The price of paid apps differed strongly between the operating systems (Figure 2). The vast majority (69.7%, 212/304), were in the price range of €0.01 to €3.00. The median price varied between €1.50 and €2.30, depending on the operating system. The apps designed for both operating systems tended to be the apps with the highest price level (Figure 2). The analysis also showed that some costly apps offer free or cheaper “lite” versions with limited functionalities (5.3%, 35/656).
[view this table]
|Table 4. Price distribution of apps and annual proportions of free apps since 2008.|
[view this figure]
|Figure 2. Price distribution of paid diabetes apps available as of April 2013.|
Range of Functions/Functionality
Examining the range of functions of diabetes apps demonstrated that most were limited to one function (54.1%, 355/656). Only 185/656 (28.2%) combined two functions, and three or more functions were offered by 116/656 (17.7%) of the apps available as of April 2013 (Table 5). Apps developed exclusively for iOS tended to offer a wider range of functions compared to Android apps.
A total of 348/656 (53.0%) apps and thus the majority of diabetes apps available as of April 2013 offered a documentation function (Figure 3). By using this feature, the measured blood glucose values can be recorded and either summarized in a table or plotted as a graph. Hence, the app enables the user to monitor the disease progression.
The recording of the blood glucose values mainly occurred via manual data input. Only a small number of apps offered the option to transfer the data wirelessly and automatically from the measuring device via Bluetooth to the mobile device.
The documentation function may be linked with an analysis function, which opens up the possibility to analyze the recorded data and to graphically display the results (Multimedia Appendix 2); 117/656 (17.8%) of the diabetes apps offered this service (Figure 3). The documentation function includes the recording and monitoring of individual eating habits (eg, the bread unit intake). Some apps, additionally, log the frequency of the user’s physical activity or the individual medical therapy (type and frequency). The documentation function was frequently linked with a reminder function, which reminds the user of its periodic, pre-defined medication (11.4%, 75/656) (Figure 3). According to the holistic setting approach, some of the available apps already offered the opportunity to track the course of disease for affected family members.
In total, 226 (34.5%) of the examined diabetes apps offered an information function, including the ability to inform about the illness, its diagnosis, the course of the disease, various treatment options, medication, and secondary diseases (Figure 3). Sometimes those apps provided information on the nutrient content of diverse foods and beverages and calorie consumption during various sporting activities (Multimedia Appendix 3).
A data forwarding/communication function was offered by 204/656 (31.1%) apps. With this function, the user has the opportunity to send the recorded data via email to the attending physician, family members, and/or friends (Multimedia Appendix 4). The reports can be sent frequently or on demand. Some of the apps were connected to special diabetes forums, where the users can upload their individual blood glucose values and discuss them with other diabetes patients (name: Diabesties, developer: Ayogo Health).
Surprisingly, only 58/656 (8.8%) of the diabetes apps provided an advisory function or any other kind of therapeutic support (Figure 3). Only a limited number of apps used the recorded data to create individualized advice to optimize the patients measuring, medication, eating habits, or activity behavior. One reason may be a previously required certification as a medical product for that kind of support.
Besides the previously described functions, 95/656 (14.5%) of the apps included suggestions for recipes suitable for the needs of diabetics (Figure 3).
As an example, Multimedia Appendix 5 shows screenshots of a highly reviewed app linking a documentation, analysis, communication, and information function (name: IBG Star Diabetes Manager mg/dl, developer: Sanofi Diabetes).
[view this table]
|Table 5. Number of functions, target user groups, and popularity/user ratings of diabetes apps available as of April 2013.|
[view this figure]
|Figure 3. Range of functions of diabetes apps available as of April 2013.|
Target User Groups
The vast majority (96.0%, 630/656) of the examined apps were designed specifically for patients, 24/656 (3.7%) apps addressed both patients and physicians/qualified health personnel, and only 50/656 (7.6%) were specifically designed for the target group physicians/qualified health personnel (Table 5). One reason might be the high potential for individual treatment and management of diabetes mellitus by the patients themselves. Particularly, patients suffering from type 2 diabetes have the opportunity to impact the course of the disease in a positive manner by a change in their lifestyles [31-33]. In this regard, diabetes mellitus differs from other chronic diseases such as cancer or dementia.
Popularity and User Rating
User ratings are a highly valuable and realistic evaluation of the additional benefits of apps. It is thus remarkable that just 31/656 (11.2%) of the apps designed exclusively for iOS were rated by users. In comparison, 189/266 (71.0%) of the Android apps and 75/114 (65.8%) of the apps running on both operating systems were rated (Table 5). One reason might be the rating procedure of the Apple App Store, which is more complicated than the procedure of the Google Play Store and requires several steps to rate an app. The median of the amount of provided ratings varied between six (iOS & iOS/Android apps) and nine ratings (Android apps). With a maximum of five stars for an app evaluation, the median rating varied from 3.5 (iOS apps) to 4 (Android & iOS/Android apps) stars (Table 5). Thus, 50% of the diabetes apps earned ratings of more than 3.5 to 4 stars, corresponding to a moderate to good rating.
Not only was the lower number of rated iOS apps conspicuous, the median of awarded stars was also lower than for Android apps (Table 5). That was surprising due to the higher access restrictions (peer review-based admission procedure for new apps) by the Apple App Store compared to non-existent restrictions by the Google Play Store.
Except for the ratings, the Google Play Store gave information about the number of downloads (ie, the number of installations) as another indicator of the app popularity. This information was not given by the Apple App Store. Hence, it was not possible to compare this indicator between both operating systems. But it has been shown that the number of downloads tended to correlate with the number of ratings and awarded stars.
Connection Between Acquisition Costs and User Ratings
During the analysis, the question arose of whether there is a connection between the price of an app and the level of user ratings. The results indicated that there existed a positive correlation between the acquisition costs and the number of given stars, for the price range of €0.01 to €5.00 (Figure 4). If the price exceeded €5.00, the correlation tended to inverse and the apps received worse evaluations. However, compared to free apps, no clear differences in the number of given stars could be found.
In general, free apps were rated more frequently than paid apps. With a share of 56.5% (204/361), they received the highest number of given ratings compared to just 27.5% (28/102; price range: €0.01-€1.00) up to 41.7% (5/10; price range: €10.00-€100.00) of the paid apps. However, it has to be considered that the number of free apps was considerably higher than the number of price-intensive apps.
[view this figure]
|Figure 4. Distribution of user rating differentiated by acquisition costs as of April 2013.|
Availability of Interfaces to External Sensors/Devices
Contrary to our initial expectations, only a limited number of diabetes apps possessed an interface to an external sensor or a measuring device (eg, for the measurement of blood glucose). Predominantly, apps developed for both operating systems were able to connect with an external sensor/device (7.9%, 9/114). Rarely, iOS apps (2.5%, 7/276) offered this feature compared to Android apps (5.3%, 14/266).
The majority of apps that were able to connect to an external measuring device transmitted the data via a Bluetooth interface. This interface enabled a wireless data transfer to the mobile device or to a PC. Some of the measuring devices already offered an automated transmission of the measured values in real time. There were two options for data synchronization: (1) wireless transfer of measured values to a mobile device and synchronization with the Internet, mostly to an online patient diary (registration required), and (2) wireless transfer of measured values to a PC, transfer of data to an online platform (registration required), and synchronization with a mobile device in the second step (eg, System Health Vault via Microsoft).
Expert-Based Usability Evaluation
In total, we evaluated 66 out of 656 diabetes apps within the usability evaluation (Multimedia Appendix 6): 29 apps available exclusively for the operating system iOS, 28 apps available exclusively for the operating system Android, and 9 apps available for both operating systems.
For all main and subcriteria, we averaged the evaluations of all three experts. The values of the main criteria represent the mean of the corresponding subcriteria (Table 6). The total usability score was calculated from all categories, which were determined by means of a 5-point Likert scale.
Analyzing the results, the majority of evaluations were in the range of 3.0 to 4.0, which corresponded to a moderate to good rating of the apps included in the 10% sample. All tested apps received the best rating for the subcriteria “use of understandable semantics” and “simple comprehensibility and interpretability of displayed images and depictions” with a total average value of 4.1 (Table 6). Independent of the operating system, all apps received the worst rating for the subcriterion “fault tolerance” (2.8) followed by “simple recognizability of click-sensitive areas” (3.0). It has to be mentioned that it was only possible to evaluate fault tolerance for apps offering data input (36.4%, 24/66) (Table 6). The values determined for all other subcriteria varied between 3.1 and 4.0. Comparing the total usability score between the different operating systems, we found no clear differences with values varying between 3.3 and 3.4 for both iOS/Android apps. The worst-rated app with a usability score of 2.5 was HealthFile LIVE! (developer: WakefieldSoft LLC) for Android (Multimedia Appendix 6). The best-rated app was Diabetic Recipes Volume I (developer: ECI) for Android with a usability score of 4.1. Password-protected services were offered by an average of 18.2% of the apps.
In the second run, we evaluated the three chosen accessibility features. The results show that their operability was rather limited. The highest values were observable for the screen reader features Voice over (iOS) and Talkback (Android); 25 (86.2%) of the 29 iOS apps offered the ability to read the screen content aloud compared to 19 (67.9%) of the 28 Android apps, and just 4 (44.4%) of the 9 apps designed for both operating systems. The feature “invert colors” showed no considerable improvement of color contrast compared to the results of our evaluation without testing this feature. The results for testing the feature “large type” differed widely. While none of the iOS apps offered this feature, 11/27 (40.7%) Android apps offered contents in large font (Table 6).
While conducting our systematic review, we hypothesized that usability decreases with an increasing number of functions. Hence, we additionally investigated the relationship between the main usability criteria and the number of functions by conducting several correlation analyses. The results shown in Table 7 indicate statistically significant negative relationships between the number of functions and all usability criteria, except “fault tolerance”. Thus, the number of functions and all usability criteria were significantly negative correlated with coefficients varying between −.29 and −.25 implicating that diabetes apps offering a wider range of functions performed worse in terms of usability.
Furthermore, we analyzed the relationship between the usability score and specific functions based on the differences in functionality we found in our systematic review. Therefore, we conducted multiple linear regression analysis to control for potential confounding effects of other functions offered by the same app (Table 8). It showed significant results only for two types of functions. The presence of a documentation function reduced the usability score on average by 0.36 points while the usability score of apps offering an analysis function was on average reduced by 0.21 points. According to R2, 25% of the variance of the usability score was explained by the model. All correlation and regression analyses were conducted with the statistical software Stata 11.1.
[view this table]
|Table 6. Usability scores from expert-based usability evaluation by operating system, shown as mean values.|
[view this table]
|Table 7. Spearman’s rank correlation coefficients comparing number of functions with main usability criteria scores.|
[view this table]
|Table 8. Multiple regression analysis: relationship between usability score and functions.a|
The systematic review showed that a large number of diabetes apps are available. Providers may be entering the market as a result of the rising number of patients suffering from diabetes. For users, especially patients, it becomes increasingly difficult to find an app in this plethora of options that is suitable for one’s own needs. This problem is caused by a lack of effective search criteria and filter functions in the app stores. More frequently, apps are chosen that appear first in the search results for diabetes apps. The sorting criteria in the app stores are not apparent. New apps from relatively unknown developers could have difficulties being listed among the first results.
At the same time, many apps offered similar functionalities, mostly a documentation function, which is consistent with earlier findings of Martínez-Pérez et al , Eng et al , Chomutare et al , and Demidowich et al . Differences were found mostly in the design and the menu structure. Additionally, the majority of diabetes apps offered only one or two functions. An application that simultaneously informs and contributes to successful treatment by combining documentation, reminder, and advisory functions was not available as of April 2013. Such a multifunctional app would have a clear additional benefit, especially for newly diagnosed and elderly diabetes patients. At the same time, simple, understandable design, content, and menu navigation are needed. But several apps showed a lack of suitability and usability for its main target group diabetics, which is in accordance with the findings of Demidowich et al . Some were apparently developed without intensive cooperation or prior (usability) tests with patients or professional health care personnel. The obligation for certification as a medical product does currently not exist, even though some diabetes apps are already certified, especially those linked to an external measurement device, eg, iBGStar Diabetes Manager (iOS), Bodytel Mobile (iOS and Android), or Diabetes Companion by mySugr (iOS and Android). Peer review processes of health apps are already offered by several platforms such as iMedical Apps , JMIR mHealth , or HealthOn . This structure offers substantial and valuable support for users and their decision-making processes, but also in terms of quality assurance and improvement. In contrast, the Google Play Store, as one of the leading app stores, does not currently apply a peer review-based admission procedure for new apps. This lack of certification results in a lack of “[…] demonstrated safety and effectiveness, especially where information and trends are not just presented to patients, but used to make treatment recommendations” .
As an example, one app mainly offered labels like “after breakfast” or “after lunch” for the documentation of measured blood glucose values, which implies postprandial states. But, for most diabetics, the blood glucose values before eating are decisive to adjust the amount of insulin. Another app offered the feature to plot a graph labeled “HbA1c (glycated hemoglobin) according to day-time”. This neglects that the HbA1c is a value for long-term blood glucose. As a further example, some apps provided no option to modify, once entered, values at a later point in time. This misconception prevents a subsequent data amendment for example in the run-up of a doctors’ visit. At the same time, there is no opportunity to correct wrong values, for instance, as a result of erroneous data input.
Taking a look into the future, we expect that the data forwarding function, especially to the attending physician, will gain significantly more importance. A regular transmission of data to their physician linked with frequent feedback can be a valuable therapy support, particularly for people in rural regions that are or will become affected by a shortage of doctors [38,39]. Nevertheless, there still exist open questions concerning data security, network coverage, interoperability, documentation requirements, and coverage in health care plans, etc [38,40,41].
Additionally, the automated transmission of measured values in real time from the measuring device to the mobile device will probably spread and is an important driver for the perceived ease of use as El-Gayar et al point out . At the moment, the input of the measured blood glucose values occurs manually in most of the cases, as earlier findings of Eng et al  and Chomutare et al  confirm. Some manufacturers already offer blood glucose meters that allow real-time data transmission of measured blood glucose data via Bluetooth to a mobile device (eg, GlucoTel , iBGStar ) (Figure 5). This function simplifies the process of documenting for the patient and, at the same time, it increases the reliability of the entered data and subsequent analysis.
Notwithstanding the functions offered by diabetes apps, their effects on patients’ self-management and, accordingly, on important indicators, as for example the HbA1c value, have to be evaluated. A comprehensive, representative, and long-term study investigating these health effects is lacking so far. But different studies focusing on the outcomes of mobile phone interventions, such as SMS, point out a slightly positive influence as shown in the reviews of Holtz et al  and Free et al .
[view this figure]
|Figure 5. Glucose meters with automated transmission of blood glucose values to mobile devices.|
Expert-Based Usability Evaluation
As a supplement to our systematic review, we conducted an expert-based usability evaluation to examine the usability of currently available diabetes apps for patients aged 50 or older. Therefore, we focused on the age group with the highest diabetes prevalence. The results show moderate to good evaluations (range 3.0-4.0) for all reviewed usability criteria, which is in accordance with the results of Demidovich et al . An exception was “fault tolerance” (Table 6). The main criteria, “comprehensibility”, rated best with a score of 4.0. In particular, the elderly benefit from easy, understandable semantics and easy, comprehensible, and interpretable images and depictions, due to their frequently limited experience in handling mobile devices and apps. Hence, it can lower inhibition thresholds, especially during the first time of use, and therefore increase acceptance among this age group. The same is true for the influence of “easily understandable feedback” (3.3) and an “intuitive usability” (3.5) (main criterion “usability”). However, these two subcriteria performed worse within our evaluation. The fact that most of the diabetes apps were in English or contained English/foreign language terms (Table 3) diminished the usability especially for non-English-speaking elderly in terms of comprehensibility. This can be seen as one optimization approach for future app development that is comparatively easy to implement.
The evaluation showed moderate results for the main criterion “presentation” (3.5). Our test of three accessibility features indicated a very good operability of the screen readers, especially for Voice over (86.2%) offered by iOS. However, the operability of the features “invert colors” (3.5) and “large type” (21.2%) was rather restricted. Additionally, the minority of diabetes apps (17.8% of the iOS apps) were developed specifically for tablet PCs. However, we assess them as more suitable and user-friendly for elderly diabetes patients due to their larger display and bigger illustrations. With increasing age, cognitive and physical skills are declining, such as eyesight, visual acuity, color vision, contrast detection, and hearing . Especially elderly diabetes patients are often suffering from retinopathy. Shortcomings of diabetes apps concerning the presentation of information (color contrast, size of operating elements, option to flexibly adapt size of operating elements, etc) and the operability of accessibility features are deterring potentially interested patients from using diabetes apps from the outset. Therefore, a barrier-free access is a basic prerequisite for elderly patients to make them use diabetes apps. Against this background, all the other criteria we determined become of secondary importance in terms of usability.
The criterion “fault tolerance” rated worst with a score of 2.8 (Table 6). This means that the available diabetes apps were lacking an efficient fault management (criterion specified in Table 2). Especially inexperienced (elderly) users often have difficulties with inputting data. Some errors are unrecoverable or even cause the application to shut down, as Garcia et al already demonstrated in their analysis . These results have to be viewed with great concern due to the fact that these apps are dealing with medical parameters. This becomes particularly serious if these values provide the basis for further calculations as, for example, the required dose of insulin. Our results demonstrate once again the meaningfulness of an automated transfer of measured values from the blood glucose meter to the mobile device . Additionally, it could be helpful to limit choice to meaningful values, for example, by offering a numeric keyboard to enter blood glucose values.
Our correlation and regression analyses indicated a strong link between usability and the number and kind of functions. In particular, the number of functions and all main usability criteria were significantly negative correlated. These results cast a different light on the aforementioned outcomes of our usability test. Hence, the moderate to good usability scores applied mainly to apps offering a small range of functions. This relation inverted when we looked upon the considerably lower usability scores for multifunctional apps (Multimedia Appendix 6). Considering the special needs of elderly diabetes patients, they would benefit from a comprehensive and easily understandable support as already mentioned above. They are frequently affected by multimorbidity and polypharmacy, particularly after many years of suffering from diabetes .
Differed by functions, apps offering a documentation and an analysis function performed worse in terms of usability. This result is surprising as the documentation function is most commonly offered with a share of 53.0%. It can be a valuable support for all diabetes patients measuring and recording blood glucose level regularly. But as interviews with diabetes patients aged 50 or older (conducted in Germany in 2013) have shown, most of them prefer documentation by means of a conventional diary (results not shown). One reason they named was the aforementioned lack of usability and therefore a too complicated and time-consuming handling. Moreover, the use of these two functions is characterized by a higher level of human-technology interaction than, for example, the use of an information function. Of course, this can be accompanied by a wider scope of error sources and usability barriers.
Altogether, the potential of diabetes apps for assisting and supporting diabetes patients aged 50 or older is large. In particular, the target group aged between 50 and 60 years holds great potential as people of this age are already quite familiar with mobile devices and apps . Now, app developers are facing the challenge of taking sufficient account of the usability criteria we examined and addressing those shortcomings. There is no need for a huge number of new app functions. It is more about improving what already exists.
The systematic review and the expert-based usability test were conducted within the project “InnoMedTec”. In that project, we investigate the question: “How should a mobile application be designed to support an effective self-management for diabetes patients aged 50 or older?” Our market analysis provided the basis for a survey among diabetes patients aged 50 or older and physicians, which we conducted in the second half of 2013. Within guided interviews, we investigated the current use, acceptance promoting/inhibiting factors, potentially needed support, and concrete design features for the development of a diabetes app. Merging the results of the systematic market review and the survey, a user- and needs-oriented prototype app for diabetics aged 50 or older will be developed this year. To guarantee usability and needs orientation, the prospective users and usability experts are involved in the product development process right from the beginning. User- and expert-based usability tests are performed regularly. The results are integrated continuously in the app optimization until its finalization.
The conducted review focused exclusively on apps for the currently leading operating systems, iOS and Android. Currently available diabetes apps for other operating systems, such as Windows Phone, Blackberry OS, or Symbian, were not considered within the analysis. The app publication date was solely available for iOS apps, but not for Android apps. Here, the date of the last update served as reference value. Due to that fact, the results concerning the annually new released diabetes apps were not directly comparable.
The app information was gathered by studying the descriptions in the app stores and within the app itself. More detailed information, such as download statistics, were not available for analysis. Perhaps this information would enable more detailed results concerning the user groups, for example, differentiated by gender, age groups, or type of diabetes.
Expert-Based Usability Evaluation
Within our usability evaluation, we investigated usability criteria exclusively. We evaluated neither the quality of content and functions nor their effectiveness. Furthermore, it has to be mentioned that one usability evaluation cannot claim to cover all possible and critical usage situations that can possibly occur [18-20,49].
We would also stress that we examined a sample of all available diabetes apps, not just a sample of apps developed specifically for the elderly. Hence, many of the apps we evaluated do not claim to be particularly suitable for this age group.
Despite the huge amount of currently available diabetes apps, most of them offer a small number of similar functionalities. Patients and physicians should be directly involved during the app development to tackle the lack of usability and needs-orientation for its main target group diabetics. We think that data forwarding options and automated transmission of measured values to mobile devices will gain more importance in the future.
The usability of diabetes apps for patients aged 50 or older was moderate to good. But this result applied mainly to apps offering a small range of functions. Multifunctional apps performed considerably worse in terms of usability. Differed by functions, the documentation and analysis function indicated significantly lower usability scores. The operability of accessibility features for diabetes apps was quite limited, except for the feature “screen reader”.
This study is conducted within the InnoMedTec project (Grant no. 100098212), which is funded by the European Social Fund and the Free State of Saxony. We would like to thank Dr Lars Renner, Robert Gurke, Andrea Mathe, Tiffany Evans, and Maike Bellmann for their feedback in the writing process and careful reading. We also would like to express our thanks to our reviewers for their valuable feedback and helpful suggestions that helped improve our paper. Findings from this study have been previously shown in a poster presentation at the Diabetes Congress 2013, Leipzig (Germany), and as an oral presentation at the Telemed 2013, Berlin (Germany).
Conflicts of Interest
Multimedia Appendix 1
Dataset diabetes app review.[XLSX File (Microsoft Excel File), 157KB]
Multimedia Appendix 2
Screenshots of apps with documentation and analysis function.[PNG File, 2MB]
Multimedia Appendix 3
Screenshots of apps with information function.[PNG File, 1MB]
Multimedia Appendix 4
Screenshots of apps with communication function.[PNG File, 568KB]
Multimedia Appendix 5
Screenshots of a highly reviewed app with various functionalities.[PNG File, 1MB]
Multimedia Appendix 6
Usability scores differed by operating systems and apps.[XLSX File (Microsoft Excel File), 12KB]
- Ingraham N. Apple: 900,000 apps in the App Store, 375,000 iPad-optimized, 28 million copies of mountain lion sold. 2013. URL: http://www.theverge.com/2013/6/10/4412918/apple-stats-update-wwdc-2013 [accessed 2013-08-30] [WebCite Cache]
- MHealth Watch. Mobile health care apps growing fast in number. 2013. URL: http://mhealthwatch.com/mobile-health-care-apps-growing-fast-in-number-20052/ [accessed 2013-09-19] [WebCite Cache]
- Martínez-Pérez B, de la Torre-Díez I, López-Coronado M. Mobile health applications for the most prevalent conditions by the World Health Organization: review and analysis. J Med Internet Res 2013;15(6):e120 [FREE Full text] [CrossRef] [Medline]
- Cafazzo JA, Casselman M, Hamming N, Katzman DK, Palmert MR. Design of an mHealth app for the self-management of adolescent type 1 diabetes: a pilot study. J Med Internet Res 2012;14(3):e70 [FREE Full text] [CrossRef] [Medline]
- Eng DS, Lee JM. The promise and peril of mobile health applications for diabetes and endocrinology. Pediatr Diabetes 2013 Jun;14(4):231-238. [CrossRef] [Medline]
- Chomutare T, Fernandez-Luque L, Arsand E, Hartvigsen G. Features of mobile diabetes applications: review of the literature and analysis of current applications compared against evidence-based guidelines. J Med Internet Res 2011;13(3):e65 [FREE Full text] [CrossRef] [Medline]
- Lehnert H, Wittchen HU, Pittrow D, Bramlage P, Kirch W, Böhler S, et al. Prevalence and pharmacotherapy of diabetes mellitus in primary care. Dtsch Med Wochenschr 2005 Feb 18;130(7):323-328. [CrossRef] [Medline]
- Kirch W. Encyclopedia of Public Health: Volume 1: A - H Volume 2: I - Z. New York: Springer; 2008.
- Du Y, Heidemann C, Scheidt-Nave C, Robert-Koch-Institut. Diabetes mellitus in Deutschland. 2011. URL: http://www.rki.de/DE/Content/Gesundheitsmonitoring/Gesundheitsberichterstattung/GBEDownloadsK/2011_3_diabetes.pdf?__blob=publicationFile [accessed 2013-09-02] [WebCite Cache]
- International Diabetes Federation. Diabetes Atlas. 2013. URL: http://www.idf.org/diabetesatlas [accessed 2013-08-30] [WebCite Cache]
- Holzinger A, Searle G, Nischelwitzer A. On some aspects of improving mobile applications for the elderly. In: Stephanidis C, editor. UAHCI'07 Proceedings of the 4th International Conference on Universal Acess in Human Computer Interaction. Coping with Diversity. Berlin, Heidelberg: Springer; 2007:923-932.
- Lorenz A, Oppermann R, Zahl L, Mielke D. Personalized mobile health monitoring for elderly. In: Cheok AD, Chittaro L, editors. MobileHCI ’07 Proceedings of the 9th International Conference on Human Computer Interaction with Mobile Devices and Services. New York: ACM; 2007:297-304.
- Mallenius S, Rossi M, Tuunainen VK. Factors affecting the adoption and use of mobile devices and services by elderly people – results from a pilot study. 2010. URL: http://classic.marshall.usc.edu/assets/025/7535.pdf [WebCite Cache]
- Lorenz A, Oppermann R. Mobile health monitoring for the elderly: Designing for diversity. Pervasive and Mobile Computing 2009 Oct;5(5):478-495. [CrossRef]
- Steele R, Lo A, Secombe C, Wong YK. Elderly persons' perception and acceptance of using wireless sensor networks to assist healthcare. Int J Med Inform 2009 Dec;78(12):788-801. [CrossRef] [Medline]
- Demidowich AP, Lu K, Tamler R, Bloomgarden Z. An evaluation of diabetes self-management applications for Android smartphones. J Telemed Telecare 2012 Jun;18(4):235-238. [CrossRef] [Medline]
- Garcia E, Martin C, Garcia A, Harrison R, Flood D. Systematic analysis of mobile diabetes management applications on different platforms. In: Holzinger A, Simonic KM, editors. Information Quality in e-Health. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer; 2011:379-396.
- Sarodnick F, Brau H. Methoden der Usability Evaluation: Wissenschaftliche Grundlagen und praktische Anwendungen. 2nd ed. Bern: Verlag Hans Huber; 2011.
- Nielsen J. Usability engineering. San Francisco, Calif: Morgan Kaufmann; 1993.
- Barnum CM. Usability testing essentials: ready, set...test!. San Francisco, Calif: Morgan Kaufmann; 2011.
- Gondy L. Designing user studies in informatics. London: Springer; 2011.
- Abascal J, Arrue M, Fajardo I, Garay N. An expert-based usability evaluation of the EvalAccess web service. In: Navarro-Prieto R, Vidal JL, editors. HCI related papers of Interacción 2004. Dordrecht: Springer; 2006:1-18.
- Lee YS. Older adults’ user experience with mobile phones: identification of user clusters and user requirements. 2007. URL: http://scholar.lib.vt.edu/theses/available/etd-09172007-135013/unrestricted/Dissertation_YSL.pdf [accessed 2014-02-07] [WebCite Cache]
- Deutscher Bundestag. Sechster Bericht zur Lage der älteren Generation in der Bundesrepublik Deutschland - Altersbilder in der Gesellschaft. 2010. URL: http://www.bmfsfj.de/RedaktionBMFSFJ/Abteilung3/Pdf-Anlagen/bt-drucksache-sechster-altenbericht,property=pdf,bereich=bmfsfj,sprache=de,rwb=true.pdf [accessed 2014-02-07] [WebCite Cache]
- Caprani N, O'Connor NE, Gurrin C. Touch screens for the older user. 2012. URL: http://www.intechopen.com/books/assistive-technologies/touch-screens-for-the-older-user [accessed 2014-02-07] [WebCite Cache]
- DIN-Fachbericht 131:2003. Leitlinien für Normungsgremien zur Berücksichtigung der Bedürfnisse von älteren Menschen und von Menschen mit Behinderungen; Deutsche und englische Fassung des CEN/CENELEC-Leitfadens 6. Berlin: Beuth Verlag; 2003.
- Schmid A, Dörfler I, Dany F, Böpple O. Analyse der Akzeptanzkriterien für mobile Anwendungen im Bereich Gesundheit in der Zielgruppe 50+. In: Shire KA, Leimeister JM, editors. Technologiegestützte Dienstleistungsinnovation der Gesundheitswirtschaft. Wiesbaden: Springer Gabler; 2012:57-82.
- Renaud K, van Biljon J. Predicting technology acceptance and adoption by the elderly : a qualitative study. In: Navarro-Prieto R, Vidal JL, editors. SAICSIT '08 Proceedings of the 2008 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on IT Research in Developing Countries: Riding the Wave of Technology. New York: ACM; 2008:210-219.
- Banse M. Softwareergonomische Optimierung Touchscreen basierter Mensch-Computer-Interaktion. 2008. URL: http://touchscreen-usability.com/ [accessed 2014-02-07] [WebCite Cache]
- DIN EN ISO 9241-110:2008-09. Ergonomie der Mensch-System-Interaktion - Teil 110: Grundsätze der Dialoggestaltung. Deutsche Fassung EN ISO 9241-110:2006. Berlin: Beuth Verlag; 2008.
- Tuomilehto J, Lindström J, Eriksson JG, Valle TT, Hämäläinen H, Ilanne-Parikka P, Finnish Diabetes Prevention Study Group. Prevention of type 2 diabetes mellitus by changes in lifestyle among subjects with impaired glucose tolerance. N Engl J Med 2001 May 3;344(18):1343-1350. [CrossRef] [Medline]
- Diabetes Prevention Program (DPP) Research Group. The Diabetes Prevention Program (DPP): description of lifestyle intervention. Diabetes Care 2002 Dec;25(12):2165-2171 [FREE Full text] [Medline]
- Diabetes Prevention Program (DPP) Research Group, Knowler WC, Fowler SE, Hamman RF, Christophi CA, Hoffman HJ, et al. 10-year follow-up of diabetes incidence and weight loss in the Diabetes Prevention Program Outcomes Study. Lancet 2009 Nov 14;374(9702):1677-1686 [FREE Full text] [CrossRef] [Medline]
- iMedical Apps. 2013. URL: http://www.imedicalapps.com/ [accessed 2013-11-11] [WebCite Cache]
- JMIR mHealth. 2013. URL: https://docs.google.com/spreadsheet/viewform?formkey= dEY0YThtT2lXTXVUQ2VuUHRzSkh5a2c6MQ [WebCite Cache]
- HealthOn. 2013. URL: http://www.healthon.de/ [accessed 2013-12-10] [WebCite Cache]
- Klonoff DC. The current status of mHealth for diabetes: will it be the next big thing? J Diabetes Sci Technol 2013 May;7(3):749-758. [Medline]
- Nuffield Council on Bioethics. Medical profiling and online medicine: the ethics of 'personalised healthcare' in a consumer age. London: Nuffield Council on Bioethics; 2010.
- Ascher M. Telemedicine in diabetes care. 2011. URL: http://next.eller.arizona.edu/courses/BusinessInternationalEnvironments/Fall2011/student_papers/finalmelissaascher.pdf [accessed 2013-09-02] [WebCite Cache]
- VDE Initiative Mikromedizin. VDE-Positionspapier Telemonitoring zur Prävention von Diabetes-Erkrankungen. 2008. URL: https://www.vde.com/de/InfoCenter/Seiten/Details.aspx?eslShopItemID=998e16ea-eb11-42de-8f42-65e672c275dd [accessed 2014-04-01] [WebCite Cache]
- World Health Organization. Telemedicine: opportunities and developments in member states: report on the Second Global Survey on eHealth 2009. Geneva: World Health Organization; 2010.
- El-Gayar O, Timsina P, Nawar N, Eid W. Mobile applications for diabetes self-management: status and potential. J Diabetes Sci Technol 2013;7(1):247-262 [FREE Full text] [Medline]
- BodyTel Europe GmbH. Glucotel. 2013. URL: http://www.bodytel.com/glucotel.html [accessed 2013-08-30] [WebCite Cache]
- Sanofi Diabetes. iBGStar. 2013. URL: http://www.ibgstar-shop.de/ [accessed 2013-08-30] [WebCite Cache]
- Holtz B, Lauckner C. Diabetes management via mobile phones: a systematic review. Telemed J E Health 2012 Apr;18(3):175-184. [CrossRef] [Medline]
- Free C, Phillips G, Felix L, Galli L, Patel V, Edwards P. The effectiveness of M-health technologies for improving health and health services: a systematic review protocol. BMC Res Notes 2010;3:250 [FREE Full text] [CrossRef] [Medline]
- Wittchen HU, Group DS. Diabetes in Deutschland. DETECT: Eine bundesweite Studie in 3.500 Hausarztpraxen und 55.000 Patienten. 2005. URL: http://www.detect-studie.de/publikationen/DETECT_EXUBERA.pdf [accessed 2014-02-07] [WebCite Cache]
- Online Publishers Association. A portrait of today’s smartphone user. 2012. URL: http://onlinepubs.ehclients.com/images/pdf/MMF-OPA_--_Portrait_of_Smartphone_User_--_Aug12_%28Public%29.pdf [accessed 2014-02-07] [WebCite Cache]
- Nielsen J. Usability inspection. In: Plaisant C, editor. CHI ’94 Conference Companion, Human Factors in Computing Systems, Celebrating Interdependence. New York, NY: ACM Press; 1994:413-414.
|Apps: mobile applications|
|HbA1c: glycated hemoglobin|
|Edited by G Eysenbach; submitted 19.09.13; peer-reviewed by C Lauckner, B Martínez-Pérez, T Chomutare; comments to author 30.10.13; revised version received 24.02.14; accepted 13.03.14; published 09.04.14|
Please cite as:
Arnhold M, Quade M, Kirch W
Mobile Applications for Diabetics: A Systematic Review and Expert-Based Usability Evaluation Considering the Special Requirements of Diabetes Patients Age 50 Years or Older
J Med Internet Res 2014;16(4):e104
END, compatible with Endnote
BibTeX, compatible with BibDesk, LaTeX
RIS, compatible with RefMan, Procite, Endnote, RefWorks
Add this article to your Mendeley library
Add this article to your CiteULike library
Copyright©Madlen Arnhold, Mandy Quade, Wilhelm Kirch. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 09.04.2014.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.