Smartphones are rapidly being adopted into the medical field. A recent survey found that 79% of medical students and 75% of postgraduate trainees owned smartphones [1]. One important use of smartphones is to aid in diagnosis, prognosis, and treatment of medical conditions. Apps can aid in diagnosis by providing a reference to staging systems, such as the severity staging of chronic obstructive pulmonary disease (COPD), or can provide rapid access to published algorithms in decision making. These reference or decision support functions that perform minimal calculations are typically considered to be at low risk of causing errors [2].

There are increasing numbers of clinical scoring systems that can include calculations, such as determining the severity of liver disease (Model for End-Stage Liver Disease, MELD), the likelihood of having a pulmonary embolism (Wells’ Score for Pulmonary Embolism), and risk stratification in acute coronary syndrome (the thrombolysis in myocardial infarction, TIMI, score for non-ST elevation myocardial infarction, NSTEMI) [3-5]. Smartphone applications can make calculating these scores easier by providing information rapidly after performing a calculation using patient-specific data. Indeed, medical calculation apps are one of the most-used apps by doctors, often used several times per day [1]. Medical calculation apps can be considered to be of higher complexity because they do not just present previously published information, but may perform complex calculations based on user input. This increases the risk of error.

Health care professionals rely on decision-making aids such as medical apps, yet their accuracy has not been verified. The American Food and Drug Administration (FDA) has attempted to eliminate the distribution of faulty apps related to health care [6]. It is critical that apps used in clinical settings are accurate because the scoring results can impact a clinician’s decision. Unfortunately, there is limited literature on the accuracy of smartphone medical calculators with the current evidence being highly specialized [7,8]. The purpose of our study was to assess the accuracy of general medical calculating apps on smartphones.

The inclusion and exclusion criteria enabled us to come up with a list of frequently downloaded apps that were relevant to internal medicine. Fourteen (0.87%) smartphone apps were tested out of the 1603 smartphone apps found during initial research (Figure 1; Table 2).

Results of testing the 14 apps by using 10 variations for each of the 13 calculating functions are shown in Table 3. Out of the 1240 tests conducted there were 17 errors; therefore, the overall accuracy was 98.6% (17/1240).

In terms of functions, 11 of 13 functions (85%) were 100% accurate on all apps. The Child-Pugh score and the MELD score were 97% and 95% accurate, respectively. For the Child-Pugh score, there were errors in scoring for 2 apps. In all 4 errors found, the errors caused a difference in score by 1 point which did not translate to a different Child-Pugh class.

Issues occurred with the MELD score calculations on multiple apps. Eight of 14 apps produced similar incorrect scores for the cases involving creatinines >4 mg/dL (353.6 μmol/L). For 1 case, this error translated to an increased score which then gave an elevated severity (from 52.6% mortality to 71.3% mortality). This same error was found in 8 apps. These errors appeared to be because of incomplete application of United Network for Organ Sharing (UNOS) modifications of the original MELD scoring by the apps. The UNOS modification set a maximum allowable creatinine of 4 mg/dL (353.6 μmol/L).

The results of the study suggest that most medical calculator smartphone apps are accurate and can confidently be used in clinical settings. From an internal medicine perspective, the free apps that were 100% accurate and contained the most functions desired by internists were CliniCalc [11], Calculate by QxMD [10], and Medscape [20]. Although most of the apps provided accurate results, it is important to be cautious while using the Child-Pugh score and MELD score on certain apps, specifically.

There is a lack of evidence on the accuracy of medical calculating apps for smartphones. Information recommending medical calculating apps only provided qualitative information on the apps, without testing accuracy [24-27]. This study determines the actual accuracy of information provided by apps.

The study highlights the need for verifying medical apps before use in patient care. Although we found smartphone apps to be quite accurate, we found errors in the smartphone calculations that were clinically significant. There are efforts in the United States by the FDA to regulate medical device apps, but it is not clear if medical calculating apps are defined as medical devices in all countries [2,28]. Medical smartphone apps may be considered devices depending on the complexity of the patient information and calculation [2]. For medical apps that provide erroneous results, although downloaded from a global app store, they likely fall under legislation of the country where they are downloaded and used. The legal ramifications could be complex. Ultimately, it is likely the responsibility of the physician to determine if their calculating app is accurate. For individual physicians, testing and verifying each calculating function of each app is not reasonable. Thus, we provide physicians with clear evidence-based advice on which current apps to use.

Apps change quickly with new apps and frequent updates. We recommend that a system be put in place to verify smartphone apps that perform medical calculations to ensure they function properly. One way this can be done is by having a third party verify the accuracy of smartphone calculations. This could be conducted similar to our study but on a larger scale, with more variations and functions tested. With a list of trustworthy and validated apps, health care professionals could more confidently integrate smartphone technology into clinical settings.

There were limitations to this study. Because of the time frame of the project and the wide range of apps available, it was determined that not every function on every app could be tested. Thus, accuracy for each app may differ for other calculating functions or other test cases. Furthermore, although apps were available on multiple platforms, we only tested 1 platform for each app. Predominantly, the iOS platform was tested because most apps were available on this platform. Another limiting factor was that the focus was on apps used by general internists; therefore, results may differ for other specialties. However, this generalist approach does provide information on accuracy of smartphone medical calculation apps used by internists.

In summary, we found that most smartphone medical calculator app functions were accurate. However, some errors were noted in some functions of some apps. Given that using smartphones as medical calculators makes them a medical device, a system to verify smartphone calculation accuracy would be useful to reduce the chance of errors affecting patient care.