Published on in Vol 16, No 9 (2014): September

mHealth and Mobile Medical Apps: A Framework to Assess Risk and Promote Safer Use

mHealth and Mobile Medical Apps: A Framework to Assess Risk and Promote Safer Use

mHealth and Mobile Medical Apps: A Framework to Assess Risk and Promote Safer Use

Authors of this article:

Thomas Lorchan Lewis1 Author Orcid Image ;   Jeremy C Wyatt2 Author Orcid Image


1Warwick Medical School, University of Warwick, Coventry, United Kingdom

2Leeds Institute of Health Sciences, Faculty of Medicine, Health & Psychology, University of Leeds, Leeds, United Kingdom

*all authors contributed equally

Corresponding Author:

Thomas Lorchan Lewis, BSc (Hons), MB ChB

Warwick Medical School

University of Warwick

Medical Teaching Centre

Gibbet Hill Road

Coventry, CV4 7AL

United Kingdom

Phone: 44 7876453511

Fax:44 2476528375


The use of mobile medical apps by clinicians and others has grown considerably since the introduction of mobile phones. Medical apps offer clinicians the ability to access medical knowledge and patient data at the point of care, but several studies have highlighted apps that could compromise patient safety and are potentially dangerous. This article identifies a range of different kinds of risks that medical apps can contribute to and important contextual variables that can modify these risks. We have also developed a simple generic risk framework that app users, developers, and other stakeholders can use to assess the likely risks posed by a specific app in a specific context. This should help app commissioners, developers, and users to manage risks and improve patient safety.

J Med Internet Res 2014;16(9):e210




The use of mobile medical apps by clinicians, patients, and others has grown dramatically since the introduction of mobile phones and tablet computers. Recent studies show that mobile devices and apps can support a variety of routine medical tasks including clinical reference, drug dose calculation, patient education, accessing medical records, and clinical decision support [1-4]. Mobile phone apps have also been shown to benefit patients in a range of interventions across numerous medical specialties and treatment modalities [5-9]. Medical apps offer clinicians the ability to access medical knowledge and patient data at the point of care with unprecedented ease. However, the intersection of mobile technology, apps, and health care is currently in its most dynamic phase, meaning that there is a need to ensure that patient safety is not compromised before this field matures. For the purposes of this paper, a mobile medical app means any software application created for or used on a mobile device for medical or other health-related purposes. This paper highlights the need for risk assessment to support clinical use of mobile medical apps by critically appraising the existing literature in this field. We identify the different types of risks to which medical apps can contribute and develop a framework that brings together the usage scenarios, contextual factors, and app complexity to estimate the overall probability and severity of harm resulting from use of a mobile medical app.

Evidence of Unsafe Apps

It is important that mobile medical apps used in health care settings are accurate and reliable, especially as health care professionals and patients may make critical decisions based on information from an app. There is limited literature that addresses the accuracy of mobile medical apps, and that which exists is often highly specialized and not necessarily generalizable to all medical apps [10]. Despite this, several studies have highlighted a number of medical apps that can compromise patient safety and are potentially dangerous in clinical use. For example, certain apps designed for opioid dosage conversion or melanoma detection demonstrate dangerously poor accuracy, while a number of other medical apps do not follow evidence-based guidelines [11-14]. Such risks have led to recent calls for increased regulation before further use and adoption of some apps in clinical practice [15-17]. One issue highlighted by a small number of studies is that many app developers have little or no formal medical training and do not involve clinicians in the development process and may therefore be unaware of patient safety issues raised by inappropriate app content or functioning [18-20]. Another issue is the sheer volume and exponential growth of medical apps, meaning it is practically impossible to assess each and every medical app [21]. The narrow scope of the current evidence base means it is difficult to generalize these statements to all medical and health-related apps. There is sufficient evidence that a small subsection of medical apps presents a risk to patient safety, and therefore it is appropriate to develop a model to help assess these risks.

Regulatory Oversight

Clinicians trying to safely navigate the apps minefield have had relatively little support from regulatory agencies. The Food and Drug Administration (FDA) released their guidance only in July 2013 after a 2-year consultation period and are focusing primarily on apps that transform the mobile platform into a regulated medical device [22], which to date numbers approximately 100 apps [23]. The remainder will be subject to what the FDA calls “enforcement discretion”, that is, no regulation [24]. Other regulatory agencies such as the Medicines and Healthcare Products Regulatory Agency and the Therapeutic Goods Administration of Australia have offered limited guidance to health care practitioners by including apps under their existing regulations for medical devices [25,26]. The lack of clarity regarding when a medical app becomes a formal medical device means that many developers may not recognize that their app requires formal regulation. As a result, the vast majority of medical apps remain without any form of regulation or safety check, and some of these may present a patient safety or other risk.

To inform the safe clinical use of apps and future professional guidance and regulation, it is important to understand and then quantify the different kinds of risk posed by medical apps. It is generally accepted that two dimensions define risk [27]: (1) the probability of an event occurring that could lead to harm, and (2) the severity of the harm that is likely to follow that event.

As with many aspects of medicine, the decision to use a medical app in a particular clinical context relies on our ability to assess the risk of harm and balance it against the anticipated benefits. These judgments require health care professionals to understand the intended benefits, limitations, and risks associated with medical apps in order to make an informed app usage decision. The first step in this process is to identify the different types of risk to which medical apps can contribute, summarized (in broadly increasing order of severity) in Table 1.

There is currently no clinically relevant risk assessment framework for medical apps, so health care practitioners, patients, and app developers find it challenging to quickly assess the risks posed by a specific app. In order to develop a comprehensive risk assessment framework, and to distinguish the different kinds of risk listed in Table 1, we must understand the key variables that can influence risk in medical apps. These variables can be broken down into those risk factors that are inherent to an app and those that depend on the external context where the app is used. Risk factors inherent to an app may be reduced through appropriate regulation, while managing contextual risk factors may require a formal education program to raise awareness among app users. In our opinion, the main contextual and inherent app risk factors are listed in Table 2 below, in no particular order. Arguably many of these risk variables are applicable to many other sources of medical information such as websites or textbooks, although there are important considerations specific to mobile apps that should be recognized.

Table 1. Different types of risk that medical use of apps may contribute to, and scenarios where these may arise.
Type of risk in increasing order of severityMain stakeholder affectedSample scenario where this risk could ariseWhat can be done to manage this risk
Loss of reputationProfessional/organizationApp displays sensitive performance data about professional or serviceGood security
Loss of privacy (patient confidentiality)PatientPoor security of patient dataEncryption
Lose phone holding patient dataAvoid holding patient data on mobile device
Poor quality patient dataPatient/professional/ organization (eg, financial data)App allows bad data to be entered into patient record or retrieved from it at handoverData validation on entry and retrieval from authenticated source
Poor lifestyle or clinical decisionPatient/professionalBad patient data used in risk calculation algorithmCheck correct data retrieved
Bad knowledge or search toolCheck algorithm properly coded
Bad advice or algorithmUse proven health behavior change methods
Poor risk communication
Inappropriate but reversible clinical actionPatient/professionalPoor medication adviceTest quality of advice on sample data
Provide facility for user feedback and respond to this
Inappropriate and irreversible clinical actionPatient/professional/ organization (liability exposure)Bad algorithm controlling insulin pump, surgical robot, radiotherapy machine, etcAdopt safety critical software design and development methods
Exhaustively check design and test algorithm & user interface
Table 2. The main inherent and external (contextual) risk variables contributing to the total risk associated with mobile medical apps.
Type of risk variableSpecific risk variableExplanation
Inherent to the appIntended functionWhen the intended function of the app is inherently dangerous, eg, calculating insulin requirements or reprogramming a pacemaker, this will increase risk
Inaccurate or out of date contentApps that contain inaccurate or out-of-date content have an increased chance of causing harm
Complexity of task supported by the appApps that carry out complex tasks (eg, drug dosage calculations) have greater potential for harm due to programming errors than simple information display
Lack of feedback or failsafe mechanismApps that do not offer the user a means to report safety issues to the developers are less safe
External factors, depending on context of app useApp userUse of the app by people other than those intended by the developer may cause harm
Inappropriate app usageApps that are used inappropriately, outside their design envelope, are inherently risky
Inadequate user trainingEven when the app user is as the developer intended, risk can be increased if the user has inadequate training or knowledge to recognize when there is a patient safety hazard, eg, incorrect content or inappropriate advice from the app
Likelihood of errors being detectedApp usage in scenarios with a low error detection capacity (eg, community care versus intensive care) are likely to be riskier
App usage factor (AUF)Total number of app users multiplied by the average number of app uses per user per day. Apps with a high usage factor have a greater safety impact on the population than those with a low usage factor

The last two contextual factors are discussed in more detail here. One is the likelihood of a clinical error being detected and averted, which should be high in a well-monitored inpatient or high dependency setting but low when there is only intermittent patient contact, such as in outpatient clinics or primary care. Paradoxically, therefore, the risk of using a faulty app may be lower in an intensive care unit than in general practice. The second is the app usage factor (AUF), which links app risk to the number of users and frequency of use. Risk is proportional to the number of patients affected, so disease prevalence or similar indices of the number of people likely to be affected by an error need to be considered. We have developed the idea of the AUF to help estimate the risk impact of a particular app on a given population. It thus follows that a popular app with a high number of frequent users will have a high AUF and subsequent high impact on the population.

It is also important to consider the generic clinical safety hazards posed by the hardware, software, and sensors that make up a typical medical software application, not just mobile apps. This includes risks posed by the display, user interface, network issues, and subsequent loss of information. Each of these factors should be taken into account, so that the more complex the app, the greater the risk. Unfortunately, these risks are difficult to assess without formal training, but there is guidance for health organizations and developers that aims to address these factors in more detail [28].For the purposes of our risk assessment framework, these factors have been included within the Complexity of task variable.

Developing a formal risk assessment framework for mobile medical apps should enable us to reduce the “residual risk” (exposure to loss remaining after all other known risks have been countered, factored in, or eliminated) by recognizing and implementing a range of possible safety measures in future app development, procurement, and regulation models.

We believe that the risks posed by a specific medical app depend on three main dimensions: (1) the probability and the severity of harm, defined by the risk scenarios listed in Table 1, (2) the inherent complexity of the app, which determines how predictable that risk is, and (3) the external or contextual factors listed above.

Given the wide variety of medical apps, we believe that different approaches to risk assessment and management will be required dependent on app risk. This is illustrated in Figure 1, which shows a 2-dimensional “app-space” where an app can be located depending on its probability of harm, based on the variables above, and its complexity. According to its combined chances of harm and complexity, it will fall into one of four broad zones. Apps in Zone A require only local inspection, those in Zone B require a more formal risk assessment, and those in Zone C require professional review of a full safety case and the use of safety critical development methods. Apps that fall into Zone D should meet the criteria for formal regulation and review by governmental bodies such as the FDA due to their high probability of causing harm. It is not possible to assess the proportion of medical apps in each of the risk categories of A-C given the lack of data on medical apps available. However, based on the total number of medical apps available (approximately 20,000) [29] and the number currently regulated by the FDA (approximately 100) [23], we calculate that the proportion of apps that currently fall into risk category D is approximately 0.5%. This classification into four broad risk zones should help app users, developers, and regulators to evaluate each app using a relevant risk assessment and management model based on the zone where the app is located. It is important to note that these zones form a spectrum rather than discrete entities, hence the gray lines at the boundaries of each zone.

Perhaps the biggest threat to patient safety from medical apps is likely to result from inadequate education and knowledge of health care professionals and patients about their risks. We think in the vast majority of cases, it is probably the actions of a user resulting from a specific app that leads to harm, rather than the app itself. Therefore, an important additional strategy to minimize the risks posed by apps is to develop an educational program to raise awareness of potential patient safety and other risks following inappropriate app use. Developing a single, authoritative, coherent set of guidance and supporting educational materials will require the support of professional bodies such as the Royal Colleges. This will help avoid a confusing plethora of guidance, such as occurred when the harm resulting from some uses of social media was recognized.

In the meantime, there are a range of proposed app regulation models, many of which are highlighted in Figure 1, that may provide some form of protection against hazardous medical apps for patients and health care practitioners [30-33]. Many of these risk management methods are in the early stages of development and have not yet been formally implemented, but they offer a number of advantages for health care professionals, patients, and developers alike, offering some degree of safety check for medical apps not meeting the requirements for formal regulation. A detailed discussion of regulation and regulatory issues for mobile medical apps is beyond the scope of this paper, and interested readers are directed to the references above for further information.

Figure 1. Two-dimensional "App-space" for risk assessment of mobile medical apps with key suggesting appropriate models for app regulation.
View this figure

While the widespread use of high-quality apps by health care practitioners and patients is to be welcomed, there still remains a significant potential for harm. The risks to patient safety and professional reputation are real, and steps should be taken to mitigate these. Identification of all the different kinds of risk and of key variables that influence risk are key stages in the development of a risk assessment model, which should also take into account app complexity and the probability of harm. Education of current health practitioners about the risks posed by medical apps should start soon, before the first case reports of patients harmed by a medical app come to light. Further work should focus on the recognition and mitigation of medical app risk, as the outlook for medical apps in health care is bright once their quality and safety can be reliably assessed and managed.


We acknowledge helpful comments from the reviewers and the editor of JMIR. The authors would like to thank Warwick University for kindly covering the article processing fees for this article. This paper was not supported by any external funding.

Conflicts of Interest

Author TL is an editor for the iMedicalApps website, dedicated towards providing news on the integration of mobile technology into medical care and the reviewing of medical apps for mobile devices. Neither TL or JW consult or receive reimbursement from app developers or creators.

  1. Patel BK, Chapman CG, Luo N, Woodruff JN, Arora VM. Impact of mobile tablet computers on internal medicine resident efficiency. Arch Intern Med 2012 Mar 12;172(5):436-438. [CrossRef] [Medline]
  2. Ozdalga E, Ozdalga A, Ahuja N. The smartphone in medicine: a review of current and potential use among physicians and students. J Med Internet Res 2012;14(5):e128 [FREE Full text] [CrossRef] [Medline]
  3. Sclafani J, Tirrell TF, Franko OI. Mobile tablet use among academic physicians and trainees. J Med Syst 2013 Feb;37(1):9903 [FREE Full text] [CrossRef] [Medline]
  4. Franko OI, Tirrell TF. Smartphone app use among medical providers in ACGME training programs. J Med Syst 2012 Oct;36(5):3135-3139. [CrossRef] [Medline]
  5. Quinn CC, Clough SS, Minor JM, Lender D, Okafor MC, Gruber-Baldini A. WellDoc mobile diabetes management randomized controlled trial: change in clinical and behavioral outcomes and patient and physician satisfaction. Diabetes Technol Ther 2008 Jun;10(3):160-168. [CrossRef] [Medline]
  6. Donker T, Petrie K, Proudfoot J, Clarke J, Birch MR, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res 2013;15(11):e247 [FREE Full text] [CrossRef] [Medline]
  7. Kirwan M, Vandelanotte C, Fenning A, Duncan MJ. Diabetes self-management smartphone application for adults with type 1 diabetes: randomized controlled trial. J Med Internet Res 2013;15(11):e235 [FREE Full text] [CrossRef] [Medline]
  8. Carter MC, Burley VJ, Nykjaer C, Cade JE. Adherence to a smartphone application for weight loss compared to website and paper diary: pilot randomized controlled trial. J Med Internet Res 2013;15(4):e32 [FREE Full text] [CrossRef] [Medline]
  9. Morris R, Javed M, Bodger O, Hemington Gorse S, Williams D. A comparison of two smartphone applications and the validation of smartphone applications as tools for fluid calculation for burns resuscitation. Burns 2014 Aug;40(5):826-834. [CrossRef] [Medline]
  10. Bierbrier R, Lo V, Wu RC. Evaluation of the accuracy of smartphone medical calculation apps. J Med Internet Res 2014;16(2):e32 [FREE Full text] [CrossRef] [Medline]
  11. Haffey F, Brady RR, Maxwell S. A comparison of the reliability of smartphone apps for opioid conversion. Drug Saf 2013 Feb;36(2):111-117. [CrossRef] [Medline]
  12. Wolf JA, Moreau JF, Akilov O, Patton T, English JC, Ho J, et al. Diagnostic inaccuracy of smartphone applications for melanoma detection. JAMA Dermatol 2013 Apr;149(4):422-426 [FREE Full text] [CrossRef] [Medline]
  13. Rosser BA, Eccleston C. Smartphone applications for pain management. J Telemed Telecare 2011;17(6):308-312. [CrossRef] [Medline]
  14. Ferrero NA, Morrell DS, Burkhart CN. Skin scan: a demonstration of the need for FDA regulation of medical apps on iPhone. J Am Acad Dermatol 2013 Mar;68(3):515-516. [CrossRef] [Medline]
  15. Misra S, Lewis TL, Aungst TD. Medical application use and the need for further research and assessment for clinical practice: creation and integration of standards for best practice to alleviate poor application design. JAMA Dermatol 2013 Jun;149(6):661-662. [CrossRef] [Medline]
  16. Buijink AW, Visser BJ, Marshall L. Medical apps for smartphones: lack of evidence undermines quality and safety. Evid Based Med 2013 Jun;18(3):90-92. [CrossRef] [Medline]
  17. McCartney M. How do we know whether medical apps work? BMJ 2013;346:f1811. [Medline]
  18. Hamilton AD, Brady RR. Medical professional involvement in smartphone 'apps' in dermatology. Br J Dermatol 2012 Jul;167(1):220-221. [CrossRef] [Medline]
  19. Huckvale K, Car M, Morrison C, Car J. Apps for asthma self-management: a systematic assessment of content and tools. BMC Med 2012;10:144 [FREE Full text] [CrossRef] [Medline]
  20. Rodrigues MA, Visvanathan A, Murchison JT, Brady RR. Radiology smartphone applications; current provision and cautions. Insights Imaging 2013 Oct;4(5):555-562 [FREE Full text] [CrossRef] [Medline]
  21. van Velsen L, Beaujean DJ, van Gemert-Pijnen JE. Why mobile health app overload drives us crazy, and how to restore the sanity. BMC Med Inform Decis Mak 2013;13:23 [FREE Full text] [CrossRef] [Medline]
  22. US Food and Drug Administration. Mobile medical applications: guidance for industry and Food and Drug Administration staff. 2013.   URL: http:/​/www.​​downloads/​MedicalDevices/​DeviceRegulationandGuidance/​GuidanceDocuments/​UCM263366.​pdf [accessed 2014-09-08] [WebCite Cache]
  23. FDA issues final guidance on mobile medical apps Internet. 2013 Sep 23.   URL: [accessed 2014-09-08] [WebCite Cache]
  24. McCarthy M. FDA will not regulate most mobile medical apps. BMJ 2013;347:f5841. [Medline]
  25. Medicines and Healthcare Products Regulatory Agency (MHRA). 2010. Medical Device Technology Forum on the use of software as a medical device   URL: http:/​/www.​​Howweregulate/​NewTechnologiesForums/​DevicesNewTechnologyForum/​Forums/​CON084987 [accessed 2014-09-08] [WebCite Cache]
  26. Therapeutic Goods Administration. Regulation of medical software and mobile medical apps. 2013 Sep 25.   URL: [accessed 2014-09-08] [WebCite Cache]
  27. International Organization for Standardization (ISO). Medical Devices - Application of risk management to medical devices ISO 14971.   URL: [accessed 2014-09-08] [WebCite Cache]
  28. Health & Social Care Information Centre. Clinical Risk Management: Telehealth/Mobile Health Solutions - Implementation Guidance. 2013.   URL: [accessed 2014-09-08] [WebCite Cache]
  29. Aungst TD, Clauson KA, Misra S, Lewis TL, Husain I. How to identify, assess and utilise mobile medical applications in clinical practice. Int J Clin Pract 2014 Feb;68(2):155-162. [CrossRef] [Medline]
  30. Lewis TL. A systematic self-certification model for mobile medical apps. J Med Internet Res 2013;15(4):e89 [FREE Full text] [CrossRef] [Medline]
  31. Wyatt JC, Lewis TL, Curtis K. How to Assess and Improve the Quality and Safety of Apps for Medicine and Health Promotion? In: Med 2.0, 6th World Congr Soc Media, Mob Apps, Internet/Web 2.0 2013.: JMIR; 2013 Presented at: Medicine 2.0, 6th World Congress on Social Media, Mobile Apps, Internet/Web2.0; September 23-24, 2013; London p. 442-443.
  32. Yetisen AK, Martinez-Hurtado JL, da Cruz Vasconcellos F, Simsekler MC, Akram MS, Lowe CR. The regulation of mobile medical applications. Lab Chip 2014 Mar 7;14(5):833-840. [CrossRef] [Medline]
  33. d4. Regulation of Health Apps: A Practical Guide   URL: [accessed 2014-09-08] [WebCite Cache]

AUF: app usage factor
FDA: Food and Drug Administration

Edited by G Eysenbach; submitted 25.11.13; peer-reviewed by L Ferris, D Rhon; comments to author 03.02.14; revised version received 11.02.14; accepted 11.04.14; published 15.09.14


©Thomas Lorchan Lewis, Jeremy C Wyatt. Originally published in the Journal of Medical Internet Research (, 15.09.2014.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.