This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The use of mobile medical apps by clinicians and others has grown considerably since the introduction of mobile phones. Medical apps offer clinicians the ability to access medical knowledge and patient data at the point of care, but several studies have highlighted apps that could compromise patient safety and are potentially dangerous. This article identifies a range of different kinds of risks that medical apps can contribute to and important contextual variables that can modify these risks. We have also developed a simple generic risk framework that app users, developers, and other stakeholders can use to assess the likely risks posed by a specific app in a specific context. This should help app commissioners, developers, and users to manage risks and improve patient safety.
The use of mobile medical apps by clinicians, patients, and others has grown dramatically since the introduction of mobile phones and tablet computers. Recent studies show that mobile devices and apps can support a variety of routine medical tasks including clinical reference, drug dose calculation, patient education, accessing medical records, and clinical decision support [1-4]. Mobile phone apps have also been shown to benefit patients in a range of interventions across numerous medical specialties and treatment modalities [5-9]. Medical apps offer clinicians the ability to access medical knowledge and patient data at the point of care with unprecedented ease. However, the intersection of mobile technology, apps, and health care is currently in its most dynamic phase, meaning that there is a need to ensure that patient safety is not compromised before this field matures. For the purposes of this paper, a mobile medical app means any software application created for or used on a mobile device for medical or other health-related purposes. This paper highlights the need for risk assessment to support clinical use of mobile medical apps by critically appraising the existing literature in this field. We identify the different types of risks to which medical apps can contribute and develop a framework that brings together the usage scenarios, contextual factors, and app complexity to estimate the overall probability and severity of harm resulting from use of a mobile medical app.
Evidence of Unsafe Apps
It is important that mobile medical apps used in health care settings are accurate and reliable, especially as health care professionals and patients may make critical decisions based on information from an app. There is limited literature that addresses the accuracy of mobile medical apps, and that which exists is often highly specialized and not necessarily generalizable to all medical apps [10]. Despite this, several studies have highlighted a number of medical apps that can compromise patient safety and are potentially dangerous in clinical use. For example, certain apps designed for opioid dosage conversion or melanoma detection demonstrate dangerously poor accuracy, while a number of other medical apps do not follow evidence-based guidelines [11-14]. Such risks have led to recent calls for increased regulation before further use and adoption of some apps in clinical practice [15-17]. One issue highlighted by a small number of studies is that many app developers have little or no formal medical training and do not involve clinicians in the development process and may therefore be unaware of patient safety issues raised by inappropriate app content or functioning [18-20]. Another issue is the sheer volume and exponential growth of medical apps, meaning it is practically impossible to assess each and every medical app [21]. The narrow scope of the current evidence base means it is difficult to generalize these statements to all medical and health-related apps. There is sufficient evidence that a small subsection of medical apps presents a risk to patient safety, and therefore it is appropriate to develop a model to help assess these risks.
Regulatory Oversight
Clinicians trying to safely navigate the apps minefield have had relatively little support from regulatory agencies. The Food and Drug Administration (FDA) released their guidance only in July 2013 after a 2-year consultation period and are focusing primarily on apps that transform the mobile platform into a regulated medical device [22], which to date numbers approximately 100 apps [23]. The remainder will be subject to what the FDA calls “enforcement discretion”, that is, no regulation [24]. Other regulatory agencies such as the Medicines and Healthcare Products Regulatory Agency and the Therapeutic Goods Administration of Australia have offered limited guidance to health care practitioners by including apps under their existing regulations for medical devices [25,26]. The lack of clarity regarding when a medical app becomes a formal medical device means that many developers may not recognize that their app requires formal regulation. As a result, the vast majority of medical apps remain without any form of regulation or safety check, and some of these may present a patient safety or other risk.
The Need for a Risk Framework to Support Clinical Use of Medical Apps
To inform the safe clinical use of apps and future professional guidance and regulation, it is important to understand and then quantify the different kinds of risk posed by medical apps. It is generally accepted that two dimensions define risk [27]: (1) the probability of an event occurring that could lead to harm, and (2) the severity of the harm that is likely to follow that event.
As with many aspects of medicine, the decision to use a medical app in a particular clinical context relies on our ability to assess the risk of harm and balance it against the anticipated benefits. These judgments require health care professionals to understand the intended benefits, limitations, and risks associated with medical apps in order to make an informed app usage decision. The first step in this process is to identify the different types of risk to which medical apps can contribute, summarized (in broadly increasing order of severity) in Table 1.
There is currently no clinically relevant risk assessment framework for medical apps, so health care practitioners, patients, and app developers find it challenging to quickly assess the risks posed by a specific app. In order to develop a comprehensive risk assessment framework, and to distinguish the different kinds of risk listed in Table 1, we must understand the key variables that can influence risk in medical apps. These variables can be broken down into those risk factors that are inherent to an app and those that depend on the external context where the app is used. Risk factors inherent to an app may be reduced through appropriate regulation, while managing contextual risk factors may require a formal education program to raise awareness among app users. In our opinion, the main contextual and inherent app risk factors are listed in Table 2 below, in no particular order. Arguably many of these risk variables are applicable to many other sources of medical information such as websites or textbooks, although there are important considerations specific to mobile apps that should be recognized.
Different types of risk that medical use of apps may contribute to, and scenarios where these may arise.
Type of risk in increasing order of severity
Main stakeholder affected
Sample scenario where this risk could arise
What can be done to manage this risk
Loss of reputation
Professional/organization
App displays sensitive performance data about professional or service
Bad algorithm controlling insulin pump, surgical robot, radiotherapy machine, etc
Adopt safety critical software design and development methods
Exhaustively check design and test algorithm & user interface
The main inherent and external (contextual) risk variables contributing to the total risk associated with mobile medical apps.
Type of risk variable
Specific risk variable
Explanation
Inherent to the app
Intended function
When the intended function of the app is inherently dangerous, eg, calculating insulin requirements or reprogramming a pacemaker, this will increase risk
Inaccurate or out of date content
Apps that contain inaccurate or out-of-date content have an increased chance of causing harm
Complexity of task supported by the app
Apps that carry out complex tasks (eg, drug dosage calculations) have greater potential for harm due to programming errors than simple information display
Lack of feedback or failsafe mechanism
Apps that do not offer the user a means to report safety issues to the developers are less safe
External factors, depending on context of app use
App user
Use of the app by people other than those intended by the developer may cause harm
Inappropriate app usage
Apps that are used inappropriately, outside their design envelope, are inherently risky
Inadequate user training
Even when the app user is as the developer intended, risk can be increased if the user has inadequate training or knowledge to recognize when there is a patient safety hazard, eg, incorrect content or inappropriate advice from the app
Likelihood of errors being detected
App usage in scenarios with a low error detection capacity (eg, community care versus intensive care) are likely to be riskier
App usage factor (AUF)
Total number of app users multiplied by the average number of app uses per user per day. Apps with a high usage factor have a greater safety impact on the population than those with a low usage factor
The last two contextual factors are discussed in more detail here. One is the likelihood of a clinical error being detected and averted, which should be high in a well-monitored inpatient or high dependency setting but low when there is only intermittent patient contact, such as in outpatient clinics or primary care. Paradoxically, therefore, the risk of using a faulty app may be lower in an intensive care unit than in general practice. The second is the app usage factor (AUF), which links app risk to the number of users and frequency of use. Risk is proportional to the number of patients affected, so disease prevalence or similar indices of the number of people likely to be affected by an error need to be considered. We have developed the idea of the AUF to help estimate the risk impact of a particular app on a given population. It thus follows that a popular app with a high number of frequent users will have a high AUF and subsequent high impact on the population.
It is also important to consider the generic clinical safety hazards posed by the hardware, software, and sensors that make up a typical medical software application, not just mobile apps. This includes risks posed by the display, user interface, network issues, and subsequent loss of information. Each of these factors should be taken into account, so that the more complex the app, the greater the risk. Unfortunately, these risks are difficult to assess without formal training, but there is guidance for health organizations and developers that aims to address these factors in more detail [28].For the purposes of our risk assessment framework, these factors have been included within the Complexity of task variable.
Developing a formal risk assessment framework for mobile medical apps should enable us to reduce the “residual risk” (exposure to loss remaining after all other known risks have been countered, factored in, or eliminated) by recognizing and implementing a range of possible safety measures in future app development, procurement, and regulation models.
Bringing Together Usage Scenarios, Contextual Factors, and App Complexity to Estimate Overall Probability and Severity of Harm
We believe that the risks posed by a specific medical app depend on three main dimensions: (1) the probability and the severity of harm, defined by the risk scenarios listed in Table 1, (2) the inherent complexity of the app, which determines how predictable that risk is, and (3) the external or contextual factors listed above.
Given the wide variety of medical apps, we believe that different approaches to risk assessment and management will be required dependent on app risk. This is illustrated in Figure 1, which shows a 2-dimensional “app-space” where an app can be located depending on its probability of harm, based on the variables above, and its complexity. According to its combined chances of harm and complexity, it will fall into one of four broad zones. Apps in Zone A require only local inspection, those in Zone B require a more formal risk assessment, and those in Zone C require professional review of a full safety case and the use of safety critical development methods. Apps that fall into Zone D should meet the criteria for formal regulation and review by governmental bodies such as the FDA due to their high probability of causing harm. It is not possible to assess the proportion of medical apps in each of the risk categories of A-C given the lack of data on medical apps available. However, based on the total number of medical apps available (approximately 20,000) [29] and the number currently regulated by the FDA (approximately 100) [23], we calculate that the proportion of apps that currently fall into risk category D is approximately 0.5%. This classification into four broad risk zones should help app users, developers, and regulators to evaluate each app using a relevant risk assessment and management model based on the zone where the app is located. It is important to note that these zones form a spectrum rather than discrete entities, hence the gray lines at the boundaries of each zone.
Perhaps the biggest threat to patient safety from medical apps is likely to result from inadequate education and knowledge of health care professionals and patients about their risks. We think in the vast majority of cases, it is probably the actions of a user resulting from a specific app that leads to harm, rather than the app itself. Therefore, an important additional strategy to minimize the risks posed by apps is to develop an educational program to raise awareness of potential patient safety and other risks following inappropriate app use. Developing a single, authoritative, coherent set of guidance and supporting educational materials will require the support of professional bodies such as the Royal Colleges. This will help avoid a confusing plethora of guidance, such as occurred when the harm resulting from some uses of social media was recognized.
In the meantime, there are a range of proposed app regulation models, many of which are highlighted in Figure 1, that may provide some form of protection against hazardous medical apps for patients and health care practitioners [30-33]. Many of these risk management methods are in the early stages of development and have not yet been formally implemented, but they offer a number of advantages for health care professionals, patients, and developers alike, offering some degree of safety check for medical apps not meeting the requirements for formal regulation. A detailed discussion of regulation and regulatory issues for mobile medical apps is beyond the scope of this paper, and interested readers are directed to the references above for further information.
Two-dimensional "App-space" for risk assessment of mobile medical apps with key suggesting appropriate models for app regulation.
Conclusions
While the widespread use of high-quality apps by health care practitioners and patients is to be welcomed, there still remains a significant potential for harm. The risks to patient safety and professional reputation are real, and steps should be taken to mitigate these. Identification of all the different kinds of risk and of key variables that influence risk are key stages in the development of a risk assessment model, which should also take into account app complexity and the probability of harm. Education of current health practitioners about the risks posed by medical apps should start soon, before the first case reports of patients harmed by a medical app come to light. Further work should focus on the recognition and mitigation of medical app risk, as the outlook for medical apps in health care is bright once their quality and safety can be reliably assessed and managed.
AbbreviationsAUF
app usage factor
FDA
Food and Drug Administration
We acknowledge helpful comments from the reviewers and the editor of JMIR. The authors would like to thank Warwick University for kindly covering the article processing fees for this article. This paper was not supported by any external funding.
Author TL is an editor for the iMedicalApps website, dedicated towards providing news on the integration of mobile technology into medical care and the reviewing of medical apps for mobile devices. Neither TL or JW consult or receive reimbursement from app developers or creators.
PatelBKChapmanCGLuoNWoodruffJNAroraVMImpact of mobile tablet computers on internal medicine resident efficiency201203121725436810.1001/archinternmed.2012.4522412110172/5/436OzdalgaEOzdalgaAAhujaNThe smartphone in medicine: a review of current and potential use among physicians and students2012145e12810.2196/jmir.199423017375v14i5e128PMC3510747SclafaniJTirrellTFFrankoOIMobile tablet use among academic physicians and trainees201302371990310.1007/s10916-012-9903-623321961PMC4057035FrankoOITirrellTFSmartphone app use among medical providers in ACGME training programs2012103653135910.1007/s10916-011-9798-722052129QuinnCCCloughSSMinorJMLenderDOkaforMCGruber-BaldiniAWellDoc mobile diabetes management randomized controlled trial: change in clinical and behavioral outcomes and patient and physician satisfaction200806103160810.1089/dia.2008.02831847368910.1089/dia.2008.0283DonkerTPetrieKProudfootJClarkeJBirchMRChristensenHSmartphones for smarter delivery of mental health programs: a systematic review20131511e24710.2196/jmir.279124240579v15i11e247PMC3841358KirwanMVandelanotteCFenningADuncanMJDiabetes self-management smartphone application for adults with type 1 diabetes: randomized controlled trial20131511e23510.2196/jmir.258824225149v15i11e235PMC3841374CarterMCBurleyVJNykjaerCCadeJEAdherence to a smartphone application for weight loss compared to website and paper diary: pilot randomized controlled trial2013154e3210.2196/jmir.228323587561v15i4e32PMC3636323MorrisRJavedMBodgerOHemington GorseSWilliamsDA comparison of two smartphone applications and the validation of smartphone applications as tools for fluid calculation for burns resuscitation2014084058263410.1016/j.burns.2013.10.01524246618S0305-4179(13)00345-8BierbrierRLoVWuRCEvaluation of the accuracy of smartphone medical calculation apps2014162e3210.2196/jmir.306224491911v16i2e32PMC3936266HaffeyFBradyRRMaxwellSA comparison of the reliability of smartphone apps for opioid conversion201302362111710.1007/s40264-013-0015-023322549WolfJAMoreauJFAkilovOPattonTEnglishJCHoJFerrisLKDiagnostic inaccuracy of smartphone applications for melanoma detection2013041494422610.1001/jamadermatol.2013.2382233253021557488PMC4019431RosserBAEcclestonCSmartphone applications for pain management20111763081210.1258/jtt.2011.10110221844177jtt.2011.101102FerreroNAMorrellDSBurkhartCNSkin scan: a demonstration of the need for FDA regulation of medical apps on iPhone201303683515610.1016/j.jaad.2012.10.04523394920S0190-9622(12)01181-4MisraSLewisTLAungstTDMedical application use and the need for further research and assessment for clinical practice: creation and integration of standards for best practice to alleviate poor application design2013061496661210.1001/jamadermatol.2013.606237831501698662BuijinkAWVisserBJMarshallLMedical apps for smartphones: lack of evidence undermines quality and safety20130618390210.1136/eb-2012-10088522923708eb-2012-100885McCartneyMHow do we know whether medical apps work?2013346f181123516158HamiltonADBradyRRMedical professional involvement in smartphone 'apps' in dermatology2012071671220110.1111/j.1365-2133.2012.10844.x22283748HuckvaleKCarMMorrisonCCarJApps for asthma self-management: a systematic assessment of content and tools20121014410.1186/1741-7015-10-144231716751741-7015-10-144PMC3523082RodriguesMAVisvanathanAMurchisonJTBradyRRRadiology smartphone applications; current provision and cautions201310455556210.1007/s13244-013-0274-423912880PMC3781246van VelsenLBeaujeanDJvan Gemert-PijnenJEWhy mobile health app overload drives us crazy, and how to restore the sanity2013132310.1186/1472-6947-13-23233995131472-6947-13-23PMC3621678US Food and Drug Administration20132014-09-08http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM263366.pdf6SQq2TgYl201309232014-09-08http://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm369431.htm6SQpzUtdfMcCarthyMFDA will not regulate most mobile medical apps2013347f58412407287520102014-09-08Medical Device Technology Forum on the use of software as a medical devicehttp://www.mhra.gov.uk/Howweregulate/NewTechnologiesForums/DevicesNewTechnologyForum/Forums/CON0849876SQq3qtCyTherapeutic Goods Administration201309252014-09-08http://www.tga.gov.au/industry/devices-software-mobile-apps.htm6SQq76Z5eInternational Organization for Standardization (ISO)2014-09-08http://www.iso.org/iso/catalogue_detail?csnumber=381936SQqjQ7pzHealth & Social Care Information Centre20132014-09-08http://www.isb.nhs.uk/documents/isb-0129/amd-39-2012/0129392012tele.pdf6SQqIet5MAungstTDClausonKAMisraSLewisTLHusainIHow to identify, assess and utilise mobile medical applications in clinical practice2014026821556210.1111/ijcp.1237524460614LewisTLA systematic self-certification model for mobile medical apps2013154e8910.2196/jmir.244623615332v15i4e89PMC3636320WyattJCLewisTLCurtisKHow to Assess and Improve the Quality and Safety of Apps for Medicine and Health Promotion?2013Medicine 2.0, 6th World Congress on Social Media, Mobile Apps, Internet/Web2.0September 23-24, 2013LondonJMIR442443YetisenAKMartinez-HurtadoJLda Cruz VasconcellosFSimseklerMCAkramMSLoweCRThe regulation of mobile medical applications20140371458334010.1039/c3lc51235e244250702014-09-08Regulation of Health Apps: A Practical Guidehttp://www.d4.org.uk/research/regulation-of-health-apps-a-practical-guide-January-2012.pdf6SQqgN5pR