Background: Pediatric drug calculators (PDCs) intended for clinical use qualify as medical devices under the Medical Device Directive and the Medical Device Regulation. The extent to which they comply with European standards on quality and safety is unknown.
Objective: This study determines the number of PDCs available as mobile apps for use in the Netherlands that bear a CE mark, and explore the factors influencing the CE marking of such devices among app developers.
Methods: A scoping review of Google Play Store and Apple App Store was conducted to identify PDCs available for download in the Netherlands. CE accreditation of the sampled apps was determined by consulting the app landing pages on app stores, by screening the United Kingdom Medicines and Healthcare products Regulatory Agency’s online registry of medical devices, and by surveying app developers. The barriers to CE accreditation were also explored through a survey of app developers.
Results: Of 632 screened apps, 74 were eligible, including 60 pediatric drug dosage calculators and 14 infusion rate calculators. One app was CE marked. Of the 20 (34%) respondents to the survey, 8 considered their apps not to be medical devices based on their intent of use or functionality. Three developers had not aimed to make their app available for use in Europe. Other barriers that may explain the limited CE accreditation of sampled PDC apps included poor awareness of European regulations among developers and a lack of restrictions when placing PDCs in app stores.
Conclusions: The compliance of PDCs with European standards on medical devices is poor. This puts clinicians and their patients at risk of medical errors resulting from the largely unrestricted use of these apps.
The use of mobile health (mHealth) apps among clinicians is growing [, ]. In 2015, 60% of medical doctors in the Netherlands used at least 1 mHealth app [ ]. The widespread use of mHealth apps creates new risks for patient safety [ , ]. These risks include both technical malfunctions and misuse, either of which may lead to life-threatening medical errors [ ].
To mitigate these risks, the European Union (EU)’s 2007 Medical Device Directive (93/42/ECC) (MDD) qualifies “any…software…intended by the manufacturer to be used…for the purpose of diagnosis…or treatment…of disease” as a medical device . The MDD categorizes medical devices into 4 classes of risk (Classes I, IIa, IIb, and III) based on their technical characteristics, invasiveness, and potential for harm. Each class of risk determines a specific conformity assessment procedure for legally entering the European market. The higher the class of risk, the more stringent the conformity assessment procedure, with the overall objective being to provide adequate safeguards for users to be able to safely use the device. For example, for a Class II medical device, conformity assessment entails an evaluation of the device’s technical documentation as well as its quality management system [ ]. Depending on the device classification, conformity assessment is performed by either the manufacturer (Class I) or a European Notified Body (Class IIa and above). Once the conformity assessment is complete, medical devices obtain a CE mark, indicating their conformity with European health and safety standards, allowing them to be made available to the public within the extended single market of the European Economic Area (EEA) [ ].
In May 2017, the MDD was replaced by the Medical Device Regulation (2017/745) (MDR) . Among other changes, the MDR addresses software as a distinct item and establishes more stringent classification rules for software apps under Rule 11 [ , ]. By May 2021, all new devices placed on the European market were required to comply with the MDR. Devices already certified under the MDD may continue to be placed on the European market until May 2024, with the exception of Class I devices receiving a higher class under the MDR [ , ].
Despite increasingly binding European regulations, poor compliance of mHealth apps with EU certification requirements has been found. An examination of a sample of health apps freely available on several app stores by the Dutch Royal Institute for Public Health and the Environment reported that less than half are CE marked, as appropriate .
Pediatric drug calculators (PDCs) are tools designed to help clinicians overcome the complexities of dosing calculations in pediatrics and are increasingly used in clinical care . By allowing clinicians to calculate drug doses to be administered to children based on patient characteristics, most often their weight, PDCs constitute 1 example of medical apps potentially associated with new risks for patients [ , ]. PDCs have received little scrutiny with regard to their conformity to European standards [ , ]. In this study, we perform a scoping review of Google Play Store and Apple App Store to identify PDCs available for download in the Netherlands and determine their CE accreditation status. Barriers to CE accreditation are explored through developer surveys and interviews.
In this study, a PDC was defined as a mobile app that allows clinicians to enter information about an individual child’s weight or age in order to calculate a recommended drug dosage for that child. Programs designed to determine an infusion rate or dilution volume for a given drug dosage were also defined as PDCs.
Because PDCs perform transformation of data intended to inform treatment decisions for individual patients, they would qualify as medical devices under the MDD and the MDR [, , , ]. According to the MDD, PDCs would be classified as Class I medical devices [ ]. In line with Rule 11 of the MDR, any software “intended to provide information…used to take decisions with…therapeutic purposes” falls under Class IIa. When such decisions can cause “a serious deterioration of a person's state of health…,” the software falls under Class IIb [ ]. If the decision has the potential to “cause death or an irreversible deterioration of a person's state of health,” the software receives a class III classification [ ]. PDCs intended for clinical use would therefore be classified as Class IIa or above under the MDR.
The terms “application provider,” “manufacturer,” and “developer” have been used interchangeably in this study.
Search Strategy and Screening
A scoping review of PDCs available on app stores was performed using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology . Apps were searched for on Google Play Store (Android system, desktop version) and Apple App Store (iOS system, mobile version) between April 8 and April 19, 2020. Separate searches were performed in both app stores using the following search terms: “pediatric drug,” “pediatric drug calculator,” “neonatal drug,” and “neonatal drug calculator.” Sample searches for the terms “pediatric drug” and “pediatric drug calculator” were also conducted in both app stores. They produced identical results to the ones obtained for the previous search terms and were hence not completed. All sampled apps were deduplicated and screened by an individual reviewer (author CK). The availability of each app on Google Play Store and Apple App Store was verified independently of the initial search results.
Eligibility criteria were defined a priori. Apps were required to appear to be designed for health care professionals, including medical students, doctors, nurses, and paramedics. A PDC was required to be the main functionality or 1 of several functionalities of each app. The drug dosage calculator should have been developed for a pediatric population, with users able to calculate a drug dose for a specific weight, age, or body surface area. PDCs for oral or intravenous drugs were eligible if they covered more than 1 drug. Infusion dilution and infusion rate calculators were also included. Apps solely performing calculations for parenteral nutrition, maintenance fluids, electrolytes, or chemotherapy were excluded.
PDCs were screened based on their name, description, and screenshots available in each app store. Only apps that were freely available were downloaded.
Data Extraction and Qualitative Analysis
The name, manufacturer, and country of manufacture of each PDC were collected. Information about the type of calculations performed (drug dosage or infusion dilution or rate), the intended location of use (within or outside the EEA), and the number of downloads on Google Play Store were captured. To determine the CE marking status, we searched the PDC description and screenshots in app stores, any documentation provided on the app website, and relevant pages of the downloaded app (License, Disclaimer, About, or Terms and Conditions).
All PDC manufacturers with identifiable contact information were contacted through email (see). Developers were invited to provide information about the type of calculations performed and the intended location of use of their PDC. They were asked whether their apps were CE marked and were invited to describe their considerations in choosing whether to pursue CE marking. They were additionally asked to report any barriers encountered in the CE accreditation process. When their responses called for clarification, they were recontacted. App providers were interviewed through video calls whenever they accepted to do so.
Data obtained from PDC manufacturers were anonymized through the attribution of a numeric code and access restricted to the first author. Responses from developers were manually analyzed. Separate considerations and barriers to CE accreditation were identified from their responses and categorized through thematic inductive analysis by 1 reviewer (CK). Developers’ responses were coded against the identified themes. The coded list of barriers and considerations was discussed with 2 additional authors (JC and NA), and discrepancies were resolved through consensus.
When relevant information about CE accreditation could not be obtained from the aforementioned sources, registration of the app on the United Kingdom Medicines and Healthcare products Regulatory Agency (MHRA) was searched using the MHRA website’s search function with the app manufacturer name . At the time of data collection, the MHRA website constituted the only available repository of information related to the CE accreditation of medical devices in the EU.
Both app stores were contacted through their online contact pages to inquire about their review process for medical apps and the extent of their collaboration with European regulatory authorities.
According to the Dutch Medical Research Involving Human Subjects Act, formal ethical review was not needed. Interviewees provided informed consent through email to collect and store their anonymized responses and for these to be published. Patient consent was not applicable.
All data that informed this study are contained within the article and its supplementary files.
Public and Patient Involvement
Patients and the public were not involved in the design, conduct, reporting, dissemination plans of this research.
The lead author affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study, as planned, have been explained.
Inclusion and Classification of Apps
A total of 632 PDCs were included for screening after deduplication (seeand ). Of these, 74 (11.7%) PDCs met the inclusion criteria: 66 of the 74 (89.2%) apps were available on Google Play Store and 8 (10.8%) on Apple App Store (see ); 20 (27%) apps were available on both stores. In addition, 18 of 74 (24.3%) apps were developed in EEA countries, 60 (81.1%) included a drug dosage calculator, and 14 (18.9%) incorporated an infusion rate or infusion dilution calculator without a drug dosage calculator. The number of installations per app on Google Play Store varied from 10-100 to over 100,000; 13 of 74 (17.6%) apps had been installed over 100,000 times. Of the 74 screened PDCs, only 1 (1.4%) app was CE marked.
|Number||App name||Country||App store||Last updatea||Installations (n)b||Purchasing fee||CE marking|
|1||AnestCRITIC Crisis y Anestesia||Spain||Google Play||2019||5000-10,000||No||No|
|2||Anesthesia Assist||Portugal||Google Play||2019||50,000-100,000||No||No|
|3||Anesthesia Drugs Fast||United States||Google Play||2018||100-1000||Yes||No|
|4||Anesthesia ICC infusion calculator||Spain||Google Play||2019||5000-10,000||No||Noc|
|5||Anesthesiologist||United States||Google Play||2016||Over 100,000||No||Noc|
|6||Anesthetic drugs||India||Google Play||2019||5000-10,000||No||No|
|7||Clinical Calculator PLUS||United States||Google Play||2020||1000-5000||Yes||No|
|8||CoPE Paediatric Emergency||Denmark||Google Play||2019||1000-5000||Yes||No|
|9||Dosage Calculator||Hong Kong||Google Play||2019||Over 100,000||Yes||No|
|10||Dose calculator||Egypt||Google Play||2020||Over 100,000||No||No|
|11||Dosefinder 1||United Kingdom||Google Play||2016||5000-10,000||No||Noc|
|12||Dosis Pediatricas||—d||Google Play||2019||1000-5000||No||No|
|13||DosisPedia||Spain||Google Play, Apple App||2020||Over 100,000||No||Noc|
|14||DrDrugs: Drug Guide for Physicians - 2020 Updates||United States||Google Play||2020||1000-5000||Yes||No|
|15||Drug dosage calculations||Saudi Arabia||Google Play||2018||50,000-100,000||No||Noc|
|16||Drug Dose||Ukraine||Google Play||2016||5000-10,000||No||No|
|17||DrugCalc: Pediatric dosing calculator||Thailand||Google Play||2017||5000-10,000||No||Noc|
|18||DrugDoses||United States||Google Play, Apple App||2019||5000-10,000||Yes||Noc|
|19||Drugscape dose calculator||Jordan||Google Play||2019||10,000-50,000||No||No|
|20||Easy Drug Dose Calculator||Australia||Google Play||2018||Over 100,000||No||No|
|21||EBMcalc Pediatrics||United States||Apple App||—||—||Yes||Noc|
|22||eBroselow SafeDose||United States||Google Play, Apple App||2020||Over 100,000||No||No|
|23||EMS Calculator||United States||Apple App||—||—||Yes||No|
|24||EMS Drugs Fast||United States||Apple App||—||—||Yes||No|
|26||GIR Calc||United States||Apple App||—||—||No||No|
|27||Infinite dose: the smart dosage calculator||Egypt||Google Play||2018||10,000-50,000||No||Noc|
|28||Infusions||Colombia||Google Play, Apple App||2020||Over 100,000||No||No|
|29||Infusions - Infusions Calculator||Egypt||Google Play||2019||10,000-50,000||No||Noc|
|30||Inotropes Rate Calculator||Jordan||Google Play||2016||5000-10,000||No||Noc|
|31||Intravenous Medications Gahart||United States||Google Play||2019||5000-10,000||Yes||Noc|
|32||Kids Drug Dosage Calc - PaedRx||India||Google Play||2013||50,000-100,000||No||No|
|33||Lexicomp||United States||Google Play, Apple App||2020||Over 100,000||Yes||Noc|
|34||Medic Dose Calculator||India||Google Play||2018||5000-10,000||No||Noc|
|35||Medical Calculator||United States||Google Play||2019||500-1000||No||Noc|
|36||Mediquations Medical Calculator||United States||Google Play, Apple App||2018||10,000-50,000||Yes||No|
|37||Millidos: Pediatric Drug Dosages||Syria||Google Play||2019||10,000-50,000||No||No|
|38||MKD Dosage Calc||—||Google Play||2019||100-1000||No||No|
|39||Neomate||United Kingdom||Google Play, Apple App||2017||50,000-100,000||No||Yes|
|41||Neonatology||United Kingdom||Google Play, Apple App||2019||100-1000||Yes||No|
|42||NICU||United Kingdom||Google Play, Apple App||2019||100-1000||Yes||No|
|43||Nursing calculator||India||Google Play||2020||Over 100,000||No||Noc|
|44||Paediatric Emergencies||United Kingdom||Google Play||2019||1000-5000||No||No|
|45||Paediatric Emergency Tools||United Kingdom||Google Play, Apple App||2019||100-1000||Yes||No|
|46||palmPEDi: Pediatric Tape||United States||Google Play, Apple App||2013||5000-10,000||Yes||No|
|47||Paramedic Meds||United States||Google Play||2019||10,000-50,000||Yes||No|
|48||PedAMINES||Switzerland||Google Play, Apple App||2018||10-50||Yes||No|
|49||Ped(z) - Pediatric Calculator||Germany||Google Play, Apple App||2017||Over 100,000||No||Noc|
|51||Pedi Crisis 2.0||United States||Google Play, Apple App||2019||1000-10,000||No||No|
|52||Pedi Help||Switzerland||Google Play, Apple App||2017||50,000-10,000||No||No|
|53||Pedi Safe Medications||United States||Google Play||2016||10,000-50,000||Yes||No|
|54||Pedi Safe Pediatric Anesthesia||United States||Apple App||—||—||No||No|
|55||Pedi STAT||Canada||Google Play, Apple App||2018||Over 100,000||Yes||No|
|56||Pediatria calculadora dosis/kg||—||Google Play||2020||Over 100,000||Yes||No|
|57||Pediatric dosage calculator||Hong Kong||Google Play||2019||Over 100,000||No||No|
|58||Pediatric dose calculator||Netherlands||Google Play, Apple App||2016||50-100||Yes||No|
|59||Pediatric doses calculator||Egypt||Google Play||2020||10,000-50,000||No||No|
|60||Pediatric Gas for Anesthesia||United States||Apple App||—||—||Yes||No|
|61||Pediatric Guideline/Emergency/Pediatric child care||United Kingdom||Google Play||2020||Over 100,000||No||No|
|62||Pediatric IV calculator||Netherlands||Google Play, Apple App||2016||10-50||Yes||No|
|63||Pediatric IV dosage||—||Google Play||2014||50,000-100,000||No||No|
|64||Pediatric IV Rate||United States||Google Play||2019||50-100||No||Noc|
|65||Pediatric oral dosage||—||Google Play||2015||50,000-100,000||No||No|
|66||Pediatric pedia||Middle East||Apple App||—||—||No||No|
|67||PediRef: Pocket Pediatrics||United States||Google Play||2017||10,000-50,000||No||No|
|68||PedsGuide||United States||Google Play, Apple App||2019||1000-5000||No||No|
|70||PICU Calculator||United Kingdom||Google Play, Apple App||2019||1000-5000||No||No|
|71||PICUDoctor 5 - Cardiac Guide||Australia||Google Play||2015||10,000-50,000||Yes||Noc|
|72||RightDose||United States||Apple App||—||—||No||No|
|73||SmartPedi-Pediatric Treatment & Dose Calculator||Bangladesh||Google Play||2019||5000-10,000||No||No|
|74||UCIN-Calc Beta||Dominican Republic||Google Play||2018||5000-10,000||No||No|
aLast update on Google Play Store.
bNumber of installs on Google Play Store on May 8, 2020.
cInformation obtained from the app developer.
Of 61 app developers, 59 (96.7%) for whom contact information was available were contacted through email; 1 (1.6%) developer was additionally contacted through a video call. Responses were obtained from 20 of 59 (33.9%) providers that developed 21 apps (see). Of the 20 developers, 3 (15%) were based in the EEA. None of the apps developed by the respondents were CE marked. In addition, 2 of the 20 developers (10%) indicated that they understood that their apps qualified as Class I medical devices under the MDD but were not CE marked (developers 4 and 8), while 2 (10%) had attempted to get their apps CE marked but were unsuccessful (developers 5 and 18).
The most frequent reason for not pursuing CE accreditation provided by developers was that in their view, their apps did not qualify as medical devices (8/20, 40%). Various arguments informed this assessment. Of the 20 developers, 2 (10%) referred to the intended use of their apps, stating that the apps were designed as a reference or an educational tool for clinicians as opposed to a clinical decision-making aid (developers 11 and 15). This disclaimer was also frequently provided in the end-user licenses of sampled apps. Other developers referred to their apps’ functionality, describing them as digital documents (developer 1) or books (developer 14), which did not entail manipulation of data. In both cases, the functionality of the apps involved transformation of data. Arguments pertaining to functionality also examined the nature of the information being input into and delivered by a given app, and the weight of the result in determining the process of care. Developer 18 highlighted a difference between drug dosage calculators that could be seen as medical devices owing to their recommending a specific drug dose based on an individual patient’s characteristics, and the infusion rate or dilution calculators performing simple conversion operations on pre-established drug prescriptions. According to developers 8 and 13, the level of transparency and complexity of the computations performed by an app constituted key factors when determining whether it qualified as a medical device. Developer 13 suggested that if the calculations performed by an app are simple enough to be immediately replicable by users, then the app would not qualify as a medical device. Developer 8 suggested that even for more complex calculations, if the calculations are linked to user-accessible formulae and bibliographic support, the app should not be classified as a medical device.
Other reasons put forward by developers as justification for not CE-marking their apps included a lack of knowledge of European legislation on medical devices (3/20, 15%), the fact that their apps were not devised for use in EEA countries (3/20, 15%), and the fact that no certification was required for access to Google Play or Apple App Store (4/20, 20%). Several manufacturers described app stores as implicit arbiters for matters of regulatory compliance or safety (“My app was evaluated in the…store by the public user” or “It was very easy to place on the…store.”). Of the 20 providers, 3 (15%) indicated that Apple App Store was more restrictive than Google Play Store when granting access for PDCs; 1 (5%) developer outside the EEA argued that his app did not require testing or accreditation according to the regulations of his country (developer 15).
Several barriers to CE marking were outlined by developers. Of the 20 manufacturers, 2 (10%) indicated that the process was too complex (developers 4 and 18), and 1 (5%) said it was too costly to take on as an individual developer or a small enterprise (developer 18). This appeared more generally relevant across the sample, with multiple developers stating that they were clinicians with programming skills who developed a PDC “as a hobby” (developer 6) or “for their own use” (developer 16). An added barrier in this view concerned the lack of institutional support received by app manufacturers seeking to obtain a CE marking that were also affiliated to a hospital or a university. After receiving confirmation from national regulatory authorities that his app qualified as a Class I medical device under the MDD, developer 4 asked the relevant national health care institution for its support in the CE accreditation process. He did not obtain this support due to the institution’s concerns over the costs and associated legal liability. He shared that “developers are often left unsupported by their associated institutions...I think mostly because of a lack of experience and knowledge regarding the governance and legal implications, many institutions feel vulnerable and unwilling to engage with regulatory bodies.” Overall, this “had an unfortunate regressive effect” on the dissemination of his app. Independently of CE accreditation, 5 of 20 (25%) developers had sought alternative forms of clinical validation for their apps, for example, by national experts.
|Barriers to CE accreditation and other considerations outlined by developers on the CE accreditation process||Developera||Total developers (n)|
|No reason provided||✗b||✓c||✗||✗||✗||✓||✗||✗||✗||✓||✗||✓||✗||✗||✗||✗||✗||✗||✓||✗||5|
|App not meant for use in European countries||✗||✗||✓||✗||✗||✗||✓||✗||✗||✗||✗||✗||✗||✗||✗||✓||✗||✗||✗||✗||3|
|Not a medical device||✓||✗||✗||✗||✗||✗||✗||✓||✓||✗||✓||✗||✗||✓||✓||✗||✗||✓||✗||✓||8|
|Unaware of the EEAd medical device regulations||✗||✗||✓||✗||✓||✗||✓||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||3|
|Compliant with national regulations (non-EEA)||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✓||✗||✗||✗||✗||✗||1|
|App store not requiring certification||✗||✗||✗||✗||✗||✗||✗||✗||✓||✗||✗||✗||✓||✗||✗||✗||✓||✗||✗||✗||3|
|Discussed with national certification authorities||✗||✗||✗||✓||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||1|
|Did not receive institutional support||✗||✗||✗||✓||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||1|
|Process too complex||✗||✗||✗||✓||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✓||✗||✗||2|
|Process too costly||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✗||✓||✗||✗||1|
|App undergoing another form of validation||✓||✗||✗||✓||✓||✗||✗||✓||✗||✗||✗||✗||✗||✗||✗||✗||✗||✓||✗||✗||5|
aOf the 20 providers, 1 (5%) had developed 2 apps; we did not indicate which one in order to prevent its identification.
c✓: developer provided this specific reason.
dEEA: European Economic Area.
Neither store provided information about its review process and collaboration with European regulatory authorities. Apple App Store’s guidance states that “drug dosage calculators must come from the drug manufacturer, a hospital, university, health insurance company, pharmacy or other approved entity, or receive approval by the FDA or 1 of its international counterparts” . No such clause was found in the Google Play Store guidance [ ].
We systematically reviewed the CE accreditation of PDCs available on 2 mobile app stores in the Netherlands. Of 74 sampled PDCs, 1 (1.4%) had the appropriate CE marking in conformity with the MDD. At a time when European regulatory authorities are seeking to enhance their scrutiny of medical apps, for example, through the MDR, this study sheds a new light on several barriers to CE accreditation for eligible mHealth apps.
This study delivered several new insights. It revealed that almost all PDCs available for download in the Netherlands fail to comply with European regulations on medical devices. The only app that is certified under the MDD (Neomate; see) will likely require additional assessment due to the more stringent classification requirements of the MDR [ , , ]. The status quo with regard to CE accreditation for PDCs available on app stores is concerning, especially considering the fact that these apps are widely used by clinicians and have the potential to cause harm. Of the 74 PDCs identified on the screened stores, 13 (17.6%) had been downloaded over 100,000 times. Our findings thus echoed those of earlier studies highlighting the widespread use of mHealth apps among clinicians, including pediatricians [ , ].
Multiple reasons were identified for PDC manufacturers’ poor compliance with European regulations. First, PDC developers appeared to have varying levels of awareness of the existence of such regulations. For those manufacturers that knew about these regulations, European rule interpretation was ambivalent. Several developers argued that their apps do not qualify as medical devices according to the relevant European standards. This was true despite the clear statement by the MDD, the MDR, and associated European and European member state guidance that any software involving manipulation of data intended to be used for diagnostic or treatment purposes in individual patients qualify as a medical device [, , , , ]. The concept of intent of use seemed especially prone to a variety of interpretations by manufacturers. Many of those interviewed, as well as the end-user licenses of multiple sampled apps, indicated that their PDCs were for reference or educational purposes only. This claim, however may be in conflict with the actual use of such apps by their users, given their functionality. Although data on PDC usage by clinicians is scarce, anecdotal evidence suggests that the advice generated by such apps is frequently used to inform real patient care.
Reflecting on the functionality of their apps, some developers highlighted a difference between pediatric drug dosage calculators and calculators of infusion volumes or rates . The difference, they contended, pertained to the type of information being input into the app and the data delivered by it, as well as the complexity and transparency of the computations it ran. Although drug dosage calculators generated medication advice based on individual patient characteristics, this was not the case for infusion rate calculators that performed conversion calculations on a pre-established drug dosage. This distinction, however, does not align with MDD guidance nor with the MDR, which take the stance that any app involving transformation of data subsequently informing the treatment of an individual patient qualifies as a medical device, irrespective of the complexity of the transformation [ , , ].
In addition to disagreements on the substance of European law, another barrier hampering broader CE accreditation of eligible apps concerned the technical nature and potential costs associated with this process. According to the MDD and the MDR [, ], the onus of certification falls on providers that may lack the capacity to take on the associated liability and costs [ , ]. The challenging nature of the conformity assessment process will only increase under the MDR, given the up-classification of software apps, leading to additional evaluation requirements, including the appointment of a notified body [ , ]. In this context, a general lack of institutional support for developers seeking CE accreditation for their apps may become even more discouraging.
Another factor likely to undermine the compliance of PDC manufacturers with European standards on medical devices concerns the lack of an established process for enforcing these rules at a premarket stage. As with other European legislations, the enforcement of the MDD and the MDR is incumbent upon each EU member state . Although the Dutch Decision on Medical Devices states that the distribution and use of apps that fail to obtain a CE mark is forbidden [ ], it does not provide any enforcement means before such apps become available on app stores. Restrictive measures are unlikely to be taken unless a medical error resulting from the use of software occurs, especially if the latter leads to litigation. In this case, the responsibility for the medical error falls on both app users and the app developer [ ]. Although the MDR tightens the requirements for CE accreditation and enhances postmarket surveillance [ ], it does not fundamentally change the principle by which software manufacturers are themselves responsible for initiating the CE marking process [ ]. Effectively, the EU’s reliance on this framework in the absence of institutional support for developers and of control mechanisms at a premarket stage may have contributed to making other actors, for example, app stores, informally responsible for restricting European market access. It also implies that clinicians (or their institutions) wishing to use a PDC should themselves assess whether an app is properly accredited despite their lack of expertise in such matters [ ].
Among other measures, these findings speak to the need for making CE marking information more readily available to PDC users. This may be achieved through the planned extension of the European Database on Medical Devices (EUDAMED), scheduled to become publicly accessible in May 2022 [, ], and through the introduction of unique device identifiers for medical devices across the EEA, expected by 2024 [ ]. EEA member states may also choose to build on existing online registries of certified or evidence-based apps [ , ]. European authorities could seek to formally engage app stores as partners in the enforcement of the European MDR. At this stage, it appears that the initiative for restricting access to app stores resides with the app stores themselves, as illustrated by the various levels of restrictions described in the guidance documents of Apple App Store and Google Play Store. The finding that more PDCs were available on Google Play Store (66/74, 89.2%) than on Apple App Store (28/74, 10.8%) may suggest that differences in the stringency of requirements contributed to developers’ decisions on where to make their apps available.
This work had several limitations. Web-based PDCs that did not have a mobile interface, for example, the Dutch Paediatric Formulary calculator, which was developed in conformity with the requirements of the MDD [, ], were excluded. The restricted search functions of app stores limited the comprehensiveness of the search possible, for example, excluding paid-for apps. As a result, the list of PDCs included from those stores may not be exhaustive and may only apply to apps available for download in the Netherlands. Eight apps that were only available for purchase were excluded. Considering the potential differences between freely available apps and apps that were available for purchase and whose manufacturers may thus rely on additional finances to recover the costs associated with obtaining a CE marking, this could have led to selection bias. Despite the existence of MDD guidance stating that CE accreditation should be clearly displayed on app landing pages in the relevant stores [ ] and our cross-referencing of multiple sources, it is possible that 1 or more CE-marked PDCs were misclassified. In the absence of a mandatory statement on CE accreditation on the app stores, PDC developers were contacted directly. Additionally, 2 of the 61 (3.3%) developers could not be contacted due to missing contact information, and only 20 of the 59 (33.9%) developers contacted provided responses. This relatively low response rate was likely to introduce response bias into the qualitative component of the study. We expect therefore that those developers who responded may represent those who wish to be accessible to those with questions about their apps, and as such their responses may not be representative of all app developers.
This study demonstrates that almost no PDC currently available on two app stores accessed in the Netherlands adheres to European regulations on CE marking. In addition to the limited awareness of these norms among PDC developers, this compliance gap can be related to incorrect rule interpretation by some app manufacturers, the lack of mechanisms for verifying mHealth apps’ compliance with European medical device rules before market access, and the technical nature of the CE accreditation process for developers often lacking institutional support.
Although limited to a single category of apps, it is likely that these findings apply to a broader set of mobile devices being used in clinical settings. This lack of regulatory compliance puts both clinicians and patients at risk of medical errors resulting from the use of uncertified and, in some cases, potentially unsafe PDCs. This practice therefore undermines the potential impact of the MDD and the MDR, which strive to create a technologically safer European medical landscape, while supporting clinicians’ trust in the devices they use.
This work was supported by the National Institute for Health Research and the Wellcome Trust (Sir Henry Wellcome Fellowship).
CK collected and analyzed data and drafted and revised the paper. She is the guarantor. NA conceptualized the paper, together with CK. NA, JC, JN, and SW revised the paper. CK also attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
The corresponding author has the right to grant on behalf of all authors, and does grant on behalf of all authors, a worldwide license to the publishers and its licensees in perpetuity, in all forms, formats, and media (whether known now or created in the future), to (1) publish, reproduce, distribute, display, and store the contribution; (2) translate the contribution into other languages, create adaptations and reprints, include within collections, and create summaries, extracts, or abstracts of the contribution; (3) create any other derivative work(s) based on the contribution; (4) exploit all subsidiary rights in the contribution; (5) include electronic links from the contribution to third-party material, wherever it may be located; and (6) license any third party to do any or all of the above.
Conflicts of Interest
The authors had no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years; and had no other relationships or activities that could appear to have influenced the submitted work. NA is director of Dosium Holdings Ltd, a software company developing computerized decision support tools for medication safety. SW is director of the Dutch Pediatric Formulary and is a paid consultant for AMPharma, Khondrion. SW receives research funding from the EU (IMI JU2) and the Bill & Melinda Gates Foundation. CF is a shareholder in Dosium Holdings Ltd. The study funders were not involved in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the article for publication. The lead author confirms the independence of the contributing researchers from funders and that all authors had full access to all of the data in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.
Email to developers (email body).DOCX File , 14 KB
List of excluded apps and reasons for exclusion.DOCX File , 19 KB
- Hadjipanayis A, Klonis A, Abdel-Mannan OA, Sutcliffe AG. Smartphone use in paediatric practice: a national survey. J Mob Technol Med 2016 Mar;5(1):3-8. [CrossRef]
- Mayer MA, Rodríguez Blanco O, Torrejon A. Use of health apps by nurses for professional purposes: web-based survey study. JMIR Mhealth Uhealth 2019 Nov 1;7(11):e15195-e15112. [CrossRef]
- Ekker A, van Rest B. Medische apps, is certificeren nodig?. URL: https://www.nictiz.nl/wp-content/uploads/2013/05/13005-Whitepaper-medische-apps.pdf [accessed 2013-06-21]
- Buijink AWG, Visser BJ, Marshall L. Medical apps for smartphones: lack of evidence undermines quality and safety. Evid Based Med 2013 Jun;18(3):90-92. [CrossRef] [Medline]
- Morse SS, Murugiah MK, Soh YC, Wong TW, Ming LC. Mobile health applications for pediatric care: review and comparison. Ther Innov Regul Sci 2018 May;52(3):383-391. [CrossRef] [Medline]
- European Council. Directive 93/42/EEC of 14 June 1993 concerning medical devices. Off J Eur Union 1993;OJ L 169, 12:1-43.
- European Commission. CE Marking. URL: https://ec.europa.eu/growth/single-market/ce-marking_en [accessed 2021-06-13]
- European Parliament, European Council. Regulation 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance). Off J Eur Union 2017 May 05;OJ L 117:1-175.
- Martelli N, Eskenazy D, Déan C, Pineau J, Prognon P, Chatellier G, et al. New European regulation for medical devices: what is changing? Cardiovasc Intervent Radiol 2019 Sep;42(9):1272-1278. [CrossRef] [Medline]
- European Commission. Transition Timelines from the Directives to the Regulations Medical Devices and In Vitro Diagnostic Medical Devices. URL: https://ec.europa.eu/docsroom/documents/34907 [accessed 2021-06-13]
- European Commission. Commission Postpones Application of the Medical Devices Regulation to Prioritise the Fight against Coronavirus. URL: https://ec.europa.eu/commission/presscorner/detail/en/ip_20_589 [accessed 2021-06-13]
- van Drongelen A, de Bruijn A, Roszek B, Vonk R. Apps under the medical devices legislation. National Institute for Public Health and the Environment. 2019. URL: https://rivm.openrepository.com/handle/10029/622397 [accessed 2021-06-13]
- Murray B, Streitz MJ, Hilliard M, Maddry JK. Evaluation of an electronic dosing calculator to reduce pediatric medication errors. Clin Pediatr (Phila) 2019 Jan 08;58(4):413-416. [CrossRef]
- Hirata K, Kang A, Ramirez G, Kimata C, Yamamoto L. Pediatric weight errors and resultant medication dosing errors in the emergency department. Pediatr Emerg Care 2019;35:637-642. [CrossRef]
- Tse Y, Tuthill D. Incidence of paediatric 10-fold medication errors in Wales. Arch Dis Child 2021 Jul;106(7):656-661. [CrossRef] [Medline]
- Gordon WJ, Landman A, Zhang H, Bates DW. Beyond validation: getting health apps into clinical practice. NPJ Digit Med 2020;3(1):14-16 [FREE Full text] [CrossRef] [Medline]
- European Commission. Guidelines on the Qualification and Classification of Stand Alone Software Used in Healthcare within the Regulatory Framework of Medical Devices. URL: https://ec.europa.eu/health/sites/default/files/md_topics-interest/docs/md_meddev-guidance-216_en.pdf [accessed 2021-12-02]
- Medicines and Health Products Regulatory Agency. Public Access Database for Medical Device Registration. URL: https://aic.mhra.gov.uk/era/pdr.nsf/name?openpage&start=7601&count=200 [accessed 2021-06-13]
- Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009 Jul 21;339:b2535 [FREE Full text] [CrossRef] [Medline]
- Apple. App Store Review Guidelines Internet. URL: https://developer.apple.com/app-store/review/guidelines/ [accessed 2021-01-12]
- Google. Play Console Help. URL: https://support.google.com/googleplay/android-developer/?hl=en#topic=7071529 [accessed 2021-01-12]
- Medicines and Healthcare products Regulatory Agency. Guidance: Medical Device Stand-Alone Software Including Apps (Including IVDMDs) v1.05. 2018. URL: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/957090/Software_flow_chart_Ed_1-07b-UKCA__002__FINAL.pdf [accessed 2021-06-13]
- Ferreras Vega R. El uso de aplicaciones móviles en medicina. Rev Electron AnestesiaR 2018;10:1-3 [FREE Full text]
- Maresova P, Hajek L, Krejcar O, Storek M, Kuca K. New regulations on medical devices in Europe: are they an opportunity for growth? Adm Sci 2020;10:1-18. [CrossRef]
- European Commission. Internal Market, Industry, Entrepreneurship and SMEs. URL: https://ec.europa.eu/growth/smes/business-friendly-environment/sme-definition_en [accessed 2021-06-13]
- CAMD Europe. Competent Authorities for Medical Devices. Implementation of the new EU Medical Devices Regulations. URL: https://www.camd-europe.eu/mdr-ivdr-implementation/ [accessed 2021-06-13]
- Overheid Wettenbank. Besluit Medische Hulpmiddelen. URL: http://wetten.overheid.nl/BWBR0007307/geldigheidsdatum_23-03-2013 [accessed 2021-06-13]
- Behan R, Watson M, Pandit A. New EU medical device regulations: impact on the MedTech sector. Med Writ 2017;26:20-24 [FREE Full text]
- European Commission. European Database on Medical Devices (EUDAMED). URL: https://ec.europa.eu/growth/sectors/medical-devices/new-regulations/eudamed_en [accessed 2021-06-13]
- Fraser AG, Butchart EG, Szymánski P, Caiani EG, Crosby S, Kearney P, et al. The need for transparency of clinical evidence for medical devices in Europe. Lancet 2018;392:521-530.
- European Commission. Unique Device Identification System under the EU Medical Device Regulations 2017/745 and 2017/746. URL: https://ec.europa.eu/health/sites/default/files/md_topics-interest/docs/md_faq_udi_en.pdf [accessed 2021-06-13]
- Arts Portaal. Het overzicht van apps voor zorgprofessionals. URL: https://www.artsportaal.nl/apps/ [accessed 2021-06-13]
- Kinderformularium. Kinderformularium.nl. URL: https://www.kinderformularium.nl/ [accessed 2021-06-13]
- van der Zanden TM, Goedknegt L, De Hoog M, Mooij MG, de Wildt SN, van der Sijs IH. Development and implementation of a paediatric dosing calculator integrated in the Dutch Paediatric Formulary. Drugs Ther Perpect 2020:253-262.
|EEA: European Economic Area|
|EU: European Union|
|EUDAMED: European Database on Medical Devices|
|MDD: Medical Devices Directive|
|MDR: Medical Device Regulation|
|MHRA: Medicines and Healthcare products Regulatory Agency|
|PDC: pediatric drug calculator|
|PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses|
|CE: Conformité Europénne|
Edited by R Kukafka; submitted 22.07.21; peer-reviewed by B Nievas Soriano, C Jacob; comments to author 09.10.21; revised version received 01.11.21; accepted 02.11.21; published 13.12.21Copyright
©Charlotte Koldeweij, Jonathan Clarke, Joppe Nijman, Calandra Feather, Saskia N de Wildt, Nicholas Appelbaum. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 13.12.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.