Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 08.11.19 in Vol 21, No 11 (2019): November

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/16607, first published Oct 09, 2019.

This paper is in the following e-collection/theme issue:

    Viewpoint

    Unlocking the Power of Artificial Intelligence and Big Data in Medicine

    1Division of Medical Information Sciences, University Hospitals of Geneva, Geneva, Switzerland

    2Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland

    Corresponding Author:

    Christian Lovis, MD, MPH, FACMI

    Division of Medical Information Sciences

    University Hospitals of Geneva

    Gabrielle Perret Gentil 4

    Geneva, 1205

    Switzerland

    Phone: 41 22 37 26201

    Email: Christian.Lovis@hcuge.ch


    ABSTRACT

    Data-driven science and its corollaries in machine learning and the wider field of artificial intelligence have the potential to drive important changes in medicine. However, medicine is not a science like any other: It is deeply and tightly bound with a large and wide network of legal, ethical, regulatory, economical, and societal dependencies. As a consequence, the scientific and technological progresses in handling information and its further processing and cross-linking for decision support and predictive systems must be accompanied by parallel changes in the global environment, with numerous stakeholders, including citizen and society. What can be seen at the first glance as a barrier and a mechanism slowing down the progression of data science must, however, be considered an important asset. Only global adoption can transform the potential of big data and artificial intelligence into an effective breakthroughs in handling health and medicine. This requires science and society, scientists and citizens, to progress together.

    J Med Internet Res 2019;21(11):e16607

    doi:10.2196/16607

    KEYWORDS



    Introduction

    Most of the daily news and recently published scientific papers on research, innovations, and applications in artificial intelligence (AI) refer to what is known as machine learning—algorithms using massive amounts of data and various methodologies to find patterns, support decisions, make predictions, or, for the deep learning part, self-identify important features in data. However, AI is a complex concept to grasp, and most people have little understanding of what it really is. AI was founded as an academic discipline in 1956 and, despite its youth, already has a rich history [1,2]. In more than 60 years of exploration and progress, AI has become a large field of research and development involving multidisciplinary approaches to address many challenges, from theoretical frameworks, methods, and tools to real implementations, risk analysis, and impact measures. The definition of AI is a moving target and changes over time with the evolution of the field. Since its early days, the field of AI has allowed the development of many techniques supporting decision support and prediction, as it is usually made by humans. As early as 1958, a perceptron was expected to be able “to walk, talk, see, write, reproduce itself and be conscious of its existence,“ which led a large scientific controversy between neural network and symbolic reasoning approaches [3]. The landscape of AI research includes knowledge representation and engineering; rule-based and symbolic reasoning; temporal reasoning and planning; sensing and perception; learning; evolutionary, emerging, social behaviors; and the ability to move and manipulate objects, to name the most important [4]—deep machine learning with autonomous features extraction. It is the point of view taken in the paper, acknowledging however that, more recently, there is a trend to restrict AI to the latter, autonomous deep machine learning. As a consequence of the wide landscape, the field draws at large through philosophy, mathematics, information sciences, computer science, psychology, anthropology, social sciences, linguistics, and many others. For some experts and visionary people such as Ray Kurzweil, deep machine learning will allow building of an artificial general intelligence that is able to develop itself autonomously and to have the capacity to understand or learn any intellectual task that a human being can, and even go far beyond the limits of human intelligence [5], but most experts would agree that there are some big missing pieces and it is still a long way off, despite recent potential important advances in quantic computing [6]. A recent white paper published by the European Commission and authored by the members of the High-Level Expert Group on AI provides, in a few pages, a good overview on what AI is, its main capabilities, applicable expectations, and disciplines involved [7].

    Taking into the field of AI at large, it is important to emphasize that AI is already broadly used today in medicine. Decision support based on knowledge engineering and rule-based systems are implemented widely in computerized provider order entry (CPOE) worldwide. Advanced signal processing is implemented in pacemakers or defibrillators to take decisions, in cochlear-implants with man-machine interfaces, in electrocardiograms to provide signal analysis and automated diagnosis, etc.

    The AI field in itself is aspirational and is expected to contribute significantly to medicine, from research to citizen-centered health. Machine learning and deep learning has led most recent major breakthroughs in AI, such as sound (speech and music) recognition and image (face, radiology, pathology, dermatology, etc) recognition, and in gaming. Recently, image recognition has almost reached a level of maturity through which it can be used and developed by nonexperts in AI [8,9]. However, the hype around AI in these last few years has built high expectations and similarly high fears. There are still very few systems based on autonomous deep learning that have emerged widely in the commercial market.

    The world of AI could roughly be summarized in three sequential and superposed acts:

    Act 1: Humans teach machines to handle data and information.

    Act 2: Humans teach expertise to machines.

    Act 3: Humans teach machines to learn alone.


    Challenges

    There are many challenges that need to be addressed in the field of AI, when it comes to medicine. Most of them are not exclusive to medicine and health, but their addition makes the goals significantly much harder to reach.

    Bayesian Trap

    Medicine and health determinants, in general, are characterized by their usually fundamental Bayesian property. In the Bayesian probability approach, a prior probability is required to evaluate the strength of the prediction.

    Most of what is used in medicine, notably but not exclusively, to establish diagnosis, falls in the Bayesian approach. For example, if a person has a fever, what is the probability of it being the flu? If a person has a high measurement of blood sugar, what is the probability of it being diabetes? To illustrate the Bayesian trap, let us take a simple example—a pregnancy test. This is a simple test; it can be positive or negative. Let us imagine that we use a classical test, which has 99% sensitivity and 95% specificity. If 100 tests are performed for 100 persons and 5 turn out to be positive, the question is, know how many of them turn out to be pregnant women. This question is defined as determining the positive predictive value of a positive test, and it gives the probability of a positive test to be really signing the presence of the factor the test is testing. To answer this question, we need to know the prior probability of being pregnant in the tested population. To understand this, imagine that 100% men were tested; in this case, none of the 5 positive tests would correspond to a pregnant woman. Similarly, if all persons tested are pregnant woman, then all 5 positive tests would correspond to pregnant women. If the prior probability is around 1%, then applying the Bayesian rules returns that the probability to be pregnant when a test is reported positive is about 17%. This means that about 4 of 5 tests are false positive. At the other end, if the prior probability is around 20% (ie, a woman with several factors suggesting a potential pregnancy), the probability of a positive test to be a true positive is above 80%. Thus, less than one test out of five is a false-positive. The example shows the major consequences of the prior probability in Bayesian situations.

    These are the consequences of AI. The models must take into account prior probability in the population they are used. This should be better understood even when reporting results in the literature, often limited to specificity and sensitivity. Another consequence only becomes visible when several focused and near-to-perfect systems are used together in complex cases. For example, having many systems, each with its own false-positive rate can end up with consolidated systems that have the sum of all false-positives. This has been shown well with decision support system in CPOE, with a very high rate of false-positive alerts, especially with patients receiving complex drug therapies [10,11].

    Regulatory Labyrinth

    Most diagnostic or therapeutic means used nowadays in medicine have to go through complex regulatory frameworks to get market approval. The regulatory agencies mostly base their decisions on safety, evidence, and added value. In addition, medicoeconomic assessments are often used by health agencies according to various dimensions such as quality-adjusted life year and burden of the disease, by using indicators such as disability-adjusted life-years [12-14]. These decisions thus have economical and legal consequences, including accountability. The role of regulatory agencies is discussed, especially around topics that are getting into the market, such as in image recognition [15]. For example, a call for inputs for “Artificial Intelligence and Machine Learning in Software as a Medical Device“ has been launched by the Federal Drug Administration [16]. This is an important aspect, as regulatory agency support is an important asset in building trust for most care professionals to use medical tools and for companies to invest in robust products ready for the market. However, this requires us to define a clear regulatory framework, appropriate evaluation processes, and benchmark tools without blocking innovation [17].

    Education and Practice Gap

    Medicine is a science with numerous tools and devices, from stethoscopes to scalpels, microscope to scanners, scores, guidelines, etc. Most of these tools and devices require education and sometimes very specific certification processes for care professionals that use them, not to speak about a good experience. This should be also the case for software, algorithms, and other decision support systems. However, this is not the case. Education to use software and understand systems as important as the computerized patient records is often minimal. When it is about big data and AI, education on the topic is worse, usually inexistent. There are only very few medical schools that teach the use of AI to future health professionals. AI should become mandatory teaching in all medical schools in the world as a priority. Experts have been raising the question since 20 years, but it has received real focus only recently [18-22]. In 5-10 years, when current young students will be starting their clinical activities, machine learning based on data science will have become embedded in many activities, devices, and software and its use, misuse, and overuse and consequences on patients and accountability will depend on how users will master it [23].

    Data Quality Chiasm

    Data quality is a recurring topic of discussion when it comes to big data and analytics. One of the characteristics of the big data era is that data are often used for a purpose that differs from the one that motivated data acquisition. This is a notable difference with traditional hypothetic-deductive scientific approaches in medicine, where a hypothesis leads to a methodology design, which itself will lead to specific data acquisition. In the big data era, the primary goal of data-producing processes is often completely independent from possible use of the data. It is interesting to emphasize that long-term clinical cohorts and long-term biobanks face similar challenges. Designing long-term cohorts and building metadata framework and standard operating procedures for biobanking are important challenges, as they have to project usages that will be made years after the initial design.

    These questions have led to a consequent literature addressing the question of data quality and secondary usage of clinical data. However, most of this work tries to describe dimensions able to assess the “intrinsic” or “absolute” quality of data [24-28]. Another approach could be to adopt a “fit-for-purpose” approach, which considers only the quantitative and descriptive properties of data, allowing further processing. The “qualitative” properties of any dataset can only be assessed in conjunction with a specific secondary usage. This means that the same dataset will be appropriate to answer some scientific questions and not appropriate to answer others. The data are not “good” or “bad” by themselves; they are “good” or “bad” when used in a specific context: the “fit-for-purpose” assessment. This is one of the major objectives of the FAIR data initiative, which aims at insuring “a posteriori” data usability (see below).

    An unexpected consequence of the “data quality chiasm” is its influence on modifying acquisition processes, especially in clinical contexts. One often hears sentences such as “the quality of clinical data is not good enough for research.” As a result, there is a constant pressure to move toward more structured data acquisition processes. For example, the RECIST (Response Evaluation Criteria In Solid Tumors) guidelines are meant to standardize the radiologic evaluation criteria in solid tumors oncological trial treatments. This has been successfully developed for trials. Use of RECIST requires good experience to avoid interobserver variability, which can be as high as 20% [29-31]. This assessment has been adapted to reflect changes in radiological response, for example, in immunotherapies where the size of tumors can increase despite good therapeutic response [32]. Unfortunately, there is growing pressure to extend the use of RECIST and other similarly structured staging guidelines beyond clinical trials for all radiological staging to improve the capacity to use standard clinical care for therapeutic assessment. As a consequence, this leads to a high time pressure on operational activities of radiology departments and an increasing number of inexperienced people using these types of staging. With the progression of natural interfaces such as voice recognition and natural language processing and their increased daily use in a growing number devices, I would argue in favor of avoiding artificial structuring many data acquisition processes and keep the data in their most natural form, exploiting more natural interactions such as voice and text and developing strong natural language processing tools that can be applied to produce structured information in a postprocessing step. This will allow reprocessing of all narratives whenever needed by new structured resources required.


    Quest for Truth

    Many aspects of the landscape of artificial intelligence require a good idea of what is true. Knowledge engineering builds the graph of the “known” or the “relevant” such as it is made in SNOMED CT (Systematized Nomenclature of Medicine - Clinical Terms) or the Open Biological and Biomedical Ontology Foundry [33,34]. The same applies with rule-based techniques or symbolic reasoning, which need to be able to express rules, that is, truth in a formalized way, but also in supervised machine learning approaches, which require having training sets that express truth, at least a probabilistic truth. There are a lot of expectations in these approaches, especially when combing them [35,36], but all of them, except unsupervised deep machine learning, require some sources of truth, which leads to the fundamental question of finding the sources of truth in life sciences and the level of evidence supporting that truth. At the first glance, it seems to be a trivial question. However, the “truth” is often “lost in text” because for most of it, the sources rely on complex narratives that contextualize the messages they convey. In addition, the “truth” is very diluted. For example, with more than 2500 papers indexed daily in Medline/PubMed [37], it is nearly impossible for an expert to catch everything published in its own research field. Finally, and by nature, science is evolving, and thus, scientific “truth” of what was true once may no more be true today. For example, it was clear until recently that there are two types of lymphocytes—the B and T cells. However, a recent paper from Rizwan et al [38] describes a new type of lymphocyte, bearing characteristics of both B and T cells, which may play a role in driving autoimmunity in some diseases such as diabetes [38]. Sources of truth and their characterizations, such as the level of evidence or their context of use, are increasingly important. This should be available to all, similar to Cochrane [39], covering all area of life sciences; maintained; and in machine-readable form.


    Building Trust

    In science, trust is strongly related to building evidence. Trust is important in not only the scientific community, but also at large, to build adoption, political support, and public acceptance. In summer 2019, a survey published by the Pew Research Center showed a positive trend among the public: science acts for the good, but with concerns about integrity, transparency, and bias. Overall, 86% of Americans say they have at least “a fair amount” of confidence in scientists [40]. One of the challenges is that scientific reliability has often been confused with trustworthiness [41]. Scientific evidence can be very strong, such as for immunization or Web-based health information, but the trust can be much lower [42,43]. There are many dimensions that have been discussed in building trust in science, but they can be summarized in three concepts, one for the scientists and the organizations, one for the objects of the research, and one for the processes. Integrity is first and most important and covers scientific integrity, funding, conflict of interests, etc. Transparency must be present for the motivation, outcomes, and process. Finally, methodologies applied to handle the processes must be strong and robust. Building evidence requires many dimensions to be taken into account, such as bias, generalizability, reproducibility, and explainability. Some challenges are more difficult in big data and AI. Proper control of data acquisition and flow is usually more difficult than that in traditional controlled studies. The consequence is that the data have specific properties, which are not always well managed, such as selection biases. Sometimes, the assumptions constraining the use of analytical tools are not well understood, such as homoscedasticity for many statistical tests. In addition, deep machine learning is facing the challenges of precise reproducibility and explainability. The latter is currently the object of numerous works, trying to understand intermediate representation of data in neural networks that can predict and explain their behavior. Explainability and interpretability are often used interchangeably. Interpretability is the extent to which it is possible to predict how the system will behave, given a change in input or algorithmic parameters. On the other hand, explainability is the extent to which the internal mechanics of the deep learning system can be understood and thus explained. Molnar [44] published a very good overview of the problem in an open book available on GitHub. However, explainability might not be the best road to raise global trust in deep machine learning approaches, especially when the explanations themselves are hard to explain. Some other dimensions such as transparency, reproducibility, or uncertainty qualifications might be more effective [45]. For example, in Science in 2018, Hutson [46] reported a survey of 400 artificial intelligence papers presented at major conferences, with only 6% including code for the algorithms and 30% test data, thus considerably limiting reproducibility possibilities [46].


    FAIR Data Hope

    The FAIR Guiding Principles are guidelines to make data discoverable and processable by both humans and machines. They were first published by Wilkinson et al [47]. The FAIR Guiding Principles are based on a set of criteria listed in Textbox 1:


    Textbox 1. FAIR data criteria.
    View this box

    Several frameworks have been defined to assess and evaluate the compliance to FAIR criteria, such as the FAIR maturity tools [48,49]. As such, FAIR data do not imply that data are in the Open Data space [50]. Access can be restricted, such as in the Harvard Dataverse, and this is an important point to emphasize. There might be a lot of restriction to have data or metadata available in the Open Data space, because of national regulation, privacy protection, intellectual property, etc. FAIR data do not make data available; they make data usable under the condition that it is authorized.

    The FAIR initiative is crucial. It illustrates the movement of data from objects to assets initiated this last decade and described in the essay of Sabina Leonelli [51] recently published in Nature.

    The initiative promotes the use of rich metadata framework, compliant to standards and formal descriptions. It promotes the use of free and open resources for descriptive information and protocols. It allows us to build a framework of shareable resources that can be used for processing, without actually sharing the data in an open space. FAIR allows building of a framework that is inclusive with all data sources, including those that are subject to authorization, with clear and open protocols.


    Privacy – New Deal

    In the era of big data, privacy requires special attention. Usual paradigms of limiting access to deidentified information are becoming less effective to protect privacy. Increasing heterogeneous data sources and richness of data about each of us, associated with data linkage techniques, strongly increases the possibility of reidentification, including anonymized data [52-57]. The challenge and potential impacts are even bigger for genetic information [58-60]. There is no good technical solution that can harmonize the challenge of preserving privacy and answering the increasing need of data-driven science for accessing large genomic et phenotypic datasets, and there are many ongoing ethical and legal discussions [61-66]. Interestingly, this is not restricted to science, and the same applies to patients’ needs for health information [67]. There is a need for better global education about implications and risks of privacy, citizen, policy makers, students, research community, and all stakeholders. A recent scoping review [68] has shown that the understanding of anonymization and de-deidentification is heterogeneous in the scientific community [68]. Discrimination is one of the major risks in privacy breaches, and disclosing privacy information can have many consequences [69-71], including in reimbursement and insurance coverage [72,73]. It is important to find the right path between naïve positivism and irrational paranoia. An important step forward is to improve awareness and education of all stakeholders about privacy, technical limitations to protect it, and building regulatory barriers to avoid discrimination.


    Conclusions

    AI and big data in medicine are only in their childhood stages; they grow up fast. Whether they grow up well is still an open question that the future will answer. However, they will not grow up well without actively helping them do so. There are several important initiatives that contribute to this, such as the Global Alliance for Genomics and Health (GA4GH), an organization setting a policy and technical framework for respecting human rights to enable responsible genomic data sharing [74], or the European Union General Data Protection Regulation (GDPR) [75] that sets a completely novel privacy regulation for the European Union. Such initiatives are converging toward building a landscape that enables science while building trust in improving protection of individual rights. I invite the readers to visit the JMIR Open Access collections available on the Web on the following topics: “Big Data,” “Decision Support for Health Professionals,” and “Artificial Intelligence” [76-78].

    Conflicts of Interest

    None declared.

    References

    1. Bringsjord S, Govindarajulu N. Stanford Encyclopedia of Philosophy Archive. Stanford, CA: Metaphysics Research Lab, Stanford University; 2018. Artificial Intelligence   URL: https://plato.stanford.edu/archives/fall2018/entries/artificial-intelligence/ [accessed 2019-09-22]
    2. Wikipedia. 2019. Artificial intelligence   URL: https://en.wikipedia.org/w/index.php?title=Artificial_intelligence&oldid=916837449 [accessed 2019-09-22]
    3. Olazaran M. A Sociological Study of the Official History of the Perceptrons Controversy. Soc Stud Sci 2016 Jun 29;26(3):611-659. [CrossRef]
    4. Luger G. Artificial intelligence: structures and strategies for complex problem solving. Boston, MA: Pearson Addison-Wesley; 2009:978.
    5. Kurzweil R. The singularity is near: when humans transcend biology. New York: Viking; 2005:978.
    6. Giles M. MIT Technology Review. Google researchers have reportedly achieved "quantum supremacy"   URL: https:/​/www.​technologyreview.com/​f/​614416/​google-researchers-have-reportedly-achieved-quantum-supremacy/​ [accessed 2019-09-22]
    7. High-Level EGOAI(. European Commission. A definition of Artificial Intelligence: main capabilities and scientific disciplines   URL: https:/​/ec.​europa.eu/​digital-single-market/​en/​news/​definition-artificial-intelligence-main-capabilities-and-scientific-disciplines [accessed 2019-09-22]
    8. Godec P, Pancur M, Ilenic N, Copar A, Stražar M, Erjavec A, et al. Democratized image analytics by visual programming through integration of deep models and small-scale machine learning. Nat Commun 2019 Oct 7;10(1):7. [CrossRef]
    9. Faes L, Wagner SK, Fu DJ, Liu X, Korot E, Ledsam JR, et al. Automated deep learning design for medical image classification by health-care professionals with no coding experience: a feasibility study. The Lancet Digital Health 2019 Sep;1(5):e232-e242. [CrossRef]
    10. Carli D, Fahrni G, Bonnabry P, Lovis C. Quality of Decision Support in Computerized Provider Order Entry: Systematic Literature Review. JMIR Med Inform 2018 Jan 24;6(1):e3. [CrossRef]
    11. Jung M, Hoerbst A, Hackl WO, Kirrane F, Borbolla D, Jaspers MW, et al. Attitude of Physicians Towards Automatic Alerting in Computerized Physician Order Entry Systems. Methods Inf Med 2018 Jan 20;52(02):99-108. [CrossRef]
    12. Neumann PJ, Sanders GD. Cost-Effectiveness Analysis 2.0. N Engl J Med 2017 Jan 19;376(3):203-205. [CrossRef] [Medline]
    13. Sanders GD, Neumann PJ, Basu A, Brock DW, Feeny D, Krahn M, et al. Recommendations for Conduct, Methodological Practices, and Reporting of Cost-effectiveness Analyses. JAMA 2016 Sep 13;316(10):1093. [CrossRef]
    14. Mokdad AH, Ballestros K, Echko M, Glenn S, Olsen HE, Mullany E, et al. The State of US Health, 1990-2016. JAMA 2018 Apr 10;319(14):1444. [CrossRef]
    15. Allen B. The Role of the FDA in Ensuring the Safety and Efficacy of Artificial Intelligence Software and Devices. J Am Coll Radiol 2019 Feb;16(2):208-210. [CrossRef] [Medline]
    16. U.S. Food and Drug Administration. 2019 Jun 26. Artificial Intelligence and Machine Learning in Software as a Medical Device   URL: http:/​/www.​fda.gov/​medical-devices/​software-medical-device-samd/​artificial-intelligence-and-machine-learning-software-medical-device [accessed 2019-09-22]
    17. Parikh RB, Obermeyer Z, Navathe AS. Regulation of predictive analytics in medicine. Science 2019 Feb 22;363(6429):810-812 [FREE Full text] [CrossRef] [Medline]
    18. Chan KS, Zary N. Applications and Challenges of Implementing Artificial Intelligence in Medical Education: Integrative Review. JMIR Med Educ 2019 Jun 15;5(1):e13930 [FREE Full text] [CrossRef] [Medline]
    19. Masters K. Artificial intelligence in medical education. Med Teach 2019 Sep;41(9):976-980. [CrossRef] [Medline]
    20. Wartman SA, Combs CD. Reimagining Medical Education in the Age of AI. AMA J Ethics 2019 Feb 01;21(2):E146-E152 [FREE Full text] [CrossRef] [Medline]
    21. Data Science Institute. Training Medical Students and Residents for the AI Future   URL: https://www.acrdsi.org/Blog/Medical-schools-must-prepare-trainees [accessed 2019-09-22]
    22. Lillehaug SI, Lajoie SP. AI in medical education--another grand challenge for medical informatics. Artif Intell Med 1998 Mar;12(3):197-225. [Medline]
    23. Kolachalama VB, Garg PS. Machine learning and medical education. NPJ Digit Med 2018 Sep 27;1(1):54 [FREE Full text] [CrossRef] [Medline]
    24. Stausberg J, Bauer U, Nasseh D, Pritzkuleit R, Schmidt C, Schrader T, et al. Indicators of data quality: review and requirements from the perspective of networked medical research. GMS Med Inform Biom Epidemiol 2019;15(1):05.
    25. Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc 2013 Jan 01;20(1):144-151 [FREE Full text] [CrossRef] [Medline]
    26. Weiner MG. Toward Reuse of Clinical Data for Research and Quality Improvement: The End of the Beginning? Ann Intern Med 2009 Sep 01;151(5):359. [CrossRef]
    27. Sandhu E, Weinstein S, McKethan A, Jain SH. Secondary Uses of Electronic Health Record Data: Benefits and Barriers. The Joint Commission Journal on Quality and Patient Safety 2012 Jan;38(1):34-40. [CrossRef]
    28. Hersh W. Adding value to the electronic health record through secondary use of data for quality assurance, research, and surveillance. Am J Manag Care 2007 Jun;13(6 Part 1):277-278 [FREE Full text] [Medline]
    29. Jiang B, Zhou D, Sun Y, Wang J. Systematic analysis of measurement variability in lung cancer with multidetector computed tomography. Ann Thorac Med 2017;12(2):95-100 [FREE Full text] [CrossRef] [Medline]
    30. Bellomi M, De Piano F, Ancona E, Lodigiani AF, Curigliano G, Raimondi S, et al. Evaluation of inter-observer variability according to RECIST 1.1 and its influence on response classification in CT measurement of liver metastases. Eur J Radiol 2017 Oct;95:96-101. [CrossRef] [Medline]
    31. Yoon SH, Kim KW, Goo JM, Kim D, Hahn S. Observer variability in RECIST-based tumour burden measurements: a meta-analysis. Eur J Cancer 2016 Jan;53:5-15. [CrossRef] [Medline]
    32. Seymour L, Bogaerts J, Perrone A, Ford R, Schwartz LH, Mandrekar S, RECIST working group. iRECIST: guidelines for response criteria for use in trials testing immunotherapeutics. Lancet Oncol 2017 Mar;18(3):e143-e152 [FREE Full text] [CrossRef] [Medline]
    33. The OBO Foundry.   URL: http://www.obofoundry.org/ [accessed 2019-10-05]
    34. SNOMED International.   URL: https://www.snomed.org/snomed-international/who-we-are [accessed 2018-06-27]
    35. Skevofilakas M, Zarkogianni K, Karamanos B, Nikita K. A hybrid Decision Support System for the risk assessment of retinopathy development as a long term complication of Type 1 Diabetes Mellitus. Conf Proc IEEE Eng Med Biol Soc 2010;2010:6713-6716. [CrossRef] [Medline]
    36. Hoehndorf R, Queralt-Rosinach N. Data Science and symbolic AI: Synergies, challenges and opportunities. DS 2017 Oct 17:1-12. [CrossRef]
    37. NIH: U.S. National Library of Medicine. Detailed Indexing Statistics: 1965-2017   URL: https://www.nlm.nih.gov/bsd/index_stats_comp.html [accessed 2017-10-08]
    38. Ahmed R, Omidian Z, Giwa A, Cornwell B, Majety N, Bell DR, et al. A Public BCR Present in a Unique Dual-Receptor-Expressing Lymphocyte from Type 1 Diabetes Patients Encodes a Potent T Cell Autoantigen. Cell 2019 May;177(6):1583-1599.e16. [CrossRef]
    39. Cochrane.   URL: https://www.cochrane.org/ [accessed 2019-10-05]
    40. Pew Research Center: Science & Society. Trust and Mistrust in Americans’ Views of Scientific Experts   URL: https:/​/www.​pewresearch.org/​science/​2019/​08/​02/​trust-and-mistrust-in-americans-views-of-scientific-experts/​ [accessed 2019-10-05]
    41. Kerasidou A. Trust me, I'm a researcher!: The role of trust in biomedical research. Med Health Care Philos 2017 Mar;20(1):43-50 [FREE Full text] [CrossRef] [Medline]
    42. Larson HJ, Clarke RM, Jarrett C, Eckersberger E, Levine Z, Schulz WS, et al. Measuring trust in vaccination: A systematic review. Hum Vaccin Immunother 2018 Jul 03;14(7):1599-1609 [FREE Full text] [CrossRef] [Medline]
    43. Sbaffi L, Rowley J. Trust and Credibility in Web-Based Health Information: A Review and Agenda for Future Research. J Med Internet Res 2017 Jun 19;19(6):e218 [FREE Full text] [CrossRef] [Medline]
    44. Interpretable Machine Learning. 2019.   URL: https://christophm.github.io/interpretable-ml-book/ [accessed 2019-10-18]
    45. Begoli E, Bhattacharya T, Kusnezov D. The need for uncertainty quantification in machine-assisted medical decision making. Nat Mach Intell 2019 Jan 7;1(1):20-23. [CrossRef]
    46. Hutson M. Artificial intelligence faces reproducibility crisis. Science 2018 Feb 16;359(6377):725-726. [CrossRef] [Medline]
    47. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 2016 Mar 15;3(1):160018 [FREE Full text] [CrossRef] [Medline]
    48. GitHub. FAIR Metrics 2019   URL: https://github.com/FAIRMetrics/Metrics [accessed 2019-10-05]
    49. Union POTE. Publications Office of the European Union. Turning FAIR into reality: final report and action plan from the European Commission expert group on FAIR data   URL: https:/​/publications.​europa.eu/​en/​publication-detail/​-/​publication/​7769a148-f1f6-11e8-9982-01aa75ed71a1 [accessed 2019-07-04]
    50. Harvard University. 2019. Towards an integrated and FAIR Research Data Management @Harvard Internet   URL: https:/​/scholar.​harvard.edu/​mercecrosas/​presentations/​towards-integrated-and-fair-research-data-management-harvard [accessed 2019-04-10]
    51. Leonelli S. Data — from objects to assets. Nature 2019 Oct 15;574(7778):317-320. [CrossRef]
    52. Malin B. Re-identification of familial database records. AMIA Annu Symp Proc 2006:524-528 [FREE Full text] [Medline]
    53. El Emam K, Jonker E, Arbuckle L, Malin B. A systematic review of re-identification attacks on health data. PLoS One 2011 Dec 2;6(12):e28071 [FREE Full text] [CrossRef] [Medline]
    54. Dankar FK, El Emam K, Neisa A, Roffey T. Estimating the re-identification risk of clinical data sets. BMC Med Inform Decis Mak 2012 Jul 09;12(1):66 [FREE Full text] [CrossRef] [Medline]
    55. Benitez K, Malin B. Evaluating re-identification risks with respect to the HIPAA privacy rule. J Am Med Inform Assoc 2010;17(2):169-177 [FREE Full text] [CrossRef] [Medline]
    56. Barth-Jones D. The 'Re-identification' of Governor William Weld's Medical Information. Soc Sci Res Netw 2012:19.
    57. Emam KE, Dankar FK, Vaillancourt R, Roffey T, Lysyk M. Evaluating the Risk of Re-identification of Patients from Hospital Prescription Records. Can J Hosp Pharm 2009 Jul;62(4):307-319 [FREE Full text] [Medline]
    58. Kulynych J, Greely HT. Clinical genomics, big data, and electronic medical records: reconciling patient rights with research when privacy and science collide. J Law Biosci 2017 Apr;4(1):94-132 [FREE Full text] [CrossRef] [Medline]
    59. Mascalzoni D, Paradiso A, Hansson M. Rare disease research: Breaking the privacy barrier. Appl Transl Genom 2014 Jun 01;3(2):23-29 [FREE Full text] [CrossRef] [Medline]
    60. Zaaijer S, Gordon A, Speyer D, Piccone R, Groen S, Erlich Y. Rapid re-identification of human samples using portable DNA sequencing. eLife Internet 2019:6. [CrossRef]
    61. Ienca M, Ferretti A, Hurst S, Puhan M, Lovis C, Vayena E. Considerations for ethics review of big data health research: A scoping review. PLoS ONE 2018 Oct 11;13(10):e0204937. [CrossRef]
    62. Loukides G, Denny JC, Malin B. The disclosure of diagnosis codes can breach research participants' privacy. J Am Med Inform Assoc 2010;17(3):322-327 [FREE Full text] [CrossRef] [Medline]
    63. Malin B, Benitez K, Masys D. Never too old for anonymity: a statistical standard for demographic data sharing via the HIPAA Privacy Rule. J Am Med Inform Assoc 2011;18(1):3-10 [FREE Full text] [CrossRef] [Medline]
    64. El Emam K, Moher E. Privacy and anonymity challenges when collecting data for public health purposes. J Law Med Ethics 2013 Mar;41 Suppl 1:37-41. [CrossRef] [Medline]
    65. Thorogood A, Zawati M. International Guidelines for Privacy in Genomic Biobanking (or the Unexpected Virtue of Pluralism). J Law Med Ethics Internet 2015;43:4. [CrossRef]
    66. O'Neill L, Dexter F, Zhang N. The Risks to Patient Privacy from Publishing Data from Clinical Anesthesia Studies. Anesth Analg 2016 Dec;122(6):2017-2027. [CrossRef] [Medline]
    67. Househ M, Grainger R, Petersen C, Bamidis P, Merolli M. Balancing Between Privacy and Patient Needs for Health Information in the Age of Participatory Health and Social Media: A Scoping Review. Yearb Med Inform 2018 Aug;27(1):29-36 [FREE Full text] [CrossRef] [Medline]
    68. Chevrier R, Foufi V, Gaudet-Blavignac C, Robert A, Lovis C. Use and Understanding of Anonymization and De-Identification in the Biomedical Literature: Scoping Review. J Med Internet Res 2019 May 31;21(5):e13484. [CrossRef]
    69. Dixon P. A Failure to "Do No Harm" -- India's Aadhaar biometric ID program and its inability to protect privacy in relation to measures in Europe and the U.S. Health Technol (Berl) 2017;7(4):539-567 [FREE Full text] [CrossRef] [Medline]
    70. Rengers JM, Heyse L, Otten S, Wittek RPM. "It's Not Always Possible to Live Your Life Openly or Honestly in the Same Way" - Workplace Inclusion of Lesbian and Gay Humanitarian Aid Workers in Doctors Without Borders. Front Psychol 2019 Feb 27;10:320 [FREE Full text] [CrossRef] [Medline]
    71. Stangl AL, Lloyd JK, Brady LM, Holland CE, Baral S. A systematic review of interventions to reduce HIV-related stigma and discrimination from 2002 to 2013: how far have we come? J Int AIDS Soc 2013 Nov 13;16(3 Suppl 2):18734 [FREE Full text] [Medline]
    72. Bélisle-Pipon J, Vayena E, Green RC, Cohen IG. Genetic testing, insurance discrimination and medical research: what the United States can learn from peer countries. Nat Med 2019 Aug;25(8):1198-1204. [CrossRef] [Medline]
    73. Bardey D, De Donder P, Mantilla C. How is the trade-off between adverse selection and discrimination risk affected by genetic testing? Theory and experiment. J Health Econ 2019 Sep 30;68:102223. [CrossRef] [Medline]
    74. Global Alliance for Genomics & Health.   URL: https://www.ga4gh.org/ [accessed 2019-10-05]
    75. EU GDPR.ORG.   URL: https://eugdpr.org/ [accessed 2019-10-05]
    76. JMIR Medical Informatics. E-collection 'Big Data'   URL: https://medinform.jmir.org/themes/183
    77. JMIR Medical Informatics. E-collection 'Decision Support for Health Professionals'   URL: https://medinform.jmir.org/themes/186
    78. Journal of Medical Internet Research. E-collection 'Artificial Intelligence (other than Clinical Decision Support)'   URL: https://www.jmir.org/themes/797


    Abbreviations

    AI: artificial intelligence
    CPOE: computerized provider order entry
    GA4GH: Global Alliance for Genomics and Health
    GDPR: General Data Protection Regulation
    RECIST: Response Evaluation Criteria In Solid Tumors
    SNOMED CT: Systematized Nomenclature of Medicine - Clinical Terms


    Edited by G Eysenbach; submitted 09.10.19; peer-reviewed by A Benis, D Gunasekeran; comments to author 13.10.19; revised version received 18.10.19; accepted 20.10.19; published 08.11.19

    ©Christian Lovis. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 08.11.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.