Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 15.12.20 in Vol 22, No 12 (2020): December

This paper is in the following e-collection/theme issue:

    Original Paper

    Status and Recommendations of Technological and Data-Driven Innovations in Cancer Care: Focus Group Study

    1FORTH-ICS, Heraklion, Greece

    2Audi Konfuzius-Institut Ingolstadt Lab, Technische Hochschule Ingolstadt, Ingolstadt, Germany

    3School of Interdisciplinary Informatics, University of Nebraska, Omaha, NE, United States

    4Interdisziplinäres Brustzentrum, Helios Klinikum München West, Munich, Germany

    5Vicomtech, Health Research Institute, San Sebastian, Spain

    6University of Strathclyde, Glasgow, United Kingdom

    7eHealth Lab, Institute of Applied Biosciences - Centre for Research & Technology Hellas, Thessaloniki, Greece

    8School of Computing, Ulster University, Newtownabbey, United Kingdom

    9Salumedia Labs & Adhera Health, Inc, Sevilla, Spain

    10Academic Unit of Radiology, Department of Infection, Immunity and Cardiovascular Disease, University of Sheffield, Sheffield, United Kingdom

    11Universidad de Sevilla, Seville, Spain

    12School of Computing and Engineering, University of West London, London, United Kingdom

    13Department of Computer Science, Aberystwyth University, Aberystwyth, United Kingdom

    *all authors contributed equally

    Corresponding Author:

    Haridimos Kondylakis, BSc, MSc, PhD

    FORTH-ICS

    N Plastira 100

    Vassilika Vouton

    Heraklion, 70013

    Greece

    Phone: 30 2810391449

    Email: kondylak@gmail.com


    ABSTRACT

    Background: The status of the data-driven management of cancer care as well as the challenges, opportunities, and recommendations aimed at accelerating the rate of progress in this field are topics of great interest. Two international workshops, one conducted in June 2019 in Cordoba, Spain, and one in October 2019 in Athens, Greece, were organized by four Horizon 2020 (H2020) European Union (EU)–funded projects: BOUNCE, CATCH ITN, DESIREE, and MyPal. The issues covered included patient engagement, knowledge and data-driven decision support systems, patient journey, rehabilitation, personalized diagnosis, trust, assessment of guidelines, and interoperability of information and communication technology (ICT) platforms. A series of recommendations was provided as the complex landscape of data-driven technical innovation in cancer care was portrayed.

    Objective: This study aims to provide information on the current state of the art of technology and data-driven innovations for the management of cancer care through the work of four EU H2020–funded projects.

    Methods: Two international workshops on ICT in the management of cancer care were held, and several topics were identified through discussion among the participants. A focus group was formulated after the second workshop, in which the status of technological and data-driven cancer management as well as the challenges, opportunities, and recommendations in this area were collected and analyzed.

    Results: Technical and data-driven innovations provide promising tools for the management of cancer care. However, several challenges must be successfully addressed, such as patient engagement, interoperability of ICT-based systems, knowledge management, and trust. This paper analyzes these challenges, which can be opportunities for further research and practical implementation and can provide practical recommendations for future work.

    Conclusions: Technology and data-driven innovations are becoming an integral part of cancer care management. In this process, specific challenges need to be addressed, such as increasing trust and engaging the whole stakeholder ecosystem, to fully benefit from these innovations.

    J Med Internet Res 2020;22(12):e22034

    doi:10.2196/22034

    KEYWORDS



    Introduction

    Background

    The morbidity and mortality associated with cancer are rapidly increasing globally because of population growth and aging, reflecting the changes in the prevalence and distribution of major risk factors of cancer [1]. These global trends have resulted in more people living with or beyond cancer. As such, there is a greater need to improve and optimize cancer care services throughout diagnosis, treatment, rehabilitation, and end-of-life care. Modern technologies, often enabled by the availability of big data and advanced analytics, have demonstrated the potential to enhance the current level of quality of cancer care, for example by improving information access, informing and sharing clinical decision making with patients, and facilitating communication and support for coliving with the illness.

    Pioneering research is now being conducted in the field of cancer care technology, resulting in the development of novel solutions to a diverse spectrum of problems in this area. However, the process of evaluating these innovations and their operation within a real-world context is at a less advanced stage. The clinical assessment of technology poses many challenges and is influenced by numerous variables. As a result, there is an emphasis on the early evaluation of technology by including key stakeholders throughout the design and development phases (sometimes referred to as a cocreation process).

    Objectives

    Evidently, as researchers involved in the creation of novel technologies for cancer care, we must consider and share both the innovative concepts being developed as well as how they are being assessed and accepted in real-world or clinical settings.

    To this end, an international workshop was convened to consider the current status of technological and data-driven innovations in cancer care, to identify key challenges and opportunities, and to formulate recommendations aimed at accelerating the rate of progress in the data-driven management of cancer. The two workshop instances led to a series of publications. This paper discusses key topics arising from the workshops and subsequent discussions among their participants.


    Methods

    An international expert consensus-building workshop named Tech4Cancer [2] was held in two instances; the first was held in June 2019 in Cordoba, Spain, along with the computer-based medical systems [3] international conference, and the second was held in October 2019 in Athens, Greece, along with the Institute of Electrical and Electronics Engineers (IEEE) Bioinformatics and Bioengineering [4] international conference. The workshops attracted significant international participation; more than 40 participants attended these events, where they focused on discussing global trends for technological and data-driven innovations in cancer care.

    The workshop was supported by four European Union (EU) projects, namely BOUNCE, CATCH ITN, DESIREE, and MyPal, funded by Horizon 2020 (H2020). The BOUNCE project [5,6] considers clinical, cancer-related biological, lifestyle, and psychosocial parameters to predict individual resilience trajectories throughout the cancer continuum. Eventually, the target is to increase resilience in breast cancer survivors and help them remain in the workforce and have a better quality of life. DESIREE [7], on the other hand, aims to provide a web-based software ecosystem for personalized, collaborative, and multidisciplinary management of primary breast cancer by specialized breast units, from diagnosis to therapy and follow-up.

    MyPal [8] aims to foster palliative care for people with cancer by leveraging patient-reported outcome (PROM) systems through their adaptation to the personal needs of the person with cancer and his or her caregivers [9]. In this regard, MyPal designed two novel eHealth interventions based on electronic PROMs to offer advanced palliative care services to adult patients with cancer and children with cancer. MyPal interventions are delivered via a sophisticated eHealth platform developed by the project. The interventions and the platform itself will be assessed through two multicenter clinical studies (one for the case of adults and one for the case of children), which will take place throughout Europe. CATCH ITN [10] is a PhD training network, with academic and industry partners across Ireland, Spain, and Denmark. Research within CATCH ITN focuses on the development and use of technology to improve the quality of life of individuals living with and beyond cancer.

    The primary deliverable from this workshop was a set of articles that summarize key issues already published by IEEE. During the paper presentation, several topics were identified through discussion among the participants. After the second instance of the workshop, participants were invited to participate in a focus group discussing status, challenges, opportunities, and recommendations in the area of technological and data-driven cancer management; the outcome is reported in this study.

    The writing group met by teleconference, and participants were asked to propose topics according to their interest and expertise and to select topics that they could actively contribute to. Several topics depicting challenges, opportunities, and recommendations were selected. Subsequently, leaders for each topic were identified and a structure was proposed. A formal consensus process was not used; however, the structured and open discussions did not reveal any fundamental disagreements about the nature of the topics, although the discussions supported the refinement and specificity of topics. Once all contributions were collected, a homogenization and integration process led to a final draft, where all participants commented and discussed, leading to the final submitted version.


    Results

    Patient Engagement and Participatory Design

    A fundamental requirement for the effectiveness of any eHealth intervention, including interventions addressed to patients with cancer and survivors, is a certain level of engagement by the patient [11], who is the main beneficiary of the intervention [12]. The lack of engagement of patients with eHealth interventions is linked to low motivation and adherence to the intervention, leading to high dropout rates and eventually affecting treatment outcomes and effectiveness [13]. The issue of patient engagement has cast a shadow over the enthusiasm for the potential of eHealth [14].

    Formulating a concrete definition of patient engagement in the context of eHealth is a challenge by itself [14], as there is a lack of consensus on what the term engagement entails and because different fields (eg, computer science, psychology, and behavioral health) conceptualize it differently. A working definition comes from the Canadian Institutes for Health Research in the framework of the Strategy for Patient-Oriented Research: patient engagement is the meaningful and active collaboration of the patient in governance, priority setting, research, and knowledge translation [15]. Some researchers have defined engagement in terms of the level of activation of the patient in the management of their own health [16]. Certain definitions rely on psychological processes related to user perceptions and experience, whereas others consider engagement as a purely behavioral construct, synonymous with intervention usage [17]. Consequently, engagement is often used interchangeably with adherence, which refers to whether the intervention is used as intended by its developers [18,19]. This is the definition of patient engagement that is primarily considered in the context of this study.

    The situation is not significantly better when it comes to the assessment of patient engagement. The lack of consensus in the conceptualization of user engagement makes the design of appropriate universal or even widely applicable assessment instruments difficult. Instead, most research efforts propose solutions for assessing user engagement for very specific purposes, modalities, and contexts pertaining to eHealth [11,20,21]. Focusing on mobile health (mHealth), a prominent subfield of eHealth, a recent scoping review analyzed 41 studies and developed a library of 7 analytic indicators to evaluate effective engagement with consumer mHealth apps for chronic conditions, namely the (1) number of measures recorded, (2) frequency of interactions logged, (3) number of features accessed, (4) number of log-ins or sessions logged, (5) number of modules or lessons started or completed, (6) time spent engaging with the app, and (7) number or content of pages accessed [22]; this work is of particular relevance to care for cancer, which currently tends to be considered a chronic disease [23].

    The scientific literature sheds some light on the techniques that have been employed to achieve, maintain, and improve patient engagement. According to a psychology study [24], patient engagement techniques are categorized as follows:

    1. Behavioral techniques, such as motivational interviewing, goal setting, and planning, which are related to patient actions when managing their health condition.
    2. Cognitive techniques, such as question-asking tasks and psychoeducation sessions, which are related to patient thoughts and received information concerning their health condition.
    3. Emotional techniques, such as positive psychology exercises and expressive writing tasks, which are related to experienced patient feelings and emotions when adjusting to their new health condition.

    Most interventions developed for older adults employ patient engagement techniques from behavioral and/or cognitive categories, but not all categories [24]. The latter is also the conclusion of a systematic review of eHealth for patient engagement [25]. Moving beyond the psychology-oriented categorization, other patient engagement techniques, such as shared decision making and brief negotiated interviewing, have also been employed; in fact, interventions have been designed to integrate both techniques for engagement optimization [26]. Personalization has also been adopted in some studies as a patient engagement technique [16,27].

    However, the main barrier in building a critical mass of literature on patient engagement with eHealth systems is the fact that very few studies address or report the topic. For instance, a systematic review on published trials discovered that only 23 of 2777 reviewed trials reported any patient engagement activities [28]. The literature review that was conducted in the framework of a different study focusing on older patients concluded that interventions to engage patients are generally scant and often poorly described [24]. In addition, in a recent systematic and mapping review on eHealth interventions employing electronic patient-reported outcomes (ePROs) for palliative cancer care conducted in the framework of the MyPal project [29], 17 of 24 eligible ePRO-based palliative care interventions for patients with cancer did not take patient engagement into account in the development or evaluation of the proposed digital health intervention. This may have been one of the reasons for the high attrition rates reported by the studies. Furthermore, four other studies recognized—mostly retrospectively—the need for patient engagement, and some attempted to measure it. Only 3 remaining studies actively attempted to optimize patient engagement during the design of the eHealth intervention or system.

    Opportunities

    In contrast to the other challenges discussed in this work, the solution to patient engagement may not be rooted in technology itself but rather related to the way technology is designed. To this end, there is a growing body of research demonstrating the value of cocreation and participatory design in the development of novel digital health services, including services for cancer care. Participatory design is one of the pillars of the revolutionary predictive, personalized, preventive, and participatory (P4) cancer medicine [30,31], and it requires the active involvement of users in the design or development phase of an eHealth system. Participatory design can provide a unique perspective on user acceptability, system usability, and the feasibility of the overall effort [32]. By involving a representative sample of a population of patients with cancer in the design or development of an eHealth system or service, one can eventually build an innovative solution that is expected to have good engagement with the target population.

    The main barrier is a lack of research culture for considering the involvement of the end user (ie, in our case, the patient with cancer) in the design process. Despite this barrier, participatory design is expected to become the norm in eHealth technology development in the upcoming years.

    Recommendations

    The MyPal project (see the Methods section) is an example of a research effort that has committed to a participatory design approach, implemented primarily via a series of focus group discussions on eHealth interventions to be developed with the participation of all the involved stakeholders (patients as well as their informal carers and treating health care professionals) [33].

    The use of the participatory approach as early as possible in the design of innovative technological solutions for patients with cancer presents a good opportunity to improve patient engagement [34]. The methodological tools, coming mostly from the field of qualitative research (eg, semistructured focus group discussions), are mature enough to support this design paradigm in eHealth, as these have been extensively validated in more generic software engineering environments. The employed participatory design process has driven the development of the MyPal platform for palliative cancer care and has provided a series of generalizable guidelines or recommendations for the successful application of participatory design processes for patients with cancer engagement, which are as follows:

    1. Participatory design should start as early as possible in the development lifecycle of an eHealth system or service, and it should rely on established methodological tools.
    2. Representative samples of the intended patient populations need to be selected for participation in the co-design and cocreation activities. This is especially important for heterogeneous patient populations.
    3. Participatory design findings should be fused with other sources of knowledge (eg, a screening of unmet patient needs from cancer care in the scientific literature).

    These recommendations can complement pre-existing published efforts to deliver more generic guidelines for developing engaging eHealth technologies. For example, the work presented in Karekla et al [14] specified 10 recommendations for researchers and clinicians interested in developing an engaging eHealth system. These recommendations are organized around 4 dimensions, namely a priori theoretical planning, human-computer interaction, tailoring and targeting to user groups, and active assessment of use. The eHealth technology development framework presented in Gemert-Pijnen et al [35] can also serve as a source of akin recommendations. Although the main goal of the framework is to improve eHealth technology uptake, some of the 6 working principles it introduces are applicable to the pursuit of patient engagement, especially the principle advocating for persuasive design techniques.

    Small Data Analytics

    Data are fuel for any machine learning (ML) project [36]. It is well known that deep supervised learning algorithms are particularly data hungry—not only do they need a lot of data samples but the data also have to be manually annotated beforehand. However, big data sets with annotations (labeling, structuring, etc) are very rare, as proper annotations must be done by experts, and this is very expensive. Therefore, annotated data sets are mostly small data. Annotating data in a less costly manner remains to be a key challenge. A related challenge is to reduce the dependence of ML on annotation.

    ML, especially deep learning, can effectively learn with big data. However, it cannot effectively learn with small data because of various issues, for example, overfitting, noise, outliers, and sampling bias, which can render the learned model effectively useless. Effective learning with small data is a challenge.

    Opportunities

    The annotation problem can be addressed via the use of annotation tools or services in many ways, such as (1) providing annotation tools so that annotation can be performed more effectively and easily, existing tools include Lionbridge artificial intelligence (AI) [37] and Computer Vision Annotation Tool [38]; (2) outsourcing the annotation task to an annotation service provider such as Amazon Mechanical Turk [39]; and (3) enabling an expert to teach their ML model while building and annotating their data set. However, none of these approaches solve the annotation problem scientifically. There are ways to deal with learning with small data problems, such as data augmentation, transfer learning, regularization, and visualization. However, these methods require skilled people, and their effectiveness is limited.

    Another research question is regarding the procedure to add the value of ML results under the constraint of the available knowledge. Moreover, as knowledge and the latest clinical evidence, such as clinical practice guidelines (CPGs), are usually in paper-based formats and written in natural language, another research question arises regarding the procedure to automatically represent knowledge in a structured and computerized manner.

    Recommendations

    To solve the annotation problem scientifically, a desirable approach is to design a new ML algorithm that requires minimal feedback from human experts. This will only be possible if domain-specific constraints can be imposed on the learning process. This will reduce the model space as well as the variance in learning. Thus, one research question is on the procedure to reduce model space by domain-specific constraints.

    An interesting approach to solve the learning with small data problem is to use domain knowledge in the learning process or knowledge-based learning. ML requires data as well as knowledge (common sense and domain-specific knowledge) implicitly or explicitly. When there are a lot of data, ML requires a small amount of knowledge; when there is not much data, ML requires a large amount of knowledge to reduce the model search space. Model-based ML can be seen as an example of knowledge-based learning, where knowledge can be specified by experts in the form of variables and their dependencies. This approach has been successfully demonstrated in various case studies. One weakness, however, is that formalizing knowledge is not a straightforward task, and it requires capturing experience from clinicians through inverse engineering and making clinical statements as explicit as possible for a computerized system. A desirable approach is a knowledge-based learning algorithm that (1) can be easily extended by data-driven findings and (2) uses standardized terminologies to provide interoperability and eases the updating and maintenance of the latest evidence. For example, within the DESIREE project, a digital breast cancer patient was formalized as a knowledge model ontology. The ontology employed standardized terminologies to identify univocally all identified clinical terms and procedures, which included the knowledge reported in standardized guidelines such as the National Comprehensive Cancer Network (NCCN) [40] or the European Society for Medical Oncology [41] guidelines in a computerized format, to be provided as recommendations through a decision support system (DSS) for multidisciplinary tumor boards during the decision-making process. Although formalization of knowledge through a model is almost a mandatory task in the very first steps of technical developments, it is a very time-consuming and costly task.

    Integration and Data Management

    The multitude of successful instantiations of digital interventions has opened exciting new directions for acquiring, delivering, and sharing data, and has already proven the potential to leverage cost-effective, patient-centered cancer care applications [42-44]. Embedding patients into the iterative design process of digital intervention has been shown to enable developers to increase the relevance and effectiveness of the intervention [42]. Such treatments should be created with the acknowledgment of patients’ knowledge, attitudes, beliefs, preferences, and expectations of therapeutic outcome [45,46]. Thus, systems that support clinicians would benefit from a redesign that aligns cancer care more completely with patients’ needs and interests [47,48]. ML, through its pervasive impact [49], has the potential to provide a supportive tool for such a redesign task, given the powerful data understanding, generalization capabilities, and robustness [50].

    Opportunities

    Guided by the current state of the art, we identified a set of opportunities for digital interventions related to patient-centered cancer care. We have considered the most cited systematic reviews in the last 3 years (ie, to extract the innovation opportunities and technological limitations) and older systematic reviews (ie, studies from 2011 to tackle the initial adoption and strategies). These opportunities tackle the inherent challenges that we identified in practice. The first challenge is data collection and integration. Here, the core opportunity relates to exploring and exploiting the aggregation of heterogeneous and distributed data, including personal, professional, and health-related information [51]. Such systems are typically deployed as a platform to integrate retrospective, prospective, and day-to-day care data. However, data interpretation and translation remain to be an open challenge, whereas there is also a constant need for standard data formats such as Digital Imaging and Communications in Medicine and other standards applicable to electronic patient record or electronic health record (EHR) data. To address this, there is a clear opportunity to interpret patient information for the patient themselves by translating the clinical findings into an intuitive representation. This would ensure that the delivery of health information is done via a user-friendly interface.

    When considering clinical data heterogeneity and the specificity of patient data, data projection is another challenge that is present in any technological intervention. We delineate a good opportunity to build individualized clinical recommendations based on the data interpretation projected in actions upon the patient. This initiative will be based on the patient’s risk profile and evidence-based guidelines (see the Extracting Patient Portraits section).

    Finally, with regard to data sparsity, such clinical setups face the challenge of data completeness and augmentation. In contrast to clinical trial experience, data completeness improves with longitudinal care. This approach may be a solution to minimizing missing data of PROMs in research or clinical care settings in support of learning health care systems capable of augmenting the data [52]. To maintain the validity of the intervention, we believe that there is a good opportunity to consider and consolidate clinical data with the comprehension of lived experiences of patients, centered on patients [53].

    Recommendations

    On the basis of the identified challenges and opportunities, we established a set of recommendations for the systematic development of appropriate patient-centered digital interventions that ensure usefulness, adoption, and sustainability in cancer care [54,55]. This process can be extended with learning and generalization capabilities. ML algorithms excel at such tasks and constitute invaluable tools for any digital intervention.

    Next, we map the challenges that we identified before practical recommendations. These recommendations can serve as a reference design for technological interventions when looking at integration and data management.

    When addressing data collection and integration, we recommend embedding continuous user feedback and iterative prototyping in the intervention. This can be achieved by exploiting the multimodal nature of patient data (ie, personal, behavioral, professional, and health-related). ML techniques (ie, deep learning and hybrid neural networks) can fuse heterogeneous data in a common representation (ie, efficiently using very large data sets containing health care use data, clinical data, and data from personal devices and many other sources), as demonstrated by the recent deep learning systems used on multi-omics data sets to drive precision oncology care [56].

    In terms of data interpretation and translation, we recommend the use of tools to extract and represent the medical substrate by synthesizing only relevant aspects in a declarative way. ML techniques (ie, deep learning—recurrent networks with word embeddings and distributed representations) can handle very large and sparse data (eg, device data may only be available for a small subset of individuals) to capture the sequential character of the data and are suitable for modeling context dependencies in inputs [57]. Such systems, which incorporate word embeddings encoding syntactic and polarity information in the language followed by deep neural network architectures, are already used to extract and normalize parameters within oncology care data.

    To address the challenges in the area, we recommend the development of clinical projections (ie, mappings) from individualized patient recommendations to therapy plans that embed temporal, procedural, and reasoning processes. ML techniques, such as temporal hierarchical task networks, can dynamically generate personalized therapy plans for oncology patients [58], following a deliberative hierarchical planning process driven by the procedural knowledge described in oncology protocols [59]. Such instantiations use mappings to attach reasoning and procedural knowledge representation as well as their interpretation in a temporal planning process. The planning process allows us to obtain temporally annotated therapy plans that support decisions of oncologists. Moreover, such an ML technique offers the ability to deal with complex temporal and resource constraints, typical in cancer care.

    Finally, tackling data augmentation, we recommend incorporating the lived experiences of the patients [60]. ML techniques, such as contextualized word embeddings, are suitable for improved text augmentation independent of any task-specific knowledge or rules and can process structured questionnaires for patients who, for instance, developed chemotherapy-induced peripheral neuropathy [61].

    Extracting Patient Portraits

    Individual biomedical and nonbiomedical patient characteristics should guide any provided chronic care—digital or not. These insights are used to develop and validate patient portraits that can be employed in practice to determine optimal treatment strategies for subgroups of patients with similar cancer care needs and preferences. Building a patient portrait is hence an endeavor that follows a bottom-up approach that includes patient profiling based on patient phenotype algorithms [62-64]; intelligent patient profiling for the decision support of cancer treatment by exploiting clinical and genomic data [65]; personalized predictive modeling and risk factor identification using patient similarity [66]; and individualization beyond biomedical factors to also include demographic, socioeconomic, and psychological aspects [67]. To build a complete description of the multiple dimensions describing each patient, we identified a series of opportunities that describe all previously reviewed work.

    Opportunities

    The series of opportunities we identified target the selection of relevant heterogeneous and multimodal data correlated with the diagnosis. To extract the opportunities and frame our recommendations, we systematically analyzed a series of studies ranging from genomic data and phenotypes to demographics and psychological data. This methodology allowed us to capture the most relevant dimensions for extracting a patient profile. More precisely, the focus was on exploiting the correlations among multiple data sources to build a digital patient portrait consistent along all dimensions.

    Such an initiative requires powerful data mining and ML algorithms, which can provide an efficient and compressed representation of a patient's digital profile, subsequently guiding therapeutic schemes.

    One challenge identified relates to data relevance. Here, there is an opportunity to identify relevant genetic, phenotypical, physiological, lifestyle, and medical data correlated with the diagnosis. Exploiting such an opportunity can improve a patient's profile and the overall effect of the intervention, especially in progression-free survival [68]. Such studies demonstrate the feasibility of intelligent patient profiling that can select, within a clinically relevant time frame, a beneficial treatment for patients with no other treatment options.

    Given the data deluge describing each patient, exploring and exploiting data correlations is another challenge we identified in the context of extracting a patient’s profile. This challenge provides a clear opportunity to exploit the correlations among the multiple identified modalities toward building an individual or personalized patient digital portrait that consistently captures all dimensions of the patient’s disease evolution.

    Such an exploration unveils another challenge, namely multimodal data fusion [69], and the opportunity and potential that fusing extracted knowledge have toward a personalized therapeutic scheme [70]. Such systems, by combining data describing complementary perspectives on the same biological phenomena, can (1) separate correlated from discordant data, (2) extract the most informative features, and (3) estimate disease progression.

    The last challenge we identified as being crucial in any effort to extract a patient’s profile is the possibility of embedding individualized data (eg, age, gender, ethnicity, health conditions, and social position) in patient cohorts [71]. Initial efforts were made to include family history for risk assessment and early detection of cancer; however, adherence to the study was low because of the limited technological support. This challenge brings along the opportunity to determine the population of interest and use this information in the process of portrayal.

    Recommendations

    On the basis of the identified opportunities and related challenges, we established a set of recommendations supporting a digital intervention design that (1) exploits available data, (2) extracts underlying correlations, and (3) integrates the multitude of representations in a structured object guiding therapeutic interventions in cancer care.

    When addressing the challenge of identifying data relevance, we recommend selecting the data or feature subset that best characterizes the statistical property of a certain variable (ie, a certain patient) subject to the constraint that these data or features are as mutually dissimilar to each other as possible but as marginally similar to a certain class of patients as possible. For this task, ML tools such as minimum redundancy feature selection (ie, Minimum Redundancy Maximum Relevance) [72]) can be used to accurately identify the characteristics of patient features or data and narrow down their relevance. Such techniques provide an integrative approach to patient-centered data and demonstrate the potential of feature selection in data analysis and predictive patient-specific outcomes [73].

    To address data correlations, we recommend the use of ML models, such as long short-term memory (LSTM) [74] networks, which can be used for their ability to effectively model varying length time series data and capture long-term dependencies and correlations. From modeling the patient life expectancy from medical records [75] to complete patient trajectories [76], modeling the disease trajectory and care processes, assumes mining electronic medical records that are episodic and irregular in time. Such models capture long-term temporal dependencies and are well suited to modeling clinical data because the evidence of certain conditions may be spread apart over several hours or days, and important symptoms may present early on in a patient’s trajectory.

    The next challenge identified is the fusion of the available multimodal data. Our recommendation is to use a data-driven feature learning class of approaches. Typically, they are based on deep networks that can directly learn the hidden characteristics of the data from different sources. As such, we recommend, for instance, the use of deep neural networks to extract features from genomic and clinical data [77], convolutional neural networks to extract features from pathology images [78], and recurrent neural networks for text and medical records data [79].

    Finally, the last identified challenge is the possibility of embedding individualization data in the patient profile. We recommend performing individual cognitive interviews and focus groups with patients to learn about their relevant needs, experiences, fears, aspirations, and expectations.

    From the ML point of view, a solution for developing personalized patient embeddings that is capable of processing such data is a combination of well-proven autoencoder methods with extensions to some of the metrics to account for data sparsity and multimodality [80]. Such studies also provide a methodology describing how these networks can be designed, built, and applied to tasks of integrative analyses of heterogeneous cancer data.

    A patient portrait that can capture complex relationships in physiological signals, nonbiomedical data, and personal data embedding is key to accurately predict the stages of interventions for different patients and is necessary for successful personalized therapy.

    Learning Patient Disease Trajectory for Personalized Diagnosis

    Cancer is remarkably heterogeneous across individuals. This heterogeneity makes treatment difficult for caregivers because they cannot accurately predict how the disease will progress to guide treatment decisions. Therefore, tools that help to predict the individualized trajectory of cancer can help improve the quality of health care [81,82]. Given the assessment of the current state of the patient (ie, patient profile, physiology, neuroimaging, blood biomarkers, and physiologic testing) along with the therapeutic scheme, a digital intervention would use ML or predictive systems to infer disease evolution or remission to be able to guide subsequent therapy scheme planning [83]. Such a trajectory can also support the detection of behavior change in patients [84].

    Opportunities

    A significant need relates to making disease predictions by leveraging baseline information and additional time-dependent clinical markers as they are collected. Such an approach is the focal challenge of personalized medicine: integrative analysis of heterogeneous data from an individual’s medical history to improve cancer care. We identified several key challenges and associated opportunities linked to this.

    The first challenge relates to the fact that markers in clinical data are irregularly and sparsely sampled. Here, we identified a valuable opportunity for handling data and choosing specific latent variable models to summarize and extract information from the irregularly sampled and sparse data. This should simultaneously ensure sidestepping the issue of jointly modeling the data-generating processes [85]. Such systems should build a temporal representation of care trajectories in the form of a time-ordered state sequence. Moreover, in addition to the routine identification of key dates and events in patients’ care trajectories, such systems should identify initially fragmented data across numerous sources.

    Another challenge identified relates to the learning of a disease trajectory and is linked to the inherent computational complexity. Imposed by the clinical setup, we identified an opportunity to predict the entire disease progression trajectory from the observed patient records without many training labels on the ground-truth stages that a patient acquired, in mechanistic models of disease progression. A joint approach is prone to alleviate the inherent variability in prediction.

    This challenge opens the stage for the next challenging point, namely continuous adaptation and updates in the face of disease progression heterogeneity. In handling such a challenge, there is a clear opportunity for continuous-time adaptation and updates to new observations and new data (markers). This provides novel computational methods for predicting, for instance, disease phenotype from molecular and genetic profiling [86].

    Finally, another challenge we identified refers to the observed versus latent data artifacts. Addressing such a challenge demands tools for capturing latent factors in disease expression and not only observed features as a crucial aspect for patient-tailored cancer therapies. We further elaborate this in the following Recommendations section.

    Recommendations

    In this section, we match the challenges to practical recommendations in designing digital interventions when predicting disease trajectories for patients with cancer.

    To handle irregularly and sparsely sampled markers in clinical data, we recommend, from the ML perspective, the use of discriminative models that condition on marker histories instead of jointly modeling them. Such an approach will not be sensitive to miss-specified dependencies across marker types and inherent irregularities and sparsity. For example, functional data analysis [87] can be employed for sequences of measurements that are assumed to be samples from an underlying continuous function. However, coefficient estimates can have a high variance in time series.

    However, the task of predicting the disease trajectory comes with its inherent computational complexity. To address this challenge, we believe that an ideal candidate would be a machine model that grows linearly in the number of marker types included in the model. This makes such a task applicable to cancer prognosis, where many different markers are recorded over time. Generative models can account for disease trajectory shapes using components at the population, subpopulation, and individual levels, which simultaneously allows for heterogeneity across and within individuals and enables statistical strength to be shared across observations at different resolutions of the data [88]. Moreover, such systems can learn accurate and interpretable structured representations for disease trajectories by adapting their attention weights that determine the dependence of future disease states on past medical history.

    Independent of the prediction model, the challenge is to continuously adapt and update in the face of progression heterogeneity. From the ML point of view, we recommend the use of a model capable of being applied dynamically in continuous time and updated as soon as any new data are available (eg, hidden Markov models). Such approaches can model the transition of disease stages or states, which implies that the progression is continuous, and the transition probability to the future state relies only on the current state and the time span. Instantiation of such causality-based ML was used to infer the underlying somatic staging of tumors from next-generation sequencing data [89].

    Moving away from the modeling decision, the last challenge lies in the observed versus latent data artifacts. We believe that a powerful tool is an ML model that accounts for latent factors and covariates influencing disease expression, as standard regression models rely on observed features alone to explain variability [90]. For example, LSTM models over physiological word inputs from health records significantly improve performance as their representation encodes important information about what is normal for each physiological value or is more robust to sparseness in the physiological data.

    Technological Interventions in Cancer Rehabilitation

    Over the past decades, early diagnosis, new drugs, and more personalized treatment have led to impressive increases in survival rates of patients with cancer. However, the most mitigating side effects of commonly used therapies are a severe problem in oncology, leading to dose reduction, treatment delay, or discontinuation [91,92].

    Opportunities

    With the increasing number of cancer survivors, more attention is being paid to persistent sequelae of tumor treatment and supportive measures [93-95] used as adjuncts to mainstream cancer care to control symptoms and enhance well-being [96-100]. The broad literature overview allowed us to identify a series of challenges and the associated opportunities that digital interventions could offer in supportive care.

    The first challenge we identified is the identification of therapeutic sequelae. This challenge offers the opportunity to develop interventions capable of assessing what deficits (sensory, motor, and/or cognitive) a specific patient has as a consequence of cancer therapy.

    The next challenge arises when the intervention needs to quantify the magnitude of therapy sequelae. Here, we identify a clear opportunity to measure the level of deficit or dysfunction induced by the therapy. This is crucial in (1) designing the follow-up therapy scheme, (2) choosing a rehabilitation strategy, and (3) determining the therapy trajectory and dosage.

    The last challenge we identified is the adaptive parametrization of rehabilitation. This challenge brings a valuable opportunity to take steps toward personalized treatment, namely, to parameterize the rehabilitation scheme according to the specific deficit type and level to drive rehabilitation.

    Recommendations

    To cope with patient sensory, motor, and cognitive deficit variability, it is necessary to perform a precise assessment of the 3 different dimensions. We believe that the 3 main challenges we identified as high-potential opportunities for digital interventions are also good candidates for ML algorithms. This technology can learn underlying correlations in patient data and generalize for robust prediction [101]. The mapping from the challenge to the recommendation is provided in the remainder of this section.

    When tackling the identification of therapy sequelae, we recommend exploiting and mining large sets of structured and unstructured data describing a patient to identify correlations among various data types and how they map to a certain type of dysfunction. We propose the use of semisupervised techniques (eg, transductive support-vector machines [102]) that will only require a limited number of labels to generalize well.

    Moreover, to address the magnitude of the therapy sequelae, we recommend the use of deep learning models, especially convolutional neural networks. This is because such networks are capable of learning relevant feature representation from unstructured data, such as pathological images or medical records. This, subsequently, allows deep learning methods to achieve good results in tasks such as regression, detection, and segmentation, which underlie the magnitude estimation.

    Finally, addressing the challenge of adaptive parametrization of rehabilitation, we recommend using methods that combine learning capabilities for regressing arbitrary nonlinear functions (ie, deep learning—encoding the type of deficit covariance with the magnitude) and adaptation through guided searches in parameter spaces (ie, reinforcement learning—finding the best parameters of the rehabilitation scheme—dosage, type, and length that best fit the regressed function).

    Addressing Current Interoperability Challenges

    Patient data needed for the provision of the best treatment are often scattered across different systems rather than stored in a single location. Technologies that facilitate care coordination through interoperability are improving, but a seamless flow of information from one care setting to another still requires more progress. Without having a full picture, it is difficult to provide the best care in an era where cancer is considered to be a chronic disease, and there is an increased demand for consistent follow-up in terms of monitoring and early management of symptoms that indicate that cancer might have returned.

    Interoperability is a primary consideration to achieve communication among applications, medical devices, and health care providers [103], although the growing demand for secondary use of clinical and administrative data increases the pressure to solve relevant challenges [104].

    The currently popular use of EHRs has alleviated some of the barriers in using data from medical records for research, although fully interoperable electronic medical record systems are not yet a reality. Several efforts to develop and apply standards in the collection, extraction, and integration of data by standardization bodies, governments, the research community, and industry are in progress [105] with the aim to establish and adopt clinically relevant, integrated standards covering the entire oncology sector. Organizations such as Health Level Seven International [106] and Personal Connected Health Alliance Continua [107] help to deliver standards-based, open specifications that can support the flow of data from the point of capture into EHRs in the same format and coded content. Cancer-specific apps that exploit platforms and interoperability standards, such as SMART [108] and Fast Healthcare Interoperability Resources [109], are emerging [110-112].

    Standards, such as those developed in the United States and Canada, to guide EHR vendors and public health central cancer registries in the implementation of standardized electronic reporting [113-115] can be used with third-party terminologies.

    It is a fact that some data resolution may be lost during the process of mapping EHR fields to a formally described abstraction layer, which may be alleviated by the use of knowledge models such as ontologies as a knowledge background mapper; however, these common interfaces support queries across EHRs or the extraction of patient data in the same format to allow the merging of patient sets between numerous institutions.

    Opportunities

    Health information technology (IT) brings clinical data and patient information together and guides oncologists in making evidence-based care decisions that lead to improved outcomes. The potential benefits of interoperable interconnected tools and health systems are particularly important for oncology [116], as providing cancer-wide care depends on access to accurate and complete information as well as extensive coordination among patients, caregivers, and diverse provider groups through treatment and survivorship. Connecting the EHR has the potential to support diagnosis assistance for complex patients [117].

    Data must be able to provide a complete look at the patient’s medical history so that physicians can see what cancer medicines and treatments did or did not work. Clinicians also need to be able to avoid recommending the same procedure twice, prescribing a medicine the patient already tried, or missing results from a diagnostic test.

    The need for consolidation and standardization efforts to create interoperable solutions [105] and the need for the cancer informatics community for national initiatives for data standardization and large-scale multidisciplinary research collaborations are timely and critical. In addition to supporting cancer care, cancer-related standards will help improve surveillance and research. Similar to migration from paper to electronic records, the shift toward data interoperability between EHR vendors may require policy changes [118,119].

    Recommendations

    As already proposed by [120], enabling interoperability among institutions and individuals that support care delivery across the cancer continuum is considered essential. Doing so requires developing, testing, disseminating, and adopting technical standards for information related to cancer care across the continuum to optimize the flow of information to serve the needs of caregivers, patients, and providers. To achieve this, standard open application programming interface platforms should be developed and used to facilitate the development of cancer-related apps.

    As mentioned in [121], standards and protocols that aim to enhance the interoperability of different data sets are a highly relevant field for policy action. Incentives for the promotion of standards for the interoperability of clinical data, such as EHR or genome experiment data, should be developed, promoted, and incentivized to allow for the pooling of data, and comparison of system-level research should be provided for the comparison of research outcomes.

    The open use and sharing of big data, without compromising patients’ rights to privacy and confidentiality, should be promoted.

    Patient-Clinician Shared Decision-Making Processes

    Current clinical care is gradually moving toward patient-clinician shared decision making, as patient involvement can provide insights into best health states or outcomes in each case, apart from establishing a partnership that will help clinicians understand their patients’ preferences [122]. In a complex disease, such as cancer, making a clinical decision is a difficult task because of the trade-off between the level of treatment benefits and the impact that adverse events or symptoms could have on the patient’s life. This is especially crucial in cases where the expected medical outcomes are similar for different clinical procedures (ie, also referred to as equipoise), requiring an individualized and personalized health care process involving the clinician and the patient in a close interaction for the best decision making [123]. Nevertheless, actual CPGs still do not include or barely mention the impact of patient preferences along with clinical evidence.

    Opportunities

    It is necessary to overcome many barriers to ensure success in the inclusion of patient preferences during the decision-making process and to track their impact in the care services provided [124]. First, patient preferences should be considered as population knowledge that follows some general trends and not just as individual one-off cases or subjective and variable factors. This would allow including them along with the clinical evidence reported in the studies to identify preference-sensitive decisions (eg, decisions having lifelong implications or an uncertain benefit to the patient, unclear or conflicting evidence, risk of having side effects that negatively affect the patient’s quality of life) of high levels of uncertainty about the best clinical procedure to follow [125].

    Recommendations

    As a complex disease, cancer involves many clinical specialties, with the need of creating a clear taxonomy (ie, systematic categorization) for patients’ preferences that will serve as a standardization over all involved disciplines (eg, analysts, clinical psychologists). This kind of approach would help in the harmonization of the different points of view on the measurement of patients’ preferences by labeling and extracting this information in a processable and understandable way [126,127]. Finally, with regard to the previous points, building a methodology to synthesize the current knowledge on preference trends to be able to describe preference-based decisions along with clinical-based evidence would have a great impact on health care performance, as it has been proven that preferences strongly influence the decision-making process [128,129].

    Assessment of Clinical Evidence-Based Recommendations, Including PROMs

    As presented in the previous sections, the evidence used in medical practice is based on clinical guidelines, which are often used as the evidence basis for the clinical DSS (CDSS) [130]. These guidelines are developed by multidisciplinary groups and are based on a systematic review of the scientific evidence, and their recommendations are explicitly linked to the supporting evidence and graded according to the strength of that evidence [131]. As reported by Harbor et al [131], the levels of evidence of the recommendations are based on study design and the methodological quality of individual studies, which is called the quality of evidence. Therefore, the latest approach also considers the strength of the recommendations, which is a trade-off between benefits and harms of a treatment, considering the quality of evidence [131,132]. Several oncology guidelines, such as the NCCN CPGs in Oncology [133] applied in DESIREE, provide quality of evidence measure, and it is up to the clinician to consider the strength of the recommendation depending on the specific patient situation. However, the subjective measures of patients with cancer, such as fatigue affected by oncological treatments, have not been systematically considered during the decision-making process. Making this information available during the decision-making process also provides other relevant and real-world quality measures to the treatment recommendation to best support clinicians and patients in their treatment decisions.

    Opportunities

    CDSSs provide the opportunity not only to quickly access the latest available evidence but also to incorporate new sources of information that can support the decision-making process. At the same time, the new IT era enables the acquisition of PROMs using questionnaires or even more sophisticated wearables that can measure activity, sleep, or other vital signs that can be translated to patient outcomes. These automatic ubiquitous technologies increase the knowledge required by patients. Processing these data and deducing the desired results will be very helpful. For example, tracking the daily activities of a user provides estimates of how active a patient is, which can be correlated with their depression and/or fatigue levels (variables that are gathered using questionnaires such as the European Platform for Cancer Research—Quality of Life Questionnaire with 30 items). Other techniques could also aid the acquisition of patient outcome information, such as the Ecological Momentary Assessment tests (short questionnaires that are sent to the patient frequently to obtain updates on their status). Thus, the collected data could be used to assess how good the treatment given was for each patient, not in the scope of a randomized control trial but in the real-world environment, without having a specific patient population, but the whole population.

    Recommendations

    In this context, in the DESIREE project focusing on primary breast cancer, new ways of including PROMs to assess guideline recommendations were explored. The aim of including PROMs does not conflict with the quality of evidence and strength of recommendation measurements but provides other quality attributes that contribute to the decision-making process, considering the patient status reported by the patients themselves.

    There are some specifications that should be considered for this quality assessment, which could be named questionnaire based–patient-reported outcome measures, and they include (1) consideration of standard questionnaires for each medical case, such as the International Consortium for Health Outcomes Measurement questionnaires [134], (2) definition of specific protocols to assess the results of the questionnaires, and (3) determination of how these results should be used during the decision-making process.

    Ambiguity on Clinical Guidelines Used for Clinical Decision Support

    When implementing CPGs, several characteristics must be considered to ensure both good health care quality levels and clinicians’ satisfaction. They must assure the validity and reliability of their clinical content, along with their clinical applicability in real clinical settings, and must be clear when defining the procedures to be followed in the current clinical performance procedures within a health care system [135,136]. Nevertheless, several barriers cause the dissemination of the guidelines to be tedious and difficult, mainly because of the ambiguity of the knowledge defined in them. The lack of awareness and familiarity with the recommendations provided in the guideline, a lack of agreement due to different clinical interpretations and simplification of the clinical knowledge reported in the guidelines, and the lack of outcome expectancy are some of the reported barriers that cause a lower adherence of clinicians to guidelines and have an effect on their implementation, compliance, and adherence in real clinical settings [137,138].

    Opportunities

    Actual trends move toward highly interactive computerized systems focusing on intuitively presenting complex clinical cases, where clinicians may access and check computerized clinical data and take away insights from all of this information in a more natural and intuitive way, alleviating the ambiguities of the guidelines through a data-driven approach guided by previous practice [139]. The digital implementation of the CPGs provides evidence-based decision support (ie, computer-interpretable guidelines [CIGs]) [140]. To achieve this, the knowledge available has to be formalized in a manner that is correct and of good quality, by following a consistent and adequate methodological workflow of the clinical processes and objectives, trying to reduce or provide some solutions to the paper-based guidelines ambiguities [141].

    Recommendations

    The proposed directions for realizing these needs include the promotion of standardized clinical terminology that facilitates the understanding and univocal interpretation of the clinical data to be analyzed and the clinical knowledge formalized in the CIGs (eg, the Breast Imaging Reporting and Database System standard for breast anomalies [142]); to be effective, clinical guidelines need to be integrated with the care flow and should provide patient-specific advice when and where needed. Hence, their formalization as CIGs should make it possible to develop CIG-based DSSs, which have a better chance of affecting clinician behavior than narrative guidelines. This will help in providing optimal personalized guideline-based recommendations, avoiding ambiguities, and at a reasonable cost and implementation effort [143].

    Up-to-Date Clinical Evidence Guidelines for CDSS

    Clinical guidelines are tedious to develop, and it is even more difficult to interpret and put them into a computer-interpretable way. This usually requires the close collaboration of knowledge engineers and medical experts. A closely related and important issue is the guideline development process or how CPG development working groups are composed. Usually, these teams comprise quality auditors or managers who are guided by their opinions, interests, and experience and intend to formalize evidence, seeking appropriateness of the provided recommendations but ignore the iterative and causal reasoning of clinicians [144]. Depending on the clinical context and according to the approaches followed for developing, disseminating, and implementing them in practice, CPGs can be more or less successful when reporting the latest clinical evidence [145]. Thus, the rapid advances in clinical practice have made the task of updating the guidelines used for the CDSS more challenging. In particular, in oncology, the number of discoveries is increasing rapidly. As reported by Beatty [146], drug label information and indications do not always keep up with meaningful advances in oncology resources. However, clinicians may be aware of the potential benefits of a particular therapy. Hence, Beatty [146] reports that continuous vigilance (in the form of continuing medical education and literature review) is a survival attribute for medical oncologists. Consequently, there is also a challenge to update the CIGs and have up-to-date guideline-based CDSS.

    Opportunities

    In this context, it is crucial to have tools supporting the easy updating of CIGs used for CDSS by clinicians themselves or knowledge engineers who do not necessarily have to understand the technology and/or the programming language used. Thus, interfaces that are easy to use and easy to understand are required for this purpose. This limitation was highlighted in the DESIREE project; however, this is an issue not only for cancer care but also for other clinical specialties.

    Recommendations

    Therefore, we propose an authoring tool for CPG formalization [147] that should at least fulfill the following requirements: (1) enable the input of guidelines’ information in an easy and visual manner, such as the form of rules or flowcharts, and (2) enable the modification of CIGs previously formalized in the system. In addition, future work should focus on providing a tool for detecting modifications in guidelines as well as semiautomate the formalization of guidelines, using natural language processing.

    Trust and Reliance on Cancer Care

    The change in both hardware and software over the past decade has been remarkable. Equally noteworthy has been the ever-increasing internet speed and the accompanying growth in the demand for connectivity. This growth and development have increased our ability to take on challenging projects to improve early diagnosis and improve the quality of cancer care. However, most discussions relating to health data and associated analytical tools often emphasize data privacy and security at the expense of other topics. These talks often overlook the dynamic nature of both health data and the software used in the analysis of those data. In addition, the popularity of internet-based apps and the use of such tools by patients for self-diagnosis necessitate a call to action. There is a need to examine the reliability and trustworthiness of health-related computational tools used in diagnosis and DSS by health care providers. Here, we will focus on the reliance and reliability of computational tools with an emphasis on cancer care and diagnosis.

    Opportunities

    Reliance and reliability of information in health care are incredibly important. These attributes are particular when it comes to genetic information associated with cancer, which is highly critical in the development of optimal clinical intervention strategies. With an increasing number of people falling victims to medical misinformation and propaganda on the internet [148], advocacy to develop reporting and assessment standards has been long overdue. The internet undoubtedly provides patients with countless sources of health information related to cancer diagnosis and care, some of which represent the gold standard, whereas other sources remain to be of ill repute. The internet promotes decentralization and web democratization of access. Having an open-source or access model certainly helps to promote this agenda. These models provide help and are available to many without the control of smaller powerful broker or agents. However, it challenges the core process of software development, which includes specification, design, development, verification, validation, and management. Although inspection or peer review is the method of choice to check for static processes and software testing a true validation and verification method to check the system, the former fails to check emergent properties such as reliability and performance.

    There is no turning back from this path of dependence on the tools and information available via the internet. People have a reasonable expectation of establishing trust and validation of these tools. Initiatives such as the Secure Socket Layer certification system, implemented for encrypting sensitive information sent across the internet or the use of digital badge to indicate or attest to adhere to an acceptable standard and/or individual skill competency (seal of approval from prominent institutions) or the use of the message-digest algorithm 5 (MD5) hash to ensure data integrity work to build and maintain that trust. Health IT requires a method to assure the credibility of the results generated by various computational tools available on the web. Unfortunately, none of the previously listed solutions are applicable to health software because of the dynamic nature of data as well as the software (as many deviate from the original specifications for which acceptable test and peer review of the result exist).

    Recommendations

    Having reliable information alone is not sufficient for people to construct their foundation of trust. People may have a distrust on the reliability of official information on certain topics because of previous unpleasant experiences. One conclusion that can be drawn from our experience in developing relevant computational tools over the past decade is the questionable credibility of the results obtained from such tools. A review of the literature shows a great deal of volatility in the availability of health IT resources and shows that providing explanations for software errors is an acceptable approach. Building on this notion, a certificate that discloses who is responsible and what tests are done or can be done to validate or test the trustworthiness of the output, something along the line of the MD5 checksum, could be envisioned as an acceptable solution to address this issue. Such a certificate could include the versions of the data and the software in a report to help explain the deviation from the previous version. This could be seen as a reasonable step in the right direction before the availability of peer-approved permanent solutions.

    Trust in Computer-Aided Diagnosis Systems

    An important issue emerging for decision support in medical diagnosis is the trust that clinicians might have in the outputs generated by a computer-aided diagnostic system. This is a highly relevant issue for tissue characterization in general and image-based tumor classification in particular. Hence, it should be considered whether the metrics such as accuracy (ACC) and area under the curve (AUC) correlate well with confidence in the algorithms used in computer-aided design (CAD) systems. If the CAD system provides a recommendation with moderate or low confidence, then a radiologist may deem the recommendation to be useless, even if the classifier being used has a high overall value for ACC and/or AUC. However, if the confidence is much higher, the clinician may deem the recommendation more useful in supporting their decision making. Therefore, building CAD systems in which clinicians have confidence is essential if those CAD systems are to be adopted and are to play a role in fully exploiting the range of digital information available for assisting diagnosis.

    Recent studies discuss the failure of CAD systems in terms of a lack of trust by clinicians in the outputs that the CAD systems generate [149-151]. When designing and evaluating an algorithm within a CAD system, technical developers tend to report metrics such as ACC and AUC to demonstrate the performance of a classification method. However, such metrics do not measure the degree of confidence in individual recommendations made by the CAD system. For example, a CAD system may produce a high AUC value very close to the optimal value of 1, but most of the cases might be classified with low confidence (as measured by the likelihood or probability value associated with the recommended class). Indeed, studies have shown that the use of CAD systems that incorporate inadequate metrics can be detrimental to diagnosis [152].

    Opportunities

    Exploratory studies for breast mass classification [153] and classification of microcalcifications in mammograms [154] using data sets taken from the Curated Breast Imaging Subset of DDSM [155] showed that although most classifiers produce similar overall ACC and AUC values, their performances differ significantly in terms of confidence measure. High ACC or AUC does not provide a full indication of the confidence level of a CAD system. This aspect of the usability of CAD methods is a key but overlooked challenge. Most cancer CAD systems have used ACC and AUC as the main evaluation metrics; however, the probability outputs of the classifier must also be considered and harnessed to measure the degree of certainty of the system in its decision making. Hence, if clinicians are to have confidence in the support provided by a CAD system, the classifier that is chosen to be implemented as part of the system should not necessarily be the one that performs best overall using the standard metrics. CAD support systems must embody reliable confidence measures as one of their key elements [156]. It is therefore essential that the domain of trust in CAD systems should be explored to incorporate trust into the initial classifier design when such algorithms are to be embedded into a cancer CAD system.

    Recommendations

    To fully exploit the range of digital information available for assisting diagnosis, it is important to identify and implement specific actions to increase the trust of physicians in cancer CAD systems and overcome the barriers to adoption of such systems. Most cancer CAD systems have used ACC and AUC as the main performance evaluation metrics; however, confidence measures must also be considered, as the traditionally used metrics are inadequate for informing clinicians in terms of the confidence that they might place on the recommendations provided. Besides, research in the design of classifiers that are incorporated into CAD systems is essential if future CAD systems are to be trusted by clinicians and adopted as a valued and reliable, and indeed routine, element of the cancer diagnosis process.

    Regulatory Roadmap for Validating the Effectiveness of AI–Based Models for Clinical Decision Making

    Data and image analysis algorithms usually require regulatory approval by the Food and Drug Administration (FDA) in the United States and conformity assessment leading to a Conformité Européenne mark in the EU [157], under the new Medical Device Regulation. Both EU and US regulations classify medical devices according to risk classes, and those that fall under certain risk classes are required to undergo clinical trials to be marketed [158]. To date, these guidelines target systems with so called locked algorithms, that is, algorithms whose functions do not change, rather than adaptive ones, that is, whose behavior can change over time based on new data [159]. Adaptive AI–based tools have the potential to autonomously “adapt and optimize device performance in real-time to continuously improve healthcare for patients.” The FDA has already approved several AI- or ML-based software as a medical device (SaMD), but these have only included algorithms that are locked before marketing [160].

    Opportunities

    Regulations that aim to validate AI in a safe and transparent way must consider doing so without compromising the potential of AI. Moreover, characteristics of AI that may be seen as risks such as biases in data, its blackbox nature, and model degradation, among others, have to be included in the validation process to better ensure safety and build public trust [161]. The FDA recently proposed an approach that aimed to “have tailored, pragmatic, and least burdensome regulatory oversight” while validating the continued safety, effectiveness, and performance of SaMDs. The framework recommends (1) establishing clear expectations of quality systems and good ML practices, for example, demonstrating analytical and clinical validation, (2) creating SaMD prespecification sheets and algorithm change protocols to qualify for a premarket review on the safety and effectiveness of the AI-based tool, (3) preparing an approach for AI- or ML-based software modifications (eg, data for retraining), and (4) implementing mechanisms that support transparency and real-world performance monitoring.

    In the EU, there are no regulatory guidelines that specifically cover AI in health care. Nevertheless, draft ethics guidelines have been published by the High-Level Expert Group on AI in April 2019, proposing 7 key requirements that AI systems should meet to be realized as trustworthy AI: human agency and oversight, including fundamental rights; technical robustness and safety; privacy and data governance; transparency, diversity, nondiscrimination, and fairness [162,163]; societal and environmental well-being; and accountability. The guidelines also propose technical and nontechnical methods to implement the 7 key requirements and recommend that these should be continuously evaluated and addressed throughout the AI system’s life cycle.

    Recommendations

    The frameworks enumerated above are, however, still being piloted or discussed. To properly define a regulatory framework for users, stakeholders and use cases (data flows) should be identified and defined. New regulatory frameworks should be built to provide guidance for the validation or qualification of AI tools within different scenarios and pathways (for nonclinical or preclinical use or clinical use), taking into consideration the adaptive nature of AI-based tools. The framework should consolidate input from scientific experts and health authorities and should take into consideration published relevant guidelines, for example, from the High-Level Expert Group on AI, the Medical Device Coordination Group and relevant implementing EU legislation, and international regulations (eg, from the FDA).

    Table 1 summarizes the various topics discussed, the related works for each topic, the opportunities, and the recommendations from the workshops.

    Table 1. Topics, opportunities, and recommendation in cancer care.
    View this table

    Discussion

    This paper presents the key topics that were discussed as part of two international workshops on the current status of technological and data-driven innovations in cancer care. Key challenges and opportunities have been identified, and several recommendations have been made to facilitate the acceleration of progress in the data-driven management of cancer. The workshops presented the work that was being conducted in four Horizon 2020 EU–funded projects, namely BOUNCE, CATCH ITN, DESIREE, and MyPal. These projects provide a rich landscape of the challenges and opportunities of the current state of the art of new technologies in cancer care. The authors have presented these issues and discussed recommendations that can be used for further research as well as practical implementation of such tools in cancer.

    Acknowledgments

    The work presented in this paper is part of the BOUNCE project, which has received funding from the EU’s H2020 Research and Innovation Program under grant agreement number 777167.

    Conflicts of Interest

    None declared.

    References

    1. Lin L, Yan L, Liu Y, Yuan F, Li H, Ni J. Incidence and death in 29 cancer groups in 2017 and trend analysis from 1990 to 2017 from the Global Burden of Disease Study. J Hematol Oncol 2019 Sep 12;12(1):96 [FREE Full text] [CrossRef] [Medline]
    2. Technological and data-driven innovations in cancer care. tech4Cancer.   URL: https://www.tech4cancer.com/ [accessed 2020-11-21]
    3. Technological and Data-Driven Innovations in Cancer Care. CBMS 2019. 2020 Nov 21.   URL: https://www.uco.es/congresos/cbms2019/tech4cancer/ [accessed 2020-11-24]
    4. 2020 Nov 21.   URL: https:/​/bibe2019.​ics.forth.gr/​participation/​technological-and-data-driven-innovations-in-cancer-care/​ [accessed 2020-11-21]
    5. Kondylakis H. Developing a Data Infrastructure for Enabling Breast Cancer Women to BOUNCE Back. 2019 Jun 5 Presented at: IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS); June 5, 2019; Cordoba, Spain. [CrossRef]
    6. Kourou K. Computational Models for Predicting Resilience Levels of Women with Breast Cancer. In: Mediterranean Conference on Medical and Biological Engineering and Computing. 2019 Sep 26 Presented at: XV Mediterranean Conference on Medical and Biological Engineering and Computing (MEDICON); 2019; Coimbra, Portugal. [CrossRef]
    7. Desiree-project.   URL: http://www.desiree-project.eu/ [accessed 2020-11-21]
    8. MyPal.   URL: https://mypal-project.eu/ [accessed 2020-11-21]
    9. Maramis C, Karamanidou C, Schera F. Using Electronic Patient Reported Outcomes to Foster Palliative Cancer Care: the MyPal approach. 2020 Oct 28 Presented at: IEEE International Conference on Bioinformatics and BioEngineering; Oct 28-30, 2019; Athens, Greece. [CrossRef]
    10. CATCH-ITN.   URL: https://www.catchitn.eu/ [accessed 2020-11-21]
    11. Short CE, DeSmet A, Woods C, Williams SL, Maher C, Middelweerd A, et al. Measuring engagement in eHealth and mHealth behavior change interventions: viewpoint of methodologies. J Med Internet Res 2018 Nov 16;20(11):e292 [FREE Full text] [CrossRef] [Medline]
    12. Kondylakis H, Koumakis L, Tsiknakis M, Kiefer S. Personally managed health data: barriers, approaches and a roadmap for the future. J Biomed Inform 2020 Jun;106:103440. [CrossRef] [Medline]
    13. Christensen H, Griffiths KM, Korten A. Web-based cognitive behavior therapy: analysis of site usage and changes in depression and anxiety scores. J Med Internet Res 2002 Feb;4(1):e3 [FREE Full text] [CrossRef] [Medline]
    14. Karekla M, Kasinopoulos O, Neto DD, Ebert DD, Van Daele T, Nordgreen T, et al. Best practices and recommendations for digital interventions to improve engagement and adherence in chronic illness sufferers. Eur Psychol 2019 Jan;24(1):49-67. [CrossRef]
    15. Canada's Strategy for Patient-Oriented Research. Canadian Institute of Health Research. 2011.   URL: https://cihr-irsc.gc.ca/e/44000.html [accessed 2020-11-21]
    16. Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res 2004 Aug;39(4 Pt 1):1005-1026 [FREE Full text] [CrossRef] [Medline]
    17. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med 2016 Dec 13:254-267. [CrossRef] [Medline]
    18. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen J. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012 Nov 14;14(6):e152 [FREE Full text] [CrossRef] [Medline]
    19. Sieverink F, Kelders SM, van GJE. Clarifying the concept of adherence to eHealth technology: systematic review on when usage becomes adherence. J Med Internet Res 2017 Dec 06;19(12):e402 [FREE Full text] [CrossRef] [Medline]
    20. Craig LR, Tada Y, Hilfiker S, Baur C. The assessment of user engagement with eHealth content: the eHealth engagement scale. J Comput Mediat Commun 2010 Jun 01;15(4):681. [CrossRef]
    21. Dewar AR, Bull TP, Malvey DM, Szalma JL. Developing a measure of engagement with telehealth systems: the mHealth Technology Engagement Index. J Telemed Telecare 2017 Feb;23(2):248-255. [CrossRef] [Medline]
    22. Pham Q, Graham G, Carrion C, Morita PP, Seto E, Stinson JN, et al. A library of analytic indicators to evaluate effective engagement with consumer mHealth apps for chronic conditions: scoping review. JMIR Mhealth Uhealth 2019 Jan 18;7(1):e11941 [FREE Full text] [CrossRef] [Medline]
    23. Phillips JL, Currow DC. Cancer as a chronic disease. Collegian 2010;17(2):47-50. [CrossRef] [Medline]
    24. Menichetti J, Graffigna G. 'PHE in Action': development and modeling of an intervention to improve patient engagement among older adults. Front Psychol 2016;7:1405 [FREE Full text] [CrossRef] [Medline]
    25. Barello S, Triberti S, Graffigna G, Libreri C, Serino S, Hibbard J, et al. eHealth for patient engagement: a systematic review. Front Psychol 2015 Jan;6:2013 [FREE Full text] [CrossRef] [Medline]
    26. Lauffenburger JC, Lewey J, Jan S, Nanchanatt G, Makanji S, Ferro CA, et al. Rationale and design of the ENhancing outcomes through goal assessment and generating engagement in diabetes mellitus (ENGAGE-DM) pragmatic trial. Contemp Clin Trials 2017 Aug;59:57-63. [CrossRef] [Medline]
    27. Rise MB, Eriksen L, Grimstad H, Steinsbekk A. The long-term effect on mental health symptoms and patient activation of using patient feedback scales in mental health out-patient treatment. A randomised controlled trial. Patient Educ Couns 2016 Jan;99(1):164-168. [CrossRef]
    28. Fergusson D, Monfaredi Z, Pussegoda K, Garritty C, Lyddiatt A, Shea B, et al. The prevalence of patient engagement in published trials: a systematic review. Res Involv Engagem 2018 May 22;4:17. [CrossRef]
    29. Karamanidou C, Natsiavas P, Koumakis L, Marias K, Schera F, Schäfer M, et al. Electronic patient-reported outcome-based interventions for palliative cancer care: a systematic and mapping review. JCO Clin Cancer Inform 2020 Jul;4:647-656 [FREE Full text] [CrossRef] [Medline]
    30. Hood L, Friend SH. Predictive, personalized, preventive, participatory (P4) cancer medicine. Nat Rev Clin Oncol 2011 Mar;8(3):184-187. [CrossRef] [Medline]
    31. Kondylakis H, Koumakis L, Mazzocco K, Tsiknakis M, Marias K. Participatory aspects of ICT infrastructures for cancer management. In: P5 eHealth: An Agenda for the Health Technologies of the Future. Cham: Springer; Nov 30, 2019:A.
    32. Nieroda ME, Lophatananon A, McMillan B, Chen L, Hughes J, Daniels R, et al. Online decision support tool for personalized cancer symptom checking in the community (REACT): acceptability, feasibility, and usability study. JMIR Cancer 2018 Jul 04;4(2):e10073 [FREE Full text] [CrossRef] [Medline]
    33. Karamanidou C, Maramis C, Stamatopoulos K, Koutkias V. Development of a ePRO-based palliative care intervention for cancer patients: a participatory design approach. Stud Health Technol Inform 2020 Jun 16;270:941-945. [CrossRef] [Medline]
    34. Saunders R, Stuckhardt L. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington (DC): National Academies Press (US); 2013.
    35. van Gemert-Pijnen J, Nijland N, van LM, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011 Dec;13(4):e111 [FREE Full text] [CrossRef] [Medline]
    36. Ehl C. Data — the Fuel for Artificial Intelligence. Medium.   URL: https://medium.com/@cehl/data-the-fuel-for-artificial-intelligence-ed90bf141372 [accessed 2020-11-21]
    37. LionBridge AI.   URL: https://lionbridge.ai/ [accessed 2020-11-21]
    38. openvinotoolkit / cvat. GitHub.   URL: https://github.com/opencv/cvat [accessed 2020-11-21]
    39. Amazon Mechanical Turk.   URL: https://www.mturk.com/ [accessed 2020-11-21]
    40. National Comprehensive Cancer Network.   URL: https://www.nccn.org/professionals/physician_gls/default.aspx [accessed 2020-11-21]
    41. ESMO Guidelines. ESMO.   URL: https://www.esmo.org/guidelines [accessed 2020-11-21]
    42. Dwamena F, Holmes-Rovner M, Gaulden C, Jorgenson S, Sadigh G, Sikorskii A. Interventions for providers to promote a patient-centred approach in clinical consultations. Cochrane Database Syst Rev 2012;12:CD003267. [CrossRef]
    43. Wolpin S, Halpenny B, Whitman G, McReynolds J, Stewart M, Lober W, et al. Development and usability testing of a web-based cancer symptom and quality-of-life support intervention. Health Informatics J 2014 Jan 09;21(1):10-23. [CrossRef]
    44. Klasnja P, Civan HA, Unruh K, Pratt W. Blowing in the wind: unanchored patient information work during cancer care. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2010 Apr Presented at: CHI '10; April 2010; Atlanta p. 193-202. [CrossRef]
    45. Mummah SA, Robinson TN, King AC, Gardner CD, Sutton S. IDEAS (Integrate, Design, Assess, and Share): a framework and toolkit of strategies for the development of more effective digital interventions to change health behavior. J Med Internet Res 2016 Dec 16;18(12):e317 [FREE Full text] [CrossRef] [Medline]
    46. Birckhead B, Khalil C, Liu X, Conovitz S, Rizzo A, Danovitch I, et al. Recommendations for methodology of virtual reality clinical trials in health care by an international working group: iterative study. JMIR Ment Health 2019 Jan 31;6(1):e11973 [FREE Full text] [CrossRef] [Medline]
    47. Bender JL, Yue RY, To MJ, Deacken L, Jadad AR. A lot of action, but not in the right direction: systematic review and content analysis of smartphone applications for the prevention, detection, and management of cancer. J Med Internet Res 2013 Dec 23;15(12):e287 [FREE Full text] [CrossRef] [Medline]
    48. Krist AH, Woolf SH. A vision for patient-centered health information systems. J Am Med Assoc 2011 Jan 19;305(3):300-301 [FREE Full text] [CrossRef] [Medline]
    49. Mulder I, Schikhof Y, Vastenburg M, Card A, Dunn T, Komninos A, et al. Designing with care: the future of pervasive healthcare. IEEE Pervasive Comput 2009 Oct;8(4):85-88. [CrossRef]
    50. Bergeson SC, Dean JD. A systems approach to patient-centered care. J Am Med Assoc 2006 Dec 20;296(23):2848-2851. [CrossRef] [Medline]
    51. Rance B, Canuel V, Countouris H, Laurent-Puig P, Burgun A. Integrating heterogeneous biomedical data for cancer research: the CARPEM infrastructure. Appl Clin Inform 2016;7(2):260-274 [FREE Full text] [CrossRef] [Medline]
    52. Wysham NG, Wolf SP, Samsa G, Abernethy AP, LeBlanc TW. Integration of electronic patient-reported outcomes into routine cancer care: an analysis of factors affecting data completeness. JCO Clin Cancer Inform 2017 Nov;1:1-10 [FREE Full text] [CrossRef] [Medline]
    53. König IR, Malley J, Pajevic S, Weimar C, Diener HC, Ziegler A. Patient-centered yes/no prognosis using learning machines. Int J Data Min Bioinform 2008;2(4):289-341 [FREE Full text] [CrossRef] [Medline]
    54. Spector J, Merrill M, Elen J, Bishop M. Handbook of Research on Educational Communications and Technology: A Project of the Association for Educational Communications and Technology. United States: Springer; 2014.
    55. Ruland CM, White T, Stevens M, Fanciullo G, Khilani SM. Effects of a computerized system to support shared decision making in symptom management of cancer patients: preliminary results. J Am Med Inform Assoc 2003;10(6):573-579 [FREE Full text] [CrossRef] [Medline]
    56. Nicora G, Vitali F, Dagliati A, Geifman N, Bellazzi R. Integrated multi-omics analyses in oncology: a review of machine learning methods and tools. Front Oncol 2020;10:1030 [FREE Full text] [CrossRef] [Medline]
    57. Zhao B. Clinical data extraction and normalization of cyrillic electronic health records via deep-learning natural language processing. JCO Clin Cancer Inform 2019 Sep;3:1-9 [FREE Full text] [CrossRef] [Medline]
    58. González-Ferrer A, ten Teije A, Fdez-Olivares J, Milian K. Automated generation of patient-tailored electronic care pathways by translating computer-interpretable guidelines into hierarchical task networks. Artif Intell Med 2013 Feb;57(2):91-109. [CrossRef]
    59. Fdez-Olivares J, Cózar J, Castillo L. Oncotheraper: Clinical decision support for oncology therapy planning based on temporal hierarchical tasks networks. 2008 Presented at: Workshop on Knowledge Management for Health Care Procedures; 2008; Patras, Greece p. 25-41. [CrossRef]
    60. Williams F, Jeanetta SC. Lived experiences of breast cancer survivors after diagnosis, treatment and beyond: qualitative study. Health Expect 2016 Jun;19(3):631-642. [CrossRef] [Medline]
    61. Tanay MA, Armes J. Lived experiences and support needs of women who developed chemotherapy‐induced peripheral neuropathy following treatment for breast and ovarian cancer. Eur J Cancer Care 2019 Feb 20;28(3):e13011. [CrossRef]
    62. Kirk R, Villanueva MT, Hutchinson L. Patient profiling in breast cancer—is knowledge power? Nat Rev Clin Oncol 2011 Dec 21;9(1):1-1. [CrossRef]
    63. Halpern Y, Horng S, Choi Y, Sontag D. Electronic medical record phenotyping using the anchor and learn framework. J Am Med Inform Assoc 2016 Jul;23(4):731-740 [FREE Full text] [CrossRef] [Medline]
    64. Liao KP, Cai T, Savova GK, Murphy SN, Karlson EW, Ananthakrishnan AN, et al. Development of phenotype algorithms using electronic medical records and incorporating natural language processing. Br Med J 2015 Apr 24;350:h1885 [FREE Full text] [CrossRef] [Medline]
    65. Goletsis Y, Exarchos T, Giannakeas N, Fotiadis D. Intelligent patient profiling for diagnosis, staging and treatment selection in colon cancer. 2008 Presented at: IEEE International Conference on BioInformatics and BioEngineering; October 8-10, 2008; Athens, Greece p. 1-6. [CrossRef]
    66. Ng K, Sun J, Hu J, Wang F. Personalized predictive modeling and risk factor identification using patient similarity. AMIA Jt Summits Transl Sci Proc 2015;2015:132-136 [FREE Full text] [Medline]
    67. Elissen AM, Hertroijs DF, Schaper N, Vrijhoef HJ, Ruwaard D. Profiling patients' healthcare needs to support integrated, person-centered models for long-term disease management (profile): research design. Int J Integr Care 2016 Apr 29;16(2):1 [FREE Full text] [CrossRef] [Medline]
    68. Tuxen IV, Rohrberg KS, Oestrup O, Ahlborn LB, Schmidt AY, Spanggaard I, et al. Copenhagen prospective personalized oncology (CoPPO)-clinical utility of using molecular profiling to select patients to phase I trials. Clin Cancer Res 2019 Feb 15;25(4):1239-1247 [FREE Full text] [CrossRef] [Medline]
    69. Lahat D, Adali T, Jutten C. Multimodal data fusion: an overview of methods, challenges, and prospects. Proc IEEE 2015 Sep;103(9):1449-1477. [CrossRef]
    70. Yuan Y, Savage RS, Markowetz F. Patient-specific data fusion defines prognostic cancer subtypes. PLoS Comput Biol 2011 Oct;7(10):e1002227 [FREE Full text] [CrossRef] [Medline]
    71. Rubinstein WS, Acheson LS, O'Neill SM, Ruffin MT, Wang C, Beaumont JL, Family Healthware Impact Trial (FHITr) Group. Clinical utility of family history for cancer screening and referral in primary care: a report from the Family Healthware Impact Trial. Genet Med 2011 Nov;13(11):956-965 [FREE Full text] [CrossRef] [Medline]
    72. Radovic M, Ghalwash M, Filipovic N, Obradovic Z. Minimum redundancy maximum relevance feature selection approach for temporal gene expression data. BMC Bioinformatics 2017 Jan 3;18(9):1. [CrossRef]
    73. El-Manzalawy Y, Hsieh T, Shivakumar M, Kim D, Honavar V. Min-redundancy and max-relevance multi-view feature selection for predicting ovarian cancer survival using multi-omics data. BMC Med Genomics 2018 Sep 14;11(Suppl 3):71 [FREE Full text] [CrossRef] [Medline]
    74. Zhang C, Biś D, Liu X, He Z. Biomedical word sense disambiguation with bidirectional long short-term memory and attention-based neural networks. BMC Bioinformatics 2019 Dec 02;20(Suppl 16):502 [FREE Full text] [CrossRef] [Medline]
    75. Beeksma M, Verberne S, van den Bosch A, Das E, Hendrickx I, Groenewoud S. Predicting life expectancy with a long short-term memory recurrent neural network using electronic medical records. BMC Med Inform Decis Mak 2019 Feb 28;19(1):36 [FREE Full text] [CrossRef] [Medline]
    76. Pham T, Tran T, Phung D, Venkatesh S. Predicting healthcare trajectories from medical records: a deep learning approach. J Biomed Inform 2017 May;69:218-229 [FREE Full text] [CrossRef] [Medline]
    77. Tranchevent L, Azuaje F, Rajapakse JC. A deep neural network approach to predicting clinical outcomes of neuroblastoma patients. BMC Med Genomics 2019 Dec 20;12(Suppl 8):178 [FREE Full text] [CrossRef] [Medline]
    78. Khosravi P, Kazemi E, Imielinski M, Elemento O, Hajirasouliha I. Deep convolutional neural networks enable discrimination of heterogeneous digital pathology images. EBioMedicine 2018 Jan;27:317-328 [FREE Full text] [CrossRef] [Medline]
    79. Xiao C, Choi E, Sun J. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. J Am Med Inform Assoc 2018 Oct 01;25(10):1419-1428 [FREE Full text] [CrossRef] [Medline]
    80. Simidjievski N, Bodnar C, Tariq I, Scherer P, Andres Terre H, Shams Z, et al. Variational autoencoders for cancer data integration: design principles and computational practice. Front Genet 2019;10:1205 [FREE Full text] [CrossRef] [Medline]
    81. Suresh H, Hunt N, Johnson A, Celi L, Szolovits P, Ghassemi M. Clinical intervention prediction and understanding with deep neural networks. In: Cornell University. 2017 Presented at: Machine Learning for Healthcare; 2017; Boston, Massachusetts, USA p. 322-337   URL: http://proceedings.mlr.press/v68/suresh17a.html
    82. Cheerla A, Gevaert O. Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics 2019 Jul 15;35(14):i446-i454 [FREE Full text] [CrossRef] [Medline]
    83. Wang X, Sontag D, Wang F. Unsupervised learning of disease progression models. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. 2014 Presented at: KDD '14; 2014; New York, USA p. 85-94. [CrossRef]
    84. Gong J, Naumann T, Szolovits P, Guttag J. Predicting clinical outcomes across changing electronic health record systems. 2017 Presented at: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; August 13-17, 2017; Halifax, Nova Scotia, Canada p. 1497-1505   URL: http://library.usc.edu.ph/ACM/KKD%202017/pdfs/p1497.pdf [CrossRef]
    85. Defossez G, Rollet A, Dameron O, Ingrand P. Temporal representation of care trajectories of cancer patients using data from a regional information system: an application in breast cancer. BMC Med Inform Decis Mak 2014 Apr 02;14:24 [FREE Full text] [CrossRef] [Medline]
    86. Mathew JP, Taylor BS, Bader GD, Pyarajan S, Antoniotti M, Chinnaiyan AM, et al. From bytes to bedside: data integration and computational biology for translational cancer research. PLoS Comput Biol 2007 Feb 23;3(2):e12 [FREE Full text] [CrossRef] [Medline]
    87. Ramsay JO. Functional Data Analysis. New York: Springer-Verlag; 2005:4.
    88. Alaa AM, van der Schaar M. Attentive state-space modeling of disease progression. In Advances in Neural Information Processing Systems. 2019.   URL: https://papers.nips.cc/paper/2019/hash/1d0932d7f57ce74d9d9931a2c6db8a06-Abstract.html [accessed 2020-11-24]
    89. Caravagna G, Graudenzi A, Ramazzotti D, Sanz-Pamplona R, De Sano L, Mauri G, et al. Algorithmic methods to infer the evolutionary trajectories in cancer progression. Proc Natl Acad Sci U S A 2016 Jul 12;113(28):E4025-E4034 [FREE Full text] [CrossRef] [Medline]
    90. Chung J, Kastner K, Dinh L, Goel K, Courville A, Bengio Y. A recurrent latent variable model for sequential data. In Advances in neural information processing systems. 2015.   URL: https://papers.nips.cc/paper/2015/hash/b618c3210e934362ac261db280128c22-Abstract.html [accessed 2020-11-24]
    91. Pereira S, Fontes F, Sonin T, Dias T, Fragoso M, Castro-Lopes JM, et al. Chemotherapy-induced peripheral neuropathy after neoadjuvant or adjuvant treatment of breast cancer: a prospective cohort study. Support Care Cancer 2016 Apr;24(4):1571-1581. [CrossRef] [Medline]
    92. Marshall TF, Zipp GP, Battaglia F, Moss R, Bryan S. Chemotherapy-induced-peripheral neuropathy, gait and fall risk in older adults following cancer treatment. J Cancer Res Pract 2017 Dec;4(4):134-138. [CrossRef]
    93. Duregon F, Vendramin B, Bullo V, Gobbo S, Cugusi L, Di Blasio A, et al. Effects of exercise on cancer patients suffering chemotherapy-induced peripheral neuropathy undergoing treatment: A systematic review. Crit Rev Oncol Hematol 2018 Jan;121:90-100. [CrossRef] [Medline]
    94. Ospina P, McComb A, Wiart L, Eisenstat D, McNeely M. Physical therapy interventions, other than general physical exercise interventions, in children and adolescents before, during and following treatment for cancer. Cochrane Database Syst Rev 2018:2924. [CrossRef]
    95. Axenie C, Becher A, Kurz D, Grauschopf T. Meta-Learning for Avatar Kinematics Reconstruction in Virtual Reality Rehabilitation. 2019 Presented at: IEEE 19th International Conference on Bioinformatics and Bioengineering (BIBE); October 28-30, 2019; Athens, Greece p. 617-624. [CrossRef]
    96. Streckmann F, Balke M, Lehmann HC, Rustler V, Koliamitra C, Elter T, et al. The preventive effect of sensorimotor- and vibration exercises on the onset of Oxaliplatin- or vinca-alkaloid induced peripheral neuropathies-STOP. BMC Cancer 2018 Jan 10;18(1):62 [FREE Full text] [CrossRef] [Medline]
    97. McCrary JM, Goldstein D, Sandler CX, Barry BK, Marthick M, Timmins HC, et al. Exercise-based rehabilitation for cancer survivors with chemotherapy-induced peripheral neuropathy. Support Care Cancer 2019 Feb 12;27(10):3849-3857. [CrossRef] [Medline]
    98. Zahiri M, Chen KM, Zhou H, Nguyen H, Workeneh BT, Yellapragada SV, et al. Using wearables to screen motor performance deterioration because of cancer and chemotherapy-induced peripheral neuropathy (CIPN) in adults - Toward an early diagnosis of CIPN. J Geriatr Oncol 2019 Nov;10(6):960-967 [FREE Full text] [CrossRef] [Medline]
    99. Schwenk M, Grewal GS, Holloway D, Muchna A, Garland L, Najafi B. Interactive sensor-based balance training in older cancer patients with chemotherapy-induced peripheral neuropathy: a randomized controlled trial. Gerontology 2016;62(5):553-563 [FREE Full text] [CrossRef] [Medline]
    100. Grewal G, Sayeed R, Schwenk M, Bharara M, Menzies R, Talal TK, et al. Balance rehabilitation: promoting the role of virtual reality in patients with diabetic peripheral neuropathy. J Am Podiatr Med Assoc 2013;103(6):498-507. [CrossRef] [Medline]
    101. Axenie C, Kurz D. Role of kinematics assessment and multimodal sensorimotor training for motion deficits in breast cancer chemotherapy-induced polyneuropathy: a perspective on virtual reality avatars. Front Oncol 2020;10:1419 [FREE Full text] [CrossRef] [Medline]
    102. Bruzzone L, Chi M, Marconcini M. A novel transductive SVM for semisupervised classification of remote-sensing images. IEEE Trans Geosci Remote Sensing 2006 Nov;44(11):3363-3373. [CrossRef]
    103. Kouroubali A, Koumakis L, Kondylakis H, Katehakis D. An integrated approach towards developing quality mobile health apps for cancer. In Mobile Health Applications for Quality Healthcare Delivery. 2019.   URL: http://users.ics.forth.gr/~katehaki/publications/igibook2019.pdf [accessed 2020-11-24]
    104. Kondylakis H, Koumakis L, Hänold S, Nwankwo I, Forgó N, Marias K, et al. Donor's support tool: Enabling informed secondary use of patient's biomaterial and personal data. Int J Med Inform 2017 Jan;97:282-292. [CrossRef] [Medline]
    105. Belenkaya R, Campbell W, Chen R. Data standardization in cancer: challenges and opportunities. IMIA 2018.
    106. Health Level Seven International (2020).   URL: http://www.hl7.org/ [accessed 2020-11-21]
    107. Personal Connected Health Alliance (2020).   URL: http://www.pchalliance.org/ [accessed 2020-11-21]
    108. SMART Health IT.   URL: https://smarthealthit.org/
    109. FHIR Release 4. HL7.   URL: http://www.hl7.org/fhir/ [accessed 2020-11-21]
    110. Mandel J, Kreda D, Mandl K, Kohane I, Ramoni R. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. J Am Med Inform Assoc 2016 Sep;23(5):899-908 [FREE Full text] [CrossRef] [Medline]
    111. Warner J, Rioth M, Mandl K, Mandel JC, Kreda DA, Kohane IS, et al. SMART precision cancer medicine: a FHIR-based app to provide genomic information at the point of care. J Am Med Inform Assoc 2016 Jul;23(4):701-710 [FREE Full text] [CrossRef] [Medline]
    112. Triantafyllidis A, Kondylakis H, Votis K, Tzovaras D, Maglaveras N, Rahimi K. Features, outcomes, and challenges in mobile health interventions for patients living with chronic diseases: A review of systematic reviews. Int J Med Inform 2019 Dec;132:103984. [CrossRef] [Medline]
    113. Kussaibi H, Macary F, Kennedy M, Booker D, Brodsky V, Schrader T, et al. HL7 CDA implementation guide for structured anatomic pathology reports methodology and tools. Stud Health Technol Inform 2010;160(Pt 1):289-293. [Medline]
    114. National Center for Chronic Disease PreventionHealth Promotion, Division of Cancer PreventionControl (2014). Implementation Guide for Ambulatory Healthcare Provider Reporting to Central Cancer Registries. Centers for Disease Control and Prevention.   URL: https:/​/www.​cdc.gov/​phin/​resources/​guides/​documents/​implementation_guide_for_ambulatory_healthcare_provider_reporting_to_central_cancer_registries_march_2014.​pdf [accessed 2020-11-21]
    115. HL7 CDA ® Release 2 Implementation Guide: Reporting to Public Health Cancer Registries from Ambulatory Healthcare Providers, Release 1, DSTU Release 1. HL7 International (2015). 2014.   URL: https://www.hl7.org/implement/standards/product_brief.cfm?product_id=398 [accessed 2020-11-26]
    116. McCabe RM. National cancer database: the past, present, and future of the cancer registry and its efforts to improve the quality of cancer care. Semin Radiat Oncol 2019 Oct;29(4):323-325. [CrossRef] [Medline]
    117. Gong M, Wang Z, Liu Y. Toward early diagnosis decision support for breast cancer: ontology-based semantic interoperability. J Clin Oncol 2019;35:15. [CrossRef]
    118. Chambers DA, Amir E, Saleh RR, Rodin D, Keating NL, Osterman TJ, et al. The impact of big data research on practice, policy, and cancer care. Am Soc Clin Oncol Educ Book 2019 Jan;39(39):e167-e175 [FREE Full text] [CrossRef] [Medline]
    119. Sicotte C, Clavel S, Fortin M. A cancer care electronic medical record highly integrated into clinicians' workflow: users' attitudes pre-post implementation. Eur J Cancer Care (Engl) 2017 Nov 25;26(6):e12548. [CrossRef] [Medline]
    120. Improving Cancer-Related Outcomes with Connected Health: A Report to the President of the United States from the President's Cancer Panel. Bethesda (MD): President's Cancer Panel; 2016.   URL: https://prescancerpanel.cancer.gov/report/connectedhealth/ [accessed 2020-11-24]
    121. Study on Big Data in Public Health, Telemedicine and Healthcare. European Commision. 2016.   URL: https:/​/ec.​europa.eu/​digital-single-market/​en/​news/​study-big-data-public-health-telemedicine-and-healthcare [accessed 2020-11-24]
    122. Say RE, Thomson R. The importance of patient preferences in treatment decisions--challenges for doctors. Br Med J 2003 Sep 06;327(7414):542-545 [FREE Full text] [CrossRef] [Medline]
    123. Hlatky M. Patient preferences and clinical guidelines. J Am Med Assoc 1995 Apr 19;273(15):1219-1220. [Medline]
    124. Chong CA, Chen I, Naglie G, Krahn MD. How well do guidelines incorporate evidence on patient preferences? J Gen Intern Med 2009 Aug 23;24(8):977-982 [FREE Full text] [CrossRef] [Medline]
    125. Krahn M, Naglie G. The next step in guideline development: incorporating patient preferences. J Am Med Assoc 2008 Jul 23;300(4):436-438. [CrossRef] [Medline]
    126. Bastemeijer CM, Voogt L, van Ewijk JP, Hazelzet JA. What do patient values and preferences mean? A taxonomy based on a systematic review of qualitative papers. Patient Educ Couns 2017 May;100(5):871-881. [CrossRef] [Medline]
    127. Luckmann R. Evidence-Based Medicine: How to Practice and Teach EBM, 2nd Edition: By David L. Sackett, Sharon E. Straus, W. Scott Richardson, William Rosenberg, and R. Brian Haynes, Churchill Livingstone, 2000. J Intensive Care Med 2016 Dec 07;16(3):155-156. [CrossRef]
    128. Noble S, Matzdorff A, Maraveyas A, Holm MV, Pisa G. Assessing patients' anticoagulation preferences for the treatment of cancer-associated thrombosis using conjoint methodology. Haematologica 2015 Nov 20;100(11):1486-1492 [FREE Full text] [CrossRef] [Medline]
    129. Froberg DG, Kane RL. Methodology for measuring health-state preferences--III: population and context effects. J Clin Epidemiol 1989;42(6):585-592. [CrossRef] [Medline]
    130. de Clercq PA, Blom JA, Korsten HM, Hasman A. Approaches for creating computer-interpretable guidelines that facilitate decision support. Artif Intell Med 2004 May;31(1):1-27. [CrossRef] [Medline]
    131. A new system for grading recommendations in evidence based guidelines. Br Med J.   URL: https://www.bmj.com/content/323/7308/334.1 [accessed 2020-11-21]
    132. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, GRADE Working Group. Grading quality of evidence and strength of recommendations. Br Med J 2004 Jun 19;328(7454):1490 [FREE Full text] [CrossRef] [Medline]
    133. NCCN Clinical Practice Guidelines in Oncology.   URL: https://www.nccn.org/professionals/physician_gls/default.aspx [accessed 2020-11-21]
    134. ICHOM: International Consortium for Health Outcomes Measurement.   URL: http://www.ichom.org/ [accessed 2020-11-21]
    135. Sackett DL, Rosenberg WM, Gray JAM, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. Br Med J 1996 Jan 13;312(7023):71-72 [FREE Full text] [CrossRef] [Medline]
    136. Thomas L. Clinical practice guidelines. Evid Based Nurs 1999 Apr 01;2(2):38-39. [CrossRef]
    137. Tunis SR, Hayward RS, Wilson MC, Rubin HR, Bass EB, Johnston M, et al. Internists' attitudes about clinical practice guidelines. Ann Intern Med 1994 Jun 01;120(11):956-963. [CrossRef] [Medline]
    138. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PC, et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. J Am Med Assoc 1999 Oct 20;282(15):1458-1465. [CrossRef] [Medline]
    139. Liem EB, Obeid JS, Shareck EP, Sato L, Greenes RA. Representation of clinical practice guidelines through an interactive World-Wide-Web interface. Proc Annu Symp Comput Appl Med Care 1995:223-227 [FREE Full text] [Medline]
    140. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. J Am Med Assoc 2005 Mar 09;293(10):1223-1238. [CrossRef] [Medline]
    141. Wang D, Peleg M, Tu SW, Boxwala AA, Greenes RA, Patel VL, et al. Representation primitives, process models and patient data in computer-interpretable clinical practice guidelines:. Int J Med Inform 2002 Dec;68(1-3):59-70. [CrossRef]
    142. Liberman L, Menell JH. Breast imaging reporting and data system (BI-RADS). Radiol Clin North Am 2002 May;40(3):409-30, v. [CrossRef] [Medline]
    143. Latoszek-Berendsen A, Tange H, van den Herik HJ, Hasman A. From clinical practice guidelines to computer-interpretable guidelines. A literature overview. Methods Inf Med 2010;49(6):550-570. [CrossRef] [Medline]
    144. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. Br Med J 1999 Feb 20;318(7182):527-530 [FREE Full text] [CrossRef] [Medline]
    145. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet 1993 Nov;342(8883):1317-1322. [CrossRef]
    146. Beatty PA. Coping with abundance: the burden of progress in medical oncology. Oncologist 2012 Feb 06;17(2):294-295 [FREE Full text] [CrossRef] [Medline]
    147. Muro N, Larburu N, Torres J, Kerexeta J, Artola G, Arrue M, et al. Architecture for a Multimodal and Domain-Independent Clinical Decision Support System Software Development Kit. In: Annu Int Conf IEEE Eng Med Biol Soc. 2019 Jul Presented at: 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); July 23-27, 2019; Berlin, Germany, Germany p. 1399-1404   URL: https://ieeexplore.ieee.org/abstract/document/8856459 [CrossRef]
    148. Burki T. Vaccine misinformation and social media. Lancet Digit Health 2019 Oct;1(6):e258-e2e9. [CrossRef]
    149. Hurwitz B. Legal and political considerations of clinical practice guidelines. Br Med J 1999 Mar 06;318(7184):661-664 [FREE Full text] [CrossRef] [Medline]
    150. Jorritsma W, Cnossen F, van Ooijen P. Improving the radiologist-CAD interaction: designing for appropriate trust. Clin Radiol 2015 Feb;70(2):115-122. [CrossRef] [Medline]
    151. Doi K. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput Med Imaging Graph 2007;31(4-5):198-211 [FREE Full text] [CrossRef] [Medline]
    152. Philpotts LE. Can computer-aided detection be detrimental to mammographic interpretation? Radiology 2009 Oct;253(1):17-22. [CrossRef] [Medline]
    153. Rampun A, Wang H, Scotney B, Morrow P, Zwiggelaar R. Confidence Analysis for Breast Mass Image Classification. 2018 Presented at: 25th IEEE International Conference on Image Processing (ICIP ); 2018; Athens p. 7-10. [CrossRef]
    154. Rampun A, Wang H, Scotney B, Morrow P, Zwiggelaar R. Classification of Mammographic Microcalcification Clusters with Machine Learning Confidence Levels. 2018 Jul Presented at: 14th International Workshop on Breast Imaging (IWBI ); 2018; Atlanta, USA p. 8-11. [CrossRef]
    155. Lee RS, Gimenez F, Hoogi A, Rubin D. Curated Breast Imaging Subset of DDSM. The Cancer Imaging Archive. 2016.   URL: https://wiki.cancerimagingarchive.net/display/Public/CBIS-DDSM [accessed 2020-11-26]
    156. Cahan A, Cimino JJ. A learning health care system using computer-aided diagnosis. J Med Internet Res 2017 Mar 08;19(3):e54 [FREE Full text] [CrossRef] [Medline]
    157. Artificial Intelligence and Machine Learning in Software as a Medical Device. US Food and Drug Administrations.   URL: http:/​/www.​fda.gov/​medical-devices/​software-medical-device-samd/​artificial-intelligence-and-machine-learning-software-medical-device [accessed 2020-11-21]
    158. Crico C, Renzi C, Graf N, Buyx A, Kondylakis H, Koumakis L, et al. mHealth and telemedicine apps: in search of a common regulation. Ecancermedicalscience 2018 Jul 11;12:853 [FREE Full text] [CrossRef] [Medline]
    159. Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a medical Device (SaMD). US Food and Drug Administration.   URL: https://www.fda.gov/media/122535/download [accessed 2020-11-21]
    160. Shah P, Kendall F, Khozin S, Goosen R, Hu J, Laramie J, et al. Artificial intelligence and machine learning in clinical development: a translational perspective. NPJ Digit Med 2019;2:69 [FREE Full text] [CrossRef] [Medline]
    161. Artificial Intelligence in healthcare: is Europe ready? Federation of European Academies of Medicine: FEAM.   URL: https://www.feam.eu/wp-content/uploads/AI-Summary-report-15-Apr-2019-FV.pdf [accessed 2020-11-21]
    162. Stratigi M, Kondylakis H, Stefanidis K. Fairgrecs: Fair group recommendations by exploiting personal health information. 2018 Presented at: International Conference on Database and Expert Systems Applications; August 9, 2018; Regensburg, Germany. [CrossRef]
    163. Stratigi M, Kondylakis H, Stefanidis K. Fairness in group recommendations in the health domain. 2017 Presented at: IEEE 33rd International Conference on Data Engineering (ICDE); 2017; San Diego, USA   URL: https://core.ac.uk/download/pdf/250169717.pdf [CrossRef]


    Abbreviations

    ACC: accuracy
    AI: artificial intelligence
    AUC: area under the curve
    CDSS: clinical decision support system
    CIG: computer-interpretable guideline
    CPG: clinical practice guideline
    DSS: decision support system
    EHR: electronic health record
    ePRO: electronic patient-reported outcome
    EU: European Union
    FDA: Food and Drug Administration
    H2020: Horizon 2020
    ICT: information and communication technology
    IEEE: Institute of Electrical and Electronics Engineers
    IT: information technology
    LSTM: long short-term memory
    MD5: message-digest algorithm 5
    mHealth: mobile health
    ML: machine learning
    NCCN: National Comprehensive Cancer Network
    PROM: patient-reported outcome
    SaMD: software as a medical device


    Edited by G Eysenbach; submitted 01.07.20; peer-reviewed by T Exarchos, Y Xu; comments to author 08.08.20; revised version received 02.10.20; accepted 26.10.20; published 15.12.20

    ©Haridimos Kondylakis, Cristian Axenie, Dhundy (Kiran) Bastola, Dimitrios G Katehakis, Angelina Kouroubali, Daria Kurz, Nekane Larburu, Iván Macía, Roma Maguire, Christos Maramis, Kostas Marias, Philip Morrow, Naiara Muro, Francisco José Núñez-Benjumea, Andrik Rampun, Octavio Rivera-Romero, Bryan Scotney, Gabriel Signorelli, Hui Wang, Manolis Tsiknakis, Reyer Zwiggelaar. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 15.12.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.