Published on in Vol 23, No 9 (2021): September

Preprints (earlier versions) of this paper are available at, first published .
How Should Oncologists Choose an Electronic Patient-Reported Outcome System for Remote Monitoring of Patients With Cancer?

How Should Oncologists Choose an Electronic Patient-Reported Outcome System for Remote Monitoring of Patients With Cancer?

How Should Oncologists Choose an Electronic Patient-Reported Outcome System for Remote Monitoring of Patients With Cancer?

Authors of this article:

Fabrice Denis 1 Author Orcid Image ;   Ivan Krakowski 2 Author Orcid Image


1ELSAN, Jean Bernard Institute, Le Mans, France

2Association Francophone pour les Soins Oncologiques de Support, Bègles, France

Corresponding Author:

Fabrice Denis, MD, PhD


Jean Bernard Institute

9 Rue Beauverger

Le Mans, 72000


Phone: 33 0684190480


Electronic patient-reported outcome (ePRO) systems for symptom monitoring in patients with cancer have shown quality of life and survival benefits in controlled trials. They are beginning to be used in routine oncology practice. Many software developers provide software solutions for clinicians, but how should clinicians decide which system to use? We propose a synthesis of the main questions regarding the effectiveness, safety, and functionality of an ePRO system that a clinician should ask software providers to assist in the selection of a software product in order to obtain the best value tools for their patients and their practice.

J Med Internet Res 2021;23(9):e30549



Electronic patient-reported outcome (ePRO) systems for symptom monitoring of patients with cancer have had demonstrated quality of life and survival benefits in controlled trials (Table 1). They also result in a reduction of emergency hospitalization and have favorable cost-effectiveness and clinical utility; in addition, users have good perceptions of ePRO systems [1-9].

The practical integration of these ePRO systems into patient care is increasing and the positive results of trials have contributed to their increased use. The organization of medical teams is challenging because it requires the presence of a nurse dedicated to initial alert management and a dedicated time for the physician to respond to the alert.

It is challenging for clinicians and patients to identify systems that will add value to patient care in real life because the number of ePRO tools available is rapidly increasing, making it difficult for physicians to choose just one.

Table 1. Randomized studies of remote monitoring of patients with cancer by electronic patient-reported outcome systems.
AuthorsNumber of patientsIndicationsQuestionnaires usedMulticentric trialType of cancerImproved outcome
Basch, 2017 [1] and 2016 [9]766Toxicity monitoringNCI-CTCAEaNoAllQuality of life, survival, reduced emergency use
Mir, 2020 [2]609Toxicity monitoringPRO-CTCAEbNoAllDose intensity
Basch, 2020 [3]1191Toxicity monitoring, follow-up, supportive carePRO-CTCAEYesAllQuality of life, symptom control
Absolom, 2021 [4]508Toxicity monitoringNCI-CTCAENoAllQuality of life, symptom control
Berry, 2014 [5]752Toxicity monitoring, supportive careSDS-15cYesAllSymptom control
Strasser, 2016 [6]264Toxicity monitoring, supportive careESASdYesAllSymptom control
Denis, 2019 [7]133Follow-upNot reportedYesLungSurvival
Moony, 2021 [8]252Toxicity monitoringNot reportedNoAllQuality of life, symptom control

aNCI-CTCAE: National Cancer Institute version of the Common Terminology Criteria for Adverse Events.

bPRO-CTCAE: Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events.

cSDS-15: Symptom Distress Scale-15.

dESAS: Edmonton Symptom Assessment System.

Many criteria can be used to select ePRO systems, although there is little scientific literature focused on ePRO systems in oncology. However, some criteria on effectiveness, security, and functionality are usually included in telehealth or digital therapeutics assessments [10-12].

In general, ePRO systems must meet specific criteria for both the patient’s and clinician’s benefit. Here, we propose some questions that physicians may direct toward system providers to help them choose a relevant ePRO software.

  1. Notifications must be sent to the medical team. This factor appears to be more important than sending a notification to the patient on their smartphone and allowing them to call the medical center. This was included in randomized studies of ePRO systems for patients with cancer [1-9].
  2. The solution must be a medical device with quality marking such as a Class II CE mark [13-15]. This ensures that the product complies with the essential requirements of the relevant European Union legislation in which all devices must be evaluated for clinical efficacy and any side effects, if applicable, by means of preclinical and clinical evaluation. As ePRO systems are Class IIa medical devices, manufacturers must provide full quality assurance such as product and postmarket surveillance (materiovigilance) to obtain conformity. The management of materiovigilance is included in Class IIa CE marking and is mandatory.
  3. The solution must use validated algorithms. This validation aims to assess reliability and the performance of algorithms in detecting events; algorithms’ performance should be published in journals [1-9,16,17].
  4. The solution should contain algorithms that allow adverse events monitoring of the main drugs used in oncology (ie, chemotherapy, immunotherapy, hormonotherapy, targeted therapy, radiotherapy) [1-6,8,16,17].
  5. As early supportive care improves survival in oncology, the solution should contain algorithms to detect symptoms that make a patient eligible for early supportive care [3,5,6,18].
  6. The solution should allow for follow-up of the patient to detect complications or symptomatic relapse early, which improves clinical utility and patient outcomes [3,7].
  7. Clinicians should use solutions with up-to-date algorithms, especially for patients receiving therapies such as maintenance or long-course treatment. As the main new drugs (eg, immunotherapy or targeted therapies) have only been widely used for fewer than 5 years, solutions using algorithms older than this should be avoided. This is mandatory for Class IIa CE marking of medical devices [13-15].
  8. Validated questionnaires such as the Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) items from the National Cancer Institute or the Edmonton Symptom Assessment System (ESAS) should be used in the patient questionnaire. The PRO-CTCAE measurement system was developed to capture symptomatic adverse events in patients in cancer clinical trials and is now recommended for ePRO remote monitoring of symptoms because it evaluates the symptom attributes of frequency, severity, and interference. It has also been linguistically validated in more than 30 languages [3,19]. ESAS is one of the most used patient-reported outcome scales for symptom assessment in palliative care and oncology in the past 25 years. Although it only uses unidimensional scales to assess symptom intensity, ESAS has been psychometrically validated, has been translated into numerous languages, and is freely available. A change of one point was found to be the optimal cutoff for both improvement and deterioration for all 10 symptoms included in ESAS using a sensitivity-specificity approach. As it enables rapid, pragmatic assessment of multiple symptoms simultaneously, ESAS is used extensively in the clinical setting for symptom screening and monitoring worldwide. However, it cannot be used to monitor all treatment toxicities because only 10 symptoms are assessed; the ESAS does not include digestive symptoms (eg, vomiting, diarrhea), cutaneous symptoms, fluid retention (eg, edema), or sepsis. However, its routine use was associated with a 6% increase in 1-year overall survival in a retrospective matched cohort study of 128,893 patients [20-22].

  1. A real-life study assessment of the solution should be done to confirm that performance and compliance are in line with that reported in premarket studies and to assess the postmarket security of the solution. It is especially important to assess the satisfaction of patients in a real-life study [13].
  2. In Europe, the solution must comply with the General Data Protection Regulation (GDPR) that protects and secures patient data. Data protection regulations strengthened the management of alerts and the security of data transmitted to the medical team [23].
  3. Oncologists should avoid a solution that allows clinicians to set alert thresholds themselves because the lack of data about the safety and reliability of modified thresholds can result in physician responsibility if a patient experiences an adverse event (eg, following a false negative event).
  4. Solutions that allow the creation of “homemade” algorithms should be avoided for the same reasons as described in the previous point.
  5. Oncologists should avoid solutions developed by software providers about which a critical security alert was issued by health authorities (eg, risk of a serious adverse event due to critical software dysfunction). Many countries are already publishing these notifications on their health agency’s portal (eg, the French national health security agency [24]).

  1. As not all patients have smartphones, the software should allow for the sending of forms by mail or SMS text message to patients [3].
  2. The software must allow different health professionals (physicians and nurses and their colleagues) to manage notifications [3].
  3. There should be one software for all major cancers, covering the entire course of treatment (active treatment and surveillance) to avoid the use of multiple software for one patient. This would reduce the technical burden placed on professionals and result in higher levels of satisfaction among patients and health professionals in routine use [3].
  4. A single software containing an all-in-one algorithm enabling the detection of toxicities, follow-up management, and supportive care should be prioritized to avoid the use of multiple software with different algorithms. Too many different rules triggering notifications to physicians reduce understandability among users [3].
  5. A free-text window triggering notifications when used should be integrated into the patient form to improve communication of other symptoms that are not present on the form or to allow patients to ask health professionals questions [3,7].
  6. A software that allows the integration of patient reports into the electronic health record of the health center would be useful to improve the care team’s management of notifications [4].
  7. A software that contains cancer patient education modules would assist in optimizing treatment tolerance and compliance [25].

These criteria are summarized in Figure 1.

As ePRO systems will become a standard of quality of care for patients with cancer, the use of these solutions will increase, but their efficacy, security, and functionality should be warranted by the software providers to give the same benefit to real-life patients to allow reimbursement for these tools by health authorities.

Figure 1. Criteria synthesis of ePRO systems for remote monitoring of patients with cancer by oncologists. EMR: electronic medical record; ePRO: electronic patient-reported outcome; ESAS: Edmonton Symptom Assessment System; PRO-CTCAE: Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events.
View this figure


No one received compensation for their contribution. We thank Professor Claire Snyder, PhD (Johns Hopkins School of Medicine) and Professor Ethan Basch, MD, MSc (The University of North Carolina at Chapel Hill) for their relevant suggestions.

Authors' Contributions

All authors contributed to the critical revision of the manuscript.

Conflicts of Interest

None declared.

  1. Basch E, Deal AM, Dueck AC, Scher HI, Kris MG, Hudis C, et al. Overall Survival Results of a Trial Assessing Patient-Reported Outcomes for Symptom Monitoring During Routine Cancer Treatment. JAMA 2017 Jul 11;318(2):197-198 [FREE Full text] [CrossRef] [Medline]
  2. Mir O, Ferrua M, Fourcade A, Mathivon D, Duflot-Boukobza A, Dumont SN, et al. Intervention combining nurse navigators (NNs) and a mobile application versus standard of care (SOC) in cancer patients (pts) treated with oral anticancer agents (OAA): Results of CapRI, a single-center, randomized phase III trial. JCO 2020 May 20;38(15_suppl):2000-2000. [CrossRef]
  3. Basch E, Stover AM, Schrag D, Chung A, Jansen J, Henson S, et al. Clinical Utility and User Perceptions of a Digital System for Electronic Patient-Reported Symptom Monitoring During Routine Cancer Care: Findings From the PRO-TECT Trial. JCO Clinical Cancer Informatics 2020 Oct(4):947-957. [CrossRef]
  4. Absolom K, Warrington L, Hudson E, Hewison J, Morris C, Holch P, et al. Phase III Randomized Controlled Trial of eRAPID: eHealth Intervention During Chemotherapy. JCO 2021 Mar 01;39(7):734-747. [CrossRef]
  5. Berry DL, Hong F, Halpenny B, Partridge AH, Fann JR, Wolpin S, et al. Electronic Self-Report Assessment for Cancer and Self-Care Support: Results of a Multicenter Randomized Trial. JCO 2014 Jan 20;32(3):199-205. [CrossRef]
  6. Strasser F, Blum D, von Moos R, Cathomas R, Ribi K, Aebi S, Swiss Group for Clinical Cancer Research (SAKK). The effect of real-time electronic monitoring of patient-reported symptoms and clinical syndromes in outpatient workflow of medical oncologists: E-MOSAIC, a multicenter cluster-randomized phase III study (SAKK 95/06). Ann Oncol 2016 Mar;27(2):324-332 [FREE Full text] [CrossRef] [Medline]
  7. Denis F, Basch E, Septans A, Bennouna J, Urban T, Dueck AC, et al. Two-Year Survival Comparing Web-Based Symptom Monitoring vs Routine Surveillance Following Treatment for Lung Cancer. JAMA 2019 Jan 22;321(3):306-307 [FREE Full text] [CrossRef] [Medline]
  8. Mooney K, Iacob E, Wilson CM, Lloyd J, Nielson H, Ward JH. Randomized trial of remote cancer symptom monitoring during COVID-19: Impact on symptoms, QoL, and unplanned health care utilization. JCO 2021 May 20;39(15_suppl):12000-12000. [CrossRef]
  9. Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, et al. Symptom Monitoring With Patient-Reported Outcomes During Routine Cancer Treatment: A Randomized Controlled Trial. J Clin Oncol 2016 Mar 20;34(6):557-565 [FREE Full text] [CrossRef] [Medline]
  10. Leigh S, Ouyang J, Mimnagh C. Effective? Engaging? Secure? Applying the ORCHA-24 framework to evaluate apps for chronic insomnia disorder. Evid Based Ment Health 2017 Nov 25;20(4):e20-e20. [CrossRef] [Medline]
  11. Rowland SP, Fitzgerald JE, Holme T, Powell J, McGregor A. What is the clinical value of mHealth for patients? NPJ Digit Med 2020 Jan 13;3(1):4 [FREE Full text] [CrossRef] [Medline]
  12. van Haasteren A, Vayena E, Powell J. The Mobile Health App Trustworthiness Checklist: Usability Assessment. JMIR mHealth uHealth 2020 Jul 21;8(7):e16844 [FREE Full text] [CrossRef] [Medline]
  13. Medical Devices: Guidance document - Classification of medical devices. European Commission DG Health and Consumer.   URL: [accessed 2021-08-30]
  14. Kramer DB, Xu S, Kesselheim AS. How does medical device regulation perform in the United States and the European union? A systematic review. PLoS Med 2012 Jul 31;9(7):e1001276 [FREE Full text] [CrossRef] [Medline]
  15. French-Mowat E, Burnett J. How are medical devices regulated in the European Union? J R Soc Med 2012 Apr 16;105 Suppl 1(1_suppl):S22-S28 [FREE Full text] [CrossRef] [Medline]
  16. Iivanainen S, Alanko T, Vihinen P, Konkola T, Ekstrom J, Virtanen H, et al. Follow-Up of Cancer Patients Receiving Anti-PD-(L)1 Therapy Using an Electronic Patient-Reported Outcomes Tool (KISS): Prospective Feasibility Cohort Study. JMIR Form Res 2020 Oct 28;4(10):e17898 [FREE Full text] [CrossRef] [Medline]
  17. Iivanainen S, Alanko T, Peltola K, Konkola T, Ekström J, Virtanen H, et al. ePROs in the follow-up of cancer patients treated with immune checkpoint inhibitors: a retrospective study. J Cancer Res Clin Oncol 2019 Mar 21;145(3):765-774 [FREE Full text] [CrossRef] [Medline]
  18. Temel JS, Greer JA, Muzikansky A, Gallagher ER, Admane S, Jackson VA, et al. Early Palliative Care for Patients with Metastatic Non–Small-Cell Lung Cancer. N Engl J Med 2010 Aug 19;363(8):733-742. [CrossRef]
  19. Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). NIH National Cancer Institute.   URL: [accessed 2021-08-30]
  20. Mackler E, Petersen L, Severson J, Blayney DW, Benitez LL, Early CR, et al. Implementing a Method for Evaluating Patient-Reported Outcomes Associated With Oral Oncolytic Therapy. JOP 2017 Apr;13(4):e395-e400. [CrossRef]
  21. Hui D, Bruera E. The Edmonton Symptom Assessment System 25 Years Later: Past, Present, and Future Developments. J Pain Symptom Manage 2017 Mar;53(3):630-643 [FREE Full text] [CrossRef] [Medline]
  22. Barbera L, Sutradhar R, Seow H, Mittmann N, Howell D, Earle CC, et al. The impact of routine Edmonton Symptom Assessment System (ESAS) use on overall survival in cancer patients: Results of a population-based retrospective matched cohort analysis. Cancer Med 2020 Oct 14;9(19):7107-7115 [FREE Full text] [CrossRef] [Medline]
  23. Article 35 EU GDPR 'Data protection impact assessment'.   URL: [accessed 2021-08-30]
  24. Informations de sécurité. ANSM.   URL: [accessed 2021-08-30]
  25. Larson JL, Rosen AB, Wilson FA. The Effect of Telehealth Interventions on Quality of Life of Cancer Patients: A Systematic Review and Meta-Analysis. Telemed J E Health 2018 Jun;24(6):397-405. [CrossRef] [Medline]

ePRO: electronic patient-reported outcome
ESAS: Edmonton Symptom Assessment System
NCI-CTCAE: National Cancer Institute version of the Common Terminology Criteria for Adverse Events
PRO-CTCAE: Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events

Edited by G Eysenbach; submitted 16.06.21; peer-reviewed by K Ledu, C Helissey; comments to author 08.07.21; revised version received 08.07.21; accepted 02.08.21; published 09.09.21


©Fabrice Denis, Ivan Krakowski. Originally published in the Journal of Medical Internet Research (, 09.09.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.