Published on in Vol 23, No 6 (2021): June

Preprints (earlier versions) of this paper are available at, first published .
Using the Computer-based Health Evaluation System (CHES) to Support Self-management of Symptoms and Functional Health: Evaluation of Hematological Patient Use of a Web-Based Patient Portal

Using the Computer-based Health Evaluation System (CHES) to Support Self-management of Symptoms and Functional Health: Evaluation of Hematological Patient Use of a Web-Based Patient Portal

Using the Computer-based Health Evaluation System (CHES) to Support Self-management of Symptoms and Functional Health: Evaluation of Hematological Patient Use of a Web-Based Patient Portal

Original Paper

1University Hospital of Psychiatry II, Medical University of Innsbruck, Innsbruck, Austria

2Oncotyrol – Center for Personalized Cancer Medicine, Innsbruck, Austria

3University Hospital of Psychiatry I, Medical University of Innsbruck, Innsbruck, Austria

4Internal Medicine V: Haematology & Oncology, Innsbruck University Hospital, Innsbruck, Austria

5Comprehensive Cancer Center Innsbruck, Innsbruck, Austria

6University Clinic of Medical Psychology, Medical University of Innsbruck, Innsbruck, Austria

7Evaluation Software Development GmbH, Innsbruck, Austria

Corresponding Author:

Jens Lehmann, MSc

University Hospital of Psychiatry II

Medical University of Innsbruck

Anichstraße 35

Innsbruck, AT-6020


Phone: 43 5050481551


Background: Patient portals offer the possibility to assess patient-reported outcome measures (PROMs) remotely, and first evidence has demonstrated their potential benefits.

Objective: In this study, we evaluated patient use of a web-based patient portal that provides patient information and allows online completion of PROMs. A particular focus was on patient motivation for (not) using the portal. The portal was developed to supplement routine monitoring at the Department of Internal Medicine V in Innsbruck.

Methods: We included patients with multiple myeloma and chronic lymphocytic leukemia who were already participating in routine monitoring at the hospital for use of the patient portal. Patients were introduced to the portal and asked to complete questionnaires prior to their next hospital visits. We used system access logs and 3 consecutive semistructured interviews to analyze patient use and evaluation of the portal.

Results: Between July 2017 and August 2020, we approached 122 patients for participation in the study, of whom 83.6% (102/122) consented to use the patient portal. Patients were on average 60 (SD 10.4) years old. Of patients providing data at all study time points, 37% (26/71) consistently used the portal prior to their hospital visits. The main reason for not completing PROMs was forgetting to do so in between visits (25/84, 29%). During an average session, patients viewed 5.3 different pages and spent 9.4 minutes logged on to the portal. Feedback from interviews was largely positive with no patients reporting difficulties navigating the survey and 50% of patients valuing the self-management tools provided in the portal. Regarding the portal content, patients were interested in reviewing their own results and reported high satisfaction with the dynamic self-management advice, also reflected in the high number of clicks on those pages.

Conclusions: Patient portals can contribute to patient empowerment by offering sought-after information and self-management advice. In our study, the majority of our patients were open to using the portal. The low number of technical complaints and average time spent in the portal demonstrate the feasibility of our patient portal. While initial interest was high, long-term use was considerably lower and identified as the main area for improvement. In a next step, we will improve several aspects of the patient portal (eg, including a reminder to visit the portal before the next appointment and closer PROM symptom monitoring via an onconurse).

J Med Internet Res 2021;23(6):e26022



Patient-reported outcomes (PRO) are defined as all reports about the health status given directly by the patient without interpretation of the patient’s response by a clinician or anyone else [1]. While they have long been used in clinical trials, they have, in recent decades, also progressed to enriching routine clinical care [2,3]. Driven by technological progress and an increased availability and use of the internet in the population [4], it has become easier to incorporate the patient’s perspective into clinical care using electronic patient-reported outcome measures (ePROMs).

In oncological care, PROs can support patient-clinician communication [5,6] and aid early detection of symptoms [7,8] and have been linked to a decrease in hospitalization and emergency department visits [9]. Building on the evidence base showing the benefits of PRO use at the hospital, the use of web-based solutions to assess ePROMs outside the hospital has gained traction. In the last decade, the number of web-based patient portals that enable the completion of ePROMs has risen and recent research has demonstrated the potential benefits of patient portals in large-scale clinical trials [10-13]. For example, in a randomized controlled trial by Denis et al [10], web-based symptom monitoring was associated with increased survival compared to standard imaging surveillance following treatment for lung cancer. The authors argue that web-based symptom monitoring may allow for earlier symptom detection and appropriate reaction by health care professionals (HCPs). Moreover, PRO web monitoring can be highly cost-effective [11] and reduce the administrative burden of assessments inside the hospital, as patients can complete ePROMs from home. Finally, remote assessments are especially helpful in an outpatient setting, as assessments conducted on the day of chemotherapy administration at the hospital have shown to systematically underestimate patients’ symptom burden associated with treatment [14].

Despite the benefits shown in study settings, and even though detailed guidance on how to implement PROs into clinical practice exists [15,16], electronic PRO monitoring and especially patient portals are still only occasionally adopted in routine clinical practice. This can be attributed to the limited integration into electronic health records, a lack of financial reimbursement for ePROM assessments, and a lack of standardized assessment methods, which hinder implementation in routine care [2]. Patient portals also vary considerably regarding the focus of the implementation and their goals and use of PROs [17]. More research is needed that evaluates the usability and acceptability of different applications in routine practice to extend and strengthen the evidence base in this heterogeneous and evolving field of research.

At the Department of Internal Medicine V in Innsbruck, a patient portal for outpatients with multiple myeloma (MM) and chronic lymphocytic leukemia (CLL) was developed to conduct remote PRO assessments. The primary aims for development of the portal were to enhance patient empowerment, encourage patient engagement with PROs, and reduce the administrative burden of PRO assessments inside the hospital.

In this study, we evaluated patient use of the various components of the portal and aimed to identify patient lack of motivation for not using the patient portal and potential barriers to accessibility.

Study Design

In our observational, longitudinal study, we evaluated patient use of the portal based on two data sources: semistructured interviews conducted during 3 consecutive visits to our unit after introducing eligible patients to the patient portal and system access logs recording the duration of user sessions and how often each page of the web portal was accessed.

Description of the PRO Monitoring and the Patient Portal

In June 2016, we implemented routine ePROM assessments to supplement care of outpatients with MM at the Department of Internal Medicine V in Innsbruck. The two primary aims of the implementation of ePROM assessments were to supplement the Austrian Myeloma Registry (AMR) with PRO data and enrich clinical care with the data [18]. In July 2017, patients with CLL were added as a second patient group. The use of the Computer-based Health Evaluation System (CHES) [19] enables immediate processing and graphical representation of the results. For monitoring at the hospital, patients complete PRO measures before their medical appointment, and the results are presented to the HCP prior to the consultation. The implementation and feasibility of the assessments at the hospital and use of data from patients have been evaluated in the past for the AMR [18], and more details on HCP use of the system are presented elsewhere [20]. This study builds upon our previous implementation strategy and is focused on process evaluation, refinement, and extension (as described in phase IV of the implementation process described by Sztankay et al [18]).

There are 3 main components of CHES: (1) the HCP interface (CHES.main), which presents PRO and patient data to HCPs; (2) the survey interface (CHES.nurse), where the patients complete questionnaires at the hospital; and (3) the patient portal (CHES.portal), which allows remote questionnaire completion and access to supplemental information and self-management advice. See Figure 1 for a visualization of the patient portal and its functionality. See Multimedia Appendix 1 for screenshots of the software and portal.

Figure 1. Computer-Based Health Evaluation System patient portal functions when patients are at home and in the hospital. *Thresholds for clinical importance [21] are used to highlight domains that require discussion with the health care provider. PRO: patient-reported outcome; HCP: healthcare professional.
View this figure

In this paper, we assess patient use of the patient portal, which features the following functionalities:

  • Disease-specific information on CLL and MM: diagnosis, possible symptoms, possible treatments, and links to further information and self-help groups
  • PRO assessments with the cancer-specific European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire (EORTC QLQ-C30) in combination with the disease-specific modules EORTC QLQ-MY20 for patients with MM and EORTC QLQ-CLL17 for patients with CLL. The EORTC QLQ-C30 is the most frequently used cancer-specific questionnaire [22] and can be used to measure patient symptoms, functional health, and global quality of life. The modules QLQ-MY20 and QLQ-CLL17 supplement the QLQ-C30 and cover disease-specific issues of quality of life for MM (eg, future perspective, treatment side effects) and CLL patients (eg, symptom burden due to disease and/or treatment, worries/fears regarding health and functioning).
  • PRO score review by patient: results are displayed as colored bar charts (longitudinal and cross-sectional). Results that exceed the thresholds for clinical importance [21] are colored red, and results that do not exceed these threshold are colored green (see Lehmann et al [20] for more information).
  • Self-management tools and tailored information: based on EORTC QLQ-C30 data, patients are presented with self-management tools for the symptoms and functional health domains. If a patient reports a potentially clinically important result [21], they are directed to the self-management tools (see Lehmann et al [20] for more information). Patients are reminded that in case of severe impairments, they should contact the clinical team directly.

At the outpatient clinic, patients are introduced to the portal by a PRO facilitator (ie, person responsible for the assessments at the hospital). The role of the PRO facilitator is that of a study assistant with a background in psychology or nursing who is trained in the use of PRO data and motivates both patients and HCPs to use the PRO data. Patients are given instructions for use and their personal log-in data for accessing the portal at home by the PRO facilitator.

Typically, patients are advised to complete ePROM assessments in the week before their hospital visits. Visits are scheduled at regular intervals that range between 1 week and 12 months, depending on the disease stage and treatment plan. Patients are encouraged to complete questionnaires as often as they like, even on a daily basis. Questionnaires completed within 7 days of a hospital visit are used to inform the HCP of the patients’ health status and linked to the clinical data in the AMR. Data from assessments in between hospital visits are used for research purposes and enable more continuous tracking of patients’ health status in the AMR.

Although patients participating in the routine monitoring are encouraged to use the patient portal to report PROs, they can still complete assessments at the hospital (eg, if they forgot to use the portal before their hospital visit).

Study Sample

We defined the following inclusion criteria for participation in monitoring via the patient portal: fluency in German, age 18 years or older, diagnosis of MM or CLL, current treatment at the outpatient clinic of the Department of Internal Medicine V or the Comprehensive Cancer Center Innsbruck of the Medical University of Innsbruck, consent given to routine PRO monitoring at the hospital, and completion of at least one ePROM during a prior visit to the hospital.

Patients were deemed ineligible if they had no access to a computer or the internet or lacked sufficient knowledge to log into a website using a username and password. All patients provided informed consent to use the patient portal. Patients who declined to use the portal were asked for the reason for their refusal. The study and use of patient data are covered by the ethics approval for the AMR issued by the ethics committee of the Medical University of Innsbruck (study number AN3252 266/4.2 386/5.14).

At baseline, demographics (sex, age, marital status, education level, and employment status) and experience with the internet and computer technology (frequency, duration) were collected via a questionnaire. Diagnosis and cancer stage were obtained from the hospital’s medical records.

Selection of Outcome Measures

We used different sources to evaluate patient use and perception of the patient portal. Table 1 displays the selection of outcome measures.

Table 1. Selection of outcome measures.
Questions addressedOutcome measureAssessment methodData type
How often do patients use the portal?
Is the portal feasible for use during routine clinical care?
ePROMa completion rate (number of completed ePROMs before study time points)Assessed via CHESbQuantitative data
Why do patients use (or not use) the portal?
What is their feedback on portal components?
Patient perspectives on portal components and motivation to use (or not use) the portal; accessibility barriers identifiedSemistructured interviewsQualitative and quantitative data/questions
How often do patients log into the portal?
How long do patients log into the portal in a single session?
Which pages are viewed and how often?
Patient user patterns in the portal: number and duration of log-ins per patient; portal page viewsAssessed via CHES portal log dataQuantitative data

aePROM: electronic patient-reported outcome measure.

bCHES: Computer-based Health Evaluation System.

Patient Interviews

Each patient participated in 3 semistructured face-to-face interviews conducted by authors LN, PB, and JL following a fixed schedule during 3 consecutive visits to the outpatient unit. See Figure 2 for a flowchart of the interview process. The topics covered in the interviews were as follows:

T1: Guided introduction to the portal and user log-in

  • Possible difficulties logging in
  • Navigating the portal
  • Reviewing PRO results
  • Other suggestions/remarks by patients

T2: Evaluation of use: acceptability and usability

  • Satisfaction with instructions for completion of questionnaire
  • Navigating the portal
  • Technical difficulties
  • Reviewing PRO results and accessing self-help tools
  • Relevance and usefulness of provided content

T3: Evaluation of use: acceptability and usability

  • Technical difficulties
  • Interest in content not explored during previous log-ins
  • Satisfaction with design
  • Other suggestions/remarks by patients
Figure 2. Patient interview procedure and topics.
View this figure

ePROM Completion Rate

To evaluate initial interest in the portal, we assessed the proportion of patients in routine monitoring at the hospital consenting to use the patient portal at T1. To evaluate continued use, we assessed the proportion of patients who completed ePROMs prior to hospital visits at T2 and T3; ePROMs were considered to be linked to a hospital visit if they were completed in the 7 days before the visit.

CHES Log Data

The frequency (absolute number) and duration (in minutes) of the use of the website were determined by the CHES system log. These data were gathered irrespective of the interview time points each time the patient logged into the system. Log data were collected per session but were not linked to individual patients. Therefore, patients with more sessions have a greater weight in this analysis. Log-ins that occurred on the same day were considered a single session, and durations of these sessions were summed.

Data Analysis

Sociodemographic and clinical data were analyzed at time point T1. Comparison of patients completing ePROMs via the patient portal and of those who declined to use the portal were made with t tests (for parametric variables) and chi-square tests (for nonparametric variables).

The interviewer took field notes during the interviews and for each open question. Responses from the interviews were either analyzed descriptively (for yes/no questions) or paraphrased and category-coded (for open answers) independently by two researchers (JL and PB) and harmonized by discussion in case of different coding. We translated selected quotes into English for the results section of this paper.

We also analyzed the frequency (number) of views for each portal page, excluding views on the start page, where patients are directed automatically after logging in. Further, we calculated the time patients needed to complete the questionnaires online (mean completion time per patient averaged across all patients).

Participant Enrollment and Baseline Characteristics

Recruitment began in July 2017 and was open until August 2020. During the study period, we identified 142 eligible patients already participating in the electronic PRO monitoring in the hospital, of whom 85.9% (122/142) were approached for study participation and use of the patient portal. Of those, 83.6% (102/122) consented to be included in the patient portal. Of 20 patients not willing to use the patient portal, 18 patients stated a preference of questionnaire completion only at the hospital as the reason. The complete enrollment process is shown in Figure 3.

The full sociodemographic information and clinical data are given in Table 2. There were no statistically significant differences regarding age, sex, education, time since the initial diagnosis, and type of internet use between those who agreed and those who refused to participate (all P>.09). The age range of patients included in the portal was 39 to 83 years, and the age range of patients not included in the portal was 39 to 77 years. A statistically significant difference was found for general internet use (see Table 2), with participants who consented to use the portal reporting higher internet use than those who did not consent to use the portal: of patients who used the portal, 94% (94/100) reported using the internet at least multiple times per week compared to 76% (17/22) for patients who did not use the portal.

Figure 3. Recruitment flowchart (study patient denotes patients who were also participating in other clinical studies and were not included in our study so as not to overburden the patient with clinical questionnaires).
View this figure
Table 2. Sociodemographic information.
CharacteristicIncluded in patient portal (n=102)Not included in patient portal (n=23)StatisticP value

χ2t score
Sex, n (%)a0.1.80

Female37 (37)9 (39)

Male65 (64)14 (61)
Age (years), mean (SD)59.9 (10.5)63.2 (10.5)1.33.19
Diagnosis, n (%)4.0.047

Multiple myeloma63 (62)9 (39)

Chronic lymphocytic leukemia39 (38)14 (61)
Time since diagnosis (years), mean (IQR)4.5 (0.9-6.8)4.7 (0.9-6.6)1.73.09
Highest education, n (%)0.9.84

Compulsory or lower9 (9)1 (5)

Vocational training55 (55)10 (50)

High school certificate18 (18)4 (20)

University18 (18)5 (25)

Missing datab23
Internet use (type), n (%)0.7.40

Private use only66 (65)17 (74)

Job and private use36 (35)6 (26)
Internet use (frequency), n (%)10.0.02

>Once per month2 (2)0 (0)

Multiple times a month4 (4)5 (23)

Multiple times a week30 (30)4 (17)

Daily64 (64)13 (59)

Missing datab21

aNot applicable.

bMissing values were not included in the calculation of percentages.


Interviews lasted between 10 and 30 minutes. Only 5% (5/102) patients required help from the PRO facilitator because they were not able to read their username and password (small font size). After the log-in and during the first interview, no patients reported technical difficulties or had difficulties navigating the portal or the questionnaire. Table 3 shows the number of completed questionnaires before the interviews and the reasons for noncompletion. Of the patients who participated in all 3 interviews, 37% (26/71) completed the questionnaires prior to the T2 and T3 interview, while the others completed the questionnaires only at the hospital visit.

Table 4 shows the use and evaluation of the portal as reported by patients who completed questionnaires prior to the interviews. Patients who completed questionnaires prior to their follow-up appointments reported using the portal in different ways: the percentage of patients reading additional portal content (eg, disease-specific information) declined over time from 71% (27/38) at T2 to 41% (13/32) at T3. The self-management tools were used and deemed useful by 50% (17/34) of patients at the second and 42% (13/31) at the third interview.

Table 3. Questionnaire completion rate and reasons for not completing questionnaires in the patient portal before hospital visits.
Completion rate or reason for noncompletionInterview 1 (n=102), n (%)Interview 2 (n=84), n (%)Interview 3 (n=71), n (%)
Questionnaire completed before appointment

Yes102 (100)38 (45)32 (45)

Noa46 (55)39 (55)
Reasons not completed

Forgot25 (29)18 (25)

Technical difficulties9 (11)0 (0)

Loss of log-in data0 (0)3 (4)

Study patientb0 (0)1 (1)

Lack of time/motivation5 (6)6 (9)

Preferred assessment at hospital3 (4)7 (10)

Other4 (5)4 (6)

aNot applicable.

bStudy patients were also participating in other clinical studies using the same questionnaires or similar; they sometimes confused questionnaires from other studies with our study’s questionnaires.

Table 4. Use and evaluation of the patient portal as reported in the interviews.
Patient-reported behavior and evaluation of the portalCompleted questionnaires, n (%)

Interview 1 (n=102)Interview 2 (n=38)Interview 3 (n=32)
Reading (additional) portal content

Noa27 (71)13 (41)

Yes11 (29)18 (56)
Looking at one’s own results

Yes96 (96)25 (69)

No4 (4)11 (31)

Missing datab22
Self-management tools

Inspected self-management tools and found them to be useful17 (50)13 (42)

Inspected self-management tools and did not find them useful4 (12)2 (6)

Did not inspect self-management tools13 (38)16 (52)

Missing datab41
Reason for not inspecting self-management tools

No impairments reportedc6 (18)8 (26)

Lack of time1 (3)1 (3)

Other6 (18)7 (23)

aNot applicable.

bMissing values were not included in the calculation of percentages.

cIf a patient did not report impairments above the thresholds for clinical importance, the software did not suggest viewing self-management tools when looking at their own results.

Patient Comments

Patients were encouraged to provide additional comments following their answers to the interview questions. In the first interview, patients reported high satisfaction with the presentation of the results as bar charts. charts are a good way of presenting the results. I like that I can compare my results to those of other patients with cancer.
I like that I can see my results after having completed the questionnaire.

Patients who did not inspect their own results expressed that they were feeling fine and therefore had no interest in viewing their results.

Similarly, patients who were feeling fine or reported no impairments did not inspect the self-management tools, while others provided positive feedback on the self-management tools or the color-coding of questionnaire results.

I did not check the self-management advice because I am not feeling ill. Why would I check it?
I like the design. The arrow in the results [direct link from the results to the self-management information] was very helpful.
...liked the color coding [red/green] as it was simple and easy to understand.

Four patients expressed the wish for a reminder (email or text message) before the next appointment to complete the questionnaires online. Two patients mentioned that they would like to be able to choose their password or username themselves (which the software currently does not allow). Three patients also explicitly reported a decline in motivation toward the end of the study, caused by a lack of sufficient feedback from physicians who did not discuss their PRO results during the consultations.

CHES Log Data

Over the study period, we registered a total of 796 sessions (ie, log-ins by patients). Of the patients who logged into the patient portal, 27% (28/102) logged in once, 18% (18/102) logged in twice, 9% (9/102) logged in 3 times, 15% (16/102) logged in 4 to 6 times, 14% (14/102) logged in 7 to 10 times, and 17% (17/102) logged in more than 10 times (total range of 1-57).

The mean duration of a session was 9.4 (median 6, range 1-90) minutes. It took patients on average 2.9 minutes to complete the EORTC QLQ-C30, 1.8 minutes to complete the EORTC QLQ-MY20, and 1.5 minutes to complete the EORTC QLQ-CLL17. This adds up to an average questionnaire completion time of 4.7 minutes for patients with MM and 4.4 minutes for patients with CLL.

During an average session, patients viewed 5.3 (median 4, range 1-28) different pages. Figure 4 shows how often specific components of the patient portal were viewed. A total of 3487 views were registered. The most frequently viewed self-management pages were those providing information on dyspnea (89 views), diarrhea (80 views), cognitive functioning (64 views), and emotional functioning (49 views). The least frequently viewed self-management pages were those providing information on obstipation (11 views), role functioning (12 views), social functioning (21 views), and pain (18 views). Multimedia Appendix 2 shows the view count on all self-management pages.

Figure 4. Page views in the patient portal by category (excluding home page). Categories marked with an asterisk combine multiple pages and subpages. QOL: quality of life; MM: multiple myeloma; CLL: chronic lymphocytic leukemia; Q&A: questions and answers.
View this figure

Principal Findings

In our study, we evaluated the use of the patient portal in patients with MM and CLL participating in routine electronic PRO monitoring. We found that in those patients who were eligible to use the patient portal, the majority (84%) were willing to use the portal. Only a few users reported difficulties logging in, but no users reported problems navigating the survey. However, we found that considerably fewer patients (37%) than initially included adopted a continued use of the portal across all 3 study time points. On average, patients spent 9.4 minutes in the portal per session.

Uptake of Home Monitoring

Our recruitment rate (ie, patients consenting to try the patient portal) is high compared to rates found in other studies. For example, other feasibility or usability studies report that between 21% and 64% [23-25] of patients were willing to use patient portals. Our high recruitment rate is most likely a consequence of the fact that patients were already participating in electronic PRO monitoring and so had already approved some form of monitoring. We would like to note the high inclusion rate for PRO monitoring at the hospital, which was, as reported previously, 94% of all MM patients treated in the department [18]. Our study emphasizes the possible synergy between assessments inside the hospital and home monitoring; patients who had already participated in electronic PRO monitoring inside the hospital were open to also using a patient portal. Moreover, the high acceptance of the patient portal might have been induced by the opportunity to get accustomed to ePROMs at the hospital with the help of a PRO facilitator before being introduced to the patient portal. We hypothesize that such a stepwise approach can reduce potential reservations about using electronic measures and contribute to user empowerment.

The adoption of home monitoring by a considerable proportion of patients also meant that those patients did not need to be assessed at the hospital. In this way, the patient portal reduced the administrative burden of assessing those patients’ PROs at the hospital and allowed us to allocate those resources elsewhere.

Long-Term Use of the Portal

Over the course of the study, slightly less than half of the participants used the portal prior to their follow-up appointments. The main reason reported by patients was that they had forgotten to report their PRO data using the patient portal. Especially for patients with an infrequent appointment schedule (intervals between hospital visits of up to 12 months), forgetting to use the portal is, in fact, not surprising. This issue might be addressed by implementing an automated reminder (email or text message) as has been done with similar home monitoring systems [24,26]. For example, the AmbuFlex system in Denmark has, in the past, used different forms of reminders, including letters, emails, and text messages. They ultimately implemented communication via a national secure email program, which accounts for 93.2% of communication to patients and secures high completion rates [27].

Additionally, it is important to note that due to our routine monitoring approach, ePRO data from patients who did not use the portal prior to the follow-up appointments were not lost. Instead, those patients were invited to complete the ePROMs at the hospital as was done with other patients who did not use the portal.

While only a few of our patients directly mentioned a decline in motivation to complete ePROMs due to HCPs not picking up on the results during the consultations, this is in fact a frequent problem of PRO implementations in clinical settings [3,26,28,29]. For patients, there is little perceived benefit of completing the questionnaires if the results are not reviewed and discussed by HCPs. Instead, sharing their health status via questionnaires might even become burdensome. Therefore, it is important to engage HCPs with the concept of PROs and train them in the use of PRO results to prevent PRO data from becoming meaningless busywork that hinders clinical practice instead of enhancing it [28]. This requires the education and training of HCPs, which can be achieved by conducting specialized training programs [30]. A buy-in strategy may be used to increase HCP engagement with PROs, for example, via the adoption of the user interface to HCP needs and preferences, regular meetings, or analyses of PRO data in the registry upon request of HCPs. Another important approach is to make PRO data comprehensible and actionable for HCPs (eg, by providing advice on how to react to results) [26,31,32]. In our system, this is done, for example, by using thresholds for clinical importance for the EORTC QLQ-C30 [21] that facilitate the interpretation of patient PRO data.

Finally, the literature shows that in order to provide a sound theoretical basis for sustainable PRO solutions in routine care settings, an implementation science approach can be followed [33]. Implementation science can help identify barriers (which are often similar across contexts) and enablers (which often depend on the given hospitals’ context) of patient-reported outcome measure (PROM) implementations.

User Patterns in the Portal

As has been found in other studies [13,17,25], the majority of our patients found the display of self-management advice linked with their results to be valuable. A recent review of electronic systems to measure PROs found that less than one-third (29%) of published electronic systems include features that provide tailored automated advice to patients, and less than half (41%) provide general educational information [17]. In a qualitative study evaluating another eHealth application, tailored feedback and advice was rated as appealing by most participants [34]. Participants valued the option of accessing information remotely between appointments and having a low threshold to receiving such information (compared to having to consult their treating physician or having to search for information) [34]. These are notions that echo the high number of views of pages with self-management advice we found in our study.

Nevertheless, despite the approval for self-management advice we found in our study, a recent clinical trial showed that, while patient portals can contribute to improved health-related quality of life [13], their measurable benefits on patient activation (knowledge, skills, and confidence for self-management) could not be shown in a diverse sample of cancer survivors [13]. However, previous evidence suggested that such effects may be more pronounced in newly diagnosed patients [25,35]. Another randomized controlled trial showed that weekly PRO monitoring along with the provision of tailored self-management advice, compared to usual care, significantly enhanced self-efficacy in patients with cancer [36].


Our study was designed to be carried out during routine care with minimal interruption of the clinical workflow. Therefore, interview time points were integrated into the patients’ hospital visits. This resulted in varying intervals between interviews (between 1 week and 1 year in a few cases), which may have influenced the results. However, we point out that these are real-world visitation schedules, and any application designed for routine care should be evaluated accordingly. Similarly, we included more males than females in our study. This reflects the epidemiology of MM and CLL, and the male/female ratio was comparable to data from the AMR.

Second, the CHES log data did not allow for detailed analysis of individual user page-view patterns but only for analysis of overall page views and duration and frequency of sessions. This means that patients logging on to the portal more frequently have a greater weight in the analysis of page views. Moreover, page view numbers have to be interpreted with care as they provide no information on whether the pages were actually read.

Another limitation is that the interviews were conducted by the authors (LN, PB, and JL), who introduced patients to the portal. Therefore, some patients might have been reluctant to voice criticism in the interviews, even though we actively encouraged patients to also report negative feedback.

Finally, we consider some sample-related limitations: We included only German-speaking patients in the patient portal. While the EORTC questionnaires are available in a large number of languages (and can be completed at the hospital [20]), translating and updating the content of the portal would require considerable resources. This results in a potential bias, and our findings may not be generalizable to patients with other first languages and limited German language proficiency. We are aware that this can create an imbalance in the provision of care, as this systematically excludes certain patient groups. In fact, patients not speaking the primary language of the country might profit most from receiving disease-specific information and from being able to report symptoms online in their native language. Moreover, we observed a selection bias due to patients’ age (patients with MM were on average 11 years younger than the mean age of patients in the AMR), as older patients might be less proficient using the internet and were consequently not included in our study. While this is an important limitation to consider, its impact should decrease over time, as the population’s internet capabilities have been steadily increasing [37] and should further increase in the years to come.

Future Steps

Following the insights gained in this study, we are currently in the process of updating our monitoring procedure and software. One important step is to more closely monitor patients’ PROMs to swiftly identify deteriorations and act accordingly; we aim to have a trained onconurse monitor patient results and check on patients in case of deteriorations. If necessary, the nurse can schedule an earlier appointment or alert the treating HCP. Another planned improvement is to implement an email or text message reminder to improve PROM completion rates.


Our study shows that a patient portal enabling remote PRO data assessment can complement routine electronic PRO implementation at the hospital by reducing the burden of administration for the clinical team and offering an additional way for patients to engage with PROs. We found that the majority of patients were open to using the patient portal and interested in assessments from home. The low number of technical problems and absence of complaints demonstrate the general user-friendliness of our portal. While initial uptake was high, fewer patients adopted regular use of the portal prior to their appointments. To increase long-term participation rates, further motivational (eg, increasing HCP engagement with the PRO data) and technical (eg, email reminder) measures are needed.


We thank the patients who participated in this study for their valuable time and input, without which the study would not have been possible. We thank the Department of Internal Medicine V in Innsbruck and the Comprehensive Cancer Center Innsbruck for their support with the project. We thank Janny Sauter for her help in the collection of data.

Authors' Contributions

JL and PB wrote and edited the manuscript and collected and analyzed the data. BH, GR, JMG, LW, MS, RW, WWeyrer, and WWillenbacher conceptualized the study, supported the implementation of the patient portal, and edited the manuscript. LN conceptualized the study, supported the implementation of the patient portal, assisted in data collection, and edited the manuscript. All authors read and approved the final version of the manuscript.

Conflicts of Interest

BH and GR have intellectual property rights to the software tool CHES. All remaining authors declare that they have no conflict of interest.

Multimedia Appendix 1

Examples of pages in the patient portal.

PDF File (Adobe PDF File), 1495 KB

Multimedia Appendix 2

Total view count in the portal and specific view count for the self-management pages.

PNG File , 104 KB

  1. Patient-reported outcome measures: use in medical product development to support labeling claims. US Food and Drug Administration. 2009 Dec.   URL: [accessed 2020-08-05]
  2. Basch E. Patient-reported outcomes: harnessing patients' voices to improve clinical care. N Engl J Med 2017 Jan 12;376(2):105-108. [CrossRef] [Medline]
  3. Jensen RE, Snyder CF, Abernethy AP, Basch E, Potosky AL, Roberts AC, et al. Review of electronic patient-reported outcomes systems used in cancer clinical care. J Oncol Pract 2014 Jul;10(4):e215-e222. [CrossRef]
  4. Digital Economy and Society Index (DESI) 2020: use of internet services. European Commission.   URL: [accessed 2020-09-03]
  5. Detmar SB, Muller MJ, Schornagel JH, Wever LDV, Aaronson NK. Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial. JAMA 2002 Dec 18;288(23):3027-3034. [CrossRef] [Medline]
  6. Velikova G, Booth L, Smith AB, Brown PM, Lynch P, Brown JM, et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol 2004 Feb 15;22(4):714-724. [CrossRef]
  7. Basch E, Abernethy AP. Supporting clinical practice decisions with real-time patient-reported outcomes. J Clin Oncol 2011 Mar 10;29(8):954-956. [CrossRef] [Medline]
  8. Takeuchi EE, Keding A, Awad N, Hofmann U, Campbell LJ, Selby PJ, et al. Impact of patient-reported outcomes in oncology: a longitudinal analysis of patient-physician communication. J Clin Oncol 2011 Jul 20;29(21):2910-2917. [CrossRef]
  9. Barbera L, Sutradhar R, Seow H, Earle C, Howell D, Mittmann N, et al. Impact of standardized edmonton symptom assessment system use on emergency department visits and hospitalization: results of a population-based retrospective matched cohort analysis. J Oncol Pract 2020 Sep;16(9):e958-e965. [CrossRef]
  10. Denis F, Basch E, Septans A, Bennouna J, Urban T, Dueck AC, et al. Two-year survival comparing web-based symptom monitoring vs routine surveillance following treatment for lung cancer. JAMA 2019 Jan 22;321(3):306-307 [FREE Full text] [CrossRef] [Medline]
  11. Lizée T, Basch E, Trémolières P, Voog E, Domont J, Peyraga G, et al. Cost-effectiveness of web-based patient-reported outcome surveillance in patients with lung cancer. J Thorac Oncol 2019 Jun;14(6):1012-1020 [FREE Full text] [CrossRef] [Medline]
  12. Basch E, Deal AM, Dueck AC, Scher HI, Kris MG, Hudis C, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA 2017 Dec 11;318(2):197-198 [FREE Full text] [CrossRef] [Medline]
  13. van der Hout A, van Uden-Kraan CF, Holtmaat K, Jansen F, Lissenberg-Witte BI, Nieuwenhuijzen GAP, et al. Role of eHealth application Oncokompas in supporting self-management of symptoms and health-related quality of life in cancer survivors: a randomised, controlled trial. Lancet Oncol 2020 Jan;21(1):80-94. [CrossRef] [Medline]
  14. Giesinger JM, Wintner LM, Zabernigg A, Gamper E, Oberguggenberger AS, Sztankay MJ, et al. Assessing quality of life on the day of chemotherapy administration underestimates patients' true symptom burden. BMC Cancer 2014 Oct 10;14:758 [FREE Full text] [CrossRef] [Medline]
  15. Wintner LM, Sztankay M, Aaronson N, Bottomley A, Giesinger JM, Groenvold M, EORTC Quality of Life Group. The use of EORTC measures in daily clinical practice: a synopsis of a newly developed manual. Eur J Cancer 2016 Nov;68:73-81 [FREE Full text] [CrossRef] [Medline]
  16. LeRouge C, Austin E, Lee J, Segal C, Sangameswaran S, Hartzler A, et al. ePROs in Clinical Care: Guidelines and Tools for Health Systems. Seattle: CERTAIN, University of Washington; 2020.
  17. Warrington L, Absolom K, Conner M, Kellar I, Clayton B, Ayres M, et al. Electronic systems for patients to report and manage side effects of cancer treatment: systematic review. J Med Internet Res 2019 Jan 24;21(1):e10875 [FREE Full text] [CrossRef] [Medline]
  18. Sztankay M, Neppl L, Wintner LM, Loth FL, Willenbacher W, Weger R, et al. Complementing clinical cancer registry data with patient reported outcomes: a feasibility study on routine electronic patient-reported outcome assessment for the Austrian Myelome Registry. Eur J Cancer Care (Engl) 2019 Nov;28(6):e13154 [FREE Full text] [CrossRef] [Medline]
  19. Holzner B, Giesinger JM, Pinggera J, Zugal S, Schöpf F, Oberguggenberger AS, et al. The Computer-based Health Evaluation Software (CHES): a software for electronic patient-reported outcome monitoring. BMC Med Inform Decis Mak 2012;12:126 [FREE Full text] [CrossRef] [Medline]
  20. Lehmann J, Wintner LM, Sztankay M, Willenbacher W, Weger R, Weyrer W, et al. Patient-reported outcomes and psycho-oncological screening in hematology: a practical example of routine electronic monitoring. Mag Eur Med Oncol 2020 Jul 03;13(3):285-293. [CrossRef]
  21. Giesinger JM, Loth FLC, Aaronson NK, Arraras JI, Caocci G, Efficace F, EORTC Quality of Life Group. Thresholds for clinical importance were established to improve interpretation of the EORTC QLQ-C30 in clinical practice and research. J Clin Epidemiol 2020 Feb;118:1-8 [FREE Full text] [CrossRef] [Medline]
  22. Efficace F, Collins GS, Cottone F, Giesinger JM, Sommer K, Anota A, et al. Patient-reported outcomes as independent prognostic factors for survival in oncology: systematic review and meta-analysis. Value Health 2021 Feb;24(2):250-267. [CrossRef] [Medline]
  23. Kuijpers W, Groen WG, Oldenburg HS, Wouters MW, Aaronson NK, van Harten WH. eHealth for breast cancer survivors: use, feasibility and impact of an interactive portal. JMIR Cancer 2016 May 10;2(1):e3 [FREE Full text] [CrossRef] [Medline]
  24. Girgis A, Durcinoska I, Levesque JV, Gerges M, Sandell T, Arnold A, PROMPT-Care Program Group. eHealth system for collecting and utilizing Patient Reported Outcome Measures for Personalized Treatment and Care (PROMPT-Care) among cancer patients: mixed methods approach to evaluate feasibility and acceptability. J Med Internet Res 2017 Dec 02;19(10):e330 [FREE Full text] [CrossRef] [Medline]
  25. Duman-Lubberding S, van Uden-Kraan CF, Jansen F, Witte BI, van der Velden LA, Lacko M, et al. Feasibility of an eHealth application "OncoKompas" to improve personalized survivorship cancer care. Support Care Cancer 2016 May;24(5):2163-2171 [FREE Full text] [CrossRef] [Medline]
  26. Wu AW, White SM, Blackford AL, Wolff AC, Carducci MA, Herman JM, PatientViewpoint Scientific Advisory Board. Improving an electronic system for measuring PROs in routine oncology practice. J Cancer Surviv 2016 Jun;10(3):573-582 [FREE Full text] [CrossRef] [Medline]
  27. Hjollund NHI. Fifteen years' use of patient-reported outcome measures at the group and patient levels: trend analysis. J Med Internet Res 2019 Sep 30;21(9):e15856 [FREE Full text] [CrossRef] [Medline]
  28. Thestrup Hansen S, Kjerholt M, Friis Christensen S, Hølge-Hazelton B, Brodersen J. Haematologists' experiences implementing patient reported outcome measures (PROMs) in an outpatient clinic: a qualitative study for applied practice. J Patient Rep Outcomes 2019 Dec 28;3(1):74 [FREE Full text] [CrossRef] [Medline]
  29. Taarnhøj GA, Lindberg H, Dohn LH, Omland LH, Hjøllund NH, Johansen C, et al. Electronic reporting of patient-reported outcomes in a fragile and comorbid population during cancer therapy: a feasibility study. Health Qual Life Outcomes 2020 Jul 11;18(1):225 [FREE Full text] [CrossRef] [Medline]
  30. Santana MJ, Haverman L, Absolom K, Takeuchi E, Feeny D, Grootenhuis M, et al. Training clinicians in how to use patient-reported outcome measures in routine clinical practice. Qual Life Res 2015 Jan 15;24(7):1707-1718. [CrossRef]
  31. Snyder CF, Blackford AL, Wolff AC, Carducci MA, Herman JM, Wu AW, PatientViewpoint Scientific Advisory Board. Feasibility and value of PatientViewpoint: a web system for patient-reported outcomes assessment in clinical practice. Psychooncology 2013 Apr;22(4):895-901 [FREE Full text] [CrossRef] [Medline]
  32. Hughes EF, Wu AW, Carducci MA, Snyder CF. What can I do? Recommendations for responding to issues identified by patient-reported outcomes assessments used in clinical practice. J Support Oncol 2012;10(4):143-148 [FREE Full text] [CrossRef] [Medline]
  33. Stover AM, Haverman L, van Oers HA, Greenhalgh J, Potter CM, ISOQOL PROMs/PREMs in Clinical Practice Implementation Science Work Group. Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res 2020 Jul 10:1. [CrossRef] [Medline]
  34. Lubberding S, van Uden-Kraan CF, Te Velde EA, Cuijpers P, Leemans CR, Verdonck-de Leeuw IM. Improving access to supportive cancer care through an eHealth application: a qualitative needs assessment among cancer survivors. J Clin Nurs 2015 May;24(9-10):1367-1379. [CrossRef] [Medline]
  35. Girgis A. The role of self-management in cancer survivorship care. Lancet Oncol 2020 Jan;21(1):8-9. [CrossRef]
  36. Absolom K, Warrington L, Hudson E, Hewison J, Morris C, Holch P, et al. Phase III randomized controlled trial of eRAPID: eHealth intervention during chemotherapy. J Clin Oncol 2021 Mar 01;39(7):734-747. [CrossRef] [Medline]
  37. Austrian Internet Monitor.   URL: [accessed 2021-02-08]

AMR: Austrian Myeloma Registry
CHES: Computer-Based health Evaluation System
CLL: chronic lymphocytic leukemia
EORTC QLQ–C30: European Organisation for Research and Treatment of Cancer Quality of Life Questionnaire
ePROM: electronic patient-reported outcome measure
HCP: health care professional
MM: multiple myeloma
PRO: patient-reported outcome
PROM: patient-reported outcome measure

Edited by R Kukafka; submitted 25.11.20; peer-reviewed by L Schougaard, T Sandell, S Oerlemans; comments to author 20.01.21; revised version received 08.02.21; accepted 30.04.21; published 08.06.21


©Jens Lehmann, Petra Buhl, Johannes M Giesinger, Lisa M Wintner, Monika Sztankay, Lucia Neppl, Wolfgang Willenbacher, Roman Weger, Walpurga Weyrer, Gerhard Rumpold, Bernhard Holzner. Originally published in the Journal of Medical Internet Research (, 08.06.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.