In our observational, longitudinal study, we evaluated patient use of the portal based on two data sources: semistructured interviews conducted during 3 consecutive visits to our unit after introducing eligible patients to the patient portal and system access logs recording the duration of user sessions and how often each page of the web portal was accessed.

In June 2016, we implemented routine ePROM assessments to supplement care of outpatients with MM at the Department of Internal Medicine V in Innsbruck. The two primary aims of the implementation of ePROM assessments were to supplement the Austrian Myeloma Registry (AMR) with PRO data and enrich clinical care with the data [18]. In July 2017, patients with CLL were added as a second patient group. The use of the Computer-based Health Evaluation System (CHES) [19] enables immediate processing and graphical representation of the results. For monitoring at the hospital, patients complete PRO measures before their medical appointment, and the results are presented to the HCP prior to the consultation. The implementation and feasibility of the assessments at the hospital and use of data from patients have been evaluated in the past for the AMR [18], and more details on HCP use of the system are presented elsewhere [20]. This study builds upon our previous implementation strategy and is focused on process evaluation, refinement, and extension (as described in phase IV of the implementation process described by Sztankay et al [18]).

There are 3 main components of CHES: (1) the HCP interface (CHES.main), which presents PRO and patient data to HCPs; (2) the survey interface (CHES.nurse), where the patients complete questionnaires at the hospital; and (3) the patient portal (CHES.portal), which allows remote questionnaire completion and access to supplemental information and self-management advice. See Figure 1 for a visualization of the patient portal and its functionality. See Multimedia Appendix 1 for screenshots of the software and portal.

In this paper, we assess patient use of the patient portal, which features the following functionalities:

At the outpatient clinic, patients are introduced to the portal by a PRO facilitator (ie, person responsible for the assessments at the hospital). The role of the PRO facilitator is that of a study assistant with a background in psychology or nursing who is trained in the use of PRO data and motivates both patients and HCPs to use the PRO data. Patients are given instructions for use and their personal log-in data for accessing the portal at home by the PRO facilitator.

Typically, patients are advised to complete ePROM assessments in the week before their hospital visits. Visits are scheduled at regular intervals that range between 1 week and 12 months, depending on the disease stage and treatment plan. Patients are encouraged to complete questionnaires as often as they like, even on a daily basis. Questionnaires completed within 7 days of a hospital visit are used to inform the HCP of the patients’ health status and linked to the clinical data in the AMR. Data from assessments in between hospital visits are used for research purposes and enable more continuous tracking of patients’ health status in the AMR.

Although patients participating in the routine monitoring are encouraged to use the patient portal to report PROs, they can still complete assessments at the hospital (eg, if they forgot to use the portal before their hospital visit).

We defined the following inclusion criteria for participation in monitoring via the patient portal: fluency in German, age 18 years or older, diagnosis of MM or CLL, current treatment at the outpatient clinic of the Department of Internal Medicine V or the Comprehensive Cancer Center Innsbruck of the Medical University of Innsbruck, consent given to routine PRO monitoring at the hospital, and completion of at least one ePROM during a prior visit to the hospital.

Patients were deemed ineligible if they had no access to a computer or the internet or lacked sufficient knowledge to log into a website using a username and password. All patients provided informed consent to use the patient portal. Patients who declined to use the portal were asked for the reason for their refusal. The study and use of patient data are covered by the ethics approval for the AMR issued by the ethics committee of the Medical University of Innsbruck (study number AN3252 266/4.2 386/5.14).

At baseline, demographics (sex, age, marital status, education level, and employment status) and experience with the internet and computer technology (frequency, duration) were collected via a questionnaire. Diagnosis and cancer stage were obtained from the hospital’s medical records.

We used different sources to evaluate patient use and perception of the patient portal. Table 1 displays the selection of outcome measures.

Each patient participated in 3 semistructured face-to-face interviews conducted by authors LN, PB, and JL following a fixed schedule during 3 consecutive visits to the outpatient unit. See Figure 2 for a flowchart of the interview process. The topics covered in the interviews were as follows:

T1: Guided introduction to the portal and user log-in

T2: Evaluation of use: acceptability and usability

T3: Evaluation of use: acceptability and usability

To evaluate initial interest in the portal, we assessed the proportion of patients in routine monitoring at the hospital consenting to use the patient portal at T1. To evaluate continued use, we assessed the proportion of patients who completed ePROMs prior to hospital visits at T2 and T3; ePROMs were considered to be linked to a hospital visit if they were completed in the 7 days before the visit.

The frequency (absolute number) and duration (in minutes) of the use of the website were determined by the CHES system log. These data were gathered irrespective of the interview time points each time the patient logged into the system. Log data were collected per session but were not linked to individual patients. Therefore, patients with more sessions have a greater weight in this analysis. Log-ins that occurred on the same day were considered a single session, and durations of these sessions were summed.

Sociodemographic and clinical data were analyzed at time point T1. Comparison of patients completing ePROMs via the patient portal and of those who declined to use the portal were made with t tests (for parametric variables) and chi-square tests (for nonparametric variables).

The interviewer took field notes during the interviews and for each open question. Responses from the interviews were either analyzed descriptively (for yes/no questions) or paraphrased and category-coded (for open answers) independently by two researchers (JL and PB) and harmonized by discussion in case of different coding. We translated selected quotes into English for the results section of this paper.

We also analyzed the frequency (number) of views for each portal page, excluding views on the start page, where patients are directed automatically after logging in. Further, we calculated the time patients needed to complete the questionnaires online (mean completion time per patient averaged across all patients).

Recruitment began in July 2017 and was open until August 2020. During the study period, we identified 142 eligible patients already participating in the electronic PRO monitoring in the hospital, of whom 85.9% (122/142) were approached for study participation and use of the patient portal. Of those, 83.6% (102/122) consented to be included in the patient portal. Of 20 patients not willing to use the patient portal, 18 patients stated a preference of questionnaire completion only at the hospital as the reason. The complete enrollment process is shown in Figure 3.

The full sociodemographic information and clinical data are given in Table 2. There were no statistically significant differences regarding age, sex, education, time since the initial diagnosis, and type of internet use between those who agreed and those who refused to participate (all P>.09). The age range of patients included in the portal was 39 to 83 years, and the age range of patients not included in the portal was 39 to 77 years. A statistically significant difference was found for general internet use (see Table 2), with participants who consented to use the portal reporting higher internet use than those who did not consent to use the portal: of patients who used the portal, 94% (94/100) reported using the internet at least multiple times per week compared to 76% (17/22) for patients who did not use the portal.

Interviews lasted between 10 and 30 minutes. Only 5% (5/102) patients required help from the PRO facilitator because they were not able to read their username and password (small font size). After the log-in and during the first interview, no patients reported technical difficulties or had difficulties navigating the portal or the questionnaire. Table 3 shows the number of completed questionnaires before the interviews and the reasons for noncompletion. Of the patients who participated in all 3 interviews, 37% (26/71) completed the questionnaires prior to the T2 and T3 interview, while the others completed the questionnaires only at the hospital visit.

Table 4 shows the use and evaluation of the portal as reported by patients who completed questionnaires prior to the interviews. Patients who completed questionnaires prior to their follow-up appointments reported using the portal in different ways: the percentage of patients reading additional portal content (eg, disease-specific information) declined over time from 71% (27/38) at T2 to 41% (13/32) at T3. The self-management tools were used and deemed useful by 50% (17/34) of patients at the second and 42% (13/31) at the third interview.

Patients were encouraged to provide additional comments following their answers to the interview questions. In the first interview, patients reported high satisfaction with the presentation of the results as bar charts.

Patients who did not inspect their own results expressed that they were feeling fine and therefore had no interest in viewing their results.

Similarly, patients who were feeling fine or reported no impairments did not inspect the self-management tools, while others provided positive feedback on the self-management tools or the color-coding of questionnaire results.

Four patients expressed the wish for a reminder (email or text message) before the next appointment to complete the questionnaires online. Two patients mentioned that they would like to be able to choose their password or username themselves (which the software currently does not allow). Three patients also explicitly reported a decline in motivation toward the end of the study, caused by a lack of sufficient feedback from physicians who did not discuss their PRO results during the consultations.

Over the study period, we registered a total of 796 sessions (ie, log-ins by patients). Of the patients who logged into the patient portal, 27% (28/102) logged in once, 18% (18/102) logged in twice, 9% (9/102) logged in 3 times, 15% (16/102) logged in 4 to 6 times, 14% (14/102) logged in 7 to 10 times, and 17% (17/102) logged in more than 10 times (total range of 1-57).

The mean duration of a session was 9.4 (median 6, range 1-90) minutes. It took patients on average 2.9 minutes to complete the EORTC QLQ-C30, 1.8 minutes to complete the EORTC QLQ-MY20, and 1.5 minutes to complete the EORTC QLQ-CLL17. This adds up to an average questionnaire completion time of 4.7 minutes for patients with MM and 4.4 minutes for patients with CLL.

During an average session, patients viewed 5.3 (median 4, range 1-28) different pages. Figure 4 shows how often specific components of the patient portal were viewed. A total of 3487 views were registered. The most frequently viewed self-management pages were those providing information on dyspnea (89 views), diarrhea (80 views), cognitive functioning (64 views), and emotional functioning (49 views). The least frequently viewed self-management pages were those providing information on obstipation (11 views), role functioning (12 views), social functioning (21 views), and pain (18 views). Multimedia Appendix 2 shows the view count on all self-management pages.

In our study, we evaluated the use of the patient portal in patients with MM and CLL participating in routine electronic PRO monitoring. We found that in those patients who were eligible to use the patient portal, the majority (84%) were willing to use the portal. Only a few users reported difficulties logging in, but no users reported problems navigating the survey. However, we found that considerably fewer patients (37%) than initially included adopted a continued use of the portal across all 3 study time points. On average, patients spent 9.4 minutes in the portal per session.

Our recruitment rate (ie, patients consenting to try the patient portal) is high compared to rates found in other studies. For example, other feasibility or usability studies report that between 21% and 64% [23-25] of patients were willing to use patient portals. Our high recruitment rate is most likely a consequence of the fact that patients were already participating in electronic PRO monitoring and so had already approved some form of monitoring. We would like to note the high inclusion rate for PRO monitoring at the hospital, which was, as reported previously, 94% of all MM patients treated in the department [18]. Our study emphasizes the possible synergy between assessments inside the hospital and home monitoring; patients who had already participated in electronic PRO monitoring inside the hospital were open to also using a patient portal. Moreover, the high acceptance of the patient portal might have been induced by the opportunity to get accustomed to ePROMs at the hospital with the help of a PRO facilitator before being introduced to the patient portal. We hypothesize that such a stepwise approach can reduce potential reservations about using electronic measures and contribute to user empowerment.

The adoption of home monitoring by a considerable proportion of patients also meant that those patients did not need to be assessed at the hospital. In this way, the patient portal reduced the administrative burden of assessing those patients’ PROs at the hospital and allowed us to allocate those resources elsewhere.

Over the course of the study, slightly less than half of the participants used the portal prior to their follow-up appointments. The main reason reported by patients was that they had forgotten to report their PRO data using the patient portal. Especially for patients with an infrequent appointment schedule (intervals between hospital visits of up to 12 months), forgetting to use the portal is, in fact, not surprising. This issue might be addressed by implementing an automated reminder (email or text message) as has been done with similar home monitoring systems [24,26]. For example, the AmbuFlex system in Denmark has, in the past, used different forms of reminders, including letters, emails, and text messages. They ultimately implemented communication via a national secure email program, which accounts for 93.2% of communication to patients and secures high completion rates [27].

Additionally, it is important to note that due to our routine monitoring approach, ePRO data from patients who did not use the portal prior to the follow-up appointments were not lost. Instead, those patients were invited to complete the ePROMs at the hospital as was done with other patients who did not use the portal.

While only a few of our patients directly mentioned a decline in motivation to complete ePROMs due to HCPs not picking up on the results during the consultations, this is in fact a frequent problem of PRO implementations in clinical settings [3,26,28,29]. For patients, there is little perceived benefit of completing the questionnaires if the results are not reviewed and discussed by HCPs. Instead, sharing their health status via questionnaires might even become burdensome. Therefore, it is important to engage HCPs with the concept of PROs and train them in the use of PRO results to prevent PRO data from becoming meaningless busywork that hinders clinical practice instead of enhancing it [28]. This requires the education and training of HCPs, which can be achieved by conducting specialized training programs [30]. A buy-in strategy may be used to increase HCP engagement with PROs, for example, via the adoption of the user interface to HCP needs and preferences, regular meetings, or analyses of PRO data in the registry upon request of HCPs. Another important approach is to make PRO data comprehensible and actionable for HCPs (eg, by providing advice on how to react to results) [26,31,32]. In our system, this is done, for example, by using thresholds for clinical importance for the EORTC QLQ-C30 [21] that facilitate the interpretation of patient PRO data.

Finally, the literature shows that in order to provide a sound theoretical basis for sustainable PRO solutions in routine care settings, an implementation science approach can be followed [33]. Implementation science can help identify barriers (which are often similar across contexts) and enablers (which often depend on the given hospitals’ context) of patient-reported outcome measure (PROM) implementations.

As has been found in other studies [13,17,25], the majority of our patients found the display of self-management advice linked with their results to be valuable. A recent review of electronic systems to measure PROs found that less than one-third (29%) of published electronic systems include features that provide tailored automated advice to patients, and less than half (41%) provide general educational information [17]. In a qualitative study evaluating another eHealth application, tailored feedback and advice was rated as appealing by most participants [34]. Participants valued the option of accessing information remotely between appointments and having a low threshold to receiving such information (compared to having to consult their treating physician or having to search for information) [34]. These are notions that echo the high number of views of pages with self-management advice we found in our study.

Nevertheless, despite the approval for self-management advice we found in our study, a recent clinical trial showed that, while patient portals can contribute to improved health-related quality of life [13], their measurable benefits on patient activation (knowledge, skills, and confidence for self-management) could not be shown in a diverse sample of cancer survivors [13]. However, previous evidence suggested that such effects may be more pronounced in newly diagnosed patients [25,35]. Another randomized controlled trial showed that weekly PRO monitoring along with the provision of tailored self-management advice, compared to usual care, significantly enhanced self-efficacy in patients with cancer [36].

Our study was designed to be carried out during routine care with minimal interruption of the clinical workflow. Therefore, interview time points were integrated into the patients’ hospital visits. This resulted in varying intervals between interviews (between 1 week and 1 year in a few cases), which may have influenced the results. However, we point out that these are real-world visitation schedules, and any application designed for routine care should be evaluated accordingly. Similarly, we included more males than females in our study. This reflects the epidemiology of MM and CLL, and the male/female ratio was comparable to data from the AMR.

Second, the CHES log data did not allow for detailed analysis of individual user page-view patterns but only for analysis of overall page views and duration and frequency of sessions. This means that patients logging on to the portal more frequently have a greater weight in the analysis of page views. Moreover, page view numbers have to be interpreted with care as they provide no information on whether the pages were actually read.

Another limitation is that the interviews were conducted by the authors (LN, PB, and JL), who introduced patients to the portal. Therefore, some patients might have been reluctant to voice criticism in the interviews, even though we actively encouraged patients to also report negative feedback.

Finally, we consider some sample-related limitations: We included only German-speaking patients in the patient portal. While the EORTC questionnaires are available in a large number of languages (and can be completed at the hospital [20]), translating and updating the content of the portal would require considerable resources. This results in a potential bias, and our findings may not be generalizable to patients with other first languages and limited German language proficiency. We are aware that this can create an imbalance in the provision of care, as this systematically excludes certain patient groups. In fact, patients not speaking the primary language of the country might profit most from receiving disease-specific information and from being able to report symptoms online in their native language. Moreover, we observed a selection bias due to patients’ age (patients with MM were on average 11 years younger than the mean age of patients in the AMR), as older patients might be less proficient using the internet and were consequently not included in our study. While this is an important limitation to consider, its impact should decrease over time, as the population’s internet capabilities have been steadily increasing [37] and should further increase in the years to come.

Following the insights gained in this study, we are currently in the process of updating our monitoring procedure and software. One important step is to more closely monitor patients’ PROMs to swiftly identify deteriorations and act accordingly; we aim to have a trained onconurse monitor patient results and check on patients in case of deteriorations. If necessary, the nurse can schedule an earlier appointment or alert the treating HCP. Another planned improvement is to implement an email or text message reminder to improve PROM completion rates.

Our study shows that a patient portal enabling remote PRO data assessment can complement routine electronic PRO implementation at the hospital by reducing the burden of administration for the clinical team and offering an additional way for patients to engage with PROs. We found that the majority of patients were open to using the patient portal and interested in assessments from home. The low number of technical problems and absence of complaints demonstrate the general user-friendliness of our portal. While initial uptake was high, fewer patients adopted regular use of the portal prior to their appointments. To increase long-term participation rates, further motivational (eg, increasing HCP engagement with the PRO data) and technical (eg, email reminder) measures are needed.