The study took place at the hematological outpatient unit of Internal Medicine V and the Comprehensive Cancer Center Innsbruck at the Medical University Hospital Innsbruck, Austria. PROMs are administered through the Computer-based Health Evaluation System (CHES) using a remote, web-based patient portal, in which patients complete PROMs before and during patient visits at the outpatient unit [21]. In the portal, patients can access their scores and corresponding self-management advice [20]. The PROMs assessment is part of the Austrian Myeloma Registry (AMR) [19]. The existing program was modified as follows: (1) monitoring before clinical encounters every few weeks was replaced by scheduled weekly monitoring with item lists tailored to treatment regimens, (2) reminders to prompt completion were installed, and (3) a trained onco-nurse regularly screened the PROM reports and was instructed to act upon clinical alerts triggered by the PROMs. See Multimedia Appendix 1 for detailed explanations of the monitoring program.

We conducted a 1-year observational longitudinal study in which patients were recruited consecutively during the study period. Patients remained on study until the predefined end date (June 30, 2022, although monitoring continued after the study) or their systemic therapy ended. The study was registered on ClinicalTrials.gov (NCT05036863) and was prepared in accordance with the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guideline (see Multimedia Appendix 2) [22].

We approached patients diagnosed with multiple myeloma treated at the outpatient unit. Patients were briefed and asked to sign written informed consent. Patients were eligible if they:

We aimed to include all eligible patients willing to participate during the 1-year study period. Following a study board discussion on November 8, 2021, the board decided to consider the latter 2 inclusion criteria optional: patients willing to participate despite reporting rarely using the internet could participate with help from relatives. Their potential benefit from the monitoring was deemed important and outweighed the extra time and assistance the electronic patient-reported outcome (ePRO) facilitator (see below) had to invest in these patients to set them up for participation.

Patients were introduced to the program by an “ePRO facilitator,” a study assistant with extensive knowledge of PROMs. Patients received patient portal login data and were instructed to complete 1 PROM assessment each week on the internet. Additionally, if patients experienced new symptoms, they could complete additional assessments at any time. If patients did not complete assessments for 7 days, automated email or text message reminders were sent. A trained onco-nurse regularly screened the completed PROM reports and acted upon clinical alerts that had been triggered based on the PROM reports. See Multimedia Appendix 1 for more information on the study procedure, PROMs used, and clinical alerts.

In brief, the symptom monitoring consisted of symptom item lists composed of the European Organisation For Research And Treatment Of Cancer (EORTC) Item Library [23], developed following an expert-based approach. Item lists covered symptoms most relevant to patients with multiple myeloma, including a core symptom set (covering pain, fatigue, nausea, vomiting, constipation, diarrhea, polyneuropathy, emotional symptom burden, sleep disturbances, and global health) and symptoms and domains specific to patients’ treatment (eg, rash, edema, fever, or adherence to oral anticancer medication). Additionally, at baseline and every 6 weeks, comprehensive, nontreatment-specific assessments were administered, which comprised the EORTC QLQ-C30 [24] and the EORTC QLQ-MY20 [25]. For all items, Answer options range from “not at all” to “very much” (4-point Likert scale, standard EORTC item answer categories).

Clinical alerts were triggered for symptoms that exceeded 50 points on a 0 to 100 scale, with higher scores indicating higher symptom burden. This threshold corresponds to a symptom burden of “quite a bit” or “very much” (see Multimedia Appendices 1 and 3 for more information) and is in line with previous similar research [26]. For symptoms measured in the EORTC QLQ-C30, we applied the “thresholds for clinical importance” [27] to trigger alerts. In response to clinical alerts, the onco-nurse documented interventions (eg, “called patient - referred to physician”) or could register that symptoms were already documented and further intervention was not necessary.

Patients received a program evaluation questionnaire after either (1) being in the symptom monitoring program for ≥6 weeks and having completed at least one remote assessment, or (2) completing 3 PROM assessments, whichever occurred first. The paper-pencil evaluation questionnaire was distributed marked with a pseudonymized ID in a prestamped envelope. The questionnaire contained questions on patients’ perceptions of the program and how it impacted their care (see Multimedia Appendix 3).

See Table 1 for the study outcomes. Primary outcomes were the assessment completion rate and patients’ perceptions of the program. Secondary outcomes were the frequency and nature of clinical alerts and interventions initiated by the onco-nurse and health care team and the data completeness of the item lists.

Patient characteristics were extracted from the electronic health records. Data were analyzed as absolute numbers with percentages and as either means with SD if data were normally distributed or medians and IQR if not normally distributed (tested using the Shapiro-Wilk test). The results from the program evaluation questionnaire, the item lists of response completeness, and the frequency of clinical alerts and interventions by the onco-nurse are reported as absolute numbers and percentages.

The completion rate is a function of the number of completed assessments divided by the total number of weeks patients were included in the study (number of days in the study divided by 7) and reported as median and IQR. To prevent an inflated completion rate, assessments that were less than 5 days apart were considered a single assessment and counted once toward the completion rate.

The study procedure and use of data are covered by the ethics approval for the AMR by the ethics committee of the Medical University of Innsbruck (number AN3252 266/4.2 386/5.14).

During the study period (July 1, 2021 to June 30, 2022), 55 patients were approached; 39 (71%) patients consented to participate (Figure 1). Four patients stopped the intervention early due to recurrent internet problems at home (n=1), personal reasons (n=2), and the termination of therapy (n=1, patient excluded from study after treatment termination). Patients were on the study for a median of 316 days (IQR 232-346 days). There were 24 (62%) male patients. Patients had a mean age of 62.8 years and a median of 1534 (IQR 695-2968) days since diagnosis. Most patients (n=20, 51%) were in their first line of therapy. The most frequently used therapy was lenalidomide maintenance therapy, received by 13 (33%) patients. During the study period, 3 patients switched therapies (Table 2). As part of their past treatment (ie, before being included in the study), most patients (32/39, 82%) had previously undergone stem cell transplantation (SCT). Of those, 97% (31/32) were autologous SCT, with only one (1/32, 3%) allogenic SCT; 3 patients (3/32, 9%) had received 2 autologous SCT. No patient received a SCT during the study period.

The 39 included patients completed 1047 questionnaire assessments (808 assessments with treatment specific item lists and 239 assessments with the EORTC QLQ-C30 and MY20). Of those, 13 assessments were excluded from the completion rate calculation as they were less than 5 days apart from another assessment. Patients completed an average of 26.7 (SD 13.4) unique assessments.

The median weekly assessment completion rate was 70.3% (IQR 41.2-89.6). During the study, it became apparent that weekly monitoring was too frequent for a few patients (eg, too burdensome, not appropriate for oral therapy which can have fewer side effects). To optimally support these patients, the study board decided to adjust the monitoring intervals for them. Two (5%) patients switched to a 2-weekly monitoring interval, 2 (5%) patients switched to a 3-weekly monitoring interval, and 3 (8%) patients switched to a monthly monitoring interval. However, we still considered a weekly interval for the completion rate for all patients as prespecified in the trial registration.

Over the course of the study, patients showed stable PROM assessment completion; additional analyses (see Multimedia Appendix 4) shows that the number of days between individual’s assessments did not increase significantly. In other words, patients did not tend to space out assessments further the longer they were longer in the study and mostly adhered to the 7-day assessment schedule.

Despite having the option to skip answers in the questionnaires, patients rarely did so; below 1% of the answers were missing for all item lists.

The program evaluation questionnaire was completed by 36/39 (92%) participants. Two (5%) patients did not receive the questionnaire as they did not complete assessments in the first 6 weeks; 1 (3%) patient did not return the questionnaire. The results are summarized in Figure 2 (for complete results, see Tables S1 and S2 in Multimedia Appendix 5).

Patients were “not at all” (75%) or only “a little” (25%) burdened by regularly completing questionnaires. Most patients (78%) felt at least “a little” to “very much” safer due to the regular PROM assessments, and 55% of patients reported a feeling of security instilled by the follow-up phone calls. In total, 78% felt that the health care team was better informed on their health status due to the monitoring program. All (100%) patients felt that contact by phone was suitable. Most patients (87%) reported that the phone calls encouraged them to complete questionnaires.

Table 3 shows the number of times each symptom was reported during the weekly monitoring, if the clinical alert threshold was exceeded, and if the symptom or consequential intervention were already documented. In total, 483/1041 (46.4%) PROM assessments included at least one symptom exceeding the thresholds. Clinical alerts were triggered for 1758 of 14,639 (12%) reported symptoms (multiple symptoms from a single assessments could trigger alerts).

The symptoms appetite loss and financial difficulties from the EORTC QLQ-C30 were not included in this analysis as they were not part of the weekly item list assessments and deemed of lower clinical relevance. The thresholds for clinical alerts were according to the Thresholds for Clinical Importance by Giesinger et al [27] for assessment using the EORTC QLQ-C30 and scoring >50 points (0-100 scale) for all other assessments or scales.

In 196 assessments, patients reported “other symptoms” in the free text item. Table S1 in Multimedia Appendix 6 shows the categorized answers; most other-symptom-reports were on additional symptoms not covered by the questionnaires, or the open answer item was used to communicate additional information to the onco-nurse.

Table 4 shows the documented responses to individual clinical alerts by the onco-nurse. For 31.2% (548/1758) of clinical alerts, the specific symptom was already documented, remained unchanged, and no further intervention was initiated based on clinical judgement by the nurse. In 19.9% (348/1758) of clinical alerts, the nurse consulted patients about their respective symptoms via phone without initiating further clinical actions. For 5% (88/1758) of clinical alerts, the nurse referred patients to the treating physician or supportive care specialists, and for 1.3% (22/1758) of them, a medication change was documented. In response to 0.4% (7/1758) of clinical alerts, an unplanned emergency visit and for 0.5% (9/1758) of additional diagnostics (eg, computerized tomography) were scheduled. Anecdotally, we report that 2 cancer recurrences were identified following diagnostics scheduled due to remote symptom reports.

Our study demonstrates that high completion rates can be obtained over a longer period of time for weekly web-based PROM monitoring in routine clinical care for patients with Multiple Myeloma. During the study, 39 patients completed just over 1000 assessments, with a median completion rate of 70%. Patients generally valued the weekly assessments and reported that the monitoring offered a sense of added security during therapy. Nurse-initiated interventions ranged from consultation and referral to self-management advice to issuing prescription orders or, in rare cases, emergency referrals.

This study was a continuation of our previous work [19,20], in which initial patient adherence to PROM monitoring was high but declined over time. We hypothesized that sending out reminders, active screening of the PROMs, and nurse-initiated clinical interventions could improve patient engagement, as reported in other studies [11,28]. We believe the implemented changes improved patient engagement and, importantly, clinical value and patients’ perceptions of the monitoring. Our completion rate is on the higher end of those reported in existing research and, often larger, clinical trials evaluating ePRO interventions: completion rates commonly vary from 41% [29] to 64.7% [16] and up to 91.5% [15]. Patients with multiple myeloma are a relatively older patient population [30]. Lower technology literacy in older people [31] may contribute to lower completion rates. Investing more resources in patient follow-up and providing alternatives to internet-based completion (eg, interactive voice response systems) may result in higher completion rates [15]. Specifically for patients with multiple myeloma, the continuous outpatient treatment at the hospital compared to transfer to follow-up care after active treatment for other types of (solid) cancers may create more relevance for long-term participation.

Our study complements evidence from large-scale clinical trials with diverse populations that provide evidence of the benefits of PRO monitoring systems [8,14,29,32], with important nuanced information on how such a monitoring system can function in routine clinical practice for patients with multiple myeloma. Contrary to clinical research settings, evidence of benefits from trials in routine clinical practice settings is just starting to emerge in the literature [33]. The symptoms that triggered most alerts in our study were dyspnea, general pain, fatigue, and back pain, which is generally similar to other studies using ePRO monitoring in populations with various diagnoses [15,34]. Specifically, back pain is a common symptom in patients with Multiple Myeloma that is linked to poorer overall health and functioning [35]. This finding underlines the value of using disease-specific questionnaires that are tailored to the symptoms patients are likely to experience, which likely contributed to patients’ satisfaction with the monitoring.

In our study, an onco-nurse was employed to review the assessments and contact the patient, or ask the physician to do so. Almost 80% of our patients felt that the health care team was better informed on their health status due to the questionnaires. A similar intervention in the larger “eSMART” randomized controlled trial for patients with different types of cancers during chemotherapy [32] also generated alerts from remote PROM assessments for mild to moderate symptoms that needed to be addressed in 8 hours and emergency alerts, for which a response time of 30 minutes was prescribed. Clinicians adhered to this in 95% and 85% of alerts, respectively. These are laudable response rates, especially for critical alerts, but they were likely also the result of a resource- and effort-intensive approach (eg, required dedicated site mobile devices for the alerts and shifts to allow the response time). Our monitoring program was adapted to the outpatient unit workflow and clinical practice, which did not have resources for 24-hour active monitoring. Nurse responses to patients’ symptom reports were neither timed nor continuously screened, and patients were made aware of that. Still, the monitoring resulted in many clinical interventions that arguably improved clinical care for participating patients, which might have been delayed (or not have happened at all) without the monitoring. Notably, following additional imaging that was initiated due to clinical alerts, 2 cancer recurrences were identified.

An interesting comparison can be made to other symptom monitoring programs that also rely on nurse-initiated telephone follow-up calls (not prompted by critical PROM responses on a web-based platform). For example, patients in the CAPRI (Impact of a Monitoring Device for Patients With Cancer Treated Using Oral Therapeutics) trial [29] who received symptom monitoring were also regularly called by nurses, even if they had not reported critical symptoms in PROMs. Most interventions that were initiated here were the result of these regularly scheduled follow-up calls. This approach is more resource intensive than our approach, in which patients were only called following PROM-triggered alerts. More research is needed to evaluate best practice approaches and determine the optimal balance between regularly calling patients and relying mostly on PROM-triggered clinical alerts.

Importantly, our findings show that the most frequently documented interventions were low-effort interventions, such as documenting an already-known symptom or calling the patient and referring them to the self-management advice for mild symptoms. This is similar to other studies, where the most frequent clinical responses were telephone consultations with the patient or a caregiver and subsequent referrals to self-care [15,16,29,32,34]. Arguably, this is already a clinical intervention in itself, as it can be important and reassuring for patients to have their symptoms acknowledged by the health care team, even if no immediate clinical action is required. More than half of patients (55%) reported that the calls that were made in response to alerts (and hence the subsequent interventions) gave them at least “quite a bit” to “very much” of a sense of security. Another notable finding was that for some diagnosis-specific (eg, back pain) and treatment-specific (polyneuropathy) symptoms, interventions were initiated more frequently. For example, while polyneuropathy was not a common symptom during monitoring (clinical alerts were sent only for 7.7% (60/781) of symptom reports), further clinical interventions were initiated for 43.3% (26/60) of those alerts. Patients may develop treatment-induced polyneuropathy during some types of systemic treatments, which can have debilitating effects on patients’ quality of life [36-38]. It is important that polyneuropathy is recognized as soon as possible by the treatment team to be able to initiate the necessary adjustments to the treatment and prevent long-term neurotoxicity. A remote symptom monitoring like ours may help with the timely identification of such key symptoms during outpatient therapy and, if possible, should therefore also be constructed to capture important therapy-specific side effects.

Clinically more complex interventions in coordination with the treating physician were necessary only in a few cases, meaning that most clinical alerts can be resolved by nurses who have dedicated time to screen the results. This is similar to another study in routine care where only 6.4% of calls initiated by nurses based on ePRO monitoring required an office evaluation within 72 hours of the report [34]. These findings highlight the crucial role of nurses in remote symptom monitoring programs to screen and triage patients’ symptom reports. However, our findings also showed that for some clinical alerts, the nurse failed to document a response more frequently than for other domains. Informal study debriefing interviews revealed that these lower response rates resulted from either a lack of documentation (ie, having discussed the alert with the patient but not documented it) or an uncertainty regarding how specific alerts should be handled. The latter was mostly the case for alerts for emotional burden; we have since improved our clinical workflow to incorporate a standardized clinical screening and pathway into the program. Creating clear pathways for what action should be taken in response to which alerts, which may differ by domain, can help with standardizing and improving care. In that way, nurse-initiated telephone screening can separate urgent care needs requiring immediate physician attention from less severe symptoms that may be managed with the help of self-management advice [33]. However, this ultimately requires staff resources and should be integrated into provider payment models [39].

When completing PROMs, patients expect that their scores will prompt the health care team to take appropriate action to manage their symptoms [40]. While results were seldom directly discussed by treating physicians, the calls and interventions initiated by the nurse in response to the clinical alerts were perceived as important by patients. For more than 3 quarters of patients, the regular assessments and calls added “a little“ to a “very much” feeling of safety, and patients felt that the health care team was better informed on their health status. This compares nicely to the 77% of patients in a similar but large-scale implementation reporting that PROM monitoring made them feel in greater control of their care and the 72% of patients who felt that it improved discussions with the care team [15]. Such findings highlight the value that ePRO monitoring brings to clinical practice from the patient perspective.

This study has several limitations. First, the monocentric study design limits the generalizability of our findings, and the sample size does not compare to larger clinical trials. Acknowledging these limitations, we also consider the study design that was tailored to 1 specific hospital unit a partial strength of our study; it allowed us to focus on the evaluation of the monitoring program in actual clinical practice and to generate insight into the detailed clinical interventions that resulted from the PROM monitoring with high-quality longitudinal data. A second limitation is that technology literacy is an important patient-level factor for PROM completion [31]. We initially included only patients with sufficient internet access and knowledge, which might have introduced bias in the sample. However, computer literacy is changing in elderly populations, and this effect will facilitate implementations in the future. Finally, even though we investigated interventions initiated in response to patient symptom reports, we cannot draw conclusions about the effects of the monitoring on outcomes such as symptom management or survival, for which randomized controlled trials are necessary.

Our study shows that high adherence to weekly, treatment-specific PROM monitoring can be achieved in routine clinical care over longer time periods. Our findings offer valuable insight into how the monitoring shaped clinical practice, how patients with multiple myeloma perceive such a program during their care, and describe first-hand experience of the clinical management and interventions that result from using such a monitoring program in routine clinical practice.