This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The benefits of electronic patient reported outcomes (PRO) questionnaires have been demonstrated in many settings, including in hospitals and patient homes. However, it remains to be investigated how melanoma patients and their treating clinicians experience the electronic self-reporting of side effects and the derived communication.
The primary objective of this study was to examine patients’ and clinicians’ experiences with an eHealth intervention for weekly monitoring of side effects during treatment with immunotherapy.
An eHealth intervention based on questions from the PRO-Common Terminology Criteria for Adverse Events (CTCAE) library was used and tested in a randomized clinical trial with patients receiving immunotherapy for malignant melanoma and clinicians at a university hospital in Denmark. On a weekly basis, patients reported their symptoms from home during the treatment via a provided tablet. The electronic patient reports were available to clinicians in the outpatient clinic. A mixed methods approach was applied to investigate the patients’ and clinicians’ experiences with the intervention. Data from patient experiences were collected in a short survey, the Patient Feedback Form. Moreover, a subset of the patients participating in the survey was interviewed about their experience. Furthermore, one focus group interview with clinicians was carried out to elucidate their views.
A total of 57 patients completed the Patient Feedback Form, and 14 patients were interviewed. The focus group interview included 5 clinicians. Overall, patients and clinicians were satisfied with the tool. They believed it enhanced patients’ awareness of side effects and increased their feeling of involvement. The patients reported that it was easy to fill out the questionnaire and that it made sense to do so. However, a minority of the patients expressed in the interviews that they did not believe that the health care professionals had seen their reports when they came to the clinic, and that the reporting did not lead to increased contact with the department.
Overall, satisfaction with the eHealth intervention was high among patients and their treating clinicians. The tool was easy to use and contributed to greater symptom awareness and patient involvement. Thus, in terms of patient and clinician satisfaction with the tool, it makes sense to continue using the tool beyond the project period.
ClinicalTrials.gov NCT03073031; https://tinyurl.com/tjx3gtu
Underreporting of symptoms by clinicians in connection with cancer therapy, particularly chemotherapy and radiotherapy, is well established [
The use of electronic PRO questions (ePROs) to monitor symptoms has proven to be feasible in connection with scheduled consultations (ie, in the waiting area in various oncology settings) [
An exploratory endpoint of PROMelanoma was to examine whether our setup of including an eHealth intervention on symptom management is implementable in clinical practice and makes the patients feel more involved in their treatment and care. Patient and clinician satisfaction with various eHealth interventions has been measured in other studies within an oncology setting to support clinical decision making and improve patient self-management [
However, there is no recipe for measuring the patient experience, and measurement is not routinely conducted in a standardized manner [
Thus, the primary objective of this study was to examine, using both qualitative and quantitative data, patients’ and clinicians’ experiences with an eHealth intervention to monitor the side effects during treatment with immunotherapy in routine clinical practice.
A mixed methods approach was employed to gain deeper insight into the feasibility of the PRO intervention for melanoma patients and their treating clinicians. For quantitative assessment, a questionnaire to measure patient satisfaction, the Patient Feedback Form [
Overview of the mixed methods study design, including a survey, individual interviews, and one focus group interview.
The survey and interviews took place at the Department of Oncology, Odense University Hospital, Denmark. The patients completed the Patient Feedback Form when they came to the outpatient clinic to receive their treatment for metastatic melanoma. The interviews also took place in the outpatient clinic in a separate room.
Common Terminology Criteria for Adverse Events (CTCAE), developed by the National Cancer Institute (NCI) for patient self-reporting [
Example of part of a patient report available to clinicians.
Patients were eligible for the qualitative part of the study if they had been enrolled in the RCT PROMelanoma. The inclusion criteria were melanoma patients, >18 years old, randomized to the intervention in PROMelanoma, and had received at least one cycle of immunotherapy. Exclusion criteria were not able or willing to comply with the study procedure (eg, fill out the electronic questionnaire) or if they did not speak Danish.
All patients in the PROMelanoma intervention group of the trial were asked to fill out the Patient Feedback Form between January 2017 and April 2019, which addressed patient satisfaction relating to the eHealth intervention. The Patient Feedback Form was developed by Basch et al [
Patients enrolled in the PROMelanoma study were contacted over the phone by the project manager and informed about this study between November 2017 and June 2018. The patients provided verbal consent and signed the written consent form in connection with the interview. We decided to use a convenience sample at the same time, taking into account the patients’ gender and age to ensure that the group was representative. The patients already had several visits scheduled in the outpatient clinic; therefore, the interviews were planned to take place on days when they were already at the hospital so as to not burden them further. If the patients were accompanied by relatives, the relatives were invited to participate in the interview. A semistructured interview guide was prepared based on the research questions, in collaboration with an expert. The interviews were carried out by the same interviewer (LT) who also carried out audio recording and transcription. The interviewer had already talked to the majority of the patients during the inclusion screening for the PROMelanoma study, but had no contact otherwise. Given that we had some knowledge about the research area in question (ie, the interviewer had worked with this patient group for more than 10 years), there were four major categories that we wished to explore: the usefulness of the eHealth solution, the questionnaire, physician-patient communication, and involvement of relatives. Thus, a directed content analysis as suggested by Hsieh and Shannon [
A focus group interview was chosen as the preferred method for clinicians, because the number of physicians and nurses caring for these patients was limited to a selected group, which made a questionnaire pointless. For the same reason, only one interview was conducted. The physicians and nurses who had the most experience with the intervention were chosen for the interviews. One author (KD) carried out the interview, who is a qualified researcher experienced in conducting focus group interviews. The interview was conducted in a semistructured manner [
All patients who were randomized to the intervention arm in the PROMelanoma study (N=70, median age 65 years, 33 men and 37 women) were expected to evaluate the eHealth intervention by filling out the Patient Feedback Form (
Evaluation of the eHealth intervention PROMelanoma in a Danish study with patients with melanoma cancer (N=57).
Patient feedback form item | Response, n (%) | ||
|
Category 1 | Category 2 | Category 3 |
1. Time it took to complete | 1 (2)a | 54 (94)b and 0 (0)c | 2 (4)d |
2. Number of times completing | 1 (2)e | 54 (94)b and 1 (2)f | 1 (2) d |
3. Easy to complete | 56 (98)g | 1 (2)h | 0 (0)d |
4. Completing was useful | 55 (96)g | 2 (4)h | 0 (0)d |
5. Easy to understand | 53 (93)g | 4 (7)h | 0 (0)d |
6. Easier to remember symptoms and side effects | 52 (91)g | 4 (7)h | 1 (2)d |
7. Improved discussions with clinician | 51 (89)g | 4 (7)h | 2 (4)d |
8. Clinician used information for my care | 48 (84)g | 6 (11)h | 3 (5)d |
9. The quality of care improved because of the questionnaire | 43 (75)g | 8 (14)h | 6 (11)d |
10. Communication with clinician improved | 45 (78)g | 6 (11)h | 6 (11)d |
11. Made me more in control of care | 50 (87)g | 6 (11)h | 1 (2)d |
12. Recommend to other patients | 57 (100)g | 0 (0)h | 0 (0)d |
13. Would like to continue responding | 57 (100)g | 0 (0)h | 0 (0)d |
aToo short.
bJust right.
cToo long.
dMissing.
eNot often enough.
fToo often.
gStrongly agree/agree.
hDisagree/strongly disagree.
In addition to filling out the Patient Feedback Form, 16 of the patients were invited to participate in an in-depth interview about their experience. One patient declined and one patient who had agreed to participate was hospitalized due to deteriorating disease before the interview was conducted. Thus, 14 interviews were conducted. The median age of the patients was 67 years (range 41-79 years), including 6 men and 8 women. Apart from one patient who had only self-reported their symptoms 3 times, the patients had reported between 6 and 24 times (weeks), and the majority (10, 71%) had reported more than 15 times. Relatives were present during 10 of the interviews. The interviews lasted on average 20 minutes (range 9-33 minutes). Nine interviews lasted for more than 20 minutes. A total of 280 minutes of interview data were available for analysis. The three themes identified from the transcripts aligned with three of the predetermined categories. However, a fourth theme (involvement of relatives) did not become a theme when the final analysis was carried out.
Overall, the patients reported that accessing and filling out the eHealth questionnaire was easy. Only two patients were not used to electronic devices upon entering the study. One of them stated “I’m pleasantly surprised. I think it is really easy to deal with” (man, 79 years old), and his wife (73 years old) added, “I did not think he could do it because he is a clown when it comes to computers…” Some of the patients, particularly the elderly, had a hard time using the touchscreen function with their fingers because they either pressed too hard or for too long. However, when they were given a touchscreen pen, which is more accurate than the fingertip, they did not have any problems. Only one patient could not do it and asked his wife to do the reporting following his instructions. Almost all of the patients experienced a request to update the operating system of the tablet while using it, but they were able to close the message easily and continued their reporting. Otherwise, there were only minor technical challenges, and the patients were very compliant and contacted the department in case of any technical problems. The majority of patients were pleased with the tablet. Only a few patients would have preferred a link instead of having to take home the tablet. As mentioned above, it was not possible to send a text message reminding the patient to fill out the questionnaire on the relevant days. However, this did not constitute a problem for the patients, who found it easy to remember because they were doing it on a fixed weekday. Two patients mentioned that a reminder text message would have been advantageous.
The patients reported that the number of items and the length of the questionnaire were appropriate and that reporting on a weekly basis was fitting. A few of the patients would have liked a free text field where they could write a comment or elaborate if the questionnaire did not adequately cover existing symptoms:
When the patients came to the outpatient clinic, two out of three of the patients who were interviewed felt that the health care professionals had in fact seen their reports and included them in their consultation: “It is like having an agenda for a meeting” (man, 66 years old)
Many of the patients explained that a strong motivation for entering the study was that they would be able to help future patients. Of course, they believed that they themselves would benefit, but being able to help others was also important. Including relatives in the reporting was not a theme. The patients did the reporting alone, apart from one patient, and it did not prompt any discussions within the family.
The participants in the focus group consisted of three doctors and two nurses. They were all women with a median age of 43 years. All of them had broad experience working with cancer patients and dealing with symptoms or side effects (6-11 years). They were also accustomed to caring for melanoma patients receiving immunotherapy. They had all seen the patient reports several times and had included them in the clinician-patient communication.
There was some discrepancy between how the patient and the clinician graded a given symptom. In some cases, the clinician did not find the symptom to be as severe as the patient. In other instances, the clinician felt that the patient had in fact neglected a symptom that they believed should have been reported: “sometimes there’s a discrepancy between what you find out when you talk to the patient and what has been reported … the two things supplement each other” (physician).
Furthermore, the inclusion of patient reporting was seen as being more time-consuming than a typical consultation due to the fact that the clinicians had to log into another system to see the report. Having the reports integrated in the electronic health records (EHRs) was stated not only to save time but also make it much easier to remember to include them in the consultation.
The clinicians agreed that the patients were better prepared when they came to the outpatient clinic, and that the patients had increased focus on their symptoms and were more alert: “I think it is an advantage that the patients become more aware of the side effects that can occur” (nurse). Moreover, the information on toxicity that had been given to the patients prior to treatment start was repeated when the patients responded to the electronic questionnaire at home. Accordingly, there was a better chance that the patients would react appropriately by contacting the department in time instead of waiting for the next scheduled consultation, which might be days or weeks ahead. Thus, having the patients call more often was seen as an advantage because it might enable earlier detection. Moreover, it was an advantage to be able to use the patient reporting as the basis of the consultation by starting with the symptoms that had bothered the patient the most: “…then I scroll down to see where it is red or yellow and that is typically where we start…” (physician). In this way, the patients took part in setting the agenda. However, according to the health care professionals, the patient reporting should be seen as a supplement and not something that could replace the clinician-patient consultation. In addition, the clinicians reported that the eHealth intervention was a valuable tool, particularly for patients who are normally slightly reluctant to contact the department unscheduled: “…it may be precisely the group of patients who are not good at self-care or at least some of them…the weakest patients who…will benefit most from self-reporting by being guided into becoming more aware of when to react to symptoms” (physician). Because the patients were encouraged to make contact if they experienced a new or worsened symptom, they might have felt that it was more legitimate to call the outpatient clinic. All of the clinicians believed that the patients with the best social resources would benefit the least from the intervention because they were sure to contact the department in agreement with the given instructions.
Overall, the clinicians had a positive attitude toward the intervention using an eHealth tool, even though there was also room for improvement in some areas.
The clinicians believed that the reporting would make the patients call the hospitals more, whereas the majority of patients did not think that they called more frequently. Some of the patients thought that their reports did not provide the clinicians with enough information; however, none of the clinicians stated this to be the case. Patients and clinicians agreed that the attention to side effects was increased and that the patients were better prepared for the consultation when they came to the outpatient clinic. The patient reports also established a shared agenda for the consultation at the outpatient clinic. Overall, the findings from the survey confirmed what had been established in the patient interviews. The patients reported that it was easy to fill out the questionnaire and that it made sense to do so. Moreover, it increased symptom awareness. Both the patients and clinicians agreed that when the report was in fact included, it helped to prioritize the problems that were most acute.
The goal of this study was to elucidate the experiences of malignant melanoma patients and their treating clinicians with an eHealth intervention. Overall, acceptance was high for both clinicians and patients, and both groups believed that it improved communication during the consultation. This is in line with previous studies showing that using PROs prompted patient-clinician dialog, streamlined consultations, and increased focus on side effects [
However, a minority of the patients in this study did not believe that the clinician had actually seen their reports when they came to the clinic. This point was primarily expressed by patients who were enrolled at the beginning of the study, when monitoring the patient reports had not yet become routine in the outpatient clinic. This improved over time as the clinicians got used to taking the reports into consideration. This finding is in line with Mooney et al [
As for the survey, patient satisfaction was extremely high for many of the questions. The three items that had the lowest scores in satisfaction (items 8, 9, and 10) deal with the inclusion of patient response in the clinic. This response is comparable with the results of other studies using the Patient Feedback Form [
The clinicians participating in the focus group interview agreed that the least resourceful patients would benefit most from the eHealth intervention, because they were usually less inclined to contact the clinic in case of any symptoms. This notion has been confirmed in other studies, which have shown that the level of patient involvement is dependent on the degree of health literacy. For example, patients with a high level of education are more inclined to be involved in medical decision making compared to patients with a low level of education [
Some of the patients also argued that the eHealth intervention was very box-like and they would have liked a space where they could write more about their symptoms instead of just checking a box. The patients in the PROMelanoma study can add other symptoms as advised by the NCI, but the patients also wished to be able to elaborate on some of the symptoms. Although this is understandable from a patient point of view, one must keep in mind that the primary aim of introducing the intervention was to increase patient awareness, hoping to reduce the number of severe side effects and improve clinical outcome. Further, it was important that it was fairly easy and not too time-consuming for the clinicians to acquire a quick outline of the reporting if it were to be implementable in the clinic. Moreover, patients had the opportunity to elaborate on the various symptoms that they experienced when they came to the clinic.
One potential limitation of the study is that a deductive approach was used by having the coding framework decided in advance, which may limit the development of new themes [
An obvious limitation is that we were only able to conduct one focus group interview with the clinicians. However, we aimed at selecting participants with vast knowledge and expertise of the subject [
Another potential limitation is that the alert function was triggered too frequently according to the majority of patients. This may be changed when designing future studies or implementing the intervention beyond the study period to avoid alert fatigue. Nevertheless, having an alert function is a good idea, as studies have shown that patients value advice on when it is appropriate to contact the hospital [
We found a high acceptance of the eHealth intervention tool among clinicians and melanoma patients being treated with immunotherapy. The tool was easy to use and contributed to greater symptom awareness and patient involvement. Thus, in terms of patient and clinician satisfaction, it makes sense to continue using the tool beyond the project period. However, it remains to be investigated whether the predominantly positive perceptions of the intervention by patients and clinicians will also be followed by a reduction in the number of severe side effects. Our RCT PROMelanoma will shed light on this aspect.
Consolidated Criteria for Reporting Qualitative Research
Common Terminology Criteria for Adverse Events
electronic health record
electronic patient reported outcomes
National Cancer Institute
patient reported outcomes
randomized controlled trial
We would like to thank the patients who participated in interviews and the survey, and–despite difficult circumstances–contributed to this study. We are also grateful to the physicians and nurses for sharing their experiences and views. We also thank Niels Henrik Hjøllund, WestChronic, Occupational Medicine, University Research Clinic, Aarhus University, who provided technical support, making it possible to carry out the study. Finally, we acknowledge William Frost, Last Word Consultancy, for language editing. The study is funded by the Danish Cancer Society and the Region of Southern Denmark. None of the funding organizations was involved in reviewing or approving the manuscript.
None declared.