This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
Due to the increasing use of shared decision-making, patients with esophagogastric cancer play an increasingly important role in the decision-making process. To be able to make well-informed decisions, patients need to be adequately informed about treatment options and their outcomes, namely survival, side effects or complications, and health-related quality of life. Web-based tools and training programs can aid physicians in this complex task. However, to date, none of these instruments are available for use in informing patients with esophagogastric cancer about treatment outcomes.
This study aims to develop and evaluate the feasibility of using a web-based prediction tool and supporting communication skills training to improve how physicians inform patients with esophagogastric cancer about treatment outcomes. By improving the provision of treatment outcome information, we aim to stimulate the use of information that is evidence-based, precise, and personalized to patient and tumor characteristics and is communicated in a way that is tailored to individual information needs.
We designed a web-based, physician-assisted prediction tool—Source—to be used during consultations by using an iterative, user-centered approach. The accompanying communication skills training was developed based on specific learning objectives, literature, and expert opinions. The Source tool was tested in several rounds—a face-to-face focus group with 6 patients and survivors, semistructured interviews with 5 patients, think-aloud sessions with 3 medical oncologists, and interviews with 6 field experts. In a final pilot study, the Source tool and training were tested as a combined intervention by 5 medical oncology fellows and 3 esophagogastric outpatients.
The Source tool contains personalized prediction models and data from meta-analyses regarding survival, treatment side effects and complications, and health-related quality of life. The treatment outcomes were visualized in a patient-friendly manner by using pictographs and bar and line graphs. The communication skills training consisted of blended learning for clinicians comprising e-learning and 2 face-to-face sessions. Adjustments to improve both training and the Source tool were made according to feedback from all testing rounds.
The Source tool and training could play an important role in informing patients with esophagogastric cancer about treatment outcomes in an evidence-based, precise, personalized, and tailored manner. The preliminary evaluation results are promising and provide valuable input for the further development and testing of both elements. However, the remaining uncertainty about treatment outcomes in patients and established habits in doctors, in addition to the varying trust in the prediction models, might influence the effectiveness of the tool and training in daily practice. We are currently conducting a multicenter clinical trial to investigate the impact that the combined tool and training have on the provision of information in the context of treatment decision-making.
Esophageal and gastric cancers rank eighth and fifth, respectively, in incidence worldwide [
Oftentimes, the choice between treatment options is based on preferences; the personal weighing of the pros and cons of the options plays a decisive role in the final decision made, and therefore, shared decision-making is needed [
Physicians may face many other challenges when informing patients with cancer on treatment and related outcomes. A vast amount of information on the possible treatment options, including their procedures and associated risks and benefits, must be communicated within the time restrictions of a consultation [
Furthermore, tailoring the type and amount of information to the individual patient’s information needs, interests, and concerns (eg, one patient wants to be informed using exact percentages, whereas another would rather get a general description) has also been shown to be a difficult skill for physicians [
Several tools have been developed to aid physicians in this task by using prediction models to generate clarifying visualizations of personalized outcome data, such as the
Therefore, we aim to develop a web-based prediction tool and supporting communication skills training to improve how physicians inform patients with esophagogastric cancer on treatment outcomes, namely, survival, side effects or complications, and HRQoL. To improve the provision of treatment outcome information, we aim for information that is evidence-based, precise, and personalized to the patient and tumor characteristics, and that is communicated in a way tailored to the individual information needs.
Introducing a change in physician-patient communication by adding a new instrument might initially result in resistance from users, as suggested by behavior change theories [
Both the tool and training were targeted at physicians in oncology who regularly conduct treatment decision-making consultations. In the development of the tool and training, we focused on patients with metastatic esophageal and gastric cancer. With regard to shared decision-making, this group is confronted with the most complex decision-making process, where personal values and preferences play a large role in deciding among multiple relevant treatment options.
The iterative development and testing of this two-part intervention occurred in several phases following the 2008 Medical Research Council framework [
Schematic representation of the development process of the tool and training. HRQoL: health-related quality of life.
The web-based, physician-assisted prediction tool named
First, prediction models were developed to ensure that the Source tool’s treatment outcome information (survival, side effects and complications, and HRQoL) was evidence-based, precise, and personalized to the individual patient and tumor characteristics. Personalized predictions for survival using this tool are based on the SOURCE prediction models [
The side effects (toxicity) of chemotherapy treatment are based on TOXView meta-analyses [
Finally, predictions of HRQoL are available from a meta-analysis and describe the change in the EORTC QLQ-C30 (European Organisation for Research and Treatment of Cancer quality of life questionnaire C30) on the global health scale for best supportive care and chemotherapy in metastatic patients up to 6 months after diagnosis [
Next, these models and meta-analyses were used to visualize treatment outcomes. For the visualizations to be easy to understand for patients, a previous systematic literature review on visual risk communication was consulted [
After opening the tool, a data entry form is shown to enter the variables needed for the prediction models.
The data entry form is dynamic and shows only relevant variables.
Survival, adverse events, and health-related quality of life outcomes are displayed in their own tabs, and only one outcome is displayed at a time.
The outcomes are displayed graphically in a screen-filling image.
The data entry form contains input validation to avoid mistakes during entry.
The data entry contains explanations of the variables.
The plots can be tailored to the patients’ and physicians’ preferences (eg, time frame and treatments to be compared).
The tool’s display language can be set to Dutch or English.
A textual summary accompanying the plots can be generated and printed so the patient can review the information at a later time.
A help function for physicians is incorporated.
A prototype of the web interface was created based on the literature guidelines and first requirements. The web-based tool was developed using the RShiny software (version 1.2.0; RStudio) supplemented with ggplot2 (version 3.2.1) to create graphs [
First, a face-to-face focus group was conducted with 6 patients with esophageal and gastric cancer and survivors from the Foundation for Patients with Cancer in the Digestive system after verbal informed consent was provided. The aim of this focus group was to obtain feedback on the tool in a group setting and promote discussions among the group members. One of the researchers acted as a moderator and presented the participants with each of the tool’s graphs. Each displayed graph was accompanied by a short oral explanation, after which the participants were asked for their opinions. Feedback on the web interface and suggestions for improvement supported by multiple participants were used to create an improved version of the tool. The focus group session was audio-recorded and analyzed according to microinterlocutor analysis to systematically evaluate the participants’ remarks [
In a second feedback round, 30-minute, semistructured, face-to-face interviews were conducted with individual patients with esophageal and gastric cancer. The main focus of these audio-recorded interviews was to determine whether the patient interpretation of the revised graphs was adequate. By conducting interviews with individual patients rather than in a group, the aim was to review patient interpretations without the influence of other patients. The interviews were conducted using a piloted script. A total of 5 outpatient participants were recruited by an oncologist at the Amsterdam University Medical Centers. Following the participants’ informed consent, 2 researchers presented the tool to patients by using fictitious predictions of treatment outcomes. Patients were asked to interpret the presented graphs and describe their meaning to assess their understanding. Thereafter, the researcher provided the correct description of the graph, and the patients’ subsequent feedback was gathered. Feedback was registered in a response matrix, including the frequency of different remarks, to establish which possible improvements could be implemented.
The third feedback round aimed to evaluate the usability of the tool when used by medical oncologists, and 3 medical oncologists at the Amsterdam University Medical Centers participated in individual face-to-face think-aloud sessions. After providing informed consent, they were asked to use the tool for two paper patient cases while stating out loud whatever came to mind. The cases described fictitious patients with esophageal or gastric cancer, including some of their clinical characteristics. Several tasks and questions about specific outcomes (eg, “What is the 1-year survival probability with best supportive care?” and “Which treatment has the best quality of life after 6 months?”) were posed to guide the use of the tool by medical oncologists. At the beginning of the think-aloud session, a video explaining the think-aloud method [
In the fourth and last round, feedback from experts was gathered by conducting semistructured interviews with 6 researchers with expertise in patient-physician interaction, shared decision-making, risk communication, medical informatics, and clinical decision support software. The experts were presented with a walkthrough of the tool and its options. Interviews were recorded and summarized to determine which possible improvements were brought forward.
Communication skills training was developed to educate physicians on informing patients with cancer in a treatment decision-making consultation using the Source tool. Due to the complexity of the skills needed, it was important to specify clear learning goals. As stated in complex learning theory, when training complex skills, the desired learning outcomes must address the following domains: knowledge, attitudes, and skills [
As physicians value time-efficient and flexible training [
First, e-learning was targeted at summarizing the evidence base for effective information provision and providing physicians with tips and tricks for clinical practice. to this end, we consulted the literature related to theories, evidence, and guidelines on the provision of information in medical practice [
An earlier study concluded that physicians value both visual attractiveness and variation between learning activities in e-learning [
Face-to-face sessions were developed based on previous experience in developing and evaluating communication skills training in oncology [
A pilot study was conducted from December 2018 to March 2019 to test both the tool and training in a real-life setting. As this pilot study targeted patients with advanced disease only, we included medical oncologists and metastatic cancer patients as study participants. The pilot study was evaluated by the Medical Ethics Review Committee of the Academic Medical Center Amsterdam (reference number: W18_278). In total, 5 medical oncology fellows (2 men and 3 women) from two university medical centers were invited to use the tool and trained according to the concept training format. After completion of the training, participating fellows were individually interviewed via telephone in a semistructured manner to gather feedback to improve both the Source tool and training. For the tool, the focus of the feedback was on opinions and experiences regarding usability and willingness to use the tool. Regarding training, feedbacks on the different components, training as a whole, and perceived utility were collected.
In addition, an experienced medical oncologist (HWMVL) conducted three treatment decision-making consultations with outpatients using the Source tool for information provision and in line with the training principles. These consultations were recorded on video after obtaining written informed consent from the patients and oncologists. To comply with ethical standards and according to the training, only information that the patient wanted to receive was disclosed to the patient. One-on-one semistructured interviews were conducted with the 3 patients by one of the researchers (LFVDW) to gather their experiences with the physician’s outcome information and the use of the Source tool.
A prototype web interface was created based on the findings of a systematic review of the effects of different types of risk communication on patients with cancer [
The final tool, Source 1.103, contains five main components. The first component, the patient information entry component, allows the oncologist to enter the patient characteristics necessary for the prediction models and meta-analyses, using supporting information, such as the definitions of TNM variables (
A screenshot of patient data entry. The data entry screen displays the fields that are necessary for the prediction models and meta-analyses. Additional information on variables such as the World Health Organization performance status is provided with a mouse-over.
The survival component outcome was visualized in two ways: an icon array displaying the survival probability at a given point in time by coloring a subset of 100 figures and a Kaplan-Meier curve (line graph) displaying the survival probability over time (
A screenshot of survival graphs. On the left, a pictograph displaying the predicted survival for best supportive care and chemotherapy after 1 year is shown. On the right, the Kaplan-Meier curve for the best supportive care is shown. The optional shaded area displays the so-called typical outcome scenario (with survival ranging from 25% to 75%).
The side effects component displays bar charts for various toxicities (
HRQoL is displayed in a line graph and shows the EORTC QLQ-C30 global health score over time (
A screenshot of the side effects bar chart. This displays the three most commonly occurring toxicities for both 5FU and CAPOX. The darker bars indicate severe toxicities, and the lighter bars indicate mild toxicities. 5FU: 5-Fluorouracil; CAPOX: capecitabine combined with oxaliplatin.
A screenshot of the HRQoL (health-related quality of life) line graph. The graph displays the HRQoL following definitive chemoradiation. The shaded area optionally displays the CI of the HRQoL line.
The four evaluation rounds of the tool resulted in several minor visual and functional adjustments to the web interface, as described in
In the pilot study, 4 of the 5 oncology fellows participated in an evaluative semistructured interview following the training. The fifth fellow did not respond to repeated invitations. All 4 oncology fellows reported that the tool was highly usable overall. Some minor suggestions were provided for improving the display of certain options, graphs, and buttons. Of the 4 oncologists, 3 reported that they would use the tool in their clinical practice. They especially valued the personalized nature of the tool’s predictions and the clear and easy-to-understand visualizations for patients. Furthermore, the inclusion of HRQoL data and the option to print a summary for the patient were found useful. The fourth fellow would like the prediction models to be further developed before using the tool. For instance, during the pilot training and interviews, several critical remarks on the prediction models were expressed, such as the lack of World Health Organization performance status as a predictor of overall survival. These comments likely reflect a possible lack of trust in the underlying models and analyses of the tool. Another barrier to using the tool that one of the fellows addressed was the fear of emotionally confronting the patient with the exact numbers. This fellow did indicate that this fear was overcome through the tailoring skills that he had acquired during the training.
A short and to-the-point e-learning module was developed to provide an overview of the theoretical background on the provision of information in a treatment decision context and introduce a web-based prediction tool. The e-learning module starts with a short peer endorsement video of the training of physicians that discusses the Source tool. Subsequently, physicians can navigate through 4 chapters. The first chapter provides an overview of the principles of effective information provision in the context of a shared decision. The second chapter introduces the physicians to the Source tool by presenting them with a tailor-made instructional video of its use and functions. Furthermore, a summary of the tool’s models and their underlying data is provided. The third chapter provides an overview of tips and tricks for informing patients about the risks and benefits of treatment (
The face-to-face component of the training consisted of two group sessions of 3.5 hours each, provided by an experienced trainer, with approximately 2 to 3 weeks in between to facilitate intermediate practice. The sessions were aimed at small groups of 2 to 6 participants, as this approach enables every participant to practice and receive personal feedback. Such a setting can also promote interactivity [
The first session covered the skills of setting the agenda of the decision-making consultation, introducing the tool and informing patients on survival outcomes. Physicians were asked to practice separate parts of the consultation while receiving feedback from other physicians and the trainer. Physicians were instructed to use the acquired knowledge and skills during their outpatient consultations before the next training session. The second session allowed for the repetition of issues addressed during the first session and sharing experiences of applying the lessons learned of the first session in clinical practice. Next, skills were addressed, again with role-play and feedback, to inform patients about treatment outcomes in terms of side effects and complications and HRQoL. This session also addressed how to conclude decision-making consultations.
In both sessions, tailoring of the amount and type of outcome information to specific patients played a significant role in both role-play practice and group discussions. Tips and tricks were discussed regarding how to determine an individual’s information needs and wants, how to fit these needs with the informational needs of the physician, and whether and how the tool could contribute to tailored information giving.
The total duration of the blended learning was 7.5 study hours, which consisted of 0.5 hours of e-learning and 7 hours of face-to-face training.
Fellows reported that they enjoyed participating in the training and specifically valued personal coaching and practical tips. Furthermore, fellows appreciated the trainers and actors. In their opinion, there was a good balance between the information provided by trainers and practical exercises. Fellows especially appreciated the feedback during the training from the actor and trainer on their role-play with the actor. Overall, the training was described as useful, and specific improvements were suggested.
A point of improvement that was brought up was the substantial time investment in the training. In particular, the pace of the e-learning and instructional video was considered too slow. Furthermore, the timing of the face-to-face sessions following a day of work was considered inconvenient. The most important adjustments to the training as a result of the fellows’ feedback were related to accelerating the pace and adding an individual booster session to the training in which the physician could receive personal feedback from one of the trainers on a full, recorded consultation.
Despite their emotions on the subject, 2 out of the 3 patients were willing to participate in a short interview about their experiences with the consultation. Both appreciated the use of the Source tool and the physicians’ explanations of possible treatment outcomes. Patients expressed differences in their experiences regarding the amount of information about treatment options and outcomes. Although one patient reported being satisfied with the amount of information, the other indicated that the amount of information was too extensive for him to memorize it all. He needed a printed summary of the tool for support. Both patients mentioned their struggle with the meaning of risk or benefit for themselves, as great uncertainty remains about their own future, despite the information provided.
Our study shows the iterative development and pilot testing of the Source tool and training. This combined intervention was developed using scientific evidence and input from physicians, patients, and experts. This process resulted in the first web-based prediction tool to inform patients with esophagogastric cancer during consultations on survival, side effects, and HRQoL of different treatment options. Furthermore, we created a supporting training to teach physicians the communication skills needed to use the tool and to provide patients with information in an evidence-based, precise, personalized, and tailored manner. Preliminary evaluation results are promising and provide valuable input for further development and testing of both elements.
Both the tool and training were valued by participating physicians and patients. Physicians especially appreciated the practical approach of the training; the multiple practice opportunities and personal feedback helped them use the tool. Nevertheless, despite their positive attitudes toward the tool and training, old habits could stand in the way of using the tool and may impede the use of learned communication techniques in clinical practice. Behavioral change theories show that many factors can contribute to but also stand in the way of learning new behaviors. Resistance could, for instance, arise as a result of a different expected outcome of the tool or because of a low tolerance for change [
During the pilot training, it was noted that in some cases, the fellows lacked trust in the prediction models used in the tool. Further steps were taken to increase the physicians’ trust. For instance, details about the underlying data and publications were added to the tool to provide more information about the methodology and sources. Model updates (such as the 2020 version of the survival model), which increases the model’s performance and sample size and includes, for instance, World Health Organization performance status as a predictor, may also increase trust in the tool [
The use of the Source tool could also be influenced by the application’s usability and how well the tool solves the patient-informing problem as perceived by patients and physicians. Therefore, iterative usability testing is necessary to achieve an acceptable level of usability. As the number of testing rounds in this study is limited compared with other studies [
On the basis of our experiences, we can provide several recommendations to aid future research in creating and evaluating web-based prediction tools with training. First, we advise involving end users, such as patients and physicians, in the early stages of development. Assumptions and implementations are often made from the perspective of developers, which may not coincide with the needs and wishes of patients or physicians. By evaluating at an early stage, it is possible to adjust the tool and training, and subsequent improvements can be implemented more seamlessly. Although not formally evaluated in this study, user research to investigate patients’ and physicians’ ideas and expectations regarding such a tool could also contribute to the usability and adoption of the tool in clinical practice. Second, evaluation by physicians may be complicated because of their busy schedules. We recommend making the feedback rounds with physicians as short as possible, planning them sufficiently in advance, and having them take place on education days. Third, as insights on data visualization and risk communication may change constantly, we recommend facilitating ongoing updates of a designed prediction tool. We also suggest that future research should use the current state of the art when designing a new tool or training. Fourth, we advise that communication on outcomes with subjective interpretations, such as HRQoL, deserves a more prominent place in communication skills training. We noticed in our training that physicians often had trouble explaining outcomes such as HRQoL. Finally, it was observed that end users sometimes had conflicting opinions regarding improvements to the tool or training. As it is not possible to cater to everyone’s wishes, we recommend weighing the pros and cons of suggestions and deciding whether a personalization option will be implemented (such as displaying survival as a pictograph and a line graph) or whether a single option will be implemented (such as the background color of the web interface).
Most of the patients and physicians who participated in this study agreed that the tool and training added value to clinical practice. However, bias may have played a role in the evaluation of the tool and training, as the evaluation only partly took place in clinical practice. To investigate this potential bias and the extent to which the combined tool and training aid in information provision in the context of treatment decision-making, we are currently performing the third phase of the Medical Research Council framework: evaluation, a multicenter effect study (registered under NCT04232735, the SOURCE trial). In this stepped-wedge trial, physicians receive training and use the Source tool both in simulated patient assessments and with outpatients. The effect that the intervention has on the outcome information provided by oncology physicians was quantitatively investigated by recording these consultations before and after the intervention and analyzing physicians’ outcome-related remarks. The primary outcome of this study is the provision of precise outcome information; secondary outcomes include the amount of tailoring to the information needs of patients, the patients’ own knowledge and opinions on the communicated outcome information, and the influence that the consultation has on patients’ emotions. As trust in the SOURCE models was found to be a potential barrier to using the tool in the pilot study, physicians’ trust in the models will be closely monitored and specifically addressed during the trial. The models included in the Source tool are being continuously improved and updated, in part, to address these issues.
In this trial, both palliative and curative patients will participate, and models aimed at potentially curable patients (cM0) will be added to the Source tool. The survival models are based on the 2020 version of the models, which include updated palliative prediction models and newly developed curative prediction models for both esophageal and gastric cancer [
We developed and evaluated a web-based tool and training to inform patients with esophageal or gastric cancer regarding treatment outcomes. Through evaluation and a pilot study, patients and physicians indicated the added value of the tool and training, and both were improved based on their feedback. The tool and training are currently being evaluated in a multicenter trial to determine their added value in clinical practice.
Detailed adaptations to the Source tool during the modeling phase.
Recommendations on risk communication displayed in the e-learning training.
Choosing Treatment Together in Cancer at the End of Life
European Organisation for Research and Treatment of Cancer quality of life questionnaire C30
health-related quality of life
Must Have, Should Have, Could Have, Won’t Have
System Usability Scale
tumor–node–metastasis
Financial support for this study was provided entirely by a grant from the Dutch Cancer Society (grant UVA 2014-7000). The funding agreement ensured the authors’ independence in designing the study, interpreting the data, and writing and publishing the report. The funder had no role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; the preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.
The authors developed and evaluated the web-based tool Source and the accompanying training described in this study. HWMVL reports grants from the Dutch Cancer Society during the conduct of the study and personal fees from Bristol Myers Squibb, Lilly, and Nordic Pharma; and grants and nonfinancial support from Bayer, Bristol Myers Squibb, Celgene, Jansen, Lilly, Nordic Pharma, Philips, Roche, and Servier outside this work. In addition, HWMVL has a consultant or advisory role with Bristol Myers Squibb, Lilly, MSD, Nordic Pharma, and Servier.