In accordance with the Medical Research Council (MRC) guidance, we started with a process evaluation focusing on the reach of the intervention and the extent to which it was used as intended [31]. Subsequently, the outcome evaluation was performed, focusing on the patients’ perceived confidence when interacting with their HCP. The research ethics committee of the Radboud University Nijmegen Medical Centre evaluated the RCT protocol and concluded that the study did not fall within the remit of the Dutch Medical Research Involving Human Subjects Act (WMO). The study is registered in the Netherlands National Trial Register (trial registration number NTR3779). Written informed consent forms were used.

The Web-based PatientTIME intervention aims to support patients in gaining more control over the communication with their HCP. The intervention development was guided by the intervention mapping framework applied in close collaboration with patients [24] and makes use of different theory-based methods: modeling, tailoring information, previsit goal setting, and listening to visit recordings. The central source of information in the intervention consists of 58 short video fragments (47-180 seconds) showing simulated patients demonstrating different communication skills during medical encounters (eg, stating the need for support, dealing with emotions, or asking questions; Figure 1) [32]. The fragments are based on communication barriers identified by the targeted population in a previous study [22]. A question prompt sheet (QPS) and an option to replay an audio recording of the user’s hospital visit were also included in the intervention. The functionality and intended use of these individual components are described in Table 1. The collaborative partners (2 hospitals, the patient association, the funding organizations, and a research institution) were listed on an information page.

PatientTIME is a stand-alone intervention; that is, patients can register and use the intervention without referral or the involvement of a professional. Several online and offline actions were organized to create awareness of the availability of PatientTIME: (1) spreading leaflets at hospital waiting rooms, (2) giving short presentations during patient information meetings arranged by Hematon and at hospitals, (3) short news flashes in Hematon’s digital newsletter and magazine, (4) distributing our own project newsletters, and (5) using social media (Twitter, forums). Additional actions were initiated during the inclusion period: (6) we sent information packages and leaflets to hematologists asking them to distribute the information among their patients, and (7) we sent short news flashes about the study to local newspapers. Patients could express their interest in participating via registration on the PatientTIME website [21] between March 2013 and May 2015. After registration, the study’s inclusion criteria were verified. Patients were eligible for participation if they were aged 18 years or older, had been diagnosed with malignant lymphoma, had at least one (follow-up) consultation with their HCP per year, had a good understanding of the Dutch language, were receiving treatment or follow-up care in a Dutch hospital, and had access to a computer with an Internet connection.

If the inclusion criteria were met, patients received a consent form with additional information by post. As soon as they had returned the informed consent form, their account was activated and an email was automatically sent to the patient with log-in details. If no consent form was returned, the patient was reminded by email after 2 weeks and by phone 1 week after that.

The intervention was implemented in the form of a 3-armed RCT with continuous recruitment and data collection. The computer assigned participants randomly to 1 of 3 groups: (1) a control group, (2) an intervention group, and (3) an intervention audio group. Software that enabled blocked randomization (block size 3) was used to balance the groups.

The 2 intervention groups differed in one component: having or not having the opportunity to listen afterward to the audio recording of their clinical consultation. For the purpose of this outcome evaluation, the 2 intervention groups were evaluated as 1 group and compared with the control group. We made this decision for practical reasons. Patients who had the opportunity to listen afterward to their consultation generally did this after completing the postvisit questionnaire (Figure 2). Therefore, the experience of replaying and listening to the consultation could not have influenced their answers. Moreover, some patients in the intervention group where we did not actively encourage them to record their consultations (group 2) still made recordings on their own initiative.

All participants were given access to a personal secure Web account with questionnaires and an integrated digital agenda. The agenda had an automatic reminder system. This system led them through the study with personalized messages and questionnaires before and after clinical consultations (Figure 2). The date of the participant’s forthcoming hospital visit determined the start of his or her participation. One week before this visit, participants received an automatic email reminder with the request to complete the previsit questionnaire on their Web account. After having completed the previsit questionnaire, the control group received no further information. The 2 intervention groups were given access to the intervention content (Figure 3). The selection of video fragments taken from the video library was based on the participant’s self-reported previsit communication preferences and needs. The intervention audio group participants were also instructed—if permitted by their HCP—to record their consultations, upload the audio recordings, and listen afterward to them. The recordings were also available to the researchers for analyses. We provided these participants with a small audio-recording device and extra information about consultation audio recordings.

The day after the consultation, patients received an automatic reminder with a link to the postvisit questionnaire. If a second and third consultation was planned within the participation year, the cycle was repeated with less comprehensive questionnaires. If necessary, a maximum of 2 reminders were sent by the researcher reminding the participant to complete the questionnaires or register a consultation date. The control group had access to the educational content of the intervention after participation.

There was no downtime during the evaluation period. PatientTIME’s content and functionality were frozen for the duration of the trial. Necessary bug fixes were made relating to questionnaires that were not always directly available after the clinical consultation.

Participants were not blinded as they could see whether they had access to the content of PatientTIME or not. As self-confidence in communication was the primary outcome measure, the power calculation was based on the primary outcome measure of the RCT, the PEPPI (Perceived Efficacy in Patient-Physician Interactions) questionnaire (see Data Collection and Measures) [26]. We expected the intervention to result in a decrease of 5 points on the sum score (reflecting higher confidence), which corresponds to an effect size of 0.70. To detect this difference with a power of .80 and alpha of .05, we needed 50 patients in the intervention group and 50 patients in the control group.

The process and outcome data were measured through (1) Web-based questionnaires (Qr, Q0, Q1, and Q2; see Figure 2), (2) log files, that is, automatically generated files mapping the interactions between the program and its users, and (3) a logbook, comprising a record of actions and interactions that was kept by the researchers.

According to the MRC guidance, the selection of dimensions that are evaluated should be adjusted to the intervention under study. A total of 7 process dimensions were selected that may influence outcome measures in this study (Table 2).

In addition to the dimensions taken from the MRC guidance, the intervention group was asked if and in what way the intervention had helped them. All “evaluative” responses (answers to the question) were coded as “positive” or “negative.” To prevent subjective interpretation of the data, 2 researchers coded all fragments. The interrater reliability was good (81% agreement). Disagreements were discussed to come to an agreement.

For the primary outcome measure, participants were asked to complete the validated, 10-item PEPPI questionnaire twice: at t0 and t2. The PEPPI questions all begin with “How confident are you in your ability to...” and assess a patient’s confidence in their ability to communicate with their HCP on a 5-point Likert scale (1 = very confident to 5 = not confident at all) [26,34]. Item scores were summed (giving a range of 10-50), whereby lower scores reflect higher self-efficacy. The PEPPI questions used at t1 were reformulated so that they could also be used as a postvisit scale, measuring efficacy experienced directly after the visit.

Descriptive statistics were used to characterize the sample and to analyze the process measures.

Differences between groups were analyzed using t tests and chi-square tests, where appropriate. To analyze short-term effects, linear regressions were conducted to test for differences in experienced efficacy measured at t1 between the control and intervention groups, adjusting for the previsit PEPPI score. To analyze long-term effects, multilevel regressions were conducted to test for differences on the PEPPI scale between the control and intervention groups at t2 and t0. The analysis of the open questions was performed using MAXQDA 11 software [35]. Other process and outcome data were analyzed with Stata 13.

A total of 146 potential participants registered (uptake) of which 97/142 (68%) patients met the inclusion criteria and signed the informed consent form (dose delivered; Figure 3). This group included 5 patients who were not diagnosed with malignant lymphoma but with a hematologic malignancy with a comparable impact (eg, chronic lymphocytic leukemia or multiple myeloma).

Of the 49 potential participants who were not included, 4 were excluded by the researchers. The remaining 45 did meet the inclusion criteria but did not return the informed consent form. Compared with these 45 patients, the 97 participants had a significantly higher level of education, were more likely to be a member of a patient organization, had a longer elapsed time between diagnosis and registration, and were more likely to have had or started treatment (Table 3).

All 97 patients in the RCT registered their first consultation date, which was a precondition for being sent the previsit questionnaire (Q0). Subsequently, the participation rate was 90% as 87/97 patients completed Q0 and were marked as participants. Compared with the nonparticipants (N=10), the 87 participants had a higher level of education and used the Internet on a more regular basis. After having completed Q1, a total of 76 participants registered a second consultation date and 52 participants a third consultation date as well. There was no significant difference between the control and intervention groups in the proportion registering multiple consultations. Because not every participant had a second or third consultation planned within the participation time span (1 year after registration), the analysis of the outcomes is based on the first consultation only.

The identified process results that facilitated correct interpretation of the outcomes were part of the delivery domain (eg, fidelity, reach; see Table 2) and user-program interaction domain (eg, exposure, use of different components; see Table 2), in addition to the benefits perceived by the patients.

Regarding the delivery domain, we concluded that the publicity actions were conducted as planned. On the whole, the intervention was indeed used as a stand-alone program; most participants did not need any help or reminders from the researchers. The questions they asked (mainly by email) concerned problems with logging in or study questionnaires that were not yet accessible. For the first consultation, 18 of the 97 randomized patients received at least one reminder to complete their previsit questionnaire.

We noted one issue regarding the “fidelity” that may have influenced outcomes. At registration, patients had to provide the date of their upcoming consultation. Of the potential participants who did this, 44/138 (32%) had their consultation planned within a fortnight. These patients were less likely to participate in the study (59%) compared with the patients who had their consultation later (76%).

Regarding the user-program interaction domain, we report the actual use of the different intervention components that may have influenced the outcomes. Of the participants assigned to the intervention group, 55/63 (87%) completed Q0 and had access to PatientTIME. This group viewed on average 6 (SD 3.5) of the 10 selected video fragments before their first consultation. Of all the fragments they watched, 76% were viewed from beginning to end. The QPS was used to write down questions, physical complaints, and concerns by 20/55 (36%) of the participants who had access to it. Of the intervention audio group 9/29 (31%) participants reported after consultation that they had recorded their visit, 13/29 (45%) reported that they had not, and 7/29 (24%) did not complete this question. Of the participants who did not record their consultation, 8/13 said that they decided that they did not want to make the recording, 2/13 said the HCP did not give permission, 2/13 said they did not want to ask the HCP and 1/13 forgot the recording device.

Regarding the “usability,” 50 intervention group participants completed the System Usability Scale questionnaire at Q1, revealing an average satisfaction score of 73 points (SD 16) on the 0-100 scale, which is considered “good” according to the study by Bangor et al [36].

As for perceived benefits, in the intervention group, 46/50 (92%) participants completed the open question “Did you benefit from the website?” at t1. The positively coded answers (59%) explained that (1) the website was instructive, insightful, or helpful for the preparation of a clinical consultation; (2) the website created awareness about the importance and role of communication; and (3) participants recognized the scenarios, which gave them the feeling that they were doing all right.

The negatively coded answers (41%) referred to (1) bugs in the questionnaire or usability problems in the website and (2) the fact that the information did not add anything to what was already known or experienced.

The results on the PEPPI scale at t0 and t2 (Table 3) were skewed to the lower scores, reflecting a group of patients with a high level of self-confidence about interacting with their HCP. The same skewed trend was visible at t1, reflecting a group of participants who were well able to reach their goals.

There were no differences on the PEPPI scale between the control and intervention groups at baseline (t0, P=.78). Directly after the consultation (t1), there was no difference between the control and intervention groups in the way they experienced their efficacy during the consultation (controlling for the previsit PEPPI score). The results remained the same when controlling for sex, the level of education, and time since diagnosis.

Comparing the PEPPI score at t2 with the PEPPI score at t0, the control group showed a small improvement in the level of perceived efficacy. This improvement (−0.38 points) was not significant (P=.69). The intervention group also showed an improved level of perceived efficacy. This improvement (−1.97 points) was significant (P=.02). The interaction effect (−1.59), however, was not significant (P=.20), indicating that the improvement solely as a result of the intervention may not be significant. In addition to this primary analysis, we did not find an association between the PEPPI outcomes and the number of video fragments watched.

In this paper, we described an integrated process and outcome evaluation of the Web-based intervention PatientTIME. PatientTIME was developed with the aim of helping patients with their communication during clinical consultations. With the process evaluation, we aimed to sketch the context in which the outcome evaluation was performed. The process evaluation was also used to obtain input on how to improve the intervention and its reach before making the intervention publicly available. In the outcome evaluation, we focused on the patients’ perceived confidence in interacting with their HCP.

As a result of the process evaluation, we identified a substantial number of delivery and interaction aspects that will help to improve the functionality of PatientTIME and its reach. Looking at the delivery, we want to highlight the recruitment process. The chosen method may resemble how patients are informed in practice about publicly available Web-based interventions, but despite the various publicity actions during the recruitment period, this approach did not appear to be enough to recruit a large, diverse group of patients for the study (see also Trial Participants section).

The patient-program interaction evaluation showed how the different PatientTIME elements (video fragments, QPS, and consultation audio recordings) were used. The core content of the intervention, the video fragments, was well used. The percentage of fragments viewed entirely (76%) may even be an underestimate (fragments were only logged as “ended” when the screen switched back to start).

The QPS was used less frequently, and slightly less compared with other studies that evaluated the use of open QPS tools in oncology care [37,38]. We do not know if the limited use was due to the design of the QPS, the fact that patients found it difficult to articulate questions, or the fact that patients preferred to make their notes on paper. Albada et al found that patients need to be motivated to use a QPS. Information about why and how to use the QPS could have been made more explicit in the PatientTIME intervention [37]. In the intervention audio group, more than half of the participants did not record their consultation on audio. There appeared to be a variety of obstacles. Other studies show that cancer patients highly value consultation audio recordings and that the majority benefit from listening to the consultation afterward [39,40]. However, in these studies the HCP facilitated the recording, whereas in our study patients were encouraged to take the initiative. We may have failed to provide sufficient information to remove existing obstacles, or the fact that it was part of a trial may have caused patients to be reluctant. To conclude, the QPS and audio recording options should be improved in the PatientTIME intervention.

More than half of the intervention group patients explained that the intervention helped them prepare for a clinical consultation, created awareness, and/or reinforced their existing communication skills. Patients who reported no benefit explained that they already had a good understanding with their HCP, had a lot of experience, or considered themselves as sufficiently participative and empowered already.

Looking at the data on confidence in interaction, we found a trend indicating that in the long run, patients with access to PatientTIME scored better on the perceived efficacy scale than patients without access. Differences were, however, small and we did not reach our aim of an improvement of 5 points. Therefore, we cannot conclude at this stage that the intervention positively influenced participants’ confidence in their ability to talk with their health professionals.

The strengths and limitations related to the study and intervention put the results in perspective and illustrate our lessons learned. We identified four important aspects.

A considerable number of patients indicated that the intervention was helpful in preparing for a clinical consultation, created awareness about the importance of communication, and reinforced their existing communication skills. These are valuable findings as such. However, at this stage we cannot conclude that PatientTIME improves patients’ confidence when interacting with health professionals. By integrating a process evaluation and an outcome evaluation in this way, we were able to demonstrate which elements of PatientTIME were used and appreciated, even though they did not lead to measurable changes in communication self-efficacy.

We identified three themes for future research. First, when evaluating the effectiveness of stand-alone Web-based interventions like PatientTIME, it may be worthwhile to consider research methods that allow the intervention to be dynamic, rather than freezing it in a certain state. In practice, Web-based interventions also need to be optimized continuously. With a more dynamic approach, intermediate test results can be used to improve the intervention and new versions of the intervention can be released during the evaluation phase. Second, when evaluating stand-alone interventions in a trial setting, the involvement of HCPs as informants might be crucial as a means of reaching a more diverse group of patients. More than half of the intervention group patients explained that the intervention helped them in the preparation for a clinical consultation, created awareness, and/or reinforced their current communication skills. Patients who reported no benefit explained that they already had a good understanding with their HCP, had a lot of experience, or considered themselves as sufficiently participative and empowered. Finally, it will be interesting to continue monitoring the use and effectiveness of PatientTIME once it is publicly available in order to provide input for other interventions and to keep PatientTIME up-to-date.