This study used 4 sources of data: implementation outcome indicators, a survey, hospital documents, and interviews. Data collection was part of a broader study on the implementation of VBHC [33]. Implementation outcome indicators were used specifically for objective 1, whereas the other data sources were used to address objectives 2 and 3.

The hospital monitored the degree to which clinicians accessed the PROM consultation room dashboard. We extracted the percentage of completed PROMs accessed by clinicians from the hospital’s monitoring system, covering the 12-month period from February 2023 to January 2024. We used the aggregated data that eventually included 70 subdepartments. Investigating subdepartmental differences in the use of PROMs was outside the scope of this study. We analyzed both the average percentage of completed PROMs accessed by clinicians during the week of the patient’s consultation and on the day of the consultation. While the former indicates the use of PROMs to prepare for consultations, the latter serves as a proxy for discussion with the patient.

Internal hospital documents were accessed through the digital workspace of the CST, to which author VvE was granted access. This workspace was established around 2020. A systematic search was conducted using the following keywords: compliance, evaluation, barrier, facilitator, challenge, plan, intervention, and strategy. This yielded 56 files. These were then screened for data pertaining to factors facilitating or impeding clinicians’ use of PROMs or strategies aimed at enhancing their use, followed by data extraction. If files referred to other files, these were also considered for inclusion. Subsequently, the list of extracted strategies was cross-validated with author MDH-A, a member of the CST, to distinguish between planned and executed strategies.

In January 2023, a collective survey, named EMC23, was digitally distributed to all 194 clinicians across the 35 subdepartments that collected PROM data from outpatients in 2022. One clinician involved in the survey design was excluded from participation. The participants remained anonymous and untraceable. The survey comprised 26 items (Multimedia Appendix 2), developed based on preliminary insight from document analysis. The items delved into clinicians’ self-reported use of PROMs, prominent factors influencing this, and their overall satisfaction with the implementation process and outcomes achieved. In addition, the respondents were asked to evaluate a subset of implemented strategies and had the opportunity to provide comments. Two reminders were sent. As shown in Table 1, 47 responses were included in the analysis.

For a process evaluation on the implementation of VBHC [33], author VvE conducted 20 semistructured interviews with individuals involved in the central implementation effort during the past decade (Table 1). Two individuals refused participation for personal reasons. Participants were identified through documents and snowball sampling. Interviews were recorded and transcribed verbatim. Given the hospital’s focus on implementing PROMs in their move to VBHC, interviewees provided valuable insights into the hospital’s efforts and experiences in enhancing clinicians’ use of PROMs. The portion of the interview guide that elicited relevant comments for this study is presented in Multimedia Appendix 3. We reanalyzed these transcripts, extracting information on barriers and facilitators for clinicians to use PROMs, strategies and their rationales, and areas for improvement. These data complemented the other sources mentioned in the Data Sources section, enriching the findings of this study.

Given the intricate nature of the implementation of PROMs, adopting a multifaceted research approach is imperative [12], typically including contextual factors, the innovation (PROMs), strategies to support effective implementation of the innovation, and implementation outcomes interact, as illustrated in Figure 2. These elements align with the objectives of this study. Therefore, our methods were plural, incorporating the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework [38]; the Expert Recommendations for Implementing Change (ERIC) taxonomy [39] and associated 9 clusters [40]; and the COM-B constructs [41]. We reviewed multiple strategy taxonomies and evaluation frameworks as summarized by Nilsen [42], selecting the aforementioned frameworks due to their alignment with this study’s objectives and data, widespread use, compatibility with the health care setting, and clarity in presenting results. These frameworks were used for post hoc, deductive data analysis; these frameworks are explained in subsequent sections.

The RE-AIM framework [38] guides the planning and evaluation of programs by examining 5 outcomes: reach, effectiveness, adoption, implementation, and maintenance. This study focused on the latter 3 outcomes at the individual level rather than at the department or hospital level. While the collection of PROMs among outpatients was a departmental choice, individual clinicians subsequently needed to adopt, implement, and maintain using PROMs. In this study, adoption denoted individual clinicians’ willingness to initiate the use of PROM data from their patients. Implementation denoted the consistency of clinicians’ use of PROMs as intended, that is, acknowledging outcomes and discussing results with the patient during their consultation to inform decisions. Maintenance referred to the clinicians’ ongoing use of PROMs. Reach and effectiveness have not been directly studied in this research but were indirectly addressed, as they are contingent upon the extent to which clinicians adopt, implement, and maintain PROM use [8,11].

We examined the outcomes adoption, implementation, and maintenance through survey data, reporting item-level frequencies for two key questions: (1) How frequently do you examine patients’ responses to PROMs? with the following response options: I never used PROMs, I stopped looking at PROMs, occasionally, as often as possible, and always; and (2) To what extent do you discuss these outcomes with the patient during the consultation? (for those who examine PROMs) with the following response options: never, occasionally, as often as possible, and always. We supplemented these data with insights from the hospital’s implementation outcomes indicators, for which no further analyses were required.

We retrospectively analyzed Erasmus MC’s implementation efforts. First, we documented all strategies applied by the hospital from 2020 to 2023 based on document and interview data, focusing on understanding why these strategies were chosen, particularly in response to barriers or facilitators. For each strategy, we aimed to deductively specify the actor, action, target of the action, temporality, dose, and implementation outcome affected and provide a justification (Multimedia Appendix 4) [43].

Besides describing the practical application of strategies at Erasmus MC, this study mapped discrete ERIC strategies [39] to the identified strategies using a coding manual developed by Fridberg et al [44]. ERIC provides a compilation of 73 discrete strategies that one can use to effectively implement an innovation in a health care setting [39], which have been mapped into 9 strategy clusters [40]. These clusters are mentioned in the Results section. The manual excluded the discrete strategy of facilitation due to overlap with other strategies. This study also included 10 strategies proposed as additions to the ERIC taxonomy: provide stakeholders with the possibility to attend educational meetings [44], recruit clinicians with competence in the innovation [44], provide stakeholders with resources [44], act as a role model [44], assess and redesign workflow [45], create web-based learning communities [45], engage community resources [45], plan for outcome evaluation [46], obtain worker feedback about the implementation plan [46], and communication as a strategy category [47].

To deepen our understanding, we included stakeholder reflections on strategies from documents, interviews, and survey data, including comments on their feasibility and perceived utility. We coded these data by mapping them to the respective strategy they pertained to, allowing us to cluster relevant information accordingly. Subsequently, we integrated these qualitative data into the results per strategy. The dataset was too limited to conduct thematic analyses on a per-strategy basis. Furthermore, we assessed whether identified barriers or facilitators in our data were addressed through these strategies. If not, these cases were labeled as missing strategies.

We observed that our data aligned effectively with the constructs of capability, opportunity, and motivation, which are components of the COM-B model [41]. Therefore, we structured our results around these constructs, finding them intuitive and straightforward for conveying our findings.

On the basis of document analysis, we identified key barriers and facilitators to clinicians’ use of PROMs. In the survey, clinicians rated the perceived influence of these factors on their PROM use (Multimedia Appendix 2). They also had the opportunity to identify additional factors.

We analyzed item-level scores for the entire survey population by grouping agree and partly agree into a single category. Likewise, disagree and partly disagree were grouped into a single category. The categories neutral and no opinion remained unchanged. Moreover, we conducted subsample analyses to explore differences among different user profiles. Clinicians were allocated to 3 groups depending on their self-reported use of PROMs: nonusers, frequent users, and occasional users. Nonusers included clinicians who reported never attempting to use PROMs or ceasing to use PROMs. Frequent users comprised those who reported using PROMs as often as possible or always. Clinicians who reported occasional use remained unchanged.

The hospital monitored that, on average, from February 2023 to January 2024, clinicians accessed PROM data for nearly 3 out of 20 patients (14%; SD 1.6%) on the day of each patient’s consultation. This data served as a proxy for discussing PROMs with the patient. During the week of the patient’s consultation, which included the use of PROM data for triage and consult preparations, this percentage was slightly higher, averaging at 19.5% (SD 2.4%). While variations in the use of PROM data were apparent across subdepartments and individual clinicians, these specifics were outside the scope of this study.

Yearly data suggested a slight, albeit minimal, increase in the percentage of completed PROMs of unique patients accessed by clinicians using the dashboard (Figure 3). It is important to note that during 2024, the number of participating subdepartments grew from 38 to 70, implying that around half of the clinicians were in the adoption and implementation stages, while the other half were potentially moving toward the maintenance stage.

Clinicians’ self-reported use of PROMs, as derived from our survey in early 2023, revealed issues across all 3 RE-AIM outcomes. Adoption issues were apparent, with around one-tenth (5/47, 11%) of the respondents never attempting to review their patients’ PROMs responses (Table 2). Implementation issues, characterized by inconsistent use of PROMs as intended, became evident, with more than half (27/47, 58%) of the clinicians examining completed PROMs infrequently. Moreover, among those who examined PROMs, less than one-third (10/35, 29%) reported always discussing PROM data with patients. Less than one-fifth of the respondents (6/35, 17%) did not engage in this activity at all. Furthermore, 15% (7/47) ceased examining PROMs, indicating a maintenance issue.

Table 3 summarizes the strategies that the hospital implemented alongside their corresponding ERIC strategies, structured according to the COM-B constructs of capability, opportunity, and motivation. The hospital used 27 (37%) out of 73 discrete ERIC strategies to enhance clinicians’ use of PROMs alongside 5 proposed additions to the ERIC taxonomy. These are elaborated upon in the subsequent section and specified in Multimedia Appendix 4.

Strategies were drawn from 8 (89%) out of 9 strategy clusters, including using evaluative and iterative strategies, providing interactive assistance, adapting and tailoring to the context, developing stakeholder interrelationships, training and educating stakeholders, supporting clinicians, engaging consumers, and changing infrastructure. No strategies from the cluster financial strategies were used. Qualitatively, it appears that clinicians who used PROMs more extensively rated the supportiveness of the initiated strategies higher than those who used PROMs less. A few strategies were found to be missing, leaving barriers unaddressed or facilitators not leveraged.

Tracking and generalizing strategy specifications, such as temporality and dose, for all strategies was challenging due to the organic and pragmatic application of strategies tailored to specific subdepartment needs and the lack of a comprehensive documented program theory. Having been developed during preceding pilot phases and, most of the strategies were available from the onset of the hospital-wide implementation program in 2020 and refined in the course of time.

To enhance clinicians’ knowledge and skills in using PROMs, the CST organized kick-off presentations at each department when PROMs were installed. Clinicians received written instructions on using the PROMs dashboard and interpreting outcomes via email, which were later supplemented with video instructions. In 2022, a pocket guide, with exemplary sentences to discuss PROMs was cocreated with clinicians (internal document dated July 14, 2022). Training possibilities, both internal and external, were highlighted. Internal training was often conducted in collaboration with the person-centered care group, focusing on using PROMs in shared decision-making and among patients with limited literacy. The latter aimed to address limitations from clinicians believing that “PROMs cannot be discussed with the patient in question” (internal document, July 14, 2022). Flexible learning resources, such as e-trainings, were developed in response to time constraints:

In addition, efforts were made to align trainings with clinicians’ accreditation requirements.

However, interviewee 19 noted that training had not been an essential component of the program since 2020; interviewee 10 emphasized a tailored approach that addressed the specific needs of individual subdepartments. The CST increasingly reached out to subdepartments with low use of PROMs. They addressed barriers in perceived capability. One such barrier is clinicians feeling unable to influence certain PROMs outcomes that are affected by external factors, such as divorces impacting the quality of life. This perception led some clinicians to disregard these data (internal document, July 14, 2022). In response, the CST emphasized the importance of empathic listening and referring patients with critical PROMs outcomes to other physicians. Furthermore, since 2022, they provided on-the-job coaching, which proved valuable:

They also organized a few peer-to-peer discussion events each year, which interviewee 20 recognized as more impactful than traditional theoretical instruction. However, attendance issues arose with clinicians not always able to participate in these sessions. While survey results indicated that around 60% (26/43) of respondents were dissatisfied with the training or coaching received, skill issues were not a prominent factor impeding their use of PROMs (Table 4).

Another set of strategies targeted clinicians’ social and physical environments to create opportunities for using PROMs. Regarding clinicians’ social context, the CST aimed to create legitimacy and enable clinicians to discuss PROM data during multidisciplinary consultations, even with colleagues who had not yet initiated PROMs. They focused on raising hospital-wide awareness about PROMs and VBHC through mass communication via email, intranet, and marketing in public spaces. One clinician described a challenge in this endeavor:

Interviewee 14 also noted difficulties arising from the organization’s size, with hard-to-reach islands.

Anticipated future strategies include providing information about PROMs to general practitioners and external providers involved in patient care. Clinicians requested this strategy to facilitate communication with external health care professionals, such as making them aware of possible referrals on the basis of PROMs scores. However, this strategy was deemed feasible once PROMs were used among all patients, as partial implementation would require excessive manual effort (internal document, November 23, 2020; interviewee 19). Furthermore, a dashboard enabling patients to review their own outcomes will be developed to increase patient initiative in discussing PROMs:

Currently, to encourage patient initiative, patients are provided with resources such as an animated video explaining how to discuss PROMs during consultations.

Regarding the physical environment, several clinicians experienced or assumed PROM use in outpatient care as incompatible with their workload:

To mitigate this issue, the hospital sought to streamline the care process and simplify PROM use, as extending available consultation time was not feasible. Interviewee 10 explained as follows:

As per interviewee 15, integration of the PROMs dashboard into the EHR was considered a major improvement. Two other features reduced the registration burden: quick copying of PROMs scores into patient notes and a referral template for easily referring patients needing further attention on the basis of PROM outcomes (internal document, May 11, 2023). However, 79% (33/42) of the survey respondents had no opinion on the usefulness of the referral letter, possibly indicating unawareness of this feature. In addition, the CST implemented a reminder by prominently displaying the PROMs completion status on the patients’ EHR front page. This also eased the workflow by eliminating the need to open the PROMs dashboard to verify completion. More than one-third (15/42, 36%) of the survey respondents perceived this feature as helpful.

Moreover, to save time, PROM data were used in triage and the developed PROMs-IT infrastructure was used to have patients report their medication and lifestyle. This extension reduced the necessity for discussions on these topics during consultations (internal document, May 11, 2023). Despite these efforts, perceived time constraints remained the most frequently cited barrier among clinicians as of early 2023 (Table 4). Loading time delays of the PROMs dashboard exacerbated these constraints, resulting in decreased commitment and temporary cessation of PROM use. These delays were resolved, and a dedicated campaign was organized to encourage the reuptake of PROMs.

The use of PROMs remained nonobligatory, focusing on genuine engagement rather than mere compliance. The inclusion of clinicians in the steering committee aimed to create “visible standard-bearers from within the healthcare sector” (internal document, May 20, 2019) to enhance buy-in. The CST frequently communicated about PROMs and VBHC to achieve and sustain interest. They shared stories highlighting the benefits of PROMs from both patients and clinicians via a monthly newsletter and intranet. One of the ideas was that “if it’s beneficial for the patient, it usually has added value for the clinician as well” (internal document, July 14, 2022). Stories of championing clinicians were noted as a key facilitator in documents and interviews (internal document, August 31, 2020; interviewee 3). However, approximately half (21/43, 49%) of the survey respondents did not feel that the stories of other health care professionals positively influenced their attitude toward working with PROMs, with only 16% (7/43)—primarily consistent PROM users—reporting that they did. In communication, the CST also emphasized the importance of clinicians’ discussion of PROMs on patient adherence to complete subsequent PROMs, supported by local evidence (internal document, June 7, 2023). Sometimes, the CST chose not to overtly reference VBHC, focusing instead on visible outcomes for clinicians, such as PROMs enhancing the patient-clinician conversation. Interviewee 3 explained as follows:

This adjusted framing aimed to address resistance among clinicians who viewed VBHC sceptically, perceiving it, for example, as a “dull, container concept” (interviewee 6) and “a hype” (survey respondent 11). As another reason, terminology shifts in the national policy created confusion and was overwhelming (interviewee 3). Nonetheless, some clinicians perceived the hospital’s communication efforts negatively. Survey respondent 60 mentioned, “extremely selective advertising,” and survey respondent 11 noted that “a hallelujah story is being told that overlooks the differences in patient populations.”

Concurrently, formal pressures increased: the executive board verbally committed to achieving VBHC, supported by video messages and integration into hospital aims. Externally, government programs supported PROMs and VBHC [48,49], and external audits increasingly evaluated PROM use (internal document, July 14, 2022). These factors directly motivated clinicians and empowered department heads to prioritize PROM use among their employees, according to interviewee 14. Since 2022, the executive board requested subdepartments to report on their VBHC implementation status thrice yearly and complimented high-performing departments (internal document, July 14, 2022). The CST proposed that the executive board explicitly request information on clinicians’ use of PROMs rather than merely their inquiries among outpatients (interviewee 19). However, this proposal has not been implemented thus far, and the reasons for this remain unknown to the CST. Survey respondent 11 expressed concerns about such an approach:

This fear may have stemmed from clinicians’ belief that PROMs do not suit all patient groups or consultations equally (internal document, July 14, 2022).

To facilitate adoption, subdepartments were enabled to try PROMs with a subset of patients. Interviewee 19 highlights the importance of clinicians developing direct experience with PROMs:

However, implementing PROMs among a subset of patients had an unintended consequence:

Forgetting to use PROMs was the second-most cited barrier by clinicians (Table 4). To stimulate sufficient PROM survey volume, several strategies aimed to enhance patient response rates to PROMs. Furthermore, the CST aimed to evidence the benefits of PROMs and their impact on professionals’ work experience, including time investment. Interviewee 6 stated the following:

However, these efforts were hindered by limitations in defining measurable outcome measures, data availability, and difficulties in consolidating local evidence and making it convincing to other disciplines. Moreover, the hospital conducted both formal and informal evaluations of PROMs among clinicians to gain insights into their experiences, motivations, and needs, subsequently adapting implementation plans on the basis of these findings. This included prioritizing disease-specific PROMs, extending training opportunities, and expanding PROM data use to areas beyond outpatient care, such as triage, research, and care pathway improvements, tapping into different clinician motivations (interviewee 17). In addition, the CST alleviated clinicians’ concerns about accountability by adapting patient communication to clarify patients’ responsibility for timely contact in critical situations via traditional means.

To achieve and maintain effective implementation, the CST provided feedback on PROM use to subdepartments. The frequency and format of feedback was tailored to departmental preferences, with some favoring nonanonymous feedback, such as “[clinician’s name] has accessed this many PROMs” (interviewee 19), for a competitive aspect, while others preferred anonymity. However, nearly half of the survey respondents (20/43, 47%) experienced infrequent feedback. Furthermore, local departments were increasingly empowered to adapt PROMs and their application to their context, addressing limitations of uniform solutions on buy-in, perceived ownership, and local fit (interviewee 18). Subdepartments could incorporate disease-specific items that were deemed critical:

Moreover, they could determine patient eligibility criteria, adjust inquiry frequency, and choose from multiple dashboard layouts (internal document, May 11, 2023). Individual clinicians were also enabled to manually exclude patients from PROMs if needed, aiming to mitigate challenges from upset patients:

Moreover, it aimed to overcome a specific drawback noted by survey respondent 4:

Overall, interviewee 20 perceived the following:

Unaddressed barriers also appeared in relation to clinicians’ capability, opportunity, and motivation. Various interviewees highlighted shortcomings in contemporary education, noting a lack of emphasis on developing the professional attitudes necessary for effectively using PROMs in VBHC. This includes insufficient focus on understanding the impact of disease on the individual, their social environment, and work (interviewee 20), as well as limitations in shared decision-making, inclusive care, and data-driven improvement work (interviewee 19). Interviewee 14 pointed out that VBHC requires clinicians to take shared responsibility for patient outcomes rather than dismissing certain issues as outside their scope, observing that this attitude is “only really embraced by a few people.” Interviewee 18 emphasized another limitation in education, stating the following:

In addition, there seem to be limitations in clinicians’ capacity for change and medical leadership (internal document, July 13, 2020). Interviewee 11 noted that clinicians are typically “quite conservative” and find changing behavior “very difficult.” Interviewee 20 observed that PROMs resonate more with nurses, who focus more on caring and benefit from longer consultation times, while physicians typically focus on curing. The CST found that departments with a dedicated nurse specialist or physician assistant achieved higher use of PROMs compared to those without such staff. Some subdepartments tasked these nurses with discussing patients’ PROM data and flagging any issues for the medical specialist who would see the patient next (internal document, May 11, 2023). However, the CST struggled to leverage this facilitator across the hospital due to budget constraints in local subdepartments to hire nurse specialists or physician assistants and the nonbillability of PROMs discussions by nurses. Overall, the CST believes that patients’ experiences of symptoms, functioning, and quality of life should be a central focus in all care activities rather than being narrowed to nurse consultations alone.

There also appeared limitations in the structural integration of PROMs for VBHC into the onboarding of new staff. Survey respondent 39 noted the following:

Interviewee 19 explained that despite 2 attempts by the program team to include PROMs in onboarding, these efforts faced rejection due to PROMs not yet being uniformly adopted across all departments.

Behavior change was further hindered by limitations in opportunity and motivation. Clinicians’ contemporary feelings of work pressure and well-being were considered to have a negative impact:

Interviewee 19 noted the challenge of change fatigue alongside “a pervasive aversion to any form of registration and data handling.” A fine line existed between meeting data needs and experiencing an overwhelming availability of data. Interviewee 11 noted a lack of tension and urgency to use PROMs:

Interviewee 18 noted the challenge from the ongoing volume-based rather than value-based payment, impeding clinicians’ opportunity to use PROMs to make value-based decisions:

Despite implementing numerous strategies (objective 2), the hospital continued to face limitations in clinicians’ use of PROMs (objective 1). Therefore, as the third objective, we aimed to broaden our understanding of the key factors influencing clinicians’ use of PROMs in early 2023. We first highlight the primary motivations driving clinicians to use PROMs, followed by the key factors contributing to suboptimal use.

Survey respondents, excluding those who never attempted using PROMs, were asked to select the most prominent reasons for using (or having used) PROMs. Response options were identified from document analysis. Respondents had the opportunity to add additional factors, but this option was not used.

Among all survey respondents, 38% (16/42) reported using PROMs primarily because of patient benefits and 36% (15/42) reported using it because of personal benefits (Table 4). Nearly one-third (12/42, 29%) experienced both types of benefits. In addition, 45% (19/42) of respondents used PROMs because they felt it was expected of them, with 31% (13/42) citing this expectation as the sole reason for using PROMs. Only 12% (5/42) used PROMs because of patient requests.

Table 4 also shows responses among 3 user profiles: nonusers, occasional users, and frequent users (refer to the Methods section). Perceived benefits were the primary driver for frequent users, while feeling expected to use PROMs was the most frequently selected reason among occasional users and those who ceased use. Frequent PROM users selected more factors compared to less frequent users, suggesting that they perceived more reasons to use PROMs.

Survey respondents were also asked to select the most prominent factors that impeded their use of PROMs, if experienced, in a multiple-choice question. Among all respondents, key issues were time constraints (24/44, 55%) alongside dashboard functioning limitations (24/44, 55%), followed by forgetting (22/44, 50%; Table 5). Limitations in reinforcement and believing to have a limited impact on improving PROMs domains were rarely selected as reasons impeding PROM use.

Disaggregating into the 3 user profiles, we found that dashboard functioning limitations impeded all groups. While all nonusers perceived time constraints, this factor was less frequently selected among the other profiles. Nonusers often cited a misalignment between PROMs and how they preferred to work as a substantial factor. Among occasional users, forgetting to use PROMs was a critical issue, a challenge also faced by current nonusers. The frequent PROM users felt impeded by the low volume of completed PROMs. Overall, those who used PROMs less frequently or not at all selected more factors compared to more frequent users.

Respondents were also asked to grade the outcomes of implementing PROMs in outpatient care and the implementation process; respectively, average scores were 4.9 and 5.4, both out of 10. Clinicians who used PROMs more frequently were more satisfied (Table 5).