This study was conducted using a sequential explanatory mixed methods study design (Figure 1). The reason we opted for this design was that this 2-phase design starts with a quantitative phase and is followed by a qualitative phase to explain and supplement the quantitative findings [9]. This approach fits best with what we intended to achieve in our study, namely, to first quantify the CVRM extractability, and then, based on these results, we aimed to explain our findings by interviewing clinicians, the end users of the dashboards.

We developed the UCC-CVRM dashboards using the data visualization software Power BI Desktop [10]. During the developmental phase of the dashboards, approximately 1 year before implementation, we had several meetings with clinicians from all participating departments to adjust the patient selection on which the data extraction for visualization in the dashboards would be based, if necessary, and to discuss the content of the dashboards. During this early phase, the initial goals and key functionalities of the dashboards were defined based on identified clinical needs and priorities. Following this internal development, a first set of prototype dashboards was presented to all participating departments. During these meetings, we invited clinicians from each department to critically assess the layout, content, and relevance of the selected indicators. Discussions focused on both the clinical usefulness of the dashboards and the patient selection. Feedback received during these sessions led to refinements in both the visual presentation and the underlying patient selection criteria.

All contact persons receiving the dashboards (ie, the clinicians involved in the development of the dashboards) also received information on the specific locations in the EHR from which the indicators were extracted, as presented in Table S1 in Multimedia Appendix 1. We sent the dashboards on a regular basis (ie, monthly or every three months, depending on the preference of the department) to clinicians of the participating departments who were appointed as contact persons for UCC-CVRM activities (Table S2 in Multimedia Appendix 2). The dashboards included information on (1) the codes used to identify the subset of patients to which CVRM guidelines apply, (2) the time frame indicating the period when the patients visited the UMC Utrecht, (3) the total number of patients included in this dashboard during the specified time frame, (4) a bar chart showing the percentage of patients for whom the 8 CVRM indicators could be extracted (so the 10-y cardiovascular risk could be calculated), (5) the percentage of patients who had values above the reference value for 6 indicators, (6) sex distribution of the patients included in the dashboard, (7) the location of the appointment in the hospital, (8) the distributions of CVRM indicators, (9) the percentage of patients using prespecified types of medication, and (10) the average number of extractable items per treating clinician (Figure 2) [11]. Pseudonymized clinician codes were presented on the x-axis of this item, so that individual treating clinicians were not aware of their own nor their colleagues’ code. Small differences in the content were made depending on the preferences of the department (Table S2 in Multimedia Appendix 2).

The primary outcome of the quantitative part of this study was the extractability of the CVRM indicators and whether the extractability changed over time after the implementation of the dashboards. Indicators were considered extractable if the value could be extracted from a predefined structured field of the EHR. To note, if performed locally in our hospital, laboratory values are automatically registered in the designated structured fields of the EHR. Thus, if a laboratory value was not extractable, no laboratory measurement was done locally in the UMC Utrecht. The other CVRM indicators can be registered in structured fields of the EHR or other fields, such as clinical notes. Thus, if these indicators were not extractable, the indicator was either not measured or was registered in a field from which the information is not extractable (eg, clinical notes).

The outcomes of the qualitative part were the perceptions, views, and explanations of clinicians related to the information in the Power BI dashboards, assessed by means of the semistructured interviews.

The study protocol was submitted to the Research Ethics Committee of the UMC Utrecht for review (23U-0181) before the initiation of both phase 1 and phase 2 of this study. The ethics committee determined that this study was exempt from the Dutch Medical Research Involving Human Subjects Act (WMO). The UMC Utrecht carried out an independent quality check to ensure compliance with legislation and regulations. Patients who objected to the (re-)use of their data for scientific research by indicating their opt-out in the UMC Utrecht’s patient portal were excluded from phase 1. For phase 2 of the study, we obtained written informed consent from all respondents before the interview. The participants in this study did not receive a compensation for their participation.

We used the COREQ (Consolidated criteria for Reporting Qualitative research) checklist as a guideline to report our qualitative research results.

Many of the respondents’ perceptions that explained the suboptimal extractability of CVRM risk factors relate to issues at a more technical level. These technical reasons are grouped into the following themes: the usability and familiarity of the EHR and the perceived incompleteness of the dashboards due to the data fields in the EHR used for data extraction.

Several operational reasons were mentioned that could explain the extractability of indicators remaining unchanged since the implementation of the dashboards. These relate to the following themes: time constraints and the habitual work practices of health care providers.

This study aimed to evaluate whether the implementation of dashboards in our LHS improved the structured registration of CVRM indicators in patients with an increased risk of CVD. The extractability of the indicators was low, and although the extractability varied between departments, we did not observe substantial changes over the period during which the dashboards were shared with the departments. From that standpoint, the dashboards have not been effective. While in-depth semistructured interviews revealed that respondents considered such dashboards as useful, disproportionate burdens were perceived by the respondents themselves or their colleagues, explaining the low and unchanged extractability of the CVRM indicators. The registration of the indicators was hindered due to organizational (ie, discussions on who was responsible for CVRM and the lack of agreements surrounding the registration), technical (ie, usability and technical knowledge of the EHR system and its functionalities, incompleteness of the dashboard), and operational issues (ie, time constraints and individual habits). Potential ways to improve the extractability included making use of the patient portal, using peer comparison, and educating about the clinical benefits of structured registration.

Several reviews evaluated the use and impact of dashboards in a health care setting by assessing, among others, before-after studies and randomized controlled trials (RCTs) [5,6,18]. However, the settings, designs, and aims of the studies included were highly heterogeneous, and the results of the systematic reviews are inconclusive [5,6,18]. The findings suggest that dashboard effectiveness may depend on clinical context, patient population, theoretical underpinning, and specific implementation features. So far, none of the studies have focused on the extractability or structured registration of CVRM indicators as the outcome of interest. Still, one of the included RCTs, a parallel arm cluster RCT, showed that a computer-guided intervention, which included decision support, audit and feedback tools, and training, improved cardiovascular risk factor screening. The impact of the intervention was compared to usual care (ie, to sites not randomized to the intervention and, thus, did not have access to the intervention tool) [19].

Respondents in our study acknowledged that recording CVRM indicators in a structured manner is valuable for clinical care and secondary use of the data, yet not all care providers seemed to support and adopt structured registration in practice. This is in line with previous research [20]. The reasons provided by the respondents of our study are echoed in other studies [21-23]. For example, Marani et al [21] compared the availability of data in structured fields of the EHR with manual chart review in diabetes patients and found that the availability of data extracted from structured fields was much lower than when the information was manually searched for in EHRs [21]. While we only extracted data from structured fields and, thus, cannot be certain that the CVRM indicators were in fact present in other parts of the EHR, it is highly likely that the information would indeed be more complete had we included unstructured sections of the EHR, an issue also mentioned by the respondents. However, to support clinical decision-making, improve patient care, and validly reuse routine care data for research and other secondary activities, routine care data should at least equal the quality of the information registered currently in the EHR. This can only happen when information is stored in a structured and standardized format in the EHR [20,23,24]. It is, therefore, desirable to improve the registration in structured fields of the EHR, through systematically creating awareness and educating medical students, specialists in training, and the present medical staff.

As mentioned, the low extractability of CVRM indicators was partly explained by the perceived organizational issues. Some clinicians did not seem to feel responsible for CVRM and felt that agreements surrounding registration are lacking. This is supported by other studies that assessed the care providers’ attitudes toward structured problem lists in the EHR [23,25]. One study found that general disagreement between specialists and primary care practitioners about who is responsible for the problem list was a major cause of the gaps in the problem lists and mentioned the need for guidelines and policies to clarify this [25]. From a technical standpoint, the user interface of the EHR was seen as a barrier to structured registration, especially for indicators that required opening additional tabs on top of the main tab. In March 2024, however, the UMC Utrecht updated its EHR system, which now has the possibility of creating customized templates that could include all fields that are deemed as necessary to complete. Respondents acknowledged that this update is expected to improve the extractability of CVRM indicators in the long term. Future studies are necessary to explore this expectation in our hospital.

Our findings indicate that there are underlying barriers to structured registration that need to be addressed and suggest that educational efforts are needed to improve care providers’ knowledge on the EHR system functionalities and the benefits of structured registration of CVRM indicators. Educational efforts have been found to be the most efficient and effective interventions to improve structured registration of routine care data in EHR records [26]. During these efforts, it is important to emphasize the direct benefit of structured registration for the care provider, rather than focusing on the secondary benefit (ie, research). Benefits for the care provider include that using a structured registration leads to better and faster decision-making [27], and that it enables the use of CVD algorithms used for clinical decision support. These clinical decision support systems (CDSSs) oftentimes require a standardized set of data to assess the patient risk and treatment possibilities [28]. These CDSSs can be linked to the EHR and automatically load the EHR data into the CDSS to provide a risk prediction during routine clinical practice [29], removing the need for manual entry, which will likely motivate care providers to pay more attention to structured and systematic registration of CVRM indicators during clinical practice. In addition, technical training on how to use all functionalities of the EHR system seems equally necessary, as it was mentioned by the respondents that not all health care providers are equally adept with the EHR system, hampering efficiency. Besides, to reduce the administrative burden on care providers, it is useful to explore the use of the hospital’s patient portal for the collection of CVRM indicators that can easily be assessed and documented by patients themselves, such as blood pressure. The portal should be linked to the EHR, and the values automatically loaded into the predefined structured fields of the EHR.

Furthermore, as mentioned by others [23,26,30], clarification and policies regarding responsibilities for structured registration and harmonization of items to be recorded are necessary. Moreover, behavioral change must be realized to gain the benefits of these dashboards. Peer comparison and allowing departments to compare their performance with other departments can be an effective strategy, as mentioned by the respondents in this study, and has proven to be successful in previous research [30-32]. While a peer comparison component was already present in our dashboards (ie, a figure showing the average number of CVRM indicators recorded in structured fields per clinician), pseudonymization of all names obscured such comparison and may have led the component to be less effective. Clinicians are expected to become more motivated to change their registration practices once we make known which code belongs to them, whereas their colleagues’ codes remain anonymous to them.

To our knowledge, we are among the first to report on a mixed methods study to evaluate the implementation of dashboards for feedback purposes to clinicians. The mixed methods design allows for combining the strengths that are inherent to the individual qualitative and quantitative approaches and to mitigate the weaknesses [33]. While there are various mixed methods study designs, we opted for the sequential explanatory design, in which the quantitative data collection and analysis precedes the qualitative part. The benefit of using this design is that it allowed for a thorough explanation of our quantitative results, as we were able to design the qualitative phase based on the quantitative results [33].

Our study has several limitations that warrant careful interpretation of the results. Regarding the comparison between the ‘before UCC-CVRM’, ‘protocolized UCC-CVRM’, and ‘routine care UCC-CVRM’, it is important to note that the time window of the data collection in the ‘before UCC-CVRM’ is somewhat different than the time window of the ‘routine care UCC-CVRM’ [2], which may explain some of the differences in the extractability between the two, making them somewhat less comparable. Besides, the extractability of the CVRM indicators in the ‘protocolized UCC-CVRM’ is based on patients who provided explicit consent only, as an informed consent procedure was in place at that time. The extractability of the CVRM indicators in patients without consent was lower [34,35].

In addition, given that this is a single-center study, the generalizability of the findings to other settings may be limited. Further research in different settings is needed to validate our results. Furthermore, although coding saturation was achieved, the relatively limited sample of interviewed clinicians, along with the underrepresentation of female clinicians among the interviewees, may have limited the diversity of perspectives and might not fully represent the views of all health care professionals or all clinical departments. A larger, more diverse sample might provide additional perspectives on the perceived barriers.

Furthermore, the results of our study may be influenced by (self-)selection bias. The majority of the respondents who were invited to participate in an interview were the primary contact persons within the cardiovascular LHS for such initiatives. Those who volunteered to participate in our study might have had particularly strong views or experiences related to the topic.

Next, the respondents work in an academic hospital and are therefore expected to actively participate in clinical research and implementation initiatives, which may have led to social desirability bias. However, we believe that this bias was limited as we took several measures to minimize the risk of social desirability bias. First, we used semistructured interviews as the data collection tool to make respondents feel at ease to express their individual views, and the interviews were held in an environment where there were no external influences, interruptions, or third parties present. Second, before their participation in the study, the respondents were assured of the confidentiality of the interviews and the pseudonymization of the transcripts. Finally, all but one of the respondents were familiar with the interviewer as they had previous contact moments during the development of the dashboards. Growing familiarity with the interviewer has previously been associated with lessening the risk of socially desirable answers [36].

Our study did not allow for comparison between health care professionals who actively used the dashboards and those who did not, as we did not track the engagement with the dashboard (eg, opening or active use of the dashboard). As a result, we cannot fully distinguish whether limited impact of the dashboard reflects ineffectiveness or a lack of use. Future research incorporating engagement metrics could help elucidate this further.

Finally, a limitation of our study is that we were unable to disentangle the effects of the dashboard’s individual components on registration practices. All participating departments received a similar multifaceted dashboard incorporating multiple design features and data elements simultaneously, except for certain specifics, as illustrated in Table S2 in Multimedia Appendix 2. As such, we cannot ascertain which specific features, if any, were most influential in changing registration behavior. Future studies should consider evaluating dashboards with varying design elements and theoretical underpinnings to better understand which components are most effective in supporting structured data registration in CVRM.

The extractability of CVRM indicators from structured fields of the EHR varied between departments and overall did not show improvement over time following the implementation of the dashboard in our LHS. This was explained by organizational, technical, and operational issues, including unclear responsibility for CVRM care, suboptimal technical knowledge of the EHR’s functionalities, limitations of the EHR system, and time constraints. These findings provide guidance on what aspects to consider for the extractability to be improved, which, in the end, will be beneficial for both clinical practice and scientific research using real-world data.