As an initial step in developing a comprehensive search strategy to identify all relevant papers on the design and related concepts of CDSS, we conducted a preliminary literature review and experimented with search terms. This process enabled us to establish an overarching understanding of the current knowledge landscape, refine the review’s focus, enhance the selection of search terms, and finalize the search strategy.

We selected 5 data sources, including 4 major electronic databases (PubMed, Web of Science, Scopus, and IEEE Xplore) and the Journal of Decision Systems, which is specific to decision support research. These sources were chosen to ensure broad coverage across health care, engineering, and design science. Related design and human-computer interaction keywords, including “user-centered design,” “design,” “user experience,” “implementation,” “evaluation,” “usability,” and “architecture,” were selected for use as search terms. Since PubMed supports MeSH (Medical Subject Headings) indexing, we also explored subject headings related to CDSS.

Based on the preliminary experimentation, we refined our selection to 4 data sources: PubMed, Web of Science, Scopus, and IEEE Xplore, due to their extensive coverage of health, engineering, and design sciences. The Journal of Decision Systems was excluded, as all its papers were already indexed in Web of Science and Scopus, making its inclusion redundant. Furthermore, the MeSH subheadings for CDSS did not sufficiently capture aspects related to design, usability, and UX, making MeSH-based searches in PubMed suboptimal. Consequently, we adopted 6 design-related UX and human-computer interaction keywords (“design,” “user experience,” “implementation,” “evaluation,” “usability,” and “architecture”) across all selected databases, including PubMed, to maintain consistency.

Regarding the scope of the search strategy, our experimentation revealed that full-text searches in PubMed and Scopus retrieved a high volume of irrelevant papers. In contrast, restricting searches to title, abstract, and keywords significantly improved specificity without omitting relevant studies, thereby reducing noise and redundancy in the search results. Based on these findings, we refined our search approach by conducting title/abstract/keyword searches in PubMed and Scopus while maintaining full-text searches in Web of Science and IEEE Xplore.

To retrieve relevant papers for the review, each of the 6 finalized keywords (“design,” “user experience,” “implementation,” “evaluation,” “usability,” and “architecture”) was combined with the phrase “Clinical Decision Support Systems” using Boolean operators (“AND” or “OR”), for example, “Clinical Decision Support Systems” AND “Implementation” for searching each of the 4 electronic database sources.

This comprehensive search, covering the period 2013-2023, was performed in November 2023 to identify English language papers. The identified papers were then exported using the bibliographic management software Zotero (Corporation for Digital Scholarship) [29], screened, and included in this review. The detailed search strategy is provided in Multimedia Appendix 1.

To screen and include eligible papers from the search results, the research team established predetermined inclusion and exclusion criteria (Textbox 1). Two researchers (AAB and JL) discussed and finalized these criteria, independently conducted data screening and extraction, assessed eligibility, reviewed included papers, and mapped study characteristics and findings. At each review stage, both researchers followed the same procedure in parallel, compared results, documented discrepancies, and resolved inconsistencies collaboratively. If consensus was not reached, a third researcher (MV) was consulted for a resolution.

Papers that did not center on a designed CDSS tool or intervention for a health condition or did not discuss the design aspects of the CDSS, such as its design approach, user interface, or integration, were excluded. Furthermore, papers that solely discussed the technical implementation or architecture of a CDSS, such as those focused on implementing machine learning techniques or comparing these methods, as well as editorials, scoping reviews, systematic reviews, books, conference abstracts, and study protocols, were also excluded. Textbox 1 shows the detailed inclusion and exclusion criteria.

Two researchers (AAB and JL) then independently screened abstracts to exclude ineligible papers before full-text review. A total of 130 papers were fully reviewed for eligibility based on the following: (1) whether the paper focused on a designed CDSS, (2) whether it described at least 1 aspect of CDSS design, evaluation, usability, implementation, UX, or architectural design, and (3) other criteria, including language (English), availability of sufficient information, and exclusion of conference abstracts.

After this full screening, 40 papers were included in our final analysis for the data extraction and analysis. Disparities that arose during the screening were usually discussed and resolved by the 2 researchers (AAB and JL) involved in the process, while unresolved disparities were referred to a third researcher (MV) for resolution.

Two researchers (AAB and JL) independently reviewed all 40 included papers and conducted data extraction. The following specific categories of information were extracted from each paper and recorded in an Excel spreadsheet:

Discrepancies between the 2 researchers were discussed and resolved collaboratively. Where consensus could not be reached, a third researcher (MV) was consulted for adjudication.

The review identified 4 primary approaches used in CDSS design. The UCD approach was the most common, applied in 23 out of 40 studies (57.5%). Other approaches included the knowledge-to-action framework (1/40, 2.5%), the agile business process development approach (1/40, 2.5%), and reflections based on the design of an existing system (2/40, 5%). Additionally, 13 studies (32.5%) did not explicitly reference a design approach but conducted usability evaluations or user testing.

UCD emphasizes active user involvement throughout the design process [34]. In contrast, the knowledge-to-action framework focuses on knowledge creation and translation, engaging end users at specific framework stages [35]. The agile business process development approach takes an integrated business process model to guide technology design and implementation [36].

Although UCD was the most widely used approach, its application varied across studies. Many studies using UCD involved users only in select design stages rather than throughout the entire process. Only a few studies fully adhered to the UCD framework by engaging users consistently across all phases of design [37-39]. As a result, while these studies claimed to use UCD, they often did not fully implement the approach as intended. For instance, some CDSS design stages lacked active end user participation, which is a core principle of UCD.

Across all approaches and frameworks, various data collection techniques were used, including interviews, focus groups, personas, questionnaires, and surveys. These techniques supported different aspects of the design process: informing the overall design [38,40-43], exploring user needs and workflow [44-46], contextualizing and implementing design [47-49], and ensuring usability, functionality, and effectiveness [50-52].

All CDSS examined in the reviewed papers underwent some form of usability evaluation or testing to assess functionality [42,51-54], feasibility [38,40,49], acceptability [37,38,47,49,51,52,55-58], accuracy [41,58-61], quality of experience [50,62-65], and usefulness [60,65,66]. Some papers reported conducting usability testing (14/40, 37.5%), with 2 studies reporting using the System Usability Scale as an assessment tool [52,67].

Usability evaluation techniques used for data collection included think-aloud protocols, questionnaires, and interviews. The theoretical underpinnings and principles that guided the usability evaluation included the task technology fit theory, the unified theory of acceptance and use of technology, user heuristics, and the sociotechnical model. In addition, analytical tools leveraged for usability evaluation included the receiver operating characteristic curve. The design approach findings are shown in Table 1.

The review also examined user interaction and engagement with CDSS, delineating the design mechanisms used for input and output access, as well as for alerting and notifying users, as illustrated in Figure 2. The primary users of CDSS in the reviewed studies were predominantly clinicians (37/40, 92.5%), with only 3 systems (3/40, 7.5%) designed for both clinicians and patients.

Primarily, CDSS uses 3 channels for data or information access and input. The first is manual entry, where users input necessary data or request recommendations [38,40,41,44,46-49,55,56,58,59,64,65,67,69]. The second is integration with EHR, where input or output data are automatically retrieved by an existing EHR system integrated with the CDSS [37,42,44,45,49,52-55,60,61,64,69-71,74]. The third is wireless transmission, where input data are sourced from other devices such as sensors via Bluetooth or similar wireless data transmission protocols [53].

The types of input data and information required by CDSS for making decisions encompassed combinations of 5 categories of data sources: vital signs data such as temperature, blood pressure, and heart rate [38,46,47,54,58,62,63,65,66]; laboratory data (test results) [41,51,53,56,59,66,73,76]; EHRs (specific or combinations of information such as clinical history, comorbidities, or prescriptions) [37,42,44,45,49,52-55,60,61,64,71,74]; radiography data (results of image scans) [59]; and symptoms of condition (specific condition–related vital symptoms, eg, signs of inflammation for fractures) [40,46,50,56,62,67,69,75].

As output, systems deliver recommendations on treatment plans, workflow, or dosage [49,54,64,74]; reports on conditions including providing explanatory education on the situation and recommending options [41,53,54,59,65,68]; and alerts as feedback [41,51,53,55,67]. To facilitate the effective relay of output feedback, ensure visibility, engagement, and adherence; various alert and notification mechanisms, including prompts such as beeps or pop-ups [46,53,55,67], color coding of recommendations to signify severity [40,41,51], and sending messages as reminders [55,64], are used.

All CDSS reported in the reviewed papers were knowledge-based, classified according to their underlying knowledge, reasoning, or support structures. No non–knowledge-based architectural designs were identified. These systems follow 1 of 4 reasoning architectures: rule-based, machine learning-based, case-based, or ontology-based. Rule-based systems use deterministic “if-then-else” rules for decision-making [77]. Machine learning–based systems develop knowledge from existing clinical data to apply it to new datasets [78,79]. Case-based reasoning systems compare new cases with stored past decisions to identify similarities and determine outcomes [80]. While ontology-based, leverage semantic web technologies to represent medical concepts and support decision-making [81]. However, there was a notable absence of ontology-based reasoning systems among the reviewed papers, likely due to the emerging nature of this category of CDSS [81].

Also, a significant trend observed in the review was the increasing use of AI and machine learning–based approaches to develop predictive decision support systems [39,43,47,48,53,57,60,61,73,75]. These AI-driven systems aim to enhance diagnostic accuracy and provide personalized recommendations based on data patterns and historical cases.

Functionally, the CDSS reviewed in the papers fall into 3 main categories: diagnostic systems, which are used for screening and identifying health conditions; treatment systems, which provide recommendations and guidance on treatment options; and dosage prescription systems, which assist in determining the appropriate dosage or combinations of drugs and treatments. The design architecture and functional scope of the reviewed CDSS are summarized in Table 2.

A notable development among some CDSS is the integration of interactive, social media–like modules that allow clinicians to share information, engage in discussions, and enhance their knowledge on supported health conditions. These collaborative features promote peer-to-peer learning and real-time knowledge exchange, enhancing the overall effectiveness of the CDSS [41,47,50,52,53,67].

An important and emerging trend in the design of reviewed CDSS is their integration with other clinical systems. We assessed whether these systems were integrated with existing digital health technologies such as EHR systems, or whether they were explicitly designed with integration in mind. This assessment is critical, given the recent shift toward the adoption of emerging digital health technologies and the development of interoperability standards such as FHIR and architectural platforms such as the Substitutable Medical Applications and Reusable Technologies [82].

Our findings indicate that most CDSS were stand-alone systems, with only 20% (8/40) of the CDSS reviewed being integrated into EHRs and existing workflows, and 17.5% (7/40) of them being intentionally designed with integration as a core feature [37,42,44,45,49,52-55,60,61,64,71,74]. However, a significant portion of the CDSS reported in the reviewed papers lacked integration considerations in their design.

A fundamental design challenge of CDSS pertains to UX and expectations. Users expressed concerns regarding how recommendations are presented and accessed [46] and inconsistencies in workflow integration [49,62]. Several studies highlighted limitations in the presentation design of CDSS recommendations, noting rigid structures that restrict users from incorporating their own perspectives [45,46,70,74]. As a result, users may feel obligated to follow system recommendations without the flexibility to adjust or choose from alternative suggestions.

Additionally, challenges arise in users’ ability to appropriately apply CDSS recommendations, prompting the need for visual aids such as video explanations to clarify their application and reduce ambiguity [46,47,65]. Also, information overload was another prevalent concern, as users may experience fatigue from the constant prompts, notifications, or input requests generated by CDSS [40,55,76].

Moreover, the design of CDSS to track usage and activity logs can evoke feelings of covert surveillance, impacting UX [55]. Furthermore, a mismatch between users’ practical workflow and the requirements of CDSS poses significant challenges [42,49,51,62,76]. For instance, CDSS may demand information that is available only several steps ahead in the operational workflow, creating a disconnect between user practices and system functionality [49,51]. Finally, other UX-related issues identified included requirements for training and technical support in the deployment and use of CDSS [38,49,75].

A critical challenge concerning CDSS revolves around the attributes of their trustworthiness. Key fundamental challenges related to their validation, including determining where to initiate the validation process, establishing a “ground truth” for validation, and defining a criteria for success in validation [54,60,64,71]. For instance, how should contextual factors be factored in, and how can learning be incorporated into the validation process? Moreover, what constitutes success: tangible improvements in patients’ clinical outcomes or the functional effectiveness of the underlying technology? If the former, how can other contextual behavioral contributors be modeled to validate success?

Another significant is ambiguity in recommendations. Some terminologies used in CDSS recommendations may be unfamiliar or interpreted differently by users, leading to potential miscommunication [46,47,65]. Furthermore, trust issues arise, as more experienced clinicians may disregard CDSS recommendations when they conflict with their own clinical judgment or opinions [45,72,74]. These challenges impact the perceived validity and reliability of CDSS, influencing user trust and adoption [39,44,56,61,71,72].

However, emerging technologies such as XAI and context-aware machine learning systems offer promising solutions to address trust and validation challenges in CDSS. XAI enhances transparency by making AI systems’ inner workings more interpretable for health care professionals, enabling them to identify potential flaws and assess the trustworthiness of system outputs [83]. Additionally, context-aware machine learning systems that integrate patient-specific and environmental factors can generate more personalized and relevant recommendations, thereby reducing ambiguity and fostering trust in CDSS [84].

The effective design and implementation of machine learning and AI-enabled CDSS hinge fundamentally on the availability, accessibility, and quality of individual patient data, extracted ideally from their EHR. However, related challenges persist in this domain. A primary issue highlighted across the reviewed papers concerns access to quality data [47,53,67]. While vast clinical datasets may be available, navigating access protocols can be cumbersome due to data access privacy concerns [85,86].

Furthermore, existing data may lack curation or could be in formats that are unsuitable for CDSS integration [71]. Moreover, quality-related issues, such as errors and missing data, can arise during the process of transforming data into formats acceptable to CDSS, such as converting paper-based clinical records into digital data formats or converting images [41]. Other related challenges highlighted included interoperability issues, where different data standards used in the collection of medical data make their uptake by CDSS difficult [71]. Despite these obstacles, the adoption of EHRs and interoperability standards such as FHIR promises opportunities for improving data access and quality assurance in CDSS [87].

The complexities inherent in the design of CDSS, coupled with integration challenges, present significant hurdles. Translating guidelines into decision rules that are implementable within CDSS presents nontrivial challenges [41,54,55], often requiring extensive data collection over extended periods.

Additionally, the diverse range of users adds another layer of complexity, particularly in designing input mechanisms that accommodate variability in user interactions [54,71]. For instance, designing a system capable of handling vague or incomplete user responses, which may occur with user input into CDSS, particularly in systems that do not support free-text entry CDSS, remains a major challenge. Moreover, modelling the clinical workflow for integration into the design of CDSS proves difficult due to the dynamic and constantly evolving nature of the clinical environment [51,71].