The study was conducted within 2 emergency medical service (EMS) regions in the Netherlands: Rotterdam-Rijnmond and Zuid-Holland Zuid. This region consists of 2 EVT-capable hospitals and 7 general hospitals, where only IVT is available. Together, these EMS organizations cover a population of approximately 1.7 million inhabitants in an area of 863 km² (Rotterdam-Rijnmond) and 720 km² (Zuid-Holland Zuid), with a total estimated 5000 suspected strokes yearly [11,14]. Approximately 250 licensed ambulance paramedics were employed across these 2 regions. The STA was implemented in January 2024 in both regions as part of the Prehospital Triage of Patients With a Suspected Acute Stroke (PRESTO-2) study and formally introduced as the standard of care for prehospital stroke triage [15]. The STA is registered as a Conformité Européenne (CE)-compliant medical device and as a Medical Device Regulation (MDR) class I software (The European Database on Medical Devices [EUDAMED], unique device identification-device identifier [UDI-DI]: 08720892217905).

Embedded within the PRESTO-2 study, this study used a sequential explanatory mixed methods design [15,16]. The quantitative phase integrated objective dashboard analytics with survey data (Figure 1). The dashboard was used to monitor actual STA engagement throughout the implementation period by tracking longitudinal data (eg, monthly triage volumes and active user rates). These insights provided a baseline of system usage, while the survey captured paramedics’ initial perceptions of the implementation process, perceived usability, and attitudes toward long-term sustainability. Combined, these quantitative findings provided preliminary insights into potential implementation challenges, which informed the subsequent qualitative phase. This second phase consisted of semistructured interviews designed to contextualize and nuance the quantitative results by exploring the contextual factors shaping the implementation process, outcomes, and workflow impact.

To evaluate paramedics’ experiences with the implementation of the STA, the study focused on 3 core concepts: the implementation process, the implementation outcomes, and the impact on clinical workflow (Table 1). These concepts were informed by established implementation science frameworks, including the Non-adoption, Abandonment, Scale-up, Spread, Sustainability (NASSS) Framework [3], Consolidated Framework for Implementation Research (CFIR) [17], and the Unified Theory of Acceptance and Use of Technology (UTAUT) [18] (Multimedia Appendix 1), which guided the operationalization of both the survey items and the interview guide. Methodological rigor was ensured by adhering to the Good Reporting of a Mixed Methods Study (GRAMMS) checklist [19], which provides a structured framework for transparent and comprehensive reporting of mixed methods research (Checklist 1).

The introduction of the STA took place through intranet announcements, e-mail, and local ambassadors, both prior to and during the initial implementation phase. Paramedics were required to complete an online e-learning module, consisting of an instructional video and text on the STA website, as a prerequisite for account activation and clinical use. Implementation was further supported by promotional activities, including monthly newsletters, incentives (eg, cake or gift cards for every 100th patient triaged), and regular visits to ambulance stations, alongside branded promotional materials (eg, branded candy jars) to enhance visibility. Local ambassadors were appointed in each ambulance region to provide support, promote the app among colleagues, and to redirect questions to the research team. In addition, a user council consisting of paramedics was regularly consulted before and during the implementation to provide practical input on development and operational aspects. Throughout the implementation process, the use of the STA was communicated as part of the standard protocol. Although nonuse did not result in formal consequences, medical managers and the research team encouraged its use. However, use of the STA remained entirely voluntary, and paramedics were never obligated to use the app.

The study population consisted of all licensed paramedics employed within the participating EMS regions. For the dashboard analysis, all active STA users during the implementation period were included. Eligibility for participation in the survey and interviews was restricted to paramedics active in clinical practice, regardless of their frequency of STA use.

The interview guide was grounded in the study’s core concepts (Table 1) and further informed by survey results. Iterative revisions within the project group led to further specification of topics and rephrasing of questions. The interview guide was pilot tested between EH and NdR to evaluate the completeness and clarity of the questions, resulting in the final version (Multimedia Appendix 4).

Participants were recruited via voluntary self-selection, following an open invitation through newsletters and the on-site visits described above, in addition to an invitation at the end of the digital survey. Snowball sampling was additionally used by asking interview participants at the end of each interview to suggest colleagues with potentially different perspectives [23], though it yielded no additional participants. Data collection continued until thematic saturation was reached, assessed through interim analysis and team discussions, after which subsequent interviews yielded no new insights.

Interviews were conducted between July 8 and September 1, 2025, by either FK or EH as the primary interviewer, with NdR present in a supporting and moderating role. MS, a trained qualitative interviewer, was present during the first 4 interviews to provide support. The interviews were conducted online in Microsoft Teams, held in Dutch, and audio-recorded, pseudonymized, and transcribed verbatim in Microsoft Word by FK, EH, NdR, and RvdW. FK, NdR, and RvdW hold a medical background (MD) and were involved in the primary PRESTO-2 clinical validation study. EH and MS have a background in implementation science and were not involved in the primary clinical trial. Interviews lasted on average 39.6 (SD 6.7; range 30.6-51.6) minutes.

Survey data and dashboard metrics were analyzed descriptively using R Studio (Posit, PBC; version 4.4.2) [24] to summarize participant characteristics, system usability (D-SUS scores), and actual usage patterns. If data were distributed normally, means and SDs were calculated. Otherwise, medians with IQR values were derived. Open-text survey responses were analyzed thematically to identify recurring topics. For cross-analysis between subgroups, such as years of experience or attitude toward technology, the Pearson chi-square test was used to assess statistical significance.

Interview transcripts were analyzed in ATLAS.ti (version 24; ATLAS.ti Scientific Software Development GmbH) using an inductive approach to thematic analysis, consisting of the following phases: familiarization with the data, generation of initial codes, searching for themes, reviewing themes, defining and naming themes, and producing the report [25,26]. Codes were derived directly from the data, while the study’s 3 core concepts (Table 1) served as a sensitizing framework informing theme development. The first 2 interviews were independently coded by EH and MS. Discrepancies were discussed until consensus was reached to calibrate the coding approach, after which EH coded the remaining transcripts. Emerging themes and subthemes were iteratively reviewed within the project group, with MS providing additional verification of the interpretations.

Data integration was performed by FK and EH. Findings from the quantitative and qualitative phases were integrated to identify convergence (confirmation), complementarity (expansion), or dissonance (divergence) between the data sources. This process facilitated the development of integrated insights from both methods. The visualization of this integration is presented in a joint display, aligning the quantitative metrics with corresponding qualitative themes.

This study was approved by the Medical Ethics Review Committee of Erasmus Medical Center, Rotterdam, the Netherlands (MEC-2024‐0760). Informed consent was obtained electronically prior to completion. Respondents who expressed interest in follow-up interviews provided their contact details in a separate, unlinked section of the survey, and written informed consent was obtained prior to the interviews. Survey participants did not receive compensation. Interview participants each received a coffee voucher worth €10 (US $11.19) as a token of appreciation for their time. Survey data were anonymized, and interview data were pseudonymized. All data were handled in compliance with Dutch data protection regulations.

A total of 2075 patients were triaged using the STA before the survey ended (1679 in ambulance region Rotterdam-Rijnmond and 396 in region Zuid-Holland Zuid), corresponding with approximately 28% (2075/7500) of the expected 7500 suspected strokes within the study period of 18 months. During this period, 1855 patients were advised to go to the nearest general hospital. In 220 (11%) cases, the STA advised bypassing the general hospital and transporting the patient directly to an EVT-capable center. In 1448 (70%) cases, paramedics mentioned that they followed the advice. STA advice was not followed in 87 (4%) cases. In 540 (26%) cases, paramedics did not mention whether the STA advice was followed, since this question in the STA was not mandatory.

On June 17, 2025, 246 of the ±250 employed paramedics had an STA account. In total, 85 (±34%) paramedics responded to the survey, of which 76 consented to participate, 24 agreed to being approached for an interview, and 9 ultimately participated in the semistructured interviews (Figure 2). A total of 7 (78%) participants that participated in the semistructured interviews were male, and the mean age was 39 (SD 7) years. Overall, 4 (44%) paramedics worked full-time, and median years of experience was 6 (IQR 6‐8). Survey demographics showed that most paramedics held a neutral to positive attitude toward technology and considered themselves average or advanced in their technology experience (Table 2).

This study aimed to investigate paramedics’ experiences with the implementation of the STA in real-world prehospital practice by evaluating the implementation process, the implementation outcomes, and its impact on clinical workflows, while considering contextual factors that may have influenced these dimensions. Despite the identified barriers in usability and clinical added value, the STA achieved substantial initial reach supported by a structured implementation strategy. However, the high self-reported acceptability in the survey did not automatically translate into consistent, routine integration into daily prehospital workflows. Instead, actual usage varied with practical and contextual factors and paramedics’ perceived added value. While the STA facilitated structured neurological assessments and enhanced communication during handovers, it simultaneously induced workflow friction for experienced users who viewed the app as an additional step in high-stakes environments.

Dashboard data indicated a decline in usage after an initial peak, reflecting the “novelty fade” phenomenon in digital health implementation [27]. This suggests that initial engagement was driven more by novelty than by structural integration. Qualitative insights indicate that the low frequency of stroke cases during shifts formed a structural barrier, consistent with reports that fewer than 5% of ambulance rides involve suspected stroke cases [28]. Literature on habit formation suggests that this lack of exposure limits paramedics from moving past the initial learning phase to an automated workflow integration before the novelty fades [29,30]. Moreover, trust in clinical decision support systems is typically built through consistent positive reinforcement over time [31]. In a low-frequency setting, a technical barrier or ambiguous clinical results may therefore carry disproportionate weight, undermining perceived reliability and long-term use [31]. This aligns with comparable findings on the operational burden of screen-based tools in time-critical prehospital settings, where balancing digital support with the demands of acute care has similarly been identified as a barrier [32]. This concern was further reinforced by the misconception that the STA was a temporary research tool, rather than standard care, creating a sense of perceived optionality, a known barrier to long-term adoption [33]. Together, these factors point to a structural tension in acute-care digital health implementation, in which reinforcement needed to build trust and routine use is inherently difficult to achieve in a low-frequency clinical context.

The findings of this study reflect a common, yet complex trajectory in digital health implementation, where high levels of acceptability and perceived appropriateness do not automatically ensure long-term sustainability [34]. However, the emergence of a core group of persistent users, the early adopters, indicates that the STA has reached a level of routinization for a subset of the population [35]. Yet, the observed decline in usage suggests that the tipping point for the early majority has not yet been fully reached [35], as members of this group may have been more sensitive to the structural barriers identified in this study. Our findings also point to a pattern of selective, strategic use across users, where the STA is reserved for cases of clinical uncertainty rather than applied universally. Rather than viewing this as a lack of adherence, this selective approach aligns with the concept of professional autonomy in high-stakes environments, where clinical decision-support tools are reserved for complex decision-making scenarios [36]. However, this approach assumes that paramedics can accurately identify a case as “unmistakable” before deciding to bypass the STA. Such judgments may be less reliable in atypical presentations, where the STA is likely to provide the greatest added value. This tension is not unique to the STA: in the ParaNASPP study [2], which evaluated a comparable prehospital stroke-recognition tool, paramedics similarly reported relying on their own clinical judgment when classic stroke symptoms were clearly present, while placing greater reliance on the tool in more ambiguous presentations.

To ensure long-term sustainability of the STA, several practical implications must be addressed. First, technical complexity should be minimized by embedding the STA directly within existing prehospital information systems, to eliminate repetitive login procedures and facilitate automatic data transfers to receiving hospitals. More broadly, this illustrates that decision-support tools are more likely to be sustained when they are embedded within existing clinical systems rather than operating as stand-alone apps. Second, the clinical relevance of the STA can be strengthened by establishing a feedback loop that provides paramedics with formal confirmation of hospital diagnoses, addressing the need for paramedics to validate both the app’s recommendations and their own clinical triage decisions against actual outcomes in order to calibrate their trust in the tool. Increasing the transparency of the STA by providing the clinical rationale underlying its recommendations could help mitigate the “black box” effect, particularly when in situations where the app’s advice differs from a paramedic’s initial judgment. Lastly, given the low frequency of stroke cases per shift, sustained promotion of the tool remains essential. Future implementation efforts should explicitly frame the STA as a specialized resource for cases of diagnostic uncertainty rather than a replacement for professional judgment. This may be particularly important, as infrequent exposure to decision-support tools may limit opportunities for paramedics to develop a clear understanding of their intended role and limitations.

Future research should prioritize longitudinal studies to evaluate the effectiveness of diverse feedback mechanisms, such as providing paramedics with real-time data on hospital diagnoses and more detailed insights into the underlying clinical rationale of the STA advice. Such feedback is essential to determine whether it enhances the perceived relative advantage and long-term adherence to the tool. Furthermore, since this study was primarily conducted in an urban setting with high hospital density, future research should explore whether the utility of the STA is more evident in rural areas, where longer transport times and fewer hospital options might lead to a larger impact on treatment times. Finally, further investigation into the receiving end of the acute stroke chain is needed to assess how neurologists perceive and use the STA data during the handover. Exploring potential changes in inter-professional dynamics and the perceived clinical authority of app-generated recommendations is needed for optimizing the integration of the STA into the broader acute care pathway.

A key strength of this study is the evaluation of the STA within a real-world prehospital setting over an extended period of 1.5 years. Digital health interventions in prehospital care are frequently evaluated in testing or development settings rather than in routine clinical practice, leaving evidence on real-world implementation and sustained use relatively scarce [6,37]. The longitudinal approach in this study captured long-term usage patterns and perspectives, providing a nuanced insight into the sustainability of the implementation process. The credibility of the results is supported by the fact that 96% of survey respondents and all interview participants had actively used the STA, ensuring that the findings reflect actual clinical experience rather than hypothetical expectations. Furthermore, because the results of the primary clinical validation study were not yet available during data collection, the risk of participants altering their feedback based on the STA’s proven clinical efficacy was minimized. Moreover, the study’s grounding in multiple implementation science frameworks enabled comprehensive operationalization of both instruments. For instance, the UTAUT provided measurable items, while CFIR’s implementation process domain structured the qualitative exploration of training, communication, and support. Ultimately, this study addresses the current lack in prehospital implementation research by providing a detailed end-user perspective on prehospital implementation through a comprehensive mixed methods approach [38].

Several limitations should be considered when interpreting our findings. First, despite efforts to include a diverse range of perspectives, selection bias may have occurred, potentially overrepresenting paramedics with a positive attitude toward the STA. A consequence of the high participation rate among active users is the potential omission of critical insights from nonusers, whose reasons for nonuse remain largely unexplored. Although the survey was actively promoted with an emphasis that feedback from all viewpoints was essential, no responses were received from this subgroup, leaving the dedicated nonuser section of the survey entirely uncompleted. Similarly, while snowball sampling was used to recruit critical users for the qualitative interviews, this subgroup remained underrepresented. Consequently, our understanding of the barriers faced by those who did not use the STA remains limited. Second, the involvement of some researchers in both this study and the primary clinical validation trial could have introduced a risk of confirmation bias, potentially leading to an overinterpretation of positive findings. To mitigate this risk, researchers who were independent of the main clinical trial were involved in conducting interviews and performing the thematic analysis. This approach brought an outsider perspective to the data, enhancing the reflexivity and interpretive rigor in the analysis. Finally, the urban geographical context of the current study regions may limit the generalizability of our results to more rural settings. In areas where the intervention center was the closest hospital, the perceived urgency of the clinical decision problem was likely reduced. This proximity may have led to lower STA compliance compared to regions with longer transport times and more complex triage choices, where the relative advantage of a decision-support tool might be more apparent.

This study demonstrates that the STA achieved substantial reach and initial adoption within a prehospital setting, facilitating structured neurological assessments and establishing a shared clinical language between paramedics and neurologists. However, the widespread acceptance did not consistently translate into routine, sustained integration into daily practice. Usage remained selective, influenced by paramedics’ professional judgment in situations of clinical uncertainty rather than applied universally. Beyond its immediate findings, this study contributes to the STA’s ongoing development and to broader efforts toward sustainable digital health implementation in prehospital emergency care. Ultimately, successful implementation depends as much on addressing organizational and human factors as on technical performance.