We have led a series of qualitative, theoretically informed case studies of different HIT implementations in the context of national formative evaluations. These included electronic health records (EHRs), clinical decision support (CDS) systems, and a combination of CDS and computerized physician order entry (CPOE) systems [4,20,21]. Our ongoing involvement as principal investigators and researchers in these various formative evaluations provided a platform for understanding HIT implementation challenges.

Table 2 shows our data set, consisting of qualitative data collected between 2009 and 2018 by our research teams that included 11 social scientists. AS was the principal investigator on two of these projects [4,20], KC was the principal investigator on one [21], and RW was a senior adviser on all three [4,20,21]. We have published several papers from these evaluations, including both primary research and sets of evaluation recommendations based on the literature [4,20,21].

KC collected some primary data (approximately 100 interviews and 60 hours of observations on [20], 40 interviews on [4], and 14 interviews on [21]). The majority of case study sites (18 of 19) included hospital settings implementing EHR and CDS/CPOE functionality in the English National Health Service (NHS).

We defined a case as an organization implementing relevant functionality within the boundaries of an organizational setting. We sampled hospitals for maximum variation in relation to geographical location, size, implementation strategy, technological systems, and governance structures [22].

We sampled individual participants through a combination of convenience and snowball approaches with key local gatekeepers facilitating initial contacts [23]. Participants in case study settings included representatives with varying degrees of seniority from a range of clinical professions (medical, nursing, pharmacy, and allied health care professionals), managerial, and IT support staff. We also collected data from relevant policymakers and system vendors, in order to gain insights into the wider market and policy dynamics in which local implementations took place.

We collected data between February 2009 and October 2018. The majority of data consisted of digitally audio-recorded semistructured qualitative interviews (mainly face-to-face, some by telephone). These interviews explored expectations and experiences of implementing, using, and developing the new technology (depending on the background of the interviewee). Although interview guides varied with the specific focus functionality examined, key issues explored included the following: current systems, strategies, and organizational setup; views on potential system benefits and barriers to achieving these; and future directions and visions.

In many cases (18 of 19 case studies), we sought to interview participants longitudinally (ie, before the implementation of the system, during the implementation, and once they had time to get used to the HIT system).

Observations were nonparticipant in nature, opportunistic, and involved attending relevant strategic meetings (where the researcher took notes) or following a particular activity (eg, doctors using a specific system). Observations explored technological deployments in real-world contexts.

Documents consisted of minutes of strategic meetings, summaries of lessons learned, and business cases. These provided insights into planned activities and local narratives surrounding implementation.

We began by conducting a review of existing frameworks and undertook a systematic literature review to explore which factors are important for the effective implementation of HIT [24]. The resulting prototype coding framework was iteratively refined over time and throughout projects. It included a number of dimensions and factors that formed the basis for coding qualitative data collected throughout case studies. In this process, we also allowed additional categories to emerge inductively [25]. Case studies were initially coded separately and then integrated iteratively across functionality (EHRs, CDS/CPOE in hospitals or what is known as ePrescribing in the United Kingdom, and CDS). This resulted in development of the prototype coding framework into a more comprehensive evaluation framework, which was synthesized to reflect the most pertinent categories and updated in light of the current literature. Here, our focus was on breadth rather than depth, aiming to produce a comprehensive overview of various stakeholder perspectives. We gave particular attention to stakeholders who were underrepresented (eg, vendors and administrative staff).

Our analysis and development of the novel framework was informed theoretically by the sociotechnical approach, structuration theory, the social shaping of technology, and the theory of the diffusion of innovations [26-29]. The final framework was agreed upon through iterative discussion.

We observed that, irrespective of the technology, systems needed to be usable, stable, and reliable (dependability), hold data securely (security), and only allow those with appropriate access rights to view confidential data (confidentiality). If systems and data within them were not dependable, users and implementers tended to lose confidence in their system choice.

However, it was also apparent that the design of technologies never occurred in isolation of the social and organizational context of use. This was exemplified through the many different ways in which technologies transformed how users worked in often unanticipated ways (eg, by making data entry more cumbersome; see the Social/human dimension, which includes “User satisfaction,” “Workload/benefits,” and “Work processes,” in Table 3), but also by how different technologies were optimized to suit organizational and user requirements over time (“Adaptability” in Table 3).

System usability was the most important prerequisite for successful adoption. If users had to navigate a large number of interfaces and had difficulty finding relevant data, and if there was a general lack of intuitiveness, this slowed down their work (“Workload” and “Work processes” in Table 3).

As a result, to not disrupt everyday delivery of care, staff had to employ workarounds (“Complete/correct use” in Table 3).

Although changes to system design could address these issues, modifications were in many instances cumbersome and lengthy (“Adaptability and flexibility” in Table 3). Vendors, in turn, struggled to find a balance between tailoring of applications to local needs (achieved through local configuration) and developing commercially viable generic system versions (“Vendors” in Table 3).

Organizations tended to employ two distinct strategies to approach system implementation. These involved both “top-down” and “bottom-up” management approaches, as well as technology strategies that involved transformative change combined with automating existing processes. “Top-down” and “bottom-up” strategies occurred at two different levels. These included the following: (1) technology design/procurement (in the English National Programme for IT (NPfIT) characterized by centralized procurement; “Political context” in the Wider macroenvironment dimension); and (2) technology implementation strategy within the organization (“Leadership and management” in the Organizational dimension).

Balancing these tensions was instrumental for the perceived success of initiatives; this is not recognized in many existing frameworks. For example, although there was a perceived need for clinicians to be heavily involved in strategy, implementation, and deployment (“Communication” and “Vision”), stakeholders also recognized that some decisions at an organizational level had to satisfy the needs of diverse stakeholders, including those outside the immediate hospital environment (“Adaptability and flexibility” in Table 3).

It was very clear throughout our work that wider structures had a significant impact on organizational processes, ways of working, and technologies (see the Macroenvironmental dimension in Table 3). These included tensions between the long time frames needed to achieve transformation (5 to 10 years) and the episodic funding schemes (2- to 3-year programs; “Timelines” in Table 3). We also observed changing policy landscapes that involved a high turnover in senior staff and accompanying changes in visions of digital care and available funding (see “Political context” in Table 3). For example, when we began our work, England’s national strategy of implementing centrally procured systems had just started [30]. However, during the CDS/CPOE evaluation, there was an increasing focus on local involvement in decision making, driven by the demise of the NPfIT and increased economic pressures resulting from the global recession. When we completed data collection, there was again a growing recognition that national guidance was crucial for promoting implementation progress and interoperability [31].

In addition, we observed shifts in market and vendor structures and the technologies themselves. These are not sufficiently accounted for in existing frameworks (see “Vendors” in the Wider macroenvironment dimension in Table 3). During the NPfIT, we observed a limited number of large vendors pushing other players out of the English/UK market, although the dynamics changed after the demise of NPfIT and the termination of associated contracts. This gave way to a more vibrant vendor landscape (although the market is still not very open to new vendors), which has important implications for organizations and users as they can only procure technologies from those that are currently available.

We also saw how technologies were refined “in use” and through close working relationships between vendors and users over long periods (“Adaptability” in the Technology dimension in Table 3), which helped vendors/system designers overcome their limited knowledge of the social context of use. We have repeatedly seen the formation of vendor/user groups and observed how these helped to actively shape designs and markets (“User input in design”).

We have developed an evaluation framework for implementers of HIT initiatives to guide implementation and optimization of functionality (Table 3). Although this draws on formative work, it can also guide summative evaluations. The TPOM framework includes key issues to consider in relation to technological, social/human, organizational, and wider macroenvironmental dimensions. Our ongoing work has shown that these dimensions are intimately related. Technologies never exist in isolation; it is therefore critical to appreciate that technological change will be accompanied by transformations in social groups, organizations, and the wider landscapes in which these are situated, including health policy, economic climates, and the development of markets.

This work has drawn on a substantive composite qualitative data set collected over a long time frame. Therefore, it helped us to assess which dimensions were relatively stable over time, and only these were included in the TPOM framework. Insights presented here are views derived following careful critical reflections on a series of evaluations.

Although we acknowledge that many dimensions could be included, we deliberately attempted to keep the themes and subthemes manageable, thereby addressing a key issue in health technology evaluation: the usability of evaluation tools. We do not claim to capture all factors that play a role in implementation, adoption, and optimization of HIT, nor do we claim that our TPOM framework will provide a recipe for success. However, its pragmatic use in implementation and evaluation activity is likely to improve processes by prompting implementers to consider the most important dimensions influencing outcomes, thereby reducing unintended consequences and maximizing value. The framework now needs to be applied prospectively to confirm its utility across settings and regions. As such, we hope that it will provide a solid foundation for other countries to develop their own evaluation frameworks.

A key challenge faced by most existing evaluation frameworks, including ours, is that they neglect to account for the dynamic relationship between social and technological dimensions of change. As this relationship is a process, it does not lend itself well to presentation in 2D pragmatic evaluation tools. We have illustrated this dynamism in the Results section, drawing on concrete examples.

Many empirical studies of HIT implementation are primarily concerned with evaluating impact and therefore emphasize quantitative measurements guided by benefits realization frameworks [32,33]. Recent evaluation frameworks have expanded this limited focus to include a more in-depth appreciation of the interplay of social and technological factors shaping implementations. However, although acknowledging the complexity of the process, these nuanced frameworks tend to concentrate on one particular local and situated aspect of technology implementation, thereby neglecting the role of wider structuring conditions in shaping developments [16,34-37].

Others have considered wider structuring conditions, but the tools developed lack intuitiveness, usability, and practical applicability. For example, some frameworks that are designed to shed light on sociotechnical processes can become abstract and difficult to apply by those without academic backgrounds (which arguably includes the vast majority of those implementing change in health system settings) [38]. Others have become so complex that they may include a myriad of relevant dimensions, but this attempt to capture everything may result in a level of complexity that undermines the usefulness of the framework as a tool [13,26]. The challenge is to avoid Lewis Carroll’s cartographer’s dilemma of needing to make a comprehensive map on the same scale as the mapped landscape, which then no longer helps users navigate [39]. Our framework has sought to address this dilemma.

There are several commonalities with existing frameworks, including a recognition of key technology, human and organizational dimensions, and their interrelationship [13-16,18]. However, TPOM is not condition- or illness-specific [13]. It is not concerned with the likelihood of the technology being adopted and its spread [13], but it is designed to help implementers of technology consider how implementation is progressing, the potential emerging risks, and what aspects therefore need attention to facilitate adoption. Implementers can apply TPOM to any HIT project at any stage of implementation. It builds on other evaluation frameworks that take into account the microcontext of use [14-16,18], to include consideration of wider macroenvironment dimensions that influence implementation and adoption. It is not concerned with management tools, but with alignment of perspectives [15]. As such, it does not provide a “recipe for success”; rather, it is a tool designed to help implementers navigate a complex landscape with many conflicting agendas and considerations. When problems or risks are identified with the help of the framework, these can be systematically targeted to facilitate implementation and adoption.

Pragmatic formative evaluation frameworks can help to understand areas for potential improvement, benefits, and ways to streamline processes associated with technology implementation in health care settings. Evaluations need to move away from simple benefit realization approaches (that attempt to identify and measure benefits at the end of an implementation) toward formative evaluations that help the stakeholders involved adjust strategy along the way. Formative evaluations are part of a shift toward an evolutionary model where evaluation is a resource for faster and more effective learning.

Our proposed framework is a guide for implementers of technological change initiatives, to assist in planning, or during implementations of HIT initiatives in health care settings. We invite those who use it to suggest changes in both content and usability, as this will help to maximize its use and application. In due course, we hope to be able to draw on a range of data collected through the lens of the framework in different settings, refine it, and develop new insights in relation to each of the dimensions.

Large transformative policy programs aimed at facilitating technology implementation beyond hospitals are likely to require different evaluation frameworks, as their effects may be harder to trace and attribute.

We have drawn on a substantial body of data to develop the TPOM framework (Table 3), which stakeholders can use to monitor change processes and, if necessary, adjust the direction of HIT implementation projects. Going forward, a key challenge is likely to be the ongoing tension between attempts to capture the dynamics, processes, and interrelationships involved in technological change; the large number of these dimensions and their complexity; and the usability of evaluation tools by those delivering care, which is linked to their potential to have impact. We encourage prospective application of the TPOM framework to determine its value.