Acceptability is the extent to which an innovation is agreeable, palatable, or satisfactory to a stakeholder [23]. For BITs, acceptability can be evaluated among a variety of stakeholders (consumers, providers, administrators, and policy makers) and for a variety of BIT aspects, from intervention content to program appearance. A recent 2018 systematic review of implementation measurement for technology-based mental health interventions found that acceptability was the outcome most frequently measured, commonly via project-specific, nonvalidated measures [28].

The most well-known model of health information technology acceptability is the Technology Acceptance Model (TAM), in which the perceived usefulness and ease of use of the technology is assessed, along with a number of associated constructs such as prior experience, output quality, and social influence [35]. The concept of technology usability clearly overlaps with acceptability and is included as a major construct in the TAM and similar models. However, usability is a broader concept, encompassing additional issues such as likelihood of error and efficiency of use. In BIT research, usability is most often measured by self-report, which may be the component of usability that shares the most conceptual overlap with acceptability. Self-report measures of usability, such as the System Usability Scale, best track the notion of perceived usability (eg, Does a user believe that a BIT will be able to help them achieve a goal?) rather than actual usability, which involves observing whether those goals are achieved [41]. Validated measures such as the Attitudes Towards Psychological Online Interventions questionnaire also include assessments of acceptability [42]. Schroder et al (2017) used this measure to assess the acceptability of a BIT for depression among both consumers and providers in a randomized controlled trial [43].

Proctor et al define adoption as the intention, decision, or initiation of use for an evidence-based practice, characterizing it at the level of the provider or organization [23]. Given the self-care nature of guided and fully automated BITs, this level of analysis is expanded to that of the consumer. The concept of adoption aligns with constructs of actual system use or behavioral intention to use in models such as TAM, for both consumers and providers [35].

The measurement of BIT adoption primarily relies on data gathered by the program, which is either passively collected (eg, log-in timestamp) or input by the consumer (eg, self-report symptom measure). However, direct observation and surveys of BIT use can occur. The amount and relative ease of data extraction from BITs can make intentions, decisions, or initiations more evident and accurate, without the need for labor-intensive data collection processes. In a large-scale naturalistic trial, Gilbody et al used log-in records to identify the number of study participants who accessed BITs for depression [22].

Appropriateness is the perceived fit or compatibility of an innovation with a practice setting or context [23]. The key concept is the perception of fit, making the measurement of appropriateness more relevant to initial phases of implementation, but not strictly limited to these phases. Fit can be assessed at the organizational level (eg, alignment with workflow and policies) or the individual level (eg, alignment with providers’ or consumers’ attitudes, needs, and background). When evaluating implementation outcomes for BITs, we add administrators, given that BIT components such as dashboards can be accessed by administrators.

It is important to clearly specify which individuals or groups are intended to use a BIT and to elicit their needs when determining appropriateness. Usability evaluations can be employed to capture expectations, cognitions, and emotional responses of potential users [44]. Formative usability assessment seeks to assess the perceived contextual fit of new products with their destination context before actual implementation. Laboratory-based usability testing is particularly relevant to the initial assessment of appropriateness. Assessing the existing health information technology infrastructure, workflow, and use, as well as the potential interoperability of the BIT at a site, are key factors later in the process [45,46]. For instance, Lyon et al (2016) evaluated school-based practitioner workflows and current technology use practices to determine the appropriateness of a digital measurement feedback system before its implementation [36].

Feasibility is the extent to which a new evidence-based practice can be successfully used or conducted in a setting. Feasibility differs from appropriateness as it is typically based on direct observation of stakeholder’s experiences and practical concerns derived during or after implementation. Traditional conceptualizations of feasibility assessment include data collected from service providers and organizations and are rooted in the assumption that new innovations pass directly through providers to consumers [23]. For BITs, feasibility must be expanded to accommodate scenarios such as those involving fully automated BITs that may reach consumers with limited mediation from providers or the health system.

Standardized measures of innovation feasibility remain applicable to BITs [47]. In addition, although assessment of the acceptability and appropriateness of BITs are associated with formative usability testing, the assessment of BIT feasibility can leverage summative testing to evaluate how well full products meet their specified objectives in the actual context of use [48,49]. Such assessment might include evaluation of user awareness of key features of a BIT, actual compatibility with provider workflows, or user workarounds to account for unaddressed design issues. As feasibility is associated with users’ actual experiences with an innovation in a specific context, in vivo usability testing is particularly relevant. Passive data collection opportunities surrounding feasibility may include frequency, time of day, or other contexts in which key BIT features are used. For instance, Lappalainen et al evaluated the feasibility of an adjunctive BIT for workplace stress reduction by determining how frequently users accessed an Web-based portal during the study [50].

Fidelity is the extent to which an intervention is used as intended and measured across several domains: protocol adherence, dose or amount delivered, and quality of delivery [23]. Protocol adherence to a BIT can be conceptualized as the functions of the program that were used, dose as the frequency of program use, and quality as whether the BIT was delivered correctly or for the intended purpose. The standards for these measurements should be based on the intentions of the BIT developers or empirical findings from efficacy testing. Therefore, fidelity to BITs suggests a match between the intended use of the program, termed expected use or clinically meaningful use, and its actual use by consumers [51]. Fidelity is traditionally measured at the provider level. However, the consumer level may be favored for BITs, especially for fully automated technologies.

BITs offer the opportunity to collect an extensive amount of passive data, which can be used to measure fidelity. Frequency metrics such as the number of log-ins, clicks, or task completions, as well as viewing times may best serve as measures of dose, whereas data representing the breadth of specific program functions used (eg, viewing of videos, completion of measures, and use of Web-based diaries) can serve as measures of adherence. Donkin et al provide a systematic review that includes measures of dose and adherence [27]. As for quality, BITs are a technology-driven platform, and they typically present information, direct activities, and make assessments in a uniform manner, which may provide better quality consistency compared with traditional human delivered care. Examples of measuring the quality dimension of fidelity include determining whether the BIT was used in a group of consumers with the appropriate condition, if the content of diary or self-report entries comply with the information requested, or whether passive sensors accurately assessed what was intended.

The measurement of certain aspects of fidelity, such as protocol adherence, may be less relevant for programs where consumers have more choice in when and how much they engage with the intervention. Determining expected use or clinically meaningful use with respect to BIT implementation is especially relevant and may be especially difficult for BITs developed as programs where users can openly navigate to browse information, activities, and assessments at their own discretion. This format differs from sequential navigation programs in which users are directed through a structured format to a specified endpoint [52]. Open navigation designs are increasingly common because of consumer desires for flexibility. This issue of clinically meaningful use and the level of fidelity associated with successful implementation is also relevant for programs used as an adjunct to provider-led care, where clinical outcomes may be associated more with provider activities than use of the BIT. A prime example is the Veterans Health Administration program PTSD Coach, a mobile phone BIT used in conjunction with provider-facilitated therapies for the treatment of posttraumatic stress disorder. The program has multiple functions, including the capability to monitor symptoms, learn relaxation techniques, and track appointments [12,13]. Lack of clarity around what constitutes expected use or clinically meaningful use might be one reason that research linking platform use and clinical outcomes has produced mixed findings [26,27].

Proctor et al identify 3 factors that drive costs in implementation research: the intervention, the setting of service delivery, and the strategy used to implement the intervention [23]. Intervention development and implementation may be blurred with respect to BITs, making these distinctions somewhat artificial. The decision to account for specific implementation costs in operational or implementation budgets will likely vary by organization but should be made clear in cost analyses.

BIT intervention costs include those associated with program development, testing, versioning, and maintenance. Program development and refinement also occurs with traditional face-to-face psychosocial interventions, but versioning and maintenance may be more applicable to BITs (eg, version 2.0 of a program) [53]. When developed outside the health system, intervention costs include the price paid by the health system for the BIT or to develop initial and subsequent versions of the BIT.

Costs associated with the context of service delivery will vary according to the implementation site’s size, complexity, overhead, and how much adaptation of the site’s current health information technology platform is needed. For instance, Quanbeck et al measured system operating costs, which they saw as distinct from implementation costs, including staff time for introducing the BIT to patients and monitoring data produced by the BIT, technical support costs, and ongoing costs such as server hosting [39].

Implementation strategy costs are those incurred directly from the strategies used to employ the BIT, and vary depending on where the BIT lies on the continuum of support defined in Figure 1 [54]. Guided BITs may require more time to educate providers and consumers, facilitate engagement, and support use. In addition, resources such as advertising materials, educational materials, registries, clinical reminders, and decision support tools may be needed. Implementation strategies for fully automated BITs likely incur fewer costs given that providers do not require training and support to facilitate BIT use. In such cases, strategy costs may focus primarily on advertisement and other materials facilitating engagement.

Penetration is the integration of a practice within a service setting and is measured within an organization. Some BITs allow for an expansion of service providers to include an array of providers who may support consumers during BIT use [23]. Other BITs may follow fully automated formats circumventing the involvement of providers and allowing for implementation outside of traditional health care systems. Thus, BITs may expand the breadth of providers for which penetration is measured and expand the reach of health care organizations beyond their brick and mortar facilities to draw in new consumer groups.

Penetration is primarily a summative implementation outcome traditionally measured as the number of providers who deliver a service among the total number of providers trained in or expected to deliver the service and/or the number of consumers engaging in a service among the number eligible to engage in a service within an organization [23]. Such traditional measurement applies well to adjunctive BITs. However, for guided BITs, penetration at the service provider level should be reconceptualized to include personnel specially trained to support BIT use, which comprises traditional health care delivery professionals as well as health technicians, administrative personnel, and peers [29]. The decision to include such adjunct providers in the larger denominator of providers, or measure penetration separately according to provider role is up to the researcher. We note that in such cases, the number of adjunct providers trained to support BIT use may be small, especially if they are centralized to provide remote support across a health care organization. Consumer-level penetration or fidelity to BIT delivery may be a more impactful measure in such cases.

Some fully automated BITs may be implemented outside the traditional health system contexts at the consumer level, either adjacent to health care systems to specifically targeted populations or disseminated worldwide as in Massive Open Online Interventions (MOOIs), where specific targeting strategies are not used [55]. In the case of targeted populations, the denominator for penetration must be selected from those for which the intervention is clinically appropriate and to which the outreach efforts for the intervention were aimed. Measuring MOOI penetration may be difficult, as the denominator cannot be precisely calculated, and estimated penetration, based on the proportion reached given the prevalence of a targeted condition, may be more appropriate.

Sustainability is the degree to which an implemented treatment is maintained, institutionalized, or integrated within a service setting. For BITs, sustainment is characterized at the level of the organization or setting. Proctor et al distinguished 3 constructs for the measurement of sustainability: passage, niche saturation, and inclusion in cycles or routines [23].

The implementation of BITs in US health care systems has primarily taken place as effectiveness trials and pilot programs. Thus, sustainment is the least well-documented BIT implementation outcome [4]. As BIT use matures, passage from research and development funding streams to permanent organizational funding will be important to measure. In addition, the integration of BITs with pre-existing organizational health information technology such as electronic health records or other patient or provider dashboards will be an important measure of niche saturation. Similarly, the addition of metrics on BIT use and clinical outcomes in routine administrative reports will be an important measure of inclusion in cycles or routines.

With respect to fully automated BITs implemented outside of traditional health care settings, the analysis of sustainment beyond simple continued adoption may be difficult because of the novel organizational cultures in these contexts. As a hypothetical example, analysis of the sustainment of a BIT aimed at reducing alcohol intake among veterans and implemented through social media by a veteran service organization may rely on analysis of the organization’s continued budget for advertising and program maintenance as a gauge of sustainment, in addition to routine adoption or fidelity measures.

An additional unique aspect of BITs is the concept of intervention versioning, whereby there is a continuous evolution of an intervention through frequent program updates [56]. Some changes are small (ie, bug fixes or cosmetic improvements), whereas others may include substantial changes to program content and function. Measurement of implementation and specifically sustainment of BIT use must account for such changes. At a minimum, the timing and content of changes should be tracked and associated with any changes in BIT use and clinical outcome.