Type 2 diabetes mellitus (T2DM) is an international public health concern, with >4.9 million people having diabetes and 13.6 million people now at an increased risk of T2DM in the United Kingdom [1]. T2DM is preventable by lifestyle modifications in diet, physical activity, and weight loss, although family history, ethnicity, and age are risk factors. Treatment for T2DM and its associated complications currently costs the National Health Service (NHS) £10 billion (US $13 billion) each year; thus, its prevention is a public health priority [2].

In response to this, the NHS Diabetes Prevention Program (NHS-DPP) was launched in 2016, a behavior change intervention for adults in England who have been identified as having elevated blood glucose levels and are thus at a higher risk of developing T2DM. The main aims of the program are to improve diet, increase physical activity, and achieve weight loss, contributing to a reduction in T2DM risk [3]. Early evaluations suggest that the program is successful in preventing the progression to T2DM [4]. However, there is a weakness in reach and retention in the face-to-face program (eg, younger adults and those from more deprived communities) [5], which a digital program may overcome. Digital interventions have the potential to address logistical challenges such as scheduling, travel, work, and childcare [6], and they have the capacity to offer wide-reaching and tailored support at a lower cost, both of which are important for large-scale behavior change [7]. For example, technology can offer more choice and convenience to patients who may not be able to engage face to face with health professionals, and interventions can be designed to address the specific needs of disadvantaged groups by tailoring those interventions to the target group [8], especially when these groups are included in the design of digital interventions [9].

In 2017, NHS England launched a pilot program of the NHS Digital DPP (NHS-DDPP) [10]. Following this pilot, a digital pathway was introduced to the program in 2019 [11], with 4 independent service providers commissioned to deliver the digital service. NHS England produced a service specification detailing the key features that should be present in the NHS-DPP [11], based on the currently available evidence [12,13]. This specification indicated the key behavior change techniques (BCTs) that should be present in the intervention [11]. BCTs are defined as the active ingredients of an intervention that promotes behavioral change (eg, setting goals, monitoring behavior, and social support) [14]. The use of BCTs to self-regulate behaviors (eg, goal setting and self-monitoring) was particularly emphasized in the specification, as the wider literature suggests that these are key to behavioral change [13].

Intervention fidelity is used to describe whether an intervention is delivered as intended [15,16]. Without an assessment of fidelity, it cannot be ascertained whether intervention effectiveness, or a lack thereof, is because of the intervention content or other factors added or omitted in the intervention delivery. A recent evaluation of the face-to-face delivery of the NHS-DPP produced a thorough fidelity assessment of the behavior change content in the program, including the assessment of intervention design [17], staff training [18], intervention delivery [19-21], and service user receipt [22]. The results of the face-to-face evaluation revealed an underdelivery of BCTs to self-regulate behaviors (eg, problem solving and goal setting) [19,21]. This suggests a gap between what the evidence base indicated as the most effective in changing health behaviors and actual program delivery, which is largely because of failures to translate the BCTs specified in the evidence base into providers’ program designs [19].

An assessment of the fidelity of the design of interventions is an important first stage in fidelity evaluation to establish what key intervention features are planned and whether these are in line with the evidence base from the outset. If a program is not designed with fidelity to the evidence base, it will have a downstream influence on its delivery and receipt. The findings from the assessment of fidelity of design for the face-to-face service indicated 19 BCTs in the NHS-DPP full program specification that should be included in the program; face-to-face providers described 74% of those BCTs in their intervention designs [17]. However, we are yet to establish the fidelity of the behavior change content in the digital offering of the NHS-DPP; that is, the extent to which the digital providers include the 19 specified BCTs in their intervention designs.

The underpinning theory of an intervention refers to the theories that inform the intervention, such as identifying theoretical constructs to be targeted [23]. Having identified these constructs allows for appropriate BCTs to be selected. Despite recommendations for clear descriptions of the underpinning theory of behavior change interventions [24], there is still a lack of literature that reports how interventions work (ie, the theoretical basis of the intervention and the rationale for selecting the BCTs designed for the intervention). To facilitate the description of different aspects of theory use, the Theory Coding Scheme (TCS) was developed [25], which is a 19-item framework providing a fine-grained assessment of the use of theory, including the extent to which all mentioned BCTs are linked to relevant constructs.

A concise way of visually presenting the theoretical basis for interventions that are informed by multiple theories could be via a logic model that uses simple diagrams to demonstrate the causal pathways between the intervention components and desired outcomes [26]. Without an explicit underpinning theory, it is not clear how the program is expected to produce the desired behavioral changes; thus, the rationale for the design of the intervention is not clear, and there is a lack of justification for why particular BCTs have been chosen for the intervention [25]. The evaluation of the face-to-face NHS-DPP found variation in the underpinning theory across face-to-face providers, with a lack of justification for the inclusion of BCTs in providers’ intervention designs [27].

Digital interventions may allow higher fidelity as staff do not have to be trained to deliver all the intervention content [7]. To date, research has primarily focused on patient engagement with digital health interventions (eg, via methods such as system usage data, qualitative measures, and self-report questionnaires) [28] and adherence to digital interventions (ie, the extent to which patients keep using the digital intervention in the desired way) [29]. Some studies have summarized the development process of digital interventions (eg, describing the theory and existing evidence to inform the development of a digital self-management intervention for people with T2DM) [30] and compared web-based intervention content against the intervention description [31]. However, there is a lack of literature examining the extent to which the planned intervention content of digital health interventions adheres to the evidence base for what works in changing health behaviors.

Assessing the fidelity of nationally implemented digital behavior change interventions is rare, and to the best of our knowledge, none to date have assessed the fidelity of design to behavior change content. This analysis aims to provide the first known independent evaluation of fidelity of design of a nationally implemented digital intervention and thus builds on previous methods set out to measure fidelity [16] by applying them in a novel way to digital intervention designs using the NHS-DDPP as an exemplar. Therefore, the objectives of this study are to (1) describe the planned BCT content and theoretical foundations of the NHS-DDPP, (2) examine variations in planned BCT content and theoretical foundations between the digital providers, and (3) evaluate the fidelity of the planned BCT content in each provider’s NHS-DDPP intervention design to the full program specification.

We conducted a document analysis of the NHS-DDPP providers’ intervention design documentation, along with qualitative interviews with program developers of each of the 4 digital providers. There were no single documents for any digital provider that offered a full and complete summary of their intervention designs; thus, interviews were used to supplement the documentation to allow researchers to gain in-depth information about the development of the NHS-DDPP and the implementation journey of the digital program. Digital providers’ intervention designs were compared with the NHS-DPP program specification.

For the purpose of this manuscript, the term digital providers indicates the 4 commercial companies commissioned to deliver the NHS-DDPP. The 4 digital providers were private service organizations, each of whom secured contracts to deliver the NHS-DDPP from 2019 to 2021. Of the 4 digital providers, 3 (75%; Oviva, Second Nature, and Liva) were in partnership with one of the face-to-face providers of the NHS-DPP to deliver the digital pathway of the program. One of the providers (WW) delivered both face-to-face and digital pathways of the NHS-DPP. In this report, providers are labeled A to D to preserve their anonymity.

To supplement the digital providers’ intervention design documentation, interviews were conducted between September 2020 and December 2020 with program developers employed by each of the 4 digital providers. Program developers were involved in the design and development of the NHS-DDPP. Participants had job titles including Head of NHS Partnerships, Head of Clinical Services, and Head of Coaching. Participants were involved in either adapting their current digital programs to be in line with the NHS Service Specification [11] or were a key contact at the digital provider best placed to describe how the behavior change content was developed for the program.

The wider program of research of which this study is a part of was reviewed and approved by the North West Greater Manchester East NHS Research Ethics Committee (reference 17/NW/0426; August 1, 2017).

The authors were in contact with the management staff of each of the 4 digital providers to obtain all relevant documentation detailing the providers’ intervention designs. All documents were emailed to the research team.

The authors contacted the management staff of each of the 4 digital providers to help identify the appropriate staff members to be interviewed. The management staff contacted the relevant individuals directly, and those individuals proactively contacted the research team, where they expressed an interest in participating in an interview. We aimed to interview professionals from different backgrounds who had different roles in program development (eg, specialists with backgrounds in behavioral science, nutrition, or physical activity, and provider leads overseeing program development) to gather a range of views and provide a comprehensive understanding of the processes involved in the design and development of each NHS-DDPP intervention. Before scheduling the interviews, the participants were sent an information sheet containing details about the research. Participants were given the opportunity to ask any questions about the research, and fully recorded verbal consent was obtained from each participant before commencing the interview. Interviews were conducted via a video call platform—Zoom (Zoom Video Communications) videoconferencing—with one of the two researchers (REH and LMM) who had previous training in qualitative data collection.

Interviews with 2 professionals took place with digital providers A and C, and interviews with 1 professional took place with digital providers B and D (n=6 interviews in total). The interviews covered the following topics:

The interviews were semistructured; the researchers initially asked open questions about the general topic, followed by a more detailed probing on specific issues. It was noted that participants were not expected to be able to answer all the interview questions, depending on their specific role and expertise in intervention development. Interviews were transcribed verbatim. Taken together, the documentation from each digital provider and interviews with program developers described the behavior change content that digital providers intended to deliver in each of their programs (providers’ intervention designs).

Across the 4 digital providers, the mean κ value for coding using the TCS was 0.90, and the mean κ values for theories and constructs mentioned were 1.00 and 0.97, respectively. Cohen κ values demonstrated strong to an almost perfect agreement [36] between coders using TCS before resolving discrepancies (Multimedia Appendix 3 provides all κ values).

The extent to which theory was used in each of the 4 providers’ intervention designs is summarized in Table 3. All digital providers mentioned theories on which their programs were based; however, none had produced an explicit logic model of how their program was expected to produce changes in behavior. Providers A, B, and D stated that their digital programs were informed by the Capability, Opportunity, Motivation–Behavior framework [37], with providers B and D describing a particular emphasis on this model. Providers A and C cited multiple theories to inform their intervention designs, including Social Cognitive Theory [38], the Transtheoretical Model [39], and Self-Determination Theory [40] (Table 4).

There was wide variation in the constructs that providers were targeting in their digital programs. All 4 digital providers mentioned the constructs of goals, motivations, and capabilities in their intervention designs. The following constructs were also mentioned in the digital designs of 75% (3/4) of providers: self-efficacy, opportunity, and social support (Table 4; constructs have been grouped based on the Theoretical Domains Framework [52]).

None of the 4 digital providers linked all of their planned BCTs (listed in Table 2) to the proposed theoretical constructs, although provider B did link all constructs and intervention functions of the Capability, Opportunity, Motivation–Behavior model (which their intervention was solely based on) to some planned BCTs, which was detailed in their intervention design documents (see item 10 in Table 3). All providers linked at least some of their planned BCTs to theoretical constructs (see item 8a in Table 3). For example, provider A described modeling, goal setting, and problem solving to drive self-efficacy, and provider C described the techniques of verbal persuasion, reviewing goals, and social support to target self-efficacy.

The digital providers of the NHS-DDPP demonstrated good fidelity to the BCT content of the NHS-DDPP design. In particular, self-regulatory BCTs, which may be most effective in changing health behaviors [13], were included in the designs of all 4 digital providers. However, providers appeared to have inconsistent use of theory to inform their intervention designs, and none had developed a logic model or provided an explicit description of how their digital programs were expected to achieve the desired health outcomes. Without a clear underpinning theory and identification of the mechanisms underlying particular BCTs, the justification for the inclusion of specific BCT content in digital providers’ programs is not clear.

Overall, digital providers planned to deliver an average of 85% of the BCT content specified in the NHS-DPP program specification. Thus, the fidelity of design in the digital offering of the NHS-DPP is, in principle, higher than that previously reported for the face-to-face offering, where the fidelity of design was 74% [17]. The planned self-regulatory BCT content is particularly encouraging, given that these BCTs are backed by evidence to suggest that such techniques are most effective in changing health behaviors such as diet and physical activity [13]. However, the inconsistent use of theory in digital programs is similar to the evaluation of face-to-face NHS-DPP, which also highlighted a lack of mapping of BCTs to theoretical constructs and a lack of a logic model produced by any of the providers [27]. Both findings could be explained by the fact that providers’ digital programs were adapted from pre-existing digital interventions already implemented before the commissioning of the NHS-DDPP service. This may be a particular problem for digital interventions.

This study provides a detailed analysis of the planned behavior change content of each of the 4 digital providers commissioned to deliver the NHS-DDPP using standardized coding frameworks [14,25], which demonstrated high κ agreement between coders. All framework response documentation obtained from each digital provider was submitted to the commissioners during service procurement; thus, they should represent an accurate description of digital providers’ intervention designs. In addition, the research team requested any other relevant documentation from each digital provider that contained information about the planned behavior change content of the NHS-DDPP. Providers were given multiple opportunities to share all relevant documentation with the research team to support the current analysis.

However, one of the providers was reluctant to share all documentation detailing the theoretical underpinning because of the commercial sensitivity of these documents. Researchers attempted to obtain as much information as possible about the theoretical foundations of providers’ programs via interviews, especially when documentation could not be shared with the research team. The interviews were able to supplement the document review and were used to fill any gaps in the information that was not present in the documentation. Despite some reluctance to share documentation with the research team from one of the digital providers, analyses from this provider were similar to those of the other 3 providers, who shared all documentation. Furthermore, all digital providers confirmed that they had not developed a logic model to explicitly map how their digital programs would achieve the desired program outcomes, nor was there an alternative presentation of how their interventions were expected to work (eg, via free text).

It should also be noted that participants who were interviewed for this study were not always necessarily directly involved in the development of the BCT content and the theoretical underpinning of the programs. Some people involved in the early stages of design and development had since moved on to other roles. Nonetheless, the research team tried to identify ≥1 relevant individual from each digital provider and aimed to interview professionals from different backgrounds to gather a range of views and provide a comprehensive understanding of the processes involved in the design and development of each NHS-DDPP intervention.

Digital interventions have the potential to be delivered with higher fidelity than face-to-face interventions [7] as they do not rely solely on facilitator-centered delivery; instead, the BCT content can be standardized in its delivery via the functions of an app or website. Therefore, the current fidelity of design results is encouraging, as drift in delivery fidelity is less likely, thus increasing the chances of the program being effective if the BCT content is proven to be engaged with effectively.

Despite the encouraging results with regard to the planned BCTs in digital providers’ program designs, there is inconsistent use of theory from providers to justify why specific BCTs have been included and how they will produce the desired behavior changes (ie, the theoretical constructs they are targeting). Given that a clear underpinning theory supports the translation of BCTs in intervention designs to intervention delivery [24-26], the absence of a clear logic model or an explicit explanation of underpinning theory describing the constructs to be targeted by specific BCTs is potentially problematic.

One of the ways in which providers could usefully describe the underpinning theory is via a logic model, which provides a concise and clear way of visually presenting interventions informed by multiple theories. However, there is no clear guidance on whether it should be the role of the providers to produce a logic model or whether it would be more appropriate for commissioners of large-scale programs to produce an initial logic model to guide providers from the outset [27]. However, the requirement of a logic model during the commissioning process of large-scale programs could ensure a clear rationale for the BCTs included in intervention designs, which may result in fewer issues in delivery and receipt fidelity in program implementation [27]. It should be noted that the inclusion of a logic model is a recommendation that has emerged since the evaluation of the NHS-DPP commenced and following the previous round of service procurement. Commissioners of the program are now increasingly focusing on the important role of logic models in the upcoming round of service procurement for the NHS-DPP.

In addition, a further 68% of the planned BCT content in the digital service was not specified in the program specification, a finding similar to the face-to-face offering of the NHS-DPP [17]. It is possible that interventions containing more techniques to help people change their diet and physical activity behaviors may be more effective [53,54]. However, without a logic model or explicit description of the underpinning theory justifying the inclusion of additional BCT content, programs cannot be tested and evaluated effectively to establish the techniques that work in digital behavior change interventions and why.

This research provides an example of a design fidelity assessment of a large-scale digital intervention. Using the present findings as a baseline, future research will assess fidelity of the delivery of the digital service and service user engagement with the program. The current analysis found that some digital providers still plan to rely on health coaches to deliver aspects of the program to service users (eg, support via telephone); thus, the nature of this delivery will depend on the skills of the health coach in a similar way as that delivered by facilitators of face-to-face sessions. This may help to increase program engagement [55] but may also pose a risk to delivery fidelity because of variations in the background or training of health coaches. This study will be critical to assess whether the plans digital providers state they will put in place (with high fidelity) translate into program content that is actually delivered and understood as intended. Ongoing qualitative work examines how the behavior change content of the program is understood by service users, an underresearched aspect of fidelity [56]. Further analysis of the interview data with program developers from this study is also planned, with the aim of qualitatively capturing the nuances of intervention development and how each of the digital providers’ programs has evolved over time.

When designing their programs, the digital providers of the NHS-DDPP demonstrated good fidelity to the BCT content of the program specification. Thus, if the NHS-DDPP is delivered with high fidelity and engaged with by service users, it should increase the effectiveness of the program in achieving a reduction in the incidence of T2DM. However, the lack of theory described in digital providers’ intervention designs is potentially problematic if providers are not explicit in how their programs will work to achieve behavior change, given that a clear underpinning theory supports the translation of BCT content in intervention design to intervention delivery.