Evidence-based online psychoeducational programs may offer an efficient, accessible, and scalable option for improving participants’ health and well-being [1-9] but only if they succeed in reaching the participants for whom they are designed [10,11]. Unfortunately, few web-based interventions are widely implemented following study completion [3,12]. Programs for informal caregivers offer a case in point [13]. Informal caregivers are family members, friends, or acquaintances (hereafter referred to as caregivers) who provide essential support to persons living with major health conditions or disabilities in the community [14,15]. Although web-based psychoeducational programs for caregivers have proliferated, the best strategies for optimizing their implementation remain unknown [7,16-19].

Among caregivers, those providing support to US veterans have some of the highest documented rates of caregiving burden, stress, depression, and neglect of self-care [20,21]. They have expressed a desire for help from the US Department of Veterans Affairs (VA) Health Care System, which is the largest integrated health care system in the United States [20]. Galvanized by a 2010 Act of the US Congress [22], the VA Caregiver Support Program (CSP) commissioned the development of a web-based psychoeducational and self-management workshop, Building Better Caregivers (BBC), for these caregivers. In earlier studies, BBC participants experienced reduced stress and depression and increased self-efficacy [23,24]. On the basis of this evidence, the VA adopted the BBC workshop nationally in 2013 at all VA medical centers in the United States and its territories [22]. However, BBC program uptake was lower than expected at many medical centers.

We seek to determine why the national web-based BBC workshop did not reach caregivers at some centers and whether uptake could be improved by performing a multistage mixed methods implementation evaluation involving national and regional VA leadership, staff and informal caregivers at local centers, and national VA CSP operations data sets. Specifically, our objectives were organized into 2 stages:

Stage 1: initial evaluation.

Stage 2: follow-up evaluation.

We employed a convergent mixed methods design for both stages of research because the implementation of a web-based caregiver workshop nationally among VA medical centers was a complex health services intervention that required multilevel processes, which we felt would be best assessed with a mixed methods approach [25]. We used the revised Promoting Action on Research Implementation in Health Services (PARIHS) framework as our implementation evaluation framework [26,27]. The PARIHS constructs include (1) evidence (eg, research evidence supporting innovation effectiveness, participant experiences); (2) context (eg, resources necessary to support staff activities, data management and evaluation capabilities); (3) facilitation (eg, skills and attributes of local staff, staff training, and mentoring and assistance from experts); and (4) successful—or unsuccessful—implementation (eg, achievement of desired outcomes of innovation, extent of uptake). These PARIHS constructs were represented throughout our mixed methods process, including in the structure of our sampling frame, survey questions, interview questions, and the analysis and interpretation of both quantitative and qualitative data. The Stanford University Institutional Review Board approved the evaluation (protocol no. 29965), and participants provided verbal consent. Caregivers received US $40 as compensation for study participation. The staff and leaders did not receive compensation.

In stage 1, we collected and assessed survey and interview data concurrently to enable planned triangulation between the data sets and results (October 2014 to January 2015). The qualitative (interview) approach was the higher priority approach because it enabled the collection of detailed, nuanced feedback on implementation strengths and weaknesses and suggestions for changes not achievable in a survey. However, the quantitative (survey) approach was integral to the implementation evaluation because it enabled anonymized structured feedback on prespecified, discrete implementation elements derived from the PARIHS constructs and subconstructs that might or might not be mentioned during interviews and that we could compare and integrate with emergent themes from the qualitative data. Surveys also generated contextualizing data on respondent characteristics (eg, age, caregiving relationship, care partner health conditions). The quantitative and qualitative analyses were performed by the study team, and the study team, the advisory board, and a national VA CSP leader integrated findings and reached consensus on strategies likely to improve workshop uptake during a half-day virtual retreat (May 2015). The study team communicated findings and recommendations to VA CSP national leaders in a final report and webinar meetings (June to July 2015).

The BBC workshop is an evidence-based, 6-week, interactive psychoeducational and self-management program designed to reduce caregiver stress and depression and increase caregiver skills and self-efficacy [23,24]. Each workshop includes 20-25 participants and is guided by 2 trained cofacilitators. During workshops, caregivers read weekly lessons, perform self-management- and caregiving-related activities to solidify new skills and interact with and receive social support from peers and facilitators. Examples of self-management and caregiving skills include techniques for stress reduction, creating weekly action plans that are realistic and achievable, problem solving when challenges occur, managing difficult care partner behaviors, communicating with family, friends, and health professionals, and planning for the future, among others. The workshop is delivered on a secure online platform developed specifically for the workshop that, at the time, supported use on a computer or tablet device and has since expanded to include smartphone delivery. The workshop is only available in English.

Workshop facilitators are caregivers of persons with chronic conditions. They receive a 5-day training that teaches them facilitation skills and about the workshop (its theoretical basis, structure, and content). Facilitators must demonstrate mastery of the material and adhere to a comprehensive workshop facilitator manual. Master facilitators monitor each workshop for fidelity. Facilitators can lead workshops remotely from anywhere in the United States and are paid to do so.

The VA adopted the BBC workshop nationally in 2013, making it available at no cost to caregivers registered at any VA medical center in the United States or its territories [22]. The VA CSP leadership and our study team at the VA Center for Innovation to Implementation identified that BBC uptake was lower than expected. Using CSP operations data, we determined that 5.74% (1265/22,022) of eligible caregivers enrolled in the workshop in the first 2 years of the rollout, but that center-specific enrollment rates varied widely—from 0% to 37%—with 10.6% (15/142) of all the centers enrolling no participants. Thus, the VA CSP and our study team partnered to perform the current implementation evaluation. As the study team, we convened an advisory board composed of representatives of key stakeholder groups and experts to advise us at all stages of the evaluation. The advisory board included a veteran caregiver, veteran care partner, expert on informal caregiver research, expert on self-management research, clinical psychologist with expertise in eHealth interventions for veterans and their families, and director of web services for the VA.

All 4 national leaders involved in the workshop rollout were invited to participate in the implementation evaluation and all agreed to do so. National leaders were defined as those overseeing or directly managing the BBC implementation at the national level—for example, the VA CSP director, national VA BBC program manager, and BBC project director at the national organization contracted to administer the workshop on behalf of the VA. To identify regional leaders, local staff, and caregivers for participation, we used stratified purposive sampling (Table 1). BBC enrollment rates at all VA medical centers (standardized by catchment size) were calculated to identify high- and low-enrollment centers and the regional networks to which they belonged. At the time, there were 142 medical centers belonging to 21 regional networks. Low-enrollment centers were defined as those in the 10th percentile of referrals during the first 2 years of workshop rollout, and high-enrollment centers were those in the 90th percentile. We identified 5 regional networks that encompassed both low- and high-enrollment centers and invited their CSP regional leaders to participate and all agreed. We then selected 2 regions for local evaluation based on geographical diversity and variation in medical center catchment size: the Northeast region with 2 paired medium-sized centers (1 high-enrollment center and 1 low-enrollment center) and the Gulf Coast with 2 paired large centers (1 high-enrollment center and 1 low-enrollment center). At selected centers, we recruited local staff most involved in the workshop rollout—all CSP social workers who worked as caregiver support coordinators at each center (hereafter referred to as coordinators) and other staff whom they identified as involved in the BBC implementation. Among the 14 local staff members approached, 13 (93%) agreed to participate. The staff member who declined had been a substitute coordinator for a short period but had returned to unrelated prior duties by the time we contacted her.

We also recruited 8 veteran caregivers per center (32 in total). Caregiver eligibility criteria were being registered with the VA as a veteran caregiver and being 18 years of age or older. Exclusion criteria included having limited English proficiency and inability to speak by phone for a telephone interview (eg, due to hearing loss). We used VA operations data to identify all registered caregivers at each center and cross-referenced these with operations data on workshop referrals and enrollments. Sampling was divided into target groups of (1) caregivers enrolled in the workshop, (2) caregivers aware of the workshop but not enrolled, and (3) caregivers unaware of the workshop. We randomly generated the order of attempted contact with caregivers and performed a maximum of 2 telephone calls per caregiver. We attempted to contact 163 caregivers and, of these, 10 (6.1%) had a nonworking phone number and 101 (62.0%) did not respond to repeated attempts. We spoke with 54 (33.1%) caregivers. Among these, <1% (1/163) was ineligible because of limited English proficiency, 1.8% (3/163) called us back and desired to enroll but could not because we had reached our enrollment targets, 11.0% (18/163) actively refused, and 19.6% (32/163) enrolled. Reasons for refusal included no time, having no one to take care of veteran or other household responsibilities during the interview, veteran having passed away, being 39 weeks pregnant, and wanting to answer questions by mail only. All potential participants (caregivers, staff, and leaders) received information on the study objectives and researcher identities before their agreement to participate.

The quantitative evaluation consisted of surveys of caregivers and local staff. Caregivers completed paper-based surveys within a week of interview completion (Multimedia Appendix 1). Questions were asked about their demographic and caregiving characteristics and veteran care partners’ demographic and health characteristics. Caregivers who participated in the workshop also indicated whether they would recommend it to others and whether it had improved their quality of life using Likert scale response options 1 to 5 that were anchored by descriptors—strongly disagree, disagree, neither disagree or agree, agree, and strongly agree—and the option of answering do not know/not applicable. The staff survey questions were adapted from the Organizational Readiness to Change Assessment (ORCA), a validated implementation sciences survey structured according to PARIHS constructs (Multimedia Appendix 2) [28]. An additional survey question asked respondents to indicate whether, “At our facility, implementation of the Building Better Caregivers program has been a success,” using Likert scale response options 1 to 5 anchored by descriptors—strongly disagree, disagree, neither disagree or agree, agree, and strongly agree—and the option of answering “do not know/not applicable.” The ORCA is designed for local staff within organizations implementing evidence-based programs. Therefore, it was completed by local staff only (ie, not by regional or national leaders). As survey respondent numbers are small, we report results with descriptive statistics rather than comparative statistics—specifically, the number (%) of respondents who gave the response of interest.

The qualitative team consisted of 4 coinvestigators, (RG, CK, AN, and VY) with training and expertise in interview-based data collection and analysis as well as content expertise in the following areas: public health and palliative care (RG), sociology and linguistics of medicine (CK), medical anthropology (AN), and caregiving, primary care, and health services research (VY). All 4 were investigators at the National VA Center for Innovation to Implementation based at the Palo Alto VA Health Care System. With the exception of VY, who worked with one of the national BBC leaders in a separate study 5 years before this study, the qualitative team did not have prior relationships with the study participants. The qualitative team codeveloped the semistructured interview guides based on PARIHS constructs and pilot tested them with 1 caregiver and 2 staff members at the Palo Alto VA Medical Center before finalizing them (Multimedia Appendix 3).

Data collection consisted of 45-min one-to-one phone-based interviews that were audio-recorded and transcribed by professional transcriptionists and field notes that were recorded by interviewers immediately following interview completion. Transcripts were not returned to the participants for comments or corrections. In caregiver interviews, we asked about caregiving responsibilities, experiences with the VA, use of online technology, and preferences for caregiver support, education, and skills training. Among caregivers who had not enrolled but were aware of the workshop, we asked about outreach or enrollment processes they had experienced, why they had not enrolled, and for any other feedback they chose to share. Among caregivers who had enrolled, we asked about reasons for enrollment and feedback on their outreach, enrollment, and workshop experiences, suggestions for improvements, and whether the workshop had impacted them. In interviews with local staff and regional and national leaders, we collected feedback corresponding to all PARIHS constructs, including asking about their role, training, activities, and observations during the BBC rollout and suggestions for improvements.

The qualitative team codeveloped the codebook. Deductive codes were based on PARIHS constructs, whereas inductive codes were developed using content analysis of transcripts of a subset of 10 interviews and accompanying field notes [29]. Following the development of the initial codebook, 3 coders (VY, RG, and AN) applied a rapid analysis matrix coding form to evaluate an additional 13 interviews, with 2 coders coding each interview (double-coding) [30]. The initial matrix coding form consisted of rows for participants, codes as column headers, and transcript quotes and notes in the pertinent cells displayed in an Excel spreadsheet (Microsoft Corp). The full qualitative evaluation team then met and reviewed the matrices, including codes, quotes, and notes. CK (noncoder) examined quotes and codes to ensure consistency and facilitated discussion of these (eg, of areas of agreement or contention or areas that required clarification), suggested modifications or additions to the codes, and elicited team agreement on the final coding form. Coders applied the final coding form to all interview data, including interviews previously coded at earlier stages. The full qualitative team met again to identify, define, and reach consensus on emergent themes. The team identified themes that addressed strengths and weaknesses of workshop implementation, similarities, and differences in participant feedback from low- and high-enrollment sites and possible strategies for optimizing uptake. Participants did not provide feedback on the findings.

Next, we convened a half-day virtual meeting with our study team, advisory board, and a CSP representative using web-based conferencing capabilities. Attendees viewed survey and interview results displayed concurrently by the relevant PARIHS construct. Nonstudy team attendees (advisory board and CSP representative) commented on and confirmed the face validity of the results. All attendees then discussed and interpreted the results and reached a consensus on the integrated study findings.

Building from integrated study findings, at the same meeting, attendees identified broad strategies likely to improve workshop uptake and agreed that these strategies should be recommended to VA BBC national leaders. The study team reported both integrated study findings and recommendations to national leaders in the form of a written report and webinar meetings.

In 2018, we performed a follow-up evaluation consisting of interviews and longitudinal workshop enrollment data from the VA CSP operations databases to determine whether new or enhanced implementation strategies had been adopted and whether workshop enrollment changed.

In spring 2018, we conducted one-on-one phone interviews with 3 of the original 4 national leaders. The 4th leader was no longer working in the same position and was unavailable for the interview. Our goal was to learn about changes that had been made in response to the initial implementation evaluation. Questions focused on whether existing implementation strategies (from the 2013-2014 timeframe) had been enhanced and new strategies had been adopted (eg, based on stage 1 recommendations) and whether the workshop itself had been maintained. We used the same codebook established from the original 2015 interviews to analyze data from the follow-up interviews.

Following interviews, we used VA CSP operations data to retrospectively evaluate longitudinal workshop enrollment for the VA as a whole from 2013 to 2018. Data were available by VA fiscal year (FY; October 1 to September 30 of the following year). FY data correspond to the following periods of interest in this study:

We graphically displayed the longitudinal data for visual inspection to determine the potential impact of expanded implementation strategies on workshop enrollment.

Thirty-two caregivers, 13 local staff, and 9 regional and national leaders participated in surveys and interviews (Table 2). Approximately half of the caregivers (15/32, 47%) and local staff (7/13, 54%) were from low-enrollment centers. A total of 17 (53%) caregivers supported younger veterans from the post-9/11 era of service. All cared for veterans at high risk for morbidity and mortality based on their health conditions. Most were White, female, above the age of 40 years, and married to their care partners. Their care partner relationships (eg, spouse, child) were similar in distribution to those reported by caregivers enrolled in BBC during the first 2 years of workshop rollout (Multimedia Appendix 4). The chronic conditions of their care partners were also broadly similar to those of caregivers enrolled in BBC (Multimedia Appendix 4). Among the local staff, 69% (9/13) were caregiver support coordinators. Among regional and national leaders, 78% (7/9) were VA based and 2 (22%) were from the national organization responsible for administering the workshops on behalf of the VA.

A total of 72% (23/32) of caregivers completed the surveys. They reported high rates of increased stress (21/23, 91%), depression symptoms (19/23, 83%), sleep deprivation (19/23, 83%), neglect of healthy physical activity (15/23, 64%), neglect of healthy eating habits (14/23, 61%), and delay of own health care (11/23, 48%) since becoming a caregiver, which are comparable with national data on veteran caregivers [20]. Among caregivers enrolled in the workshop, 71% (10/14) completed surveys. In response to questions that correspond to the PARIHS evidence construct, 100% (10/10) reported that they would recommend the workshop to others and that their quality of life had improved (8/10, 80%) or may have improved (2/10, 20%) as a result of what they had learned in the workshop.

Among the local staff, 10 (77%) completed surveys: 71% (5/7) from low-enrollment centers and 83% (5/6) from high-enrollment centers. Describing whether workshop implementation at their center had been a success, all 5 staff members from high-enrollment centers agreed, whereas none of the 5 from low-enrollment centers agreed. Nonetheless, most felt that the BBC workshop had been accepted by caregivers and was supported by research and that they had received useful informational materials for caregivers and educational materials for themselves (Table 3). In contrast, a minority reported having a regional mentor who could assist them or having a system for tracking caregivers.

During interviews, participants from low-enrollment centers identified fewer implementation strengths and a greater number of weaknesses than those from high-enrollment centers. However, both groups generated similar implementation themes. As the analyses identified common themes, we report findings from low- and high-enrollment centers together.

Our study team, advisory board, and a representative from VA CSP leadership determined that the survey and interview findings were consistent with a number of implementation strengths—specifically, that caregiver experiences with the workshop were positive, research evidence supported its use, and staff received some useful outreach materials and resources that helped them support the rollout (Table 3, last column, and Table 4, last column). Findings also converged on implementation weaknesses—that outreach materials and approaches were insufficient in multiple areas, certain staff and regional leaders did not receive necessary training or mentoring, and that data management capabilities were inconsistent or lacking. When we noted areas of apparent inconsistency between quantitative and qualitative results (Table 3, rows 3 and 4), we discussed and noted potential explanations (Table 3, footnotes f and g).

On the basis of the integrated findings, we identified multiple strategies likely to improve workshop uptake and recommended them to VA leadership in the form of a report. The strategies included investment in robust outreach and marketing capabilities, tailoring outreach strategies to all key stakeholder groups and subgroups, use of stories, testimonials, repetition, details, and diverse delivery options in marketing campaigns, recurrent training sessions and mentoring for new staff, and comprehensive data management and reporting capabilities (Table 5).

Stage 2: Findings on New and Enhanced Implementation Strategies

During follow-up interviews in 2018, VA leaders identified multiple implementation changes made in response to stage 1 findings and recommendations (Table 5, last column; see Multimedia Appendix 5 for additional details). The VA CSP contracted with a new organization with greater marketing expertise to administer the workshop, and together they developed and adopted both new and enhanced outreach approaches. They tailored new marketing materials and outreach to implementation staff (eg, new monthly newsletters for caregiver support coordinators), some VA nonimplementation staff and services in close contact with pre-9/11 veterans and their caregivers (eg, primary care social work, home health nursing), and some veteran service organizations and communities (eg, through presentations and press releases). They highlighted stories for caregivers in multiple new ways (eg, videos, quotes, photographs) and did the same for implementation staff (eg, caregiver support coordinators sharing their BBC stories with each other in videos or listservs). Outreach campaigns incorporated more repeat touches with both groups (eg, regular email invitations to caregivers), diverse forms of delivery (eg, social media posts and phone-based blurbs) and more detailed information on workshop structure and content. For instance, leadership approved and built a demonstration workshop so that staff could learn about workshop details firsthand and then pass the information to the caregivers with whom they worked. Expanded materials for new staff include recurring national training and reminders about how to access outreach resources. Finally, the workshop platform began regularly to email staff with reports on the enrollment status of caregivers at their respective medical centers.

Leaders also reported that they were unable to address some implementation weaknesses. The CSP website, which was managed by a different VA entity with limited funding, was not updated to include more information on the workshop and links to it. The VA national database of registered caregivers remained unlinked from the workshop database, and so it could not be used to generate workshop outreach invitations or other communications to caregivers. Finally, there was no plan to develop a workshop in Spanish.

Visual inspection of longitudinal enrollment data indicates a possible impact of the new and enhanced implementation strategies. During the initial BBC rollout period (2013 to 2014) and the implementation evaluation period (2015), annual enrollment was approximately 700 caregivers per VA FY (Figure 1). Following implementation changes made in late FY 2015 and early FY 2016, annual workshop enrollment increased. In FYs 2013 to 2015, cumulative enrollment was 2139, which rose to 4030 in FYs 2016 to 2018. Over 6 years, cumulative BBC enrollment reached 6169, which equates to 12% of approximately 50,000 caregivers registered with the VA.

This study expands the limited implementation science literature on best practices for health systems to use when implementing web-based psychoeducational programs [13,31]. In our evaluation of the BBC workshop rollout among VA centers nationwide, we identified implementation strengths and weaknesses, including belief in the positive impact of the workshop, use of some successful outreach approaches for caregivers, and missed opportunities to improve outreach and to support both new and experienced implementation staff. On the basis of these findings, we identified and recommended to VA leadership new or enhanced implementation strategies likely to improve workshop uptake—specifically, increased investment in outreach and marketing capabilities, tailoring outreach strategies to key stakeholder groups and subgroups, use of campaigns that are personal, repeated, detailed, and have diverse delivery options, recurrent training and mentoring for new staff, and comprehensive data management and reporting capabilities to support staff implementation efforts. We determined that the VA adopted many of these strategies and, in the years following adoption, national BBC enrollment increased.

Our evaluation has a number of limitations. We initiated this after the workshop rollout was underway, rather than prospectively. At the same time, this enabled us to quickly identify low- and high-enrollment sites for inclusion in the evaluation. Our evaluation is also limited to a single health care system. Nonetheless, the VA health care system is the largest integrated health care system in the United States and, as such, it has numerous medical centers. These demonstrated variation in implementation success as well as geography and catchment size, which enhanced the robustness of our evaluation. The VA workshop is available only in English, which precludes use by caregivers who have limited English proficiency.

Among caregiver programs, many have been funded as research projects but few, if any, have been the focus of published implementation evaluations. A recent systematic review identified 17 studies of web-based caregiver programs focused on education, health, and well-being [7]. However, we were unable to identify any with a published implementation evaluation. Among web-based caregiver programs identified by other means [32, 33], no implementation evaluations were found. Finally, we identified a process evaluation of a web-based intervention to support frail older adults that, at their discretion, could also involve their caregivers but was not designed for caregivers [34]. Thus, it remains to be seen how results from the current implementation evaluation may compare with findings from implementation evaluations of other web-based psychoeducational programs for caregivers.

Among caregivers who care for military veterans, US VA caregivers may be unique in the amount of support they receive. Since 2010, the VA and VA CSP have received substantial federal funding for caregiver programs and research. This has enabled the US VA and VA investigators to develop the BBC workshop and other VA caregiver programs (eg, Resources for Enhancing All Caregivers Health [REACH] VA); assess such programs for efficacy; fund their implementation; and in the case of this study, evaluate their implementation [22,35]. We are unaware of funding opportunities on the same scale in other countries. This may explain why evidence on the efficacy and implementation of veteran caregiver programs outside the United States is limited. In an environmental scan of resources (defined as programs and services) for families of veterans with operational stress injuries, investigators identified 42 non–United States resources but concluded, “Information on evaluation and evidence for resources is limited” [36]. To our knowledge, none were the focus of an implementation evaluation.

If one examines web-based or technology-based interventions for patients rather than caregivers, implementation data remain sparse [4,31,34,37,38]. A 2013 systematic review of studies of web-based patient self-management workshops determined that 87% “did not describe any measures taken to sustain the tested programs past designated study time periods” [31]. The remainder reported on the number of participants reached but not the strategies that were successful or unsuccessful in achieving participation. Not surprisingly, a similar 2013 review of web-based interventions for psychosocial health determined that issues of reach and adoption of web-based interventions were a crucial area for further research [4]. Four years later, a 2017 review of 268 randomized controlled trials described a proliferation of evidence-based self-guided internet health programs but concluded, “Unfortunately, most of the Internet delivered health interventions that were efficacious through RCTs were not available after the conclusion of the trials” [3] and we could not identify implementation evaluations of these same trials. Finally, in the case of eHealth services broadly defined (eg, telehealth, patient portals, web-based decision aids), others have identified a taxonomy of implementation barriers and facilitators from multiple studies and expert input [11], but there are few follow-up studies to determine whether making implementation enhancements suggested by the taxonomy does indeed boost uptake. Thus, experts have highlighted implementation research as the crucial next step in spreading the benefits of evidence-based digital health interventions to all those who might benefit [39].

This evaluation suggests that implementation research can help boost uptake of web-based psychoeducational interventions for caregivers and perhaps others. Among the results, we were most surprised by the prominence of findings related to outreach and marketing. The implementation science literature has not emphasized these concepts to any great extent, although this may be changing. This evaluation was performed before the publication of the most recent iteration of the PARIHS implementation framework known as the i-PARIHS framework [40]. i-PARIHS adds a new construct, “recipients” to existing constructs (evidence, context, and facilitation) and defines it as “the people who are affected by and influence implementation [of the intervention] at both the individual and collective team level” [40]. In this study, we found implementation could be strengthened when specific groups and subgroups of recipients (eg, caregivers, staff, community organizations) were explicitly identified for tailored outreach campaigns, which aligns with this new PARIHS construct. Implementation scientists have started to note that marketplace principles and storytelling may contribute to implementation success [41,42]. Similarly, the recent report on a US National Cancer Institute implementation accelerator program concluded that “researchers can benefit from learning the ‘language’ of business” when “trying to move their research from the lab to the real world” [43].

In this mixed methods implementation evaluation of the web-based psychoeducational BBC workshop for US veteran caregivers, we identified initial implementation strengths and weaknesses as well as multiple strategies likely to improve workshop uptake—including strong investment in outreach and marketing capabilities, tailoring of specific outreach strategies to all key stakeholder groups, use of outreach campaigns that are personal, repeated, detailed, and have diverse delivery options, ongoing training and mentoring for new implementation staff, and comprehensive data management and reporting capabilities. Upon follow-up assessment, we determined that the VA enhanced or newly adopted a number of these recommended strategies nationally and that workshop enrollment subsequently increased. Other health systems may want to deploy these strategies when implementing their web-based programs.