Background: Despite the near ubiquity of mobile phones, little research has been conducted on the implementation of mobile health (mHealth) apps to treat patients in primary care. Although primary care clinicians routinely treat chronic conditions such as asthma and diabetes, they rarely treat addiction, a common chronic condition. Instead, addiction is most often treated in the US health care system, if it is treated at all, in a separate behavioral health system. mHealth could help integrate addiction treatment in primary care.
Objective: The objective of this paper was to report the effects of implementing an mHealth system for addiction in primary care on both patients and clinicians.
Methods: In this implementation research trial, an evidence-based mHealth system named Seva was introduced sequentially over 36 months to a maximum of 100 patients with substance use disorders (SUDs) in each of three federally qualified health centers (FQHCs; primary care clinics that serve patients regardless of their ability to pay). This paper reports on patient and clinician outcomes organized according to the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework.
Results: The outcomes according to the RE-AIM framework are as follows: Reach—Seva reached 8.31% (268/3226) of appropriate patients. Reach was limited by our ability to pay for phones and data plans for a maximum of 100 patients per clinic. Effectiveness—Patients who were given Seva had significant improvements in their risky drinking days (44% reduction, (0.7-1.25)/1.25, P=.04), illicit drug-use days (34% reduction, (2.14-3.22)/3.22, P=.01), quality of life, human immunodeficiency virus screening rates, and number of hospitalizations. Through Seva, patients also provided peer support to one another in ways that are novel in primary care settings. Adoption—Patients sustained high levels of Seva use—between 53% and 60% of the patients at the 3 sites accessed Seva during the last week of the 12-month implementation period. Among clinicians, use of the technology was less robust than use by patients, with only a handful of clinicians using Seva in each clinic and behavioral health providers making most referrals to Seva in 2 of the 3 clinics. Implementation—At 2 sites, implementation plans were realized successfully; they were delayed in the third. Maintenance—Use of Seva dropped when grant funding stopped paying for the mobile phones and data plans. Two of the 3 clinics wanted to maintain the use of Seva, but they struggled to find funding to support this.
Conclusions: Implementing an mHealth system can improve care among primary care patients with SUDs, and patients using the system can support one another in their recovery. Among clinicians, however, implementation requires figuring out how information from the mHealth system will be used and making mHealth data available in the electronic health (eHealth) record. In addition, paying for an mHealth system remains a challenge.
The Use of mHealth in Primary Care
Despite the near ubiquity of mobile phones, little systematic research has been conducted on the use or implementation of mobile health (mHealth) technology in managing chronic health conditions in primary care. Although primary care patients use mHealth apps, their use is generally haphazard and self-selected [, ]. The mHealth apps available to the public vary greatly in quality, and problems such as software bugs, poor design, and limited technical support are common [ ]. Perhaps not surprisingly, given these problems, the majority of health-related apps are used only once [ ]. Primary care clinicians know very little about the mHealth systems their patients are using, and most clinicians receive no health-related information from mHealth systems [ , ]. The literature contains numerous pilot studies and descriptions of mHealth systems [ ] but only a few rigorous studies about the use of mHealth in primary care [ , ]. Implementation research studies in which primary care clinics enroll cohorts of patients using the same mHealth system in an orchestrated fashion are particularly lacking. Thus, the role mHealth can play in the US primary care system remains largely unknown [ ].
mHealth Interventions for Addiction Treatment
Although hundreds of mobile phone apps for addiction treatment are available commercially, most of these apps have not been evaluated in the peer-reviewed literature . The small number of apps that have been evaluated constitute a growing body of evidence supporting the effectiveness of mHealth in treating addiction [ - ]. Most of this evidence relates to self-help interventions [ , , ] and to texting-based monitoring and reminder systems [ , ]. Little evidence relates to comprehensive mHealth systems for addiction, which have the strongest theoretical base and the most long-lasting effects, or to mHealth interventions for addiction integrated into patients’ recovery and health plans [ ].
Barriers to the Integration of Behavioral Health in Primary Care
In this paper, we report the results of an implementation research trial funded by the National Institutes of Health–National Institute on Drug Abuse. The trial aimed to integrate behavioral health treatment into primary care. We focused on one aspect of behavioral health—addiction—that presents considerable barriers to integration, such as the inability to bill for services, mental health stigma, and primary care physicians being ill-prepared to treat behavioral health problems [, ]. Primary care operates under productivity guidelines that limit the time clinicians can spend with patients, whereas addiction treatment typically involves frequent counseling sessions. Medication is crucial in primary care, but it has a comparatively short history in addiction treatment [ ]. Primary care focuses on chronic conditions, such as diabetes and hypertension, and practitioners expect that patients’ adherence to treatment will vary over time. Behaviorial health has only recently begun to view addiction as a chronic condition [ - ]. Lapses have often resulted in discharge from treatment. Primary care treats patients one-on-one, whereas behavioral health often organizes patients into groups for treatment [ ]. Financing models and information technology (IT) infrastructure also differ greatly between the two systems of care [ ]. Finally, patients with addictions often have elevated anxiety and present frustrating behaviors to providers, such as frequently missing appointments [ ].
In this research, our premise was that mHealth could ease the integration of addiction treatment into primary care. We proposed to examine how implementing an evidence-based mHealth system for addiction could be useful to both patients and clinical staff in real-world primary care settings. The mHealth system used in the study is named Seva, a Sanskrit word meaning “selfless caring.” Its key components were previously proven effective in carefully controlled patient-level randomized clinical trials [, ]. In a randomized clinical trial conducted in patients with alcohol use disorder leaving 90-day residential care, the intervention comprising the backbone of Seva reduced risky drinking days by 57% [ ] and increased retention in treatment by 77% [ ] compared with patients in the control group.
Purpose of This Study
The study reported here sought answers to 3 broad research questions:
- How can Seva be implemented in primary care settings efficiently and effectively?
- To what extent do patients and staff accept and use Seva?
- How does Seva affect clinical care for patients and staff?
Because the study tested both clinical and implementation interventions, it is considered to be a hybrid type 2 effectiveness-implementation study . Details of the study protocol, the theoretical foundations of the implementation strategy, and a description of Seva were published previously [ ]. This paper reports quantitative and observational results. Selected qualitative results were reported separately [ ].
We made Seva available to up to 100 patients at each of 3 federally qualified health centers (FQHCs) across the United States of America. FQHCs are federally funded primary care clinics that serve mainly low-income patients. As a condition of funding, FQHCs must provide access to behavioral health services. Thus, FQHCs are in the vanguard of clinics in the United States trying to integrate behavioral health into primary care. FQHCs also serve very vulnerable patients—many in poverty and suffering from addiction. At each site, patients were enrolled over a 12-month period. After enrollment, patients had access to Seva for 12 months.
Because the focus of the study was implementation and not patient outcomes per se, the study did not randomize patients. Instead, clinicians were free to enroll any patients from their substance-using populations whom they thought might benefit from Seva based on their clinical judgment. We used the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework to organize the evaluation . The RE-AIM framework is a predominant evaluation framework in implementation research studies.
The study protocol was approved by the Medical Sciences Institutional Review Board at the University of Wisconsin–Madison (2012-0937-CP020) and is registered at ClinicalTrials.gov (NCT01963234).
Clinics were recruited in partnership with the National Association of Community Health Centers. We recruited FQHCs with established electronic health (eHealth) records to understand how Seva relates to existing clinic technology. In selecting sites, we aimed to achieve geographic reach, diversity in patient populations, and differences in organizational structures to better understand how environmental and structural factors might affect implementation. From a pool of approximately 1100 FQHCs nationally, we selected an FQHC affiliated with the University of Wisconsin as a pilot site; a relatively small, rural, freestanding FQHC with integrated behavioral health services (including addiction treatment) as a second site; and an urban FQHC that largely serves a minority population as our third site.
Clinician and Patient Recruitment
A limited number of clinician subjects consented at each site. These staff subjects worked with the research team to integrate Seva into clinical workflows during the preimplementation phase and subsequently identified, recruited, enrolled, and trained patients to use Seva and monitored patients’ use of the system.
Authorized clinicians identified potential subjects through the electronic health record (EHR). On the basis of their clinical judgment, clinicians were free to enroll patients from their substance-using populations whom they thought might benefit from Seva. The patients had to meet the following inclusion criteria: (1) aged 18 years or older, (2) meet the criteria for substance use disorder (SUD) as per the Severity of Dependence Scale, (3) have no current psychotic disorder severe enough to prevent participation, (4) have no acute medical problem requiring immediate inpatient treatment, (5) are willing to use Seva, and (6) could understand and sign a consent form in English. If a patient was incarcerated during the study, his or her participation was stopped. If he or she was still interested in participating in the study when released from jail, the person was able to rejoin the study. Patient participation was voluntary. Patient subjects were excluded if (1) their condition warranted inpatient detoxification until they were well enough to participate and (2) they were unable to understand and complete the informed consent. Couples were not recruited to the study at the same time to avoid dyadic conflict.
During a clinic appointment, the clinician asked an eligible patient if he or she was interested in learning more about the study. If the patient was interested and gave permission, the clinician notified the site coordinator that a patient was interested in hearing about the project. In the privacy of a clinic exam room or office, the site coordinator explained the study, its benefits, and potential risks of participation. The site coordinator also answered any questions the patient (subject) had. If the patient was interested in participating, he or she was asked to complete the informed consent.
The flow of patients through the study is shown in.
Training Patients on Seva
The site coordinator trained participants in person. Participants could download Seva if they had their own mobile phone. If they did not, the site coordinator gave participants a phone and basic instructions on how to use it. Then he or she explained and showed participants how to use the different services of Seva. Participants were encouraged to ask questions and were given a toll-free number to call the Seva technical support line if they had any questions. Each participant also received a user guide that showed all Seva services and explained how to use them with easy text and graphic instructions.
Clinic Rollout Process
Each site designated a change leader, a clinical leader who was the point of contact with the implementation coach and coordinated implementation activities; the site coordinator; and a change team of 4 to 8 clinical and/or administrative staff members who helped make the organizational changes that were necessary to implement Seva. The coach made an initial site visit during the 4-month period of implementation preparation to create a welcoming environment for Seva. During this visit, the coach conducted a walk-through exercise with the change team members (an exercise in which employees experience clinic processes as patients do); a workflow assessment using flowcharting; and a technical assessment of data to be gathered and procedures needed to conduct the study. Through this preparation period, the coach worked with the change leader to ensure that pretest data were collected, Seva was demonstrated to clinical teams who might serve as referral sources, and barriers to implementation were identified and rectified.
Seva offered patients a discussion board used by the patients in the study; interactive modules to teach problem-solving, self-regulation, and other skills; tools for coping with cravings and high-risk situations (eg, relaxation exercises, strategies from cognitive behavioral therapy, links to local 12-step meetings); and health tracking. For clinicians, Seva provided a Web portal with a Clinician Report containing longitudinal information generated by patients’ self-reported data about their substance use and well-being (eg, sleep, depression).
We created a detailed implementation plan tailored to each clinic. The implementation strategy involved 4 phases at each site, which were (1) initiate (bring key clinical staff together for training), (2) prepare (assure a welcoming environment for Seva), (3) improve (conduct rapid-cycle tests of ideas from the previous stages), and (4) implement (use, monitor, and sustain the technology). An organizational coach (a member of the research team experienced in coaching) helped clinics implement Seva, starting with an in-person planning and kick-off meeting. This meeting was to be followed by monthly phone calls. Implementation plans consisted of a list of scheduled activities, such as when recruitment would start and end, when patients would be trained to use Seva, and so on. The implementation plans were informed by baseline assessments of readiness for implementation, sustainability potential, and technology acceptance from the perspective of clinic staff members who worked on the implementation plan [- ].
We expected that Seva would enable primary care physicians to better manage addiction in their patients, much as they manage chronic conditions such as diabetes and asthma, by making information about a patient’s recovery readily available. Physicians and other clinicians who referred a patient to behavioral health could see through Seva how these referrals worked out. Seva would also improve outcomes by making support, information, and skills training available to patients almost anywhere and anytime.
The measures used in this study are summarized in.
|Domain and measure||Data source(s)|
|Number of Seva patients (eligible, enrolled)||EHRa, patient surveys|
|Characteristics of participating patients||Patient surveys|
|Substance use||Patient surveys|
|Health care utilization (hospitalizations, ERc visits, specialty addiction treatment)||Patient surveys|
|HIV testing rates||Patient surveys|
|HIV risk behaviors||Patient surveys|
|Characteristics of participating clinics||Clinic administrative data|
|Use of Seva by staff (including referrals)||Seva server files, referral tracking logs maintained by clinic staff|
|Use of Seva by patients||Seva server files|
|Stages of Implementation Completion||Project tracking logs maintained by research team|
|Implementation and operating costs||Observation and interviews of clinic staff; project administrative data|
|6-month follow-up on effectiveness measures||Patient surveys|
|Patient use of Seva at 12 months||Seva server files|
aEHR: electronic health record.
bQoL: quality of life.
cER: emergency room.
To assess Reach, we examined the characteristics of the participating sites and the patients they serve. We examined how many patients were eligible to use Seva in each clinic versus how many enrolled in the study over the enrollment period.
To assess Effectiveness, we considered patients’ substance use, quality of life (QoL), health care utilization, human immunodeficiency virus (HIV) risk behaviors, and HIV testing rates through self-reported data collected in surveys administered in person at baseline and by phone at 6 months. Risky drinking days were the number of days in the past 30 days on which, within a period of 2 hours, women consumed more than 3 standard drinks and men more than 4 standard drinks, corresponding to the National Institute on Alcohol Abuse and Alcoholism’s definition of binge drinking . Illicit drug use days were defined as the number of days in the past 30 days on which participants used any illicit drug. QoL was measured using the PROMIS Global Health Scale [ ].
Effectiveness outcomes are reported for differences between baseline and 6 months. To handle skew distributions, nonparametric related sample tests (Wilcoxon signed rank tests for continuous and ordinal variables and McNemar tests for binary outcomes) were conducted. P values (2-tailed) and effect sizes were reported.
We assessed Adoption at both the patient and clinic level. For patients, we assessed weekly rates of Seva use, with use defined as a patient accessing any part of Seva beyond the home page during a given week in the 12-month period when patients’ phones and service plans were paid for. We defined clinician use in two ways—by the number of log-ins by clinicians per week to the website where Seva data were available and by the number of referrals to Seva. We also tracked the total number of Seva patients enrolled and whether medical providers (doctors of medicine [MDs], residents, physician assistants [PAs], nurse practitioners [NPs], and nurses or behavioral health providers) referred patients to Seva. To assess Implementation, we used the Stages of Implementation Completion model . The stages of implementation completion are (1) engagement; (2) consideration of feasibility; (3) readiness planning; (4) staff hired and trained; (5) fidelity monitoring processes in place; (6) services and consultation begin; (7) ongoing services, consultation, fidelity monitoring, and feedback; and (8) competency. We planned to assess Maintenance in a follow-up phone survey 6 months after each patient’s 12-month intervention period ended, but this plan was abandoned, as discussed below.
We selected an FQHC in Madison, WI, affiliated with the University of Wisconsin-Madison as our pilot implementation site; a relatively small, rural, freestanding FQHC with integrated behavioral health services, including addiction treatment, in Missoula, MT, was our second site; and an urban FQHC in the Bronx, NY, that serves a largely minority population was our third site (). In all 3 clinics, a total of 3226 patients were deemed clinically appropriate to use Seva in the 12 months corresponding to the implementation period. This number of appropriate patients represents all patients with a SUD diagnostic code in the EHR. With 268 patients enrolled, the intervention reached approximately 8.31% (268/3226) of patients with substance use issues at these 3 clinics. The reach of Seva to the target population was limited by our ability to pay for phones and data plans for a maximum of 100 patients per clinic. The racial and ethnic composition of patients in the study departs somewhat from those of US adults 18 or older estimated by the Substance Abuse and Mental Health Services Administration to meet the Diagnostic and Statistical Manual of Mental Disorders, 4th Edition (DSM-IV) criteria for alcohol dependence and abuse, with study participants being 67.9% (182/268) white versus 55% nationally; 25.0% (67/268) African American/Black versus 16% nationally; 11.6% (31/268) Other versus 10% nationally; and 14.2% (38/268) Hispanic/Latino versus 20% nationally [ ]. At least one clinic indicated more patients could have been enrolled.
and show changes in patient outcomes reported from baseline to 6 months. Of 268 enrolled patients, 207 (77.2%) were included in this analysis. Effect sizes were calculated per Cohen [ ].
In the substance use domain, significant reductions were observed for the number of risky drinking days (the primary outcome in the clinical trial preceding this implementation study), which declined by 44% [(0.7-1.25)/1.25] from baseline to 6 months and illicit drug-use days, which declined by 34% [(2.14-3.22)/3.22]. Two of the three abstinence outcomes also showed significant improvements (any illicit drug use and/or any drink or drug). Significant effects were found for two of the three QoL scores (overall QoL and mental health).also shows a significant reduction in hospitalizations and a trend toward fewer emergency room (ER) visits. shows an increase in HIV screening rates. Change in the rates of HIV risk behaviors (eg, condom use) and receiving other addiction treatments appeared to be nonsignificant.
Post hoc analyses assessed the relationship between the extent of Seva use and our study outcomes at 6 months. For each outcome, we used linear or logistic regression, controlling for the value of the outcome at baseline. System use was operationalized as (1) the total number of calendar days during the first 6 months on which individuals used Seva (going beyond the main menu), and (2) the total number of Seva pages viewed in the first 6 months (excluding the main menu). These system use measures were natural log transformed to reduce skewness. We found that participants who used Seva on more days in the first 6 months showed a significant increase in alcohol abstinence (P=.02), and participants who loaded more Seva pages showed a significant increase in overall abstinence from both alcohol and drugs (P=.01), as well as reduced HIV risk behaviors (P=.02). We found no significant associations between system use and risky drinking days, illicit drug use days, health care utilization, or QoL.
Whereas Seva’s quantitative results show promise in helping primary care patients remain abstinent, other important effects can be best appreciated by directly observing how mHealth affected patient care. An exchange on the Seva discussion board in the Bronx (Site 3) illustrates how the system helped patients support one another.shows an unedited excerpt of this exchange—only names have been changed. The exchange—which occurred over a 2-hour period beginning at 4:41 AM, well before the clinic was open—illustrates how patients struggling with addiction can support one another, in real time, outside the clinic, using mHealth. As the exchange highlights, the network of patients in the Bronx even took to referring to themselves as the “Seva family.”
Mobile health apps generally have low levels of continued use—approximately 80% are abandoned after only 2 weeks . In this context, all 3 sites showed high levels of sustained patient use ( ). At the start of the study, rates of patient use—defined as accessing any part of Seva beyond the home page during each week of the 12-month implementation period—at the 3 sites ranged from 94% (90/96) to 99% (69/70). Rates of use declined, but slowly, with rates at 12 months ranging from 53% (41/78) to 60% (39/65) across the 3 sites, which mirrors the patient retention rate of 57.6% at 8 months in the randomized trial of an earlier iteration of Seva [ ].
Clinician adoption of Seva was less robust than patient adoption for two main reasons. At each clinic, clinicians worried about being responsible for data available from Seva. For example, clinicians were concerned that a patient might express suicidal thoughts in a discussion post and they would miss it. To address this concern, the clinical staff at each clinic wanted one staff member to lead the implementation and operation of Seva for the clinic. This job included monitoring patient Seva activity for the clinic and alerting clinicians of significant changes in patient status. At 2 of the 3 sites, members of the behavioral health department led the implementation and operation of Seva. At the other site, the director of innovation led implementation. Not all FQHCs have such a position. At one of the sites, the behavioral health provider who led the implementation of Seva left the clinic near the end of the patient enrollment period. Members of the research team increased their involvement at the site to pick up the slack, but staff turnover remained a vexing issue.
|Characteristics||Site 1 (Madison, WI)||Site 2 (Missoula, MT)||Site 3 (Bronx, NY)|
|Insurance of patients (%)|
|Services offered||Primary care and mental health||Primary care, mental health, and addiction||Primary care and mental health|
|PCMHb designation||Level 3 (2011)||Level 3 (2014)||Level 3 (2014)|
|Number of eligible SUDc patients||1189||961||1076|
|Patients enrolled in Seva||97||100||71|
|Mean (standard deviation)||41.61 (10.95)||42.53 (10.24)||42.66 (11.78)|
|Gender n, (%)|
|Female||52 (54)||40 (40)||36 (51)|
|Drug of choice, n (%)|
|Alcohol||34 (35)||44 (44)||27 (38)|
|Opiates||31 (32)||14 (14)||8 (11)|
|Crack cocaine||9 (9)||3 (3)||11 (16)|
|Marijuana||1 (1)||4 (4)||16 (23)|
|Methamphetamine||0 (0)||15 (15)||1 (1)|
|Multiple drugs||22 (23)||20 (20)||8 (11)|
|Ethnicity n (%)|
|Hispanic/Latino||1 (1)||2 (2)||35 (49)|
|Race, n (%)d|
|White||68 (70)||90 (90)||24 (33)|
|African American/Black||30 (31)||2 (2)||35 (50)|
|American Indian or Alaskan Native||4 (4)||8 (8)||1 (2)|
|Asian or Pacific Islander||0 (0)||1 (1)||0 (0)|
|Other||0 (0)||2 (2)||15 (21)|
aeHealth: electronic health.
bPCMH: patient-centered medical home. Three levels of recognition exist, based on practice sites meeting six standards. Level 3 clinics have the best adherence to the standards.
cSUD: substance use disorder.
dPercentages do not add to 100 because patients could select more than one race.
|Measures||Sample size||Baseline||6 months||Za,b (P value)||Effect size db,c|
|N||Mean (SD)||Mean (SD)|
|Substance use in last 30 days|
|Any drinking days||207||2.53 (6.01)||1.67 (4.69)||-2.304 (.02)||-0.228|
|Risky drinking daysd||207||1.25 (3.78)||0.70 (2.58)||-2.008 (.4)||-0.199|
|Illicit drug-use days||206||3.22 (7.57)||2.14 (6.55)||-2.499 (.01)||-0.248|
|Overall QoL||202||28.47 (6.46)||30.03 (7.11)||3.653 (<.001)||0.370|
|Physical subscale QoL||206||13.20 (3.01)||13.48 (3.11)||1.682 (.09)||0.167|
|Mental subscale QoL||204||9.75 (2.99)||10.77 (3.50)||3.892 (<.001)||0.393|
|Health care utilization in last 6 months|
|No. of hospitalizationsd||207||0.43 (1.03)||0.22 (0.65)||-3.357 (.001)||-0.335|
|No. of ERf visitsd||207||1.10 (2.79)||0.75 (1.31)||-1.911 (.06)||-0.189|
aZ, provided in the Wilcoxon sign test, is the standard normal distributed Z-value used to test the significance between outcomes reported at two time points (eg, pretest vs 6 months).
bFor Z and d values, negatives mean decreases and positives mean increases in values from baseline to 6 months.
cCalculated from effect size d. On the basis of Cohen (1988) effect size, small: d=0.2, medium: d=0.5, large: d=0.8.
dRisky drinking days, hospitalizations, and ER visits: Those who reported no such events were coded with zero in the number of days of these events.
eQoL: quality of life.
fER: emergency room.
|Measures||Sample size||Baseline||6 months||Chi-square (P value)||Odds ratioa|
|N||n (%)||n (%)|
|Substance use in last 30 days|
|Any drink (Yes)||207||64 (30.9)||51 (24.6)||3.2 (.07)||0.552|
|Illicit drug use (Yes)||206||63 (30.6)||36 (17.5)||14.38 (<.001)||0.270|
|Any drink or drug (Yes)||206||97 (47.1)||69 (33.5)||12.57 (<.001)||0.349|
|Health care utilization in last 6 months|
|Currently receive other addiction treatments (Yes)||207||89 (43)||78 (37.7)||1.639 (.20)||0.694|
|HIV in last 6 months|
|HIV risky behavior (Yes)||207||76 (36.7)||65 (31.4)||1.818 (.18)||0.667|
|HIV testing (Yes)||206||81 (39.3)||116 (56.3)||33.03 (<.001)||—b|
aNumbers lower than 1 mean reductions of the events from baseline to 6 months.
bThe odds ratio for this variable cannot be calculated because patients’ HIV testing status was considered current at 6 months if they had been tested at baseline; that is, there were zero patients considered tested at pretest and not tested at 6 months.
A second unexpected development affected clinician adoption: We never succeeded in incorporating Seva data into the EHR at each clinic. The interoperability of EHRs is a widely recognized problem. In fact, the same system (eg, Epic Systems, developed by Epic Systems Corporation) may function very differently at one clinic versus another. The technical challenge in getting Seva data into the EHR of 3 health systems proved insurmountable in the context of this implementation research project. Instead, clinicians who wanted to review Seva data had to go to a website outside the EHR.
At each site, the leaders of Seva implementation and operation logged into Seva regularly during the 24 months of enrollment and implementation. At Site 1, 2 other staff clinicians besides the clinic Seva leader from behavioral health were most involved with Seva. The 3 clinicians at Site 1—one primary care physician and 2 behavioral health staff members—logged in an average of 0.6, 0.5, and 1.0 days per week. The Seva clinic leaders at Sites 2 and 3 were also the main users of Seva. The director of innovation at Site 2 logged in an average of 1.7 days per week, and a member of the behavioral health group at Site 3 logged in an average of 4.1 days per week. At two sites, behavioral health providers made most referrals of patients to Seva: 92% (89/97) of referrals at Site 1 and 76% (54/71) at Site 3. At Site 2, on the other hand, where we observed especially strong leadership and operational support for integrating behavioral with medical health, medical providers made 92% (92/100) of referrals (see).
also shows the percent of implementation goals and milestones completed. At the first two sites, we largely executed our implementation plan successfully in the preimplementation and implementation stages. At the third site, we struggled during preimplementation, primarily because of delays in getting the institutional review board’s (IRB) approval, which cost 6 additional months. Cost, which was specified in the protocol as another element of implementation, is reported below.
|Measures||Site 1 |
|Site 2 |
|Site 3 |
|Total number of Seva patients||97||100||71|
|Total number of primary care clinical staff at sitea||70||74||27|
|Number of patients referred to Seva by primary care clinical staff out of total number of Seva patients, n (%)||8 (8)||92 (92)||17 (24)|
|Total number of behavioral health providers at siteb||10||8||11|
|Number of patients referred to Seva by behavioral health providers out of total number of Seva patients, n (%)||89 (92)||8 (8)||54 (76)|
|Time to completion of project phases (months)|
|Preimplementation (12 months planned)||12||12||18|
|Implementation (12 months planned)||18||13||16|
|Maintenance (12 months planned)||2||4||1|
|Number and percentage of implementation goals and milestones completed, n (%)|
|Preimplementation (% of monthly implementation plan milestones reached)||12 (100)||12 (100)||12 (100)|
|Implementation (% of 100 patient enrollment goal at each site)||97 (97)||100 (100)||71 (71)|
|Maintenance (% of 12 monthly coaching follow up calls completed)||2 (17)||4 (33)||1 (8)|
aClinical staff members were physicians, residents, physician assistants, nurse practitioners, registered nurses, licensed practical nurses, and medical assistants.
bBehavioral health providers were licensed medical social workers, licensed mental health counselors, licensed clinical social workers, doctorate-level psychologists, and physicians.
|Type of costs||Site 1 (Madison, WI)||Site 2 (Missoula, MT)||Site 3 (Bronx, NY)|
|Operating costs (patient and clinic)||113,636||117,150||83,177|
|Implementation costs (clinic)||9948||23,345||28,121|
|Total costs for clinic||123,584||140,495||111,298|
Our implementation plans and their execution arose from baseline assessments of readiness for implementation, sustainability potential, and technology acceptance by clinic staff members who worked on the implementation plan, as well as the information that researchers gained from directly interacting with clinic staff. The plans for implementation involved using an organizational coach to help clinics through the 4 phases (initiate, prepare, improve, and implement), using an initial in-person visit and then monthly phone calls between the coach and the clinic change team. Although the organizational coach was heavily involved in the first two phases, another member of the research team who was most familiar with operating Seva became the primary contact for the clinics starting with the third phase. This researcher monitored the Seva discussion groups and gave hands-on, practical advice in response to questions from clinicians. Instead of having monthly coaching calls, clinician-researcher contact consisted mostly of short, frequent, ad hoc phone and email communications between the main clinician users and the research team’s site coordinator in response to specific patient issues and technical questions. These were the types of problems clinics wanted help in solving, rather than helping make organizational change, which was the primary type of help offered by the coach. Thus, we adapted our implementation plan at each site to remove monthly coaching calls during the implementation phase and instead focused on the enrollment of patients as our primary implementation goal.
Maintenance, defined as continued use of Seva after the 12 months during which patient phones and data plans were paid for, was low at all 3 sites. Patients were allowed to keep their phones and could continue accessing Seva by using Wi-Fi or paying for service themselves with a new phone number. Although we did not track whether patients paid for new data plans independently, we did track Seva use after the 12 months of paid use. Use after 12 months declined gradually to zero once the last recruited patient reached the end of the 12-month period of paid phone use. The decline in use limited our ability to collect follow-up patient surveys and led us to abandon our attempts to collect the phone survey we planned to collect 6 months after each patient’s 12-month intervention period to gauge maintenance. The decline in clinician use of Seva mirrored patient decline. That is, as fewer patients used the system, or if only a few later-enrolled patients were using it, clinicians logged in less because Seva had little patient activity for them to see.
Sites did not continue to use Seva after the study for at least two reasons. First, we could not resolve issues related to transitioning from a research study to ongoing use of an mHealth system. In particular, establishing procedures for consenting patients who wanted to use their own phones to access Seva outside the research protocol, and commingling these patients with patients from the research study, proved challenging. Second, the National Institute of Health grant funding ended, and none of the 3 clinics made arrangements to pay for mobile phones and data plans afterwards. Offering Seva only to patients who can cover their own mobile phone costs could have been a condition of eligibility, but this choice would have shifted the cost to patients and restricted access for low-income patients. It may also have reduced patients’ motivation to use the system, because patients reported that receiving a mobile phone was a strong incentive to use Seva.
Without ongoing funding for patients to use the system, use of Seva declined significantly and patients ultimately became unreachable to the research staff (note that patient surveys were administered over the phone, using the number associated with the patient’s mobile phone). This led us to abandon our attempts to collect the phone survey we planned to collect 6 months after each patient’s 12-month intervention period to gauge maintenance.
This analysis addresses cost from the perspective of an FQHC clinic administrator—a comprehensive economic analysis of the study will be reported separately. This analysis includes operating costs (eg, mobile phones, data plans, clinician time for monitoring Seva use, information technology [IT] staff time) and the costs of executing the implementation strategy (eg, site visits, coaching calls). Costs were tracked for 36 months across each project phase (preimplementation, implementation, and maintenance). Costs are broken down as follows: system operating costs; implementation costs per clinic; and overall cost per patient and per clinic (see). All costs are given in US dollars. Operating costs per patient were estimated at US $1185, which covers US $200 for a mobile phone; US $720 for a voice and data plan (US $60/month x 12 months); US $135 in clinical staff time for patient identification, recruitment, and training (based on observation and interviews with staff—this 2-stage process took an average of 1.5 hours per patient x US $90/hour); and US $130 in staff time for monitoring patients’ use of Seva (based on server logs, clinical and research staff spent 0.12 hours per patient per month monitoring patients; 0.12 hours per patient x 12 months x US $90/hour). We conducted interviews with IT staff members to derive estimates of system operating costs of US $8,000/per clinic over 36 months, which covers costs such as technical support for users, server hosting, and software updates to the system. Implementation costs were estimated at US $10,350 per clinic, which includes US $8,100 for coaching time and expenses associated with site visits (3 visits per clinic at US $500/day, including travel costs of US $1200 per visit) and US $2250 for monthly follow up via email and phone (36 hours x US $62.50/hour).
Total cost per clinic averaged approximately US $124,000 across the 3 clinics; cost per patient averaged approximately US $1,400. For comparison, the average cost for an episode of outpatient addiction treatment among 21 addiction treatment programs surveyed in 2008 (the year with the most recent available cost data) was US $2325 .
To summarize our findings related to the three research questions that were the focus of the study:
RQ1: How can Seva be implemented in primary care settings efficiently and effectively?
The study offers the following 4 lessons about implementing an mHealth system in primary care:
First, plan and have a budget for working extensively with clinic IT staff to integrate mHealth data into the EHR. This very challenging task is essential for integrating mHealth into primary care because it makes the mHealth data part of rather than separate from the data clinicians expect to see as they treat patients. Second, work with clinic staff to figure out how the mHealth system will fit into the clinic’s existing workflow. All 3 clinics in this study chose to appoint one clinician to monitor Seva data and alert fellow clinicians as needed about important changes in a patient’s recovery. Wider and deeper integration would result from each clinician routinely monitoring data from the mHealth system for his or her patients, just as he or she monitors data related to other chronic conditions such as diabetes. This level of involvement may be unrealistic, however, given the time pressures on primary care staff, and assigning routine monitoring to other clinical staff has been done effectively in other studies we have conducted [, ]. Third, ensure that the questions clinical staff members have as they operate the mHealth system can get rapid responses from the mHealth developers. Most questions clinicians had in this study were day-to-day operational issues (eg, how to enter a new patient into the system) that were ideally addressed in the moment they occurred by quickly calling the research team. Fourth, to assure sustained use, address cost. Two of the 3 clinics wanted to maintain using Seva based on patient and clinician feedback and, at one site, cost savings from reduced hospital admissions and ER visits. This last clinic used the cost savings to make a case to an insurer to pay for Seva, a process still under way at this writing.
RQ2: To what extent do patients and staff accept and use Seva?
Patient use was exceptionally high compared with continued use of most mHealth apps, although patient use declined steeply after funding for the phones and data plans ended. Clinician use was low compared with patient use because, as stated above, clinicians worried about being responsible for data from Seva and they had to view Seva data in a separate website rather than in the EHR. Of the two aspects of integration we examined—the integration of behavioral health into primary care, and the integration of mHealth into the treatment of addiction in primary care—the second was more successful than the first. Although the treatment of patients suffering from addiction and the use of Seva remained mainly the province of behavioral health providers in this study, the integration of mHealth into addiction treatment was successful if judged by the high levels of patient use.
RQ3: How does Seva affect clinical care for patients and staff?
This study showed the potential of patient peer support in encouraging treatment adherence. Patient peer support is unusual in primary care. It is also a type of care that does not add to, and may reduce, clinician burden. Clinicians who used Seva were generally enthusiastic about it, as demonstrated by 2 clinics wanting to continue using the system after grant funding ended, even though these intentions have thus far gone unfulfilled. The number of clinicians involved, though, is too small to warrant generalizations.
Comparisons With Prior Work
This study—which involved 3 unaffiliated primary care clinics enrolling nearly 300 patients in the same mHealth system—is the most comprehensive implementation research trial focused on the use of mHealth in primary care yet conducted in the US health care system. Implementation research on mHealth has been focused almost entirely on developing countries in Africa, where mHealth is usually used as a replacement for standard health care and operates independently of any health care system .
Prior studies focused on providing addiction treatment to primary care patients have also been rare; we found only 2 clinical trials for primary care SUD interventions [, ]. Neither of these trials used mHealth. Both addressed illicit drug use and prescription drug abuse rather than AUD and neither had an effect compared with the control group.
We observed significant reductions in drinking and substance use among patients using Seva. These are promising reductions compared with other outpatient AUD treatments. For example, Project MATCH showed that psychosocial treatments for alcoholism are not particularly effective , although more recent studies of cognitive-behavioral therapy for SUDs have demonstrated efficacy [ ]. A Cochrane review of the effectiveness of naltrexone, a commonly administered medication for AUD, found a decreased risk of heavy drinking by 17% compared with the placebo group and a decrease in the number of drinking days of about 4% [ ].
Granted, this study was an implementation research trial not specifically designed to retest the effectiveness of Seva. A rigorously designed randomized controlled trial would be required to definitively demonstrate the effectiveness of Seva within a substance-using primary care population.
The study has some limitations. (1) The primary outcome, risky drinking days, is a self-reported measure. (2) Selection bias could have affected which patients participated in the study. Clinicians were advised to enroll any eligible patients they thought might benefit from Seva, based on their clinical judgment. It was not possible for us to tell if this led clinicians to favor some patients over others (eg, those with more education). (3) We could not retrieve racial and ethnic data for SUD patients who were eligible to participate in the study but were not enrolled, and we did not collect data about education and socioeconomic status on Seva patient surveys, limiting our understanding of the representativeness of the patient sample. (4) We lost an opportunity to learn about sustainability from the patient’s perspective because we could not reach patients for the survey we planned to collect 6 months after each patient’s 12-month intervention period. (5) The study would be hard to replicate because it reports on a mobile phone app that, like almost all such apps, changes over time rather than remaining stable. For example, since the start of this study, the app has been rebuilt to update it to current accessibility and security standards, add features (My Motivation), improve features (the automatic tailoring of feedback in response to weekly surveys), and add content for opioid addiction.
The current findings suggest that mHealth faces several formidable challenges to widespread implementation. The first is integrating mHealth with EHRs (eg, Epic Systems), described above. Clinicians in this study had to view Seva data by going to a website outside the EHR, requiring already overburdened clinicians to make extra mouse-clicks. In addition, logistical and patient privacy concerns limited our ability to use EHR data for evaluation purposes. For example, we could not obtain participants’ attendance at primary care visits, which we planned to include in the health care utilization analyses. The poor interoperability and accessibility of EHRs are likely to generalize to other researchers and mHealth developers, making research challenging to conduct and limiting the usefulness of mHealth systems for clinicians.
Another key challenge relates to the difficulty of enrolling patients with SUDs. Clinics reported that it often took multiple follow-up calls to get patients into the clinic to get informed consent and conduct training, even though patients were often excited about getting a mobile phone. The effort required to enroll patients might have dampened enthusiasm for enrollment beyond the first 100 patients that the budget provided for.
FQHCs are required by law to provide behavioral health services. We do not know how Seva might function in a primary care clinic without designated behavioral health staff. Although Seva was somewhat integrated into behavioral health, it did not deeply penetrate physicians' treatment of patients, as reported previously about Site 1  and observed in Sites 2 and 3, perhaps reflecting the deep divides between primary care and behavioral health mentioned in the Introduction [ ].
Finally, our experience has revealed several fundamental questions about the role of mHealth in primary care: Is the value of patient peer support for behavioral health sufficient to make the costs of embedding an mHealth system such as Seva into a clinic’s operations worthwhile? Might it instead suffice for patients to use mHealth systems on their own, based upon the recommendation of primary care clinicians? If integrating mHealth into clinic systems is deemed worthwhile, who bears the costs? Certain costs, such as those for mobile phones and data plans, could be borne by patients, some of whom already pay for phones and data plans out of pocket. In this study, patient use of Seva and survey follow-up rates dropped significantly when research funding stopped paying for data plans. Yet, these costs account for only about half the total cost of the system, and lower-income patients often have pay-as-you-go data plans that may not work with the data requirements of an mHealth system such as Seva. Volunteer peer mentors could potentially act as monitors, as clinicians and researchers did in this project, thereby reducing the cost to clinics. Indeed, similar volunteer roles are essential to the Alcoholics Anonymous model (eg, “sponsors”).
mHealth has the potential to transform health care, and given the enormous cost of health care, we need to make effective use of every available resource. In contrast to the seemingly inexorable rise in health care costs over time, the cost of technology tends to decrease in accordance with Moore’s Law, which posits that computing capability roughly doubles every 2 years. Our experience illustrates that mHealth can engage patients suffering from addiction in ways that benefit patients without adding substantial burden on health care providers. Although challenges remain, thoughtful deployment of mHealth could improve the treatment of addiction in primary care and might also improve the treatment of other chronic conditions that have prominent behavioral components (eg, diabetes). In so doing, mHealth could transcend the physically local and professionally controlled systems that characterize the US health care system.
This research was funded by the National Institute on Drug Abuse grant number R01-DA034279. The funder had no role in any aspect of the development, conduct, analysis, or reporting of the study. The authors wish to thank the dedicated clinicians and staff members at the 3 clinics where the study was conducted, in particular, Chantelle Thomas from Access Community Health Center in Madison, WI; Virna Little, Victoria Ward, and Tom McCarry from the Institute for Family Health, Center for Counseling, at Walton, Bronx, NY; and Mary Jane Nealon from Partnership Health Center in Missoula, MT. The project could not have succeeded without their dedication and support. We also thank Judith Ganch, Joseph Glass, Andrew Isham, Adam Maus, Helene McDowell, Klaren Pe-Romashko, and Victoria Ward for their important contributions to the project.
Conflicts of Interest
Authors DG, FM, RJ, and AQ have a shareholder interest in CHESS Mobile Health, a public benefit corporation that disseminates technology to the specialty addiction treatment system. CHESS Mobile Health did not develop Seva. The relationship between the authors and CHESS Mobile Health is managed by the authors and the University of Wisconsin–Madison’s Conflict of Interest Committee. LM is affiliated with HealthSim, LLC—a small business that developed the Web-based Therapeutic Education System (TES). The relationship is extensively managed by LM and her academic institution. All other authors declare that they have no competing interests.
- Bauer AM, Rue T, Keppel GA, Cole AM, Baldwin L, Katon W. Use of mobile health (mHealth) tools by primary care patients in the WWAMI region Practice and Research Network (WPRN). J Am Board Fam Med 2014;27(6):780-788 [FREE Full text] [CrossRef] [Medline]
- Singh K, Drouin K, Newmark LP, Lee J, Faxvaag A, Rozenblum R, et al. Many mobile health apps target high-need, high-cost populations, but gaps remain. Health Aff (Millwood) 2016 Dec 01;35(12):2310-2318. [CrossRef] [Medline]
- Wicks P, Chiauzzi E. 'Trust but verify'—five approaches to ensure safe medical apps. BMC Med 2015;13:205 [FREE Full text] [CrossRef] [Medline]
- Ades J. Mpulsemobile. Healthcare apps: the good, the bad, the ugly URL: https://mpulsemobile.com/2016/07/good-bad-ugly-healthcare-apps/ [WebCite Cache]
- Torous J, Chan S, Luo J. Psychnews.psychiatryonline. 2016. Are your patients using 'digital supplements'? URL: http://psychnews.psychiatryonline.org/doi/full/10.1176%2Fappi.pn.2016.11b16 [WebCite Cache]
- Fiordelli M, Diviani N, Schulz PJ. Mapping mHealth research: a decade of evolution. J Med Internet Res 2013;15(5):e95 [FREE Full text] [CrossRef] [Medline]
- Donker T, Petrie K, Proudfoot J, Clarke J, Birch M, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res 2013;15(11):e247 [FREE Full text] [CrossRef] [Medline]
- Steinhubl SR, Muse ED, Topol EJ. Can mobile health technologies transform health care? J Am Med Assoc 2013 Dec 11;310(22):2395-2396. [CrossRef] [Medline]
- Quanbeck A, Chih MY, Isham A, Gustafson D. Mobile delivery of treatment for alcohol use disorders: a review of the literature. Alcohol Res 2014;36(1):111-122 [FREE Full text] [Medline]
- Riper H, Spek V, Boon B, Conijn B, Kramer J, Martin-Abello K, et al. Effectiveness of E-self-help interventions for curbing adult problem drinking: a meta-analysis. J Med Internet Res 2011;13(2):e42 [FREE Full text] [CrossRef] [Medline]
- Campbell W, Hester RK, Lenberg KL, Delaney HD. Overcoming Addictions, a web-based application, and SMART Recovery, an online and in-person mutual help group for Problem drinkers, part 2: six-month outcomes of a randomized controlled trial and qualitative feedback from participants. J Med Internet Res 2016 Oct 04;18(10):e262 [FREE Full text] [CrossRef] [Medline]
- Berrouiguet S, Baca-García E, Brandt S, Walter M, Courtet P. Fundamentals for future mobile-health (mHealth): a systematic review of mobile phone and web-Based text messaging in mental health. J Med Internet Res 2016 Jun 10;18(6):e135 [FREE Full text] [CrossRef] [Medline]
- World Health Organization. E-health technologies and substance abuse URL: http://www.who.int/substance_abuse/activities/ehealth/en/ [WebCite Cache]
- Klein S, Hostetter M. The Commonwealth Fund. 2014. In focus: integrating behavioral health and primary care URL: http://www.commonwealthfund.org/publications/newsletters/quality-matters/2014/august-september/in-focus [WebCite Cache]
- Awoyinka L, Gustafson D, Johnson R. Using technology to integrate behavioral health into primary care. In: Marsch L , Lord S , Dallery J , editors. Behavioral Healthcare and Technology: Using Science-Based Innovations to Transform Practice. New York: Oxford University Press; 2014.
- Dennis M, Scott CK. Managing addiction as a chronic condition. Addict Sci Clin Pract 2007 Dec;4(1):45-55 [FREE Full text] [Medline]
- Pating DR, Miller MM, Goplerud E, Martin J, Ziedonis DM. New systems of care for substance use disorders: treatment, finance, and technology under health care reform. Psychiatr Clin North Am 2012 Jun;35(2):327-356. [CrossRef] [Medline]
- Capoccia VA, Grazier KL, Toal C, Ford 2nd JH, Gustafson DH. Massachusetts's experience suggests coverage alone is insufficient to increase addiction disorders treatment. Health Aff (Millwood) 2012 May;31(5):1000-1008 [FREE Full text] [CrossRef] [Medline]
- DeRubeis RJ, Crits-Christoph P. Empirically supported individual and group psychological treatments for adult mental disorders. J Consult Clin Psychol 1998 Feb;66(1):37-52. [Medline]
- Molfenter T. Reducing appointment no-shows: going from theory to practice. Subst Use Misuse 2013 Jun;48(9):743-749 [FREE Full text] [CrossRef] [Medline]
- Gustafson DH, McTavish FM, Chih M, Atwood AK, Johnson RA, Boyle MG, et al. A smartphone application to support recovery from alcoholism: a randomized clinical trial. J Am Med Assoc Psychiatry 2014 May;71(5):566-572 [FREE Full text] [CrossRef] [Medline]
- Bickel WK, Marsch LA, Buchhalter AR, Badger GJ. Computerized behavior therapy for opioid-dependent outpatients: a randomized controlled trial. Exp Clin Psychopharmacol 2008 Apr;16(2):132-143 [FREE Full text] [CrossRef] [Medline]
- Glass JE, McKay JR, Gustafson DH, Kornfield R, Rathouz PJ, McTavish FM, et al. Treatment seeking as a mechanism of change in a randomized controlled trial of a mobile health intervention to support recovery from alcohol use disorders. J Subst Abuse Treat 2017 Jun;77:57-66. [CrossRef] [Medline]
- Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 2012 Mar;50(3):217-226 [FREE Full text] [CrossRef] [Medline]
- Quanbeck AR, Gustafson DH, Marsch LA, McTavish F, Brown RT, Mares M, et al. Integrating addiction treatment into primary care using mobile health technology: protocol for an implementation research study. Implement Sci 2014 May 29;9:65 [FREE Full text] [CrossRef] [Medline]
- Mares M, Gustafson DH, Glass JE, Quanbeck A, McDowell H, McTavish F, et al. Implementing an mHealth system for substance use disorders in primary care: a mixed methods study of clinicians' initial expectations and first year experiences. BMC Med Inform Decis Mak 2016 Sep 29;16(1):126 [FREE Full text] [CrossRef] [Medline]
- Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327. [Medline]
- Wen K, Gustafson DH, Hawkins RP, Brennan PF, Dinauer S, Johnson PR, et al. Developing and validating a model to predict the success of an IHCS implementation: the Readiness for Implementation Model. J Am Med Inform Assoc 2010;17(6):707-713 [FREE Full text] [CrossRef] [Medline]
- Maher L, Gustafson D, Evans A. Webarchive.nationalarchives.gov.uk. Leicester, England: British National Health Service Modernization Agency; 2004. Sustainability, model and guide URL: http://webarchive.nationalarchives.gov.uk/20160805122935/http://www.nhsiq.nhs.uk/media/2757778/nhs_sustainability_model_-_february_2010_1_.pdf [accessed 2017-08-30] [WebCite Cache]
- Legris P, Ingham J, Collerette P. Why do people use information technology? A critical review of the technology acceptance model. Inf Manage 2003 Jan;40(3):191-204. [CrossRef]
- Cranford JA, McCabe SE, Boyd CJ. A new measure of binge drinking: prevalence and correlates in a probability sample of undergraduates. Alcohol Clin Exp Res 2006 Nov;30(11):1896-1905 [FREE Full text] [CrossRef] [Medline]
- Hays RD, Bjorner JB, Revicki DA, Spritzer KL, Cella D. Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items. Qual Life Res 2009 Sep;18(7):873-880 [FREE Full text] [CrossRef] [Medline]
- Chamberlain P, Brown CH, Saldana L. Observational measure of implementation progress in community based settings: the Stages of Implementation Completion (SIC). Implement Sci 2011 Oct 06;6:116 [FREE Full text] [CrossRef] [Medline]
- Center for Behavioral Health Statistics and Quality. Substance Abuse and Mental Health Services Administration. Rockville, MD: Substance Abuse and Mental Health Services Administration; 2016. 2015 National Survey on Drug Use and Health: Detailed Tables URL: https://www.samhsa.gov/data/sites/default/files/NSDUH-DetTabs-2015/NSDUH-DetTabs-2015/NSDUH-DetTabs-2015.pdf [WebCite Cache]
- Cohen J. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Lawrence Erlbaum Associates; 1988.
- Baldwin JL, Singh H, Sittig DF, Giardina TD. Patient portals and health apps: pitfalls, promises, and what one might learn from the other. Healthc (Amst) 2016 Oct 3;5(3):81-85 [FREE Full text] [CrossRef] [Medline]
- French MT, Popovici I, Tapsell L. The economic costs of substance abuse treatment: updated estimates and cost bands for program assessment and reimbursement. J Subst Abuse Treat 2008 Dec;35(4):462-469 [FREE Full text] [CrossRef] [Medline]
- DuBenske LL, Gustafson DH, Shaw BR, Cleary JF. Web-based cancer communication and decision making systems: connecting patients, caregivers, and clinicians for improved health outcomes. Med Decis Making 2010;30(6):732-744 [FREE Full text] [CrossRef] [Medline]
- Dubenske LL, Chih M, Dinauer S, Gustafson DH, Cleary JF. Development and implementation of a clinician reporting system for advanced stage cancer: initial lessons learned. J Am Med Inform Assoc 2008;15(5):679-686 [FREE Full text] [CrossRef] [Medline]
- Aranda-Jan CB, Mohutsiwa-Dibe N, Loukanova S. Systematic review on what works, what does not work and why of implementation of mobile health (mHealth) projects in Africa. BMC Public Health 2014;14:188 [FREE Full text] [CrossRef] [Medline]
- Saitz R, Palfai TP, Cheng DM, Alford DP, Bernstein JA, Lloyd-Travaglini CA, et al. Screening and brief intervention for drug use in primary care: the ASPIRE randomized clinical trial. J Am Med Assoc 2014 Aug 06;312(5):502-513 [FREE Full text] [CrossRef] [Medline]
- Roy-Byrne P, Bumgardner K, Krupski A, Dunn C, Ries R, Donovan D, et al. Brief intervention for problem drug use in safety-net primary care settings: a randomized clinical trial. J Am Med Assoc 2014 Aug 06;312(5):492-501 [FREE Full text] [CrossRef] [Medline]
- Cutler RB, Fishbain DA. Are alcoholism treatments effective? The Project MATCH data. BMC Public Health 2005 Jul 14;5:75 [FREE Full text] [CrossRef] [Medline]
- McHugh RK, Hearon BA, Otto MW. Cognitive behavioral therapy for substance use disorders. Psychiatr Clin North Am 2010 Sep;33(3):511-525 [FREE Full text] [CrossRef] [Medline]
- Rösner S, Hackl-Herrwerth A, Leucht S, Vecchi S, Srisurapanont M, Soyka M. Opioid antagonists for alcohol dependence. Cochrane Database Syst Rev 2010 Dec 08(12):CD001867. [CrossRef] [Medline]
|EHR: electronic health record|
|ER: emergency room|
|FQHC: federally qualified health center|
|IT: information technology|
|mHealth: mobile health|
|QoL: quality of life|
|RE-AIM: Reach, Effectiveness, Adoption, Implementation, and Maintenance|
Edited by G Eysenbach; submitted 12.09.17; peer-reviewed by P Wicks, K Claborn; comments to author 05.10.17; revised version received 30.11.17; accepted 22.12.17; published 30.01.18Copyright
©Andrew Quanbeck, David H Gustafson, Lisa A Marsch, Ming-Yuan Chih, Rachel Kornfield, Fiona McTavish, Roberta Johnson, Randall T Brown, Marie-Louise Mares, Dhavan V Shah. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 30.01.2018.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.