Routine outcome monitoring (ROM) is central to evidence-based health care for a range of chronic conditions [1], including addictive behaviors [2,3]. ROM is central to understanding, evaluating, and improving service delivery [4-6]. A range of clinical benefits have been identified [7-9], particularly for those people identified as not on track early in the course of treatment [10,11]. Emerging evidence suggests that providing clients with tailored feedback may be central to demonstrated improvements in client outcomes [12].

To date, ROM and feedback have been implemented in a range of mental health [13] and addiction [14,15] treatment settings but not in mutual support groups. Mutual support groups offer an important source of fee-free, accessible support to people experiencing a range of addictive behaviors. Mutual support is particularly important for people experiencing addictive behaviors, given the often long-term and nonlinear process of recovery [16]. Mutual support groups may be attended before, during, after, or in lieu of engagement with formal treatment services, providing the potential for continuity across the recovery process. Although accumulating evidence highlights the importance and benefits of participating in mutual support [17-21], a major limitation is the lack of systematically collected data evaluating the outcomes. Unlike other clinically endorsed [2,3] models of mutual support for addictive behaviors (eg, 12-step approaches), SMART Recovery groups use a trained facilitator. This provides a unique opportunity to work with group facilitators to embed ROM and personal feedback as a standard component of the groups.

Integrating ROM and tailored feedback into routine service provision is not without challenges [22,23]. Common barriers include the time burden associated with completing, scoring, interpreting, or discussing outcome assessments [22,24], as well as skepticism regarding the perceived relevance of the outcomes assessed and feedback generated [25,26]. Additional limitations include the traditionally clinician-centric nature of ROM (see studies by Carlier and van Eden [7] and Thompson et al [13] for a discussion and studies by Lambert et al [8], Goodman et al [12], and Burgess et al [27] for common instruments) and accompanying feedback [28,29]. Improved acknowledgment of the client perspective during assessment [30] and greater client involvement in the feedback process [31] are both important clinical and research priorities.

The idea of using technology to track progress within health care settings is not new, but current approaches are limited [32]. Unlike other health information technology approaches (eg, web-based platforms), mobile health (mHealth [33]) apps offer a quick, easy, interactive, and engaging platform for tracking and accessing information about health and health-related behaviors [34]. A key benefit of mHealth apps is their ability to provide timely, individualized feedback [35]. Given the ubiquity of smartphone ownership [36,37], smartphone apps can engage individuals in real time and in their natural environment and by offering moment-to-moment support as needed [38]. Indeed, a recent systematic review of digital support services highlighted that their on-demand nature is a key benefit [39].

Although not specifically designed for the purposes of ROM and feedback, mHealth apps with the capability to track a variety of health behaviors, conditions, or outcomes [40-45], including alcohol consumption, substance use, and other addictive behaviors [39,44-52], have been developed. However, a key limitation is the ever-increasing gap between the availability of mHealth apps and their scientific validation [40,52-55]. Moreover, the level of end-user involvement throughout the development process is often unclear. This is important because inadequate consideration of the needs and preferences of the end user has been implicated in mHealth attrition [56-58]. Accordingly, we worked alongside end users to develop a purpose-built mHealth app for ROM and feedback in SMART Recovery Australia (SRAU) mutual support groups (SMART Track), which was then evaluated in this study.

The aim of this stage 1 nonrandomized, single-arm pilot study is to explore the feasibility, acceptability, and preliminary outcomes of a novel mHealth ROM and feedback app (SMART Track) in mutual support groups coordinated by SRAU.

Approval was granted by the University of Wollongong and Illawarra Shoalhaven Local Health District Health and Medical Human Research Ethics Committee (2018/099; HREC/18/WGONG/34). The study was registered with the Australian New Zealand Clinical Trials Registry (ACTRN12619000686101), and a protocol was published [59]. The reporting of this study follows the CONSORT (Consolidated Standards of Reporting Trials)-EHEALTH checklist [60].

Participants were recruited from the SMART Recovery groups registered with SRAU. Detailed accounts of SMART Recovery groups have been published [61]. Briefly, SMART Recovery groups originated in the United States and are now available across 23 countries. They offer support for people experiencing a range of addictive behaviors, including substance- and non–substance-related behaviors. SMART Recovery groups focus on self-empowerment and use evidence-based techniques (eg, cognitive behavioral therapy and motivational interviewing) [62]. These groups are held in a variety of community, inpatient, outpatient, residential rehabilitation, and clinical health settings. Online support groups are also available.

We invited 20 sites in New South Wales, Australia, to participate in this study, and 14 (70%) agreed (Figure 1). To enhance generalizability, the invited sites were selected to reflect a range of geographical locations and service providers. We sought to recruit 100 study participants. A sample of this size was selected to allow estimation of the recruitment rate and 95% CI with a margin of error of no more than 7%.

Participants were eligible if they were aged at least 18 years, were currently participating in SRAU groups (either face-to-face or online), had (or were willing to obtain) an email address, and comprehended English at a level sufficient to complete the study requirements. Participants were eligible irrespective of self-reported computer or smartphone literacy, and they did not have to own a smartphone. The study sites were provided with an Android (Samsung Galaxy Tab A) tablet for on-site participant use. No restrictions were placed on concomitant care or the frequency or duration of SMART Recovery group participation. The only exclusion criterion was inability or unwillingness to provide informed consent.

A group facilitator or member of the research team provided potential participants with standardized written and verbal information at the beginning of the SMART Recovery group session. Potential participants were asked to provide their preferred contact details, and they were contacted directly by a member of the research team. To avoid any potential coercion or desirability bias arising from the working relationship between facilitators and participants, the researcher (not the facilitator) was responsible for confirming participant interest and seeking informed consent. To boost accrual, during the final month of recruitment, a web-based expression-of-interest form (displayed prominently on the SRAU website) was introduced. Potential participants could contact the research team directly through email, phone, or the web-based expression-of-interest form. All participants provided verbal or written informed consent. The participants were reimbursed (Aus $30 [US $22.02] supermarket voucher) for their time, travel, and effort associated with each interview: baseline and 8-week assessments, as well as qualitative interviews (up to a total of Aus $90 [US $66.06]).

A total of 72 participants were enrolled in this study (Figure 1). The participant characteristics were comparable with prior accounts of SRAU group characteristics [99]; the average age of the participants was 44 years (SD 11), with more men (44/72, 61%) than women (28/72, 39%; Table 3). Most of the participants were born in Australia (59/72, 81%), and 6% (4/72) reported being of Aboriginal, Torres Strait Islander, or both Aboriginal and Torres Strait Islander descent. Employment was the main source of income for almost half of the participants (35/72, 48%).

At baseline, the participants reported attending an average of 6.63 (SD 5.44) SMART Recovery meetings in the preceding 12 weeks (range 0-24). Excessive alcohol consumption was the most common primary behavior of concern, endorsed by 68% (49/72) of participants over the preceding 4 weeks. Injecting drug use (ever) was reported by 25% (18/72), and 11% (8/72) of the sample reported a recent arrest (past 3 months).

There was a significant reduction between baseline and 8-week follow-up for the SDS (mean difference 1.93, SD 3.02; 95% CI 1.12 to 2.73) and K-10 scores (mean difference 3.96, SD 8.31; 95% CI 1.75 to 6.17), but there was no change in the SURE scores (mean difference 0.11, SD 7.97; 95% CI –2.02 to 2.24).

There was a strong relationship between the clinician-administered SURE [74] (at baseline) and the app-administered SURE (week 1; r=0.89; P<.001). For the quality-of-life item, there was a moderate relationship between baseline clinician administration as part of the EUROHIS-QOL 8-item index [76] and as part of the app-administered measures 2 weeks later (r=0.61; P=.005). For the Kessler Psychological Distress Scale-6 (K-6) [75], there was a moderate relationship between baseline clinician administration as part of the K-10 [97] and app administration as part of the K-6 2 weeks later (r=0.51; P=.02). The internal consistency (Cronbach α) for the SURE and K-6 was high across time points and when collected through clinician-interviewed telephone assessments or within the app (SURE: .86 to .94; K-6: .86 to .90).

This study was designed to assess the feasibility, acceptability, and preliminary outcomes of SMART Track for ROM and feedback in SRAU. The qualitative and quantitative findings support the feasibility, acceptability, and utility of SMART Track for ROM and feedback in SRAU. The findings also provide insight into avenues for enhancing sustained engagement. SMART Recovery participants were willing to use SMART Track, demonstrated sustained use across the 8-week follow-up interval, engaged most with the two main progress screens (Urges and Overall Progress), and experienced SMART Track as useful and consistent with SMART Recovery principles and strategies. Although it is difficult to attribute it directly to the use of SMART Track or SMART Recovery, the participants also showed clinical improvement over the 8-week follow-up, specifically reductions in the severity of dependence and psychological distress. Varied rates of ROM completion, minimal use of domain-specific feedback screens, and qualitative feedback suggest that the utility of SMART Track would be improved by making minor changes to app functionality and improving attention to implementation strategies.

To put engagement with SMART Track in perspective, it is helpful to consider the rates of engagement with other mHealth apps. One of the challenges with such comparisons is the considerable variation in the metric used to capture mHealth use (eg, mean number of log-ins, sessions, modules, activities completed [100]) and the degree to which these variables are reported [101,102]. Although several systematic reviews of digital recovery support services [39], digital measurement feedback systems [32], and addiction-related mHealth apps are available [54,103-105], the focus tends to be on content, experience, or outcomes, with little to no data examining participant engagement or use. However, the use of mHealth apps by people in recovery from substance use has been shown to vary from as high as 90% in the first few weeks to as low as 18% after 6 weeks [105]. For people with mental health conditions, engagement varied according to the target mental health condition. The number of nonusers (individuals who fail to download or use the intervention) has been calculated as 41% (range 25%-58%) for apps targeting depression and 8% (range 0%-16%) for apps targeting anxiety [100]. Reduced engagement over time was common [100]. Compared with these data, participant engagement with, and sustained use of, SMART Track is at least comparable, if not higher than the available evidence.

Another useful point of comparison comes from use trends within the global app marketplace. In 2019, data derived from more than 12,000 apps demonstrated that a quarter of the users will abandon an app after one-time use [106]. In comparison, of the 94% (68/72) of the participants in this study who created a SMART Track account, more than one occasion of use was documented for 79% (57/72) of the participants. Benchmarks pertaining specifically to lifestyle-related apps (which include fitness-, health-, and travel-related apps) indicate that the average 2-month retention rate is 36% [107]. In comparison, more than half of the participants in this study used SMART Track for between 5 and 8 weeks (25/72, 35%) or longer (17/72, 23%), and week 8 ROM data were provided by 31% (22/72) of the study participants.

Uncertainty exists around what is considered a good level of mHealth use. Some addiction-related apps (eg, In My First Year of Recovery and A-CHESS mHealth interventions) have documented high levels of sustained participant engagement (78% program completion and 4-month retention, respectively [105]). In contrast, SMART Track use was intermittent, with a proportion of the participants using the app weekly, whereas others disengaged and re-engaged every few weeks. Emerging evidence suggests that engagement with digital recovery support tools may be influenced by recovery duration [39]. Accordingly, engagement with SMART Track is likely to vary widely, given that SRAU caters to people across the spectrum of recovery experiences. It is also possible that users may perceive apps as a short-term commitment [108]. Therefore, compared with the use of other digital platforms (eg, the web), app use may be shorter and more sporadic [109]. Moreover, training in SMART Track was not extensive. The researchers met with the facilitators at each site to orient them to the features of the app. The participants received an email with brief instructions and an in-app onboarding process, although analytics showed that this was used by less than 45.2% (105/232) of users who downloaded SMART Track during the study period. The exemplary use rates documented by In My First Year of Recovery and A-CHESS may have been due to the more active role of counselors in encouraging mHealth use [105] and point to the importance of improved integration of SMART Track into SMART Recovery groups. Identifying participant and contextual factors that influence engagement represents an important challenge for future research.

A few studies have examined ROM implementation report data on engagement and attrition rates [8,25,28,110,111]. Compared with recent data from the Netherlands, however, the proportion of participants using SMART Track to input ROM data is largely comparable, and in some cases, greater than traditional clinician-completed methods [112]. A 50% response rate has been recently suggested as an acceptable benchmark for ROM data and is likely sufficient to protect against bias and yield valid information about patient progress (see study by de Beurs et al [112] for a discussion). In this study, this benchmark was achieved each week across the first month of data collection, with 83% (60/72), 63% (45/72), 60% (43/72), and 53% (38/72) of the study participants completing at least one of the ROM instruments across the first 4 weeks of app use. ROM completion continued to decline during the second month of data collection, with 31% (22/72) of the sample providing ROM data during week 8. Ongoing efforts are needed to improve ROM completion and understand the participant characteristics associated with drop-off (eg, through attrition analyses). For example, given the voluntary, open-enrolling format of SMART Recovery groups and individual variation in group attendance (0-24 in this study), it would be interesting to examine whether app use varies according to group engagement.

The R&D costs of SMART Track were estimated at Aus $203,523. However, the developer offered considerable in-kind support, and the overall true cost was Aus $281,058. It is important to reiterate that these are essentially sunk costs. If SMART Track had been rolled out routinely in SMART Recovery groups, the cost of implementation would have consisted of costs related to app hosting, infrastructure, maintenance, and training. The dynamic nature of the industry makes it difficult to estimate these costs and points to the need for ongoing data collection to understand the long-term real-world feasibility of SMART Track.

To maximize participant engagement in ROM, a clear rationale for why the data are being collected and what they will be used for is essential [25]. The measures must be experienced as relevant [110] and the process deemed worthwhile [113]. Analytics revealed that few people accessed the detailed, personalized feedback provided within each of the domains listed on the summary page. Qualitative feedback highlighted a mismatch between the effort expended and satisfaction with the feedback provided. Given that the participants were asking for feedback that was already provided in the app (but not accessed), we expect that modifications to improve the visibility of these sections of the app will further enhance ROM completion. Importantly, the regular and frequent use of app self-monitoring features has been linked to a longer period of use and reduced likelihood of abandoning apps [114]. Additional features that have been linked to mHealth engagement include an esthetically pleasing interface, ease of use, degree of personalization, reinforcement (eg, rewards and reminders), communication (with peers or professionals), message presentation (including language, tone, and design), and credibility (encompassing trustworthiness and confidentiality [115]). Although these features were considered throughout the design of SMART Track (and positively evaluated as part of the quality assessment), an opportunity exists to further enhance the user experience (eg, through improved personalization, greater use of rewards, and the addition of information-sharing or communication capabilities).

Organizational resources are essential for ensuring the sustained implementation of digital interventions to reduce substance use [105], for example, ensuring leadership support; providing adequate training and resources to both staff and service users; leveraging the expertise of service users to contribute to training; and having a process in place to monitor, evaluate, provide feedback, and respond to uptake rates [116]. Training is a particularly important consideration and provides a forum to (1) build clinician knowledge and confidence in app use and features, (2) practice introducing it to participants, and (3) identify and overcome any concerns or perceived barriers to implementation of mHealth to support routine care [117]. The next steps for SMART Track include improved facilitator training and support and leveraging of participant and facilitator champions (ie, individuals who actively support the use of SMART Track).

The development of SMART Track was grounded in theory [40,57] and user-centered design [57]. Consistent with recommendations for enhancing measurement-based care, SMART Track includes both standardized and idiographic outcome assessment and harnesses technology to overcome traditional barriers to ROM (eg, scoring and providing tailored feedback [110]). Quality assessment was conducted using a psychometrically valid tool [88], and SMART Track surpassed the minimum acceptable quality benchmark (≥3 [88]) on each of the domains assessed by the uMARS, with an overall quality rating of good. This is superior to several published accounts [50]. The observed quality of SMART Track likely reflects the user-centric approach to development. However, because the ratings were collected as part of a telephone interview with the researcher, the contribution of response bias cannot be ruled out.

Evidence from mental health settings suggests that mHealth apps that have a clear purpose and simple user interface and are easy and time efficient to navigate and demonstrate were more likely to be used as part of routine practice [117]. Quantitative and qualitative data indicate that SMART Track possesses these attributes. Pending minor upgrades and improved training and support, this finding further bolsters our confidence in the routine uptake of SMART Track within SMART Recovery groups.

This study includes several limitations. In its current format, SMART Track is not suitable for people who cannot adequately read and comprehend English. The reliance on written and visual information may also compromise the suitability of SMART Track for people with vision impairment. The use of cloud functions for collecting and storing data means that SMART Track needs a reliable internet connection to function. The study’s approach to assessing engagement is consistent with recommendations for a multidimensional approach using mobile app data analytics (index of microengagement), indices of behavior change (macro level of engagement), and participant subjective experience [102]. However, differences in how various studies define engagement and use make it hard to position SMART Track within the context of existing studies. Recent guidelines for the measurement and reporting of engagement data in digital interventions may be beneficial in the future [118].

The current findings are derived from a small sample of participants who attended a limited sample of SRAU groups. We did not collect data on the number of participants attending SRAU groups across the study period; therefore, although participant characteristics are comparable with published accounts [99], generalizability is unclear. Furthermore, the short-term nature of the study makes it challenging to position the findings within the often long-term, nonlinear experience of recovery [16]. Finally, because this is a stage 1 feasibility study, our finding that participants reported reduced severity of dependence and psychological distress from baseline to 8 weeks needs to be interpreted cautiously.

The qualitative and quantitative findings support the feasibility and acceptability of SMART Track and lend insight into avenues for enhancing sustained engagement. Low rates of engagement and high rates of attrition are known challenges for services working with participants who experience substance use and mental health–related difficulties [119]. Sustained engagement with mHealth apps is notoriously difficult to achieve. In light of these challenges, our findings are promising. SMART Track offers a potential solution for ROM and feedback, particularly for people with substance use disorders who attend mutual support groups. Future research should focus on identifying relevant demographic, clinical, and contextual factors that may influence the engagement, attrition, and perceived utility of this promising mHealth app.