We published the study protocol and design previously [23]. The study was approved by the Institutional Review Board for Human Protection at the University of Pittsburgh (#19060112) and registered on ClinicalTrials.gov (NCT03367936). We informed all participants of screening procedures prior to obtaining consent and obtained in-person informed consent for the intervention study.

Recruitment, conducted in the greater community surrounding Pittsburgh, PA, commenced in August 2018 and ended in March 2020. The intervention trial was completed in April 2021. Both online and in-person methods were used. Initially, interested individuals who were regular smartphone users completed surveys and a 5-day food diary in which they needed to record at least 700 calories of food intake per day to ensure that they could SM. Once deemed eligible, individuals had an in-person assessment to verify weight and height for BMI measures. Inclusion criteria were BMI between 27 and 43 kg/m², completion of a 5-day electronic food diary, and ability to engage in moderate PA. Exclusion criteria were needing supervision of diet or PA, pregnancy, serious mental illness (eg, schizophrenia), alcohol abuse or eating disorder, and current weight loss treatment [23].

After completing the intervention consent, research staff used a randomization software program to determine group assignment that was generated using minimization with stratification by gender (male or female) and race (Black or non-Black) with equal allocation to the 2 treatment conditions (please see the CONSORT [Consolidated Standards of Reporting Trials] diagram in Figure 1). All randomized participants were included in the final analyses. We used Wadden's conservative approach for imputing missing weight data assuming that there was a weight gain.

Key staff (BB, IL) who conducted the assessments were not blinded to the treatment assignment, whereas all other personnel and investigators, including the statisticians, (LEB, SMS, ZB, BP, JK, JC, MC, IWP, YW, and MBC) were blinded to assignment. Since participants were informed of both treatment conditions, they could not be blinded.

The planned total sample size for this randomized controlled trial was determined as 530 (265 per treatment group), allowing for a statistical power of 0.80 to detect effect sizes (standardized mean differences, d) as small as d=0.301 for the mean percent weight changes at 6 and 12 months between the SM-only and SM + FB groups when using linear mixed modeling with linear contrasts at a Bonferroni-adjusted significance level of P=.025 and for at most 20% attrition [23]. Due to the COVID-19 pandemic, recruitment was stopped in March 2020 with 502 randomized participants (251 per treatment arm). With this reduced sample size, slightly larger small-to-medium effect sizes of d=0.309 would still be detectable with 0.80 power at an adjusted significance level of P=.025, allowing for up to 20% attrition.

Continuous variables are summarized as mean and SD, and descriptive statistics for categorical variables are reported as counts with percentages. Appropriate group comparative analyses were performed on participant descriptors and outcome variables at baseline by randomized treatment assignment [37]. The effect of treatment assignment on percent weight change over 12 months was examined using linear mixed modeling following intention to treat. Models included a random intercept and unstructured variance-covariance matrix for the repeated assessments, supported by Akaike information and Bayesian information criteria. The base model included fixed effects for time (baseline vs 6 months and 12 months), group (SM+FB vs SM alone), and group by time interaction. Sensitivity analyses were performed among completers only and using inverse probability weighting for dropout by 12 months with age, race, and baseline as predictors.

The effect of the percentage of FB messages opened on percent weight change from baseline to 12 months for the SM+FB group was analyzed using univariate linear regression. Additionally, the associations of monthly percentages of days adherent to the calorie goal with treatment assignment and monthly percentages of FB messages opened were analyzed using separate linear mixed models with random intercept and slope for the total sample and for the SM+FB group, respectively. We conducted sensitivity analyses on the treatment effects on monthly percentages of days adherent to the calorie goal over 12 months in the total sample and on the associations between monthly percentages of FB messages opened and monthly percentages of days adherent to the calorie goal in the SM+FB group for the varying monthly percentage of days with sufficient dietary SM data (data not shown). Here, we report the results using days with ≥50% of the calorie goal recorded or ≥15 of 30 days with sufficient dietary SM data.

Model assessment (ie, residual analyses with influence diagnostics) was performed for each fitted model; sensitivity analyses were conducted for outlying or influential observations and to explore the effect of the COVID-19 pandemic on the efficacy of treatment assignment on percent weight change (data not shown). All analyses were performed using SAS version 9.4 (SAS Institute).

Most participants were White (414/502, 82.5%), female (399/502, 79.5%) and on average 45.0 (SD 14.4) years old. Sociodemographic, clinical, and psychosocial characteristics, as well as primary outcome measurements at baseline, were similar between the treatment groups [37].

At 12 months, the overall retention was 78.5% (394/502). Retention was similar by treatment condition, with SM+FB at 80.5% (202/251) and SM at 76.5% (192/251; X²=1.18; P=.28).

Figures 2 and 3 illustrate the results from linear mixed modeling for the effect of treatment assignment on weight loss and percentage of weight loss over 12 months. On average, both groups had statistically significant weight loss over 12 months (b_{6 months}=–2.94, 95% CI –3.70 to –2.19; b_{12 months}=–2.34, 95% CI –3.10 to –1.59; F= 61.46; P<.001). The trajectory of weight change over time was similar between groups (b_{group × 6 months}=0.09, 95% CI –0.97 to –1.16; b_{group × 12 months}=0.36, 95% CI –0.70 to –1.43; F=0.24; P=.79), and there were no significant overall treatment effects on weight change (b_group=–0.32; 95% CI –3.04 to 2.40; F=0.02; P=.90). There was a significant percent weight loss from baseline in both groups (SM+FB: –2.12%, 95% CI –3.04% to –1.21%, P<.001; SM: –2.39%, 95% CI –3.32% to –1.47%, P<.001), but no difference between the groups (–0.27%; 95% CI –1.57% to 1.03%; t=–0.41; P=.68). The percentages of participants who lost ≥5% weight from baseline to 12 months were similar between the SM+FB (66/251, 26.3%) and SM (73/251, 29.1%) arms (Χ²=0.49; P=.49). Based on sensitivity analyses using inverse probability weighting, there was no significant difference in percent weight change at 12 months in the SM+FB arm (mean –3.57%, SD 20.16) and the SM arm (mean –3.53%, SD 19.94; t=0.07; P=.95). Additional analyses among completers showed no significant treatment effects on percent weight change at 12 months (SM+FB: mean –3.54%, SD 7.16; SM mean: –3.58%, SD 7.06; t= –0.07; P=.95). For the total sample, mean weight change in kilograms at 12 months was mean –2.16 kg (SD 0.27). There was no significant difference in weight change in kilograms at 12 months in the SM+FB arm (mean –1.98 kg, SD 0.38) or the SM arm (mean –2.34, SD 0.38; 2-sample t test=0.67; P=.50).

Analyses without outliers had results similar to the full-sample results. There were no significant effects for the COVID-19 pandemic on the relationship of treatment with weight change over time.

In the SM+FB arm, the median percentage of FB messages opened from baseline to 12 months was 42.19% (461/1026; q1: 234/597, 39.20%; q3: 728/1095, 66.48%) and ranged from 1.28% (14/78) to 93.70% (1026/1095). Figure 4 displays the association of the percentage of FB messages opened and percent weight change at 12 months. A 1% increase in FB messages opened was associated with a 0.10 greater percent weight loss at 12 months (b=–0.10; 95% CI –0.13 to –0.07; t=–5.90; P<.001).

Figure 5 illustrates the change in monthly percentage of days adherent to the calorie goal by treatment group over 12 months using the criteria of recording ≥50% of the calorie goal 15 days per month. The monthly percentage of days adherent to the calorie goal declined nonlinearly in both groups. Overall, the percentage of days adherent to the calorie goal was greater in the SM+FB group than in the SM group (b_group=4.43; 95% CI 0.41-8.45; F=4.67; P=.03). The rate of decline in percentage of days adherent to the calorie goal was slower in the SM+FB arm than the SM arm over 12 months (b_{group × time-linear}=–1.98, 95% CI –3.03 to –0.93, F=13.71, P<.001; b_{group × time-quadratic}=0.14, 95% CI 0.06-0.22, F=11.04, P<.001).

Figure 6 illustrates the change in monthly percentages of FB messages opened and monthly percentages of days adherent to the calorie goal over 12 months and their association in the SM+FB group, respectively. In general, monthly percentages of FB messages opened (b_time-linear=–8.34, 95% CI –9.91 to –6.78, F=110.26, P<.001; b_{time-quadratic}=0.54, 95% CI 0.27-0.81, F=15.07, P<.001; b_time-cubic=–0.02, 95% CI –0.04 to –0.009, F=10.63, P=.001) and monthly percentages of days adherent to the calorie goal declined nonlinearly over 12 months (b_time-linear=–3.37, 95% CI –5.29 to –1.45, F=11.93, P=.001; b_{time-quadratic}=0.44, 95% CI 0.10-0.79, F=6.26, P=.01; b_time-cubic=–0.02, 95% CI –0.04 to 0.001, F=3.62, P=.06), with a greater percentage of FB messages opened being associated with higher adherence to the calorie goal as shown in Figure 7 (b_FB=0.12; 95% CI 0.07 to 0.17; F=22.19; P<.001). There was no significant interaction between the percentage of FB messages opened and the polynomial time effects.

We conducted a trial of a scalable, remotely delivered, behavioral weight loss intervention and tested the efficacy of a custom-developed, theoretically based, smartphone app that provided real-time FB remotely to reinforce diet, PA, and self-weighing behaviors. We observed a small but significant percent weight change from baseline to 12 months with no significant difference between the groups, suggesting the FB provided no additional benefit beyond SM. Our findings suggest that it is feasible to deliver a 12-month, remotely delivered intervention for weight loss to a large sample even with COVID-19 pandemic restrictions. The percent weight losses observed (equivalent to an absolute weight change of 2.0-2.5 kg) were less than what is usually reported in studies of in-person group counseling or online coaching [40-42] but not different from other small trials that tested digital interventions with limited human interaction [43]. A systematic review of eHealth interventions reported similar weight losses (1.4 to 2.7 kg) [44] at postintervention [45].

In this paper, we report initial app engagement findings, specifically the number of times participants opened the app to SM or read FB messages. Our study demonstrated that without personal interaction, engagement declined to the point that it interfered with the intervention delivery. Although the FB message system worked in theory, the declining adherence to SM created a lack of data for the algorithm to select a FB message. Sending an email message after 2 weeks without evident SM might have been too long to wait to prompt re-engagement, and a phone call might have been more effective; however, we were trying to increase the study scalability. The COVID-19 shutdown ended all in-person interactions including assessments, which might have affected engagement.

We measured engagement by the percentage of FB messages opened, which was reflective of dietary SM adherence. Overall, approximately 40% (461/1026, 42.19%; IQR 45.30) of the FB messages were opened with a very wide range of 1.28% (14/78) to 93.70% (14/78) in our study. Although this is less than ideal, engagement was equal (or greater) than the 41% to 60% rate of engagement that others have reported [46-48] in mobile health studies. A recent literature review emphasized the challenge of initial and sustained engagement in mobile health studies [49]. Considering the completely remote intervention with minimal to no in-person contact, our findings are encouraging but indicate a continued need to improve sustained engagement in SM [50].

There was greater adherence to the calorie goal in the SM+FB group compared to the SM group, and a greater number of FB messages opened was associated with a greater percent weight loss in the SM+FB group, suggesting that when messages were opened, the FB messages reinforced behavior changes related to food selection. These findings are similar to those in the SMART trial which showed that the personal data assistant + FB group had better adherence to the dietary goals and was the only 1 of the 3 study groups that had a significant within-group weight loss over 24-months. In SMART, adherence to SM strongly predicted weight loss at all time points [17,29,49].

The study algorithm and FB message library for SMARTER were significantly expanded from the earlier SMART trial; however, these improvements did not compensate for the absence of the 16 in-person group sessions that were part of the SMART trial, suggesting that some form of interpersonal interaction may be needed to augment mobile health interventions. Recent studies have reported similar findings [40,42]. For example, Thomas et al [40] demonstrated that providing a monthly in-person weigh-in to accompany 5-minute skills training videos achieved weight losses comparable to the gold standard of frequent in-person group sessions over the 18-month trial. Similarly, Amagai et al’s [49] literature review suggested that coaching to provide social support is an important strategy to improve engagement.

The comparison group in SMARTER received the same treatment components (ie, a one-to-one, in-person intervention session at baseline and digital tools for SM) and achieved very similar weight loss without receiving any prompts or reminders to SM. The intent of this comparative intervention was to determine the effectiveness of the approach that thousands of individuals are using by purchasing apps and tracking devices. These results suggest that some individuals who receive individual guidance at baseline with goals for diet and PA and encouragement to SM can achieve a clinically significant weight loss under their own direction.

Both groups used the Fitbit Charge 2 for tracking diet and PA. The Fitbit provided graphical FB on dietary intake (total calories consumed and “burned”) and a weekly summary of PA; thus, the SM-only group received some automatic FB if they were syncing their device to their phone. Although this FB could be reinforcing for some individuals, it lacked the personalized component that the SMARTER FB provided since the SMARTER messages were tailored to SM data entered at that time, were positive in tone, and often provided suggestions. However, due to the lower-than-expected engagement with SM, many individuals did not open (and therefore receive) enough messages.

A recent pilot study that used a 2 × 2 factorial design provides some insights into FB (counselor-crafted vs pre-scripted [51]) and group sessions (yes or no). Participants in the group sessions were more engaged in SM and lost more weight than did those not offered group sessions; however, the group that received pre-scripted, modular FB had significantly greater weight loss than did the group that received the counselor-crafted FB while there was no consistent difference in their treatment engagement. It is not known why the group that received briefer FB lost more weight; the longer FB sent weekly was possibly perceived as burdensome. The authors and other researchers suggested that the mechanisms underlying FB are poorly defined and that the amount, timing, frequency, and framing are just a few of the dimensions that need to be further studied [52-54].

Several recent weight loss intervention studies have examined an array of digital strategies to enhance adherence to SM while reducing components of the gold standard SBT; however, several had small samples, conducted brief interventions, and had small weight losses [26,43,55]. Despite these limitations, results showed promise for further study of approaches to enhance SM adherence (eg, counseling phone calls [26] or weekly emails with structured lessons [43]). The cumulative evidence makes it difficult to determine which intervention components can be most effective in producing clinically meaningful weight loss. Specifically, it is difficult to ascertain how much of the human interventionist component can be replaced to make weight loss treatments scalable to a broader reach and lower operational costs. This critical gap in the evidence needs to be addressed in future studies, so we can broaden our reach to the millions who need weight loss treatment, particularly those who do not have access to existing clinical and commercial weight loss programs.

There are several strengths to our study: a large sample size, a rigorous randomized design with a comparable control group, a retention rate higher than that reported by shorter and similar studies [56], use of validated measures, defined adherence metrics, and an objective measure of FB messages opened. The theory-based intervention was expanded from a previously tested and efficacious FB system. Additional strengths include using an extensive remote screening system and, born of necessity due to the COVID pandemic, pivoting to a remotely delivered, objective assessment protocol with minimal data loss.

Limitations include recruitment of fewer males and minorities than targeted, which limits generalizability. The retention was slightly lower than the targeted 80%. The metric of FB messages opened does not necessarily equate to the actual number of FB messages read.

The SMARTER trial delivered customized, real-time FB to participants based on SM data and capitalized on the use of available digital technology to provide personal weight management support without ongoing human counseling. This approach is scalable, as it reduces cost and participant burden while increasing reach to those without access to SBT or who do not wish to participate in an in-person program. We hypothesized that participants in the SM+FB group would have greater weight loss than those in the SM group at 12 months; however, weight loss outcomes were similar. Our results suggest that the addition of FB to SM did not make a significant between-group difference in weight loss outcomes; however, those who remained engaged and opened more FB messages had better calorie goal adherence and weight outcomes. Moreover, one-fourth and almost one-third of each group achieved clinically significant weight loss, suggesting that for a portion of participants, the SM component of the intervention was efficacious.

Considering the unrelenting prevalence of obesity, there is a critical need for scalable interventions that can reach those most at risk and with the least resources. The evidence supports standalone, scalable digital interventions, yet the crucial challenge is the development of digital tools that will keep users engaged long enough to see positive, sustainable outcomes. In advancing the digital aspects, we also need to identify the most efficacious personal interaction components that best augment and support sustained SM and lifestyle change. Obesity is a complex, multifactorial, chronic condition that requires ongoing support and an array of treatment options that will accommodate for the diverse needs of those seeking treatment.