The NoHoW trial was an 18-month, 3-center, 2×2 factorial, single-blind RCT that evaluated the efficacy of the TK in WLM. In total, 1627 participants were recruited from 3 countries—the United Kingdom, Denmark, and Portugal and randomly assigned to 4 arms—(1) control arm (400/1627, 24.58%), (2) motivation and self-regulation arm (403/1627, 24.76%), (3) emotion regulation arm (416/1627, 25.56%), and (4) combined arm (408/1627, 25.07%). The participants were required to be aged ≥18 years, have a verified ≥5% weight loss in the last 12 months with current weight at least 5% below their highest weight, and have had a BMI of ≥25 kg/m² before weight loss. Recruitment was conducted through several channels to reach eligible individuals, for example through commercial and municipal weight loss services, registered dieticians and nutritionists, leisure centers, and local and national media coverage and advertisements. The individuals were directed to country-specific recruitment websites and completed a web-based eligibility screener. Eligible individuals were contacted for a telephonic screening interview and provided with study information. Eligible participants were invited to a clinical investigation day where informed consent was obtained before randomization. A detailed description of the study is presented in the paper by Scott et al [18].

All the participants received commercial wireless body weight scales (Fitbit Aria [Fitbit LLC]), activity trackers (Fitbit Charge 2 [Fitbit LLC]), and access to the TK website with content tailored to their respective arm. The users also had access to the Fitbit smartphone app (Fitbit LLC) and although they were asked not to use it, access could not be prevented. The TK contained a dashboard and graphs for summarizing the measurements and visualizing long-term progress and enabled simple self-assessments of mood and satisfaction with diet, sleep, activity, and weight as star ratings (1 to 5 stars), and entering free text personal notes into a diary. These parts of the TK were also available to the control arm along with the self-tracking devices. The participants in the 3 intervention arms received intervention content in the form of weekly sessions displayed in the TK as an interactive map. The motivation and self-regulation arm had 17 sessions with an estimated minimum time duration of 51.2 minutes (ranging from 20 seconds to 6 minutes and 16 seconds per session), the emotion regulation arm had 17 sessions with an estimated minimum time duration of 83.3 minutes (ranging from 40 seconds to 21 minutes and 52 seconds per session), and the combined arm had 34 sessions combining the contents of the other 2 intervention arms and having an estimated minimum time duration of 118.3 minutes (ranging from 20 seconds to 12 minutes and 32 seconds per session). Intervention participants were encouraged to complete the intervention sessions during the first 18 weeks of the trial. This was achieved by sending participants weekly emails during this time to introduce the weekly themes and remind them to visit the TK. The control arm also received weekly emails for the first 18 weeks, but they only contained links to generic weight management content. The TK was designed to provide automatic individualized feedback to the motivation and self-regulation and combined arms based on weight, activity, sleep, and use data. However, owing to an error, the emotion regulation arm received the messages instead of the motivation and self-regulation arm. The feedback was displayed in the TK as short statements (eg, “your weight management appears better when you are more active”). The TK also provided extra support for weight regain situations (weight alert), where an extra module was launched if the user was >3% above their target weight. The TK implementation was fixed for the duration of the trial, that is, no new features or content were added. Only technical errors were rectified when they were reported by the trial staff. A detailed description of the TK is presented by Marques et al [17].

Quantitative user experience data were collected through electronic questionnaires at 1 (user experience questionnaire at month 1 [UX1]), 3 (user experience questionnaire at month 3 [UX3]), 6 (user experience questionnaire at month 6 [UX6]), and 12 (user experience questionnaire at month 12 [UX12]) months. Table 1 presents the measures in each questionnaire and Multimedia Appendices 1 and 2 contain the detailed questionnaires. The TAMM model was used to create a questionnaire section aimed at measuring the acceptance of the intervention. The TAMM-based items were slightly different in the UX1 from those in the other questionnaires to capture first impression experiences. The System Usability Scale (SUS) was included as a validated measure of the system usability [19]. In addition, items measuring the overall impressions of the TK and its individual features were included. The questionnaires had voluntary open-ended items for providing free-form feedback. Furthermore, the eHealth Literacy Scale (eHEALS) was included in the baseline questionnaire of the RCT to describe the participants’ baseline capacity to engage with eHealth interventions [20].

Focus group discussions were conducted after 6 months. Focus groups were organized to provide a deeper understanding of the user experiences of the TK. A participant was eligible if they had been using the TK for ≥6 months. In each country, 1 focus group per arm was conducted, leading to a total of 12 groups. Recruitment was done by listing all the participants who had the opportunity to use the TK for ≥6 months at the time of the focus group discussions and recruiting from this list until 8 participants agreed to participate. Participants were recruited via an email that included an invitation letter and an information sheet. Groups were moderated by one or more researchers who were experienced in conducting focus groups and qualitative research and who had not taken part in the TK design or development or participated in delivering the intervention (LB [Denmark], MNS [Portugal], and AD and LR [United Kingdom]). The conversations were guided by a semistructured discussion guide, which highlighted the following 5 main topics of conversation: ways of use, user experience, perceived support in WLM, impact on weight management, and how to improve the TK (Multimedia Appendix 3). Unplanned topics of conversation were also explored based on the issues raised by the participants. Moderators aimed to ensure that no one participant dominated the conversation, and every participant was given the opportunity to contribute to discussions. Ethical approval was granted by the local institutional ethics committees at the Universities of Leeds (17-0082; February 27, 2017), Lisbon (17/2016; February 20, 2017) and the Capital Region of Denmark (H-16030495; March 8, 2017).

The TK use was captured in log files, which were used to calculate the total number of visits to the TK and the total duration of use during the 12 months of the study. The percentages of weekly users and retained users were plotted for the control and intervention arms. Retention was determined based on rolling retention, which means that a participant was considered as a retained user if they used the TK during or after a specific week. Intervention completion was calculated for the intervention arm participants based on the duration of time they spent in the sessions assigned to them versus the estimated duration of the sessions. For most sessions, if a user visited a session at least once and remained engaged for at least 33% of the estimated duration, the session was considered as completed. For sessions containing video and audio content, the threshold was increased, that is, the user was required to spend 50%-80% of the estimated duration in the session. The completion rates were calculated as the percentage of completed sessions compared with the total number of sessions in the arm. The use metrics were reported as medians and IQRs owing to the skewed distributions and nonparametric methods were used for comparisons and correlations (ie, Kruskal-Wallis test, Mann-Whitney U test, and Spearman correlation). Spearman correlations between use metrics and weight changes from baseline to 12 months were calculated for each intervention group.

The statistics of responses to the questionnaires were summarized. Quantitative questionnaire sections were descriptively summarized using means and SDs. One-way analysis of variance and Tukey post hoc test were used to determine if there were differences between the intervention arms, and Spearman correlations between the number of visits and the user experience items were calculated. The TAMM-based items were summarized as mean scores over the 4 TAMM dimensions and a total score over all the items. The eHEALS score was calculated as the sum of individual items. The SUS was scored according its guidelines [19]. Principal component analysis was conducted on the 6 items that measured the appeal of the TK features to investigate whether there were underlying or latent variables that accounted for the items. Differences between the centers were investigated by comparing use metrics, eHEALS score, and 3-month user experiences in the control and intervention arms using a general linear model. For this analysis, a logarithmic transformation was applied to the number of visits and total duration of use to normalize their distribution. Quantitative data were analyzed using MATLAB R2017a (Mathworks) and IBM SPSS Statistics version 26 (IBM Corporation). An α level of .05 was used as the threshold for statistical significance.

Focus groups were transcribed and analyzed thematically via an iterative, nonlinear, and nonprescriptive process [21,22]. This involved initial familiarization with the transcripts, reflections on similarities and differences between cases, and systematic coding of the data. After preliminary discussions across the countries on codes and themes, the initial coding framework was developed in Denmark and further iterated collaboratively to integrate the findings from Portugal and the United Kingdom. In each country, experienced researchers were involved in the process and coding was done from original transcripts by native speakers (LB and LL [Denmark], MNS [Portugal], and AD and LR [United Kingdom]). At each stage, decisions on coding and analysis were discussed and revised by all coders and where necessary, the original data sources were revisited to ensure that the decisions were grounded in the data (refer the focus group code book in Multimedia Appendix 4). The analysis was particularly focused on the research questions (ie, the strengths, weaknesses, and improvement needs), while considering unpredicted but relevant themes. Similarly, responses to the open-ended questionnaire items were also thematically analyzed separately but by using the code book of the focus group analysis as the starting point. The analyses were conducted using NVivo (version 12; QSR International).

The average eHealth literacy score measured with the eHEALS questionnaire at baseline was 30.8 (SD 5.12; range 10-40) out of a maximum of 40, with higher values signifying higher levels of eHealth literacy. There were no significant differences between the arms and no significant correlations between digital literacy levels and use (data not shown); thus, eHealth literacy is not expected to have affected the results.

The average SUS scores for measuring the usability of the TK decreased over time in the control arm, from a mean of 70.4 (SD 21.3) in the UX3 to 67.6 (SD 19.9) in the UX12; in the motivation and self-regulation arm, from 68.4 (SD 20.9) in the UX3 to 62.1 (SD 20.2) in the UX12; in the emotion regulation arm, from 73.0 (SD 19.2] in the UX3 to 67.1 (SD 19.8) in the UX12; and in the combined arm, from 69.9 (SD 20.2) in the UX3 to 66.2 (SD 19.5) the UX12. The lowest scores were obtained in the motivation and self-regulation arm and they differed significantly from those in the emotion regulation arm (P=.01) in the UX3 and from all other arms in the UX12 (control arm: P=.003; emotion regulation arm: P=.007; combined arm: P=.03).

In the control arm, the SUS scores correlated significantly with the number of visits to the TK in all the questionnaires (ρ=0.426 in the UX3, ρ=0.362 in the UX6, and ρ=0.434 in the UX12; all P<.001). In the motivation and self-regulation arm, there was a small but significant correlation only in the UX3 (ρ=0.142; P=.03). In the emotion regulation arm, there were significant correlations in all the questionnaires (ρ=0.186, P=.003 in the UX3; ρ=0.164, P=.02 in the UX6; and ρ=0.218, P=.001 in the UX12). In the combined arm, there were no significant correlations between SUS scores and number of visits.

In most questionnaires, the intervention arm participants gave the TK significantly higher ratings in overall score, likelihood of recommending, and intention to continue to use the TK compared with the control arm participants; ratings declined in all the arms during the study. The ratings mostly did not differ among the 3 intervention arms; only in the UX6, the combined arm participants gave a higher overall score than the other intervention arms and had a higher intention to continue than the emotion regulation arm (P=.01). There were small but significant correlations between the ratings and the number of visits to the TK in the control and emotion regulation arms (Table 4).

Table 5 presents the mean scores for all the items and the 4 TAMM dimensions. For the total score, the results are presented for all the arms separately, whereas in other results, the intervention arms are combined, as the total score mostly did not differ among the intervention arms. In the UX1, the post hoc tests showed that the emotion regulation arm participants gave a higher total score to the TK than the combined arm participants (P=.006) and the control arm participants (P=.02).

The participants of the intervention arms gave a higher rating for value in all the questionnaires (P<.001 in the UX1, UX6, and UX12 and P=.01 in the UX3). The ratings for ease of adoption, ease of use, and trust were mostly similar in the control and intervention arms in all the questionnaires; only in the UX6, the intervention arms had a higher score for ease of use (P=.02) and in the UX3, for trust (P=.04). All the ratings decreased during the trial, and only the ratings for trust remained above 4 in the intervention arms. In both the control and intervention arms, the highest scores were given to trust and the lowest scores to value.

When analyzing only the participants who responded to all the user experience questionnaires, the questionnaire averages and the decline during the trial were nearly identical, that is, the decline was not because of the differences in the respondent populations (data not shown).

Table 6 presents correlations between the total number of visits and TAMM total score in all the arms. The control arm showed the highest correlations between the TAMM score and TK use in all the questionnaires.

Principal component analysis was conducted on the 6 appeal items rated on each feature. In all the questionnaires, only 1 factor was found, accounting for 65.2% to 68.9% of the variance in the items in the UX12 and UX3, respectively. The component loadings were similar for all the items, and therefore, the mean of the appeal items was calculated to represent the perception of the features (Table 7).

All the feature ratings were between 3 and 4 on a 5-point scale. At 3 months, the intervention arm participants gave the highest ratings to weekly emails, graphs, and goal setting; at 6 months to graphs, weekly emails, and interactive exercises; and at 12 months, they gave the highest ratings to graphs, interactive exercises, and goal setting. The control arm participants gave the highest ratings to weekly emails and graphs at 3 and 6 months and to graphs and dashboard at 12 months.

Age and gender were added to the general linear models as covariates, as these background variables differed between the centers. There were no significant differences between the use metrics among the centers (Table 8).

There were differences in the eHEALS scores. The Portuguese intervention participants gave significantly lower scores than the participants in other countries. In addition, the SUS score differed among the centers in the intervention arms, and all other investigated user experience ratings differed among the centers in both the control and intervention arms. In all the user experience ratings, the highest scores were given by Portuguese participants and lowest by the Danish participants.

Analysis of the focus groups and open-ended questionnaire data revealed the following four main themes: (1) engagement with features, (2) use decline, (3) external factors affecting user experience, and (4) suggestions for improvements. These reflect the opportunities and challenges regarding the user acceptance of a DBCI. In the following, each theme and its subthemes are described. In the qualitative data, the comments and overall perception of the TK were very similar in the 3 countries and across intervention arms; thus, the results are not divided into arms or countries. Comments from the focus groups are marked with sex, study arm, and country; he survey responses are marked with the code “UX” and the month of the survey is added to the end of the identifier.

During the focus groups, participants identified and discussed the features and aspects they found most helpful regarding their WLM and these were also explained in the questionnaire comments.

Support for self-monitoring was perceived positively in both the focus groups and questionnaire comments. For participants, monitoring progress in their weight, sleep, and activity through graphs, dashboards, and weight alerts was identified as very important for their continued WLM, as it gave them a sense of control over their progress. A participant from Portugal commented the following:

Although participants often highlighted the usefulness of the Fitbit app in monitoring weight, they also mentioned that the visualization of their development and progress provided by the TK through graphs and dashboards was important. In contrast, some participants did not find the monitoring as helpful and were reluctant to weigh themselves often, fearing that it would be stressful and potentially detrimental to their WLM:

However, in addition to monitoring, the ability of the TK to induce self-awareness and help the participants reflect on their choices was also perceived as important. The idea that everyone must think and know what they do, regarding eating and activity, was fundamental to orient behavior in a different way:

Participants in the focus groups perceived the TK content and intervention modules as especially useful for their WLM if they prompted them to reflect on the association between behavior and weight. The participants also acknowledged the importance of emotion regulation on WLM, regardless of whether they had it in their intervention, as illustrated by the following comments:

Participants further indicated that the TK allowed them to be more conscious of the behavioral patterns linked to sedentary behaviors and bad eating habits and to know themselves better.

Thus, the various features of the TK enabled participants to learn about their own behavior and better understand the antecedents and consequences of their food-related choices, thus helping them self-regulate their behavior and respond to emotional cues.

The use of the TK declined over time. The participants often linked their declining use to various design aspects that they felt hindered their use of the TK. Most often, participants expressed frustration with the login procedure:

The focus group participants reported varying engagement with the content and sessions of the TK. Although the monitoring tools were important for progress, many of the content-heavy features failed to engage participants in the same way. A prominent issue was the time required to complete the modules as the duration often exceeded the participant’s expectations, meaning that the participants had to unwillingly take time out of their day to participate:

The issue was exacerbated by problems with responsiveness and readability when accessing the TK via a mobile phone, which forced the participants to access it via their home or work computers. This was not always convenient within a busy lifestyle full of interruptions and complications, which was an additional reason for declining use:

Although some participants felt that the information provided in the TK was interesting, motivating, and relevant, some felt there was nothing new in the information or that its presentation was not pleasing:

In the questionnaire responses, participants across the intervention arms reported that the weekly modules were tasks that they enjoyed or needed to complete. Weekly reminders were also well received and considered very important in inviting users back to the TK. Without the reminders and the tasks to complete, motivation to use the TK declined:

Interestingly, the control arm participants used the TK in a similar way as the intervention arms participants, despite having much less content:

Apart from technical issues, there were several external factors that influenced participants’ perceptions of the TK. The activity information collected via the Fitbit smartphone app was available to view on the TK, but the overall design was considered lacking, especially when compared with the Fitbit app, which was also available to all the participants alongside the activity tracker and weight scales. Furthermore, many participants were aware of other commercial weight management apps and used them as a benchmark to measure the TK. This comparison was often unfavorable, and the participants wanted future iterations of the TK to incorporate the best parts of the commercial apps or at least be compatible with them because of their convenience, accessibility, user-friendliness, and design:

Finally, study procedures such as the user experience surveys, the food recall questionnaire (INTAKE 24, which was a measurement of the NoHoW trial rather than part of the TK), and visits to the local study sites for weighing and measuring were often confused with the TK and some participants were unsure about what was meant when asked about it:

When asked for feedback on the TK

The focus group participants had many ideas to improve the TK. The most prevalent suggestion was the need to create an app, not only to make the service more competitive but also to circumvent issues with login and access and make it easier for the participants to prioritize and plan when to use it. The possibility of personalizing the TK according to one’s own use and interests was also highlighted. Similarly, the participants wanted the content to be more tailored to their specific needs, for example, information about the expected time allocation required for the activities and designing a tool that provides immediate help in tempting food situations:

The lack of interaction with other participants was also regarded as a weakness of the TK. Creating a social platform to allow the participants to interact with others, discuss their progress, and offer support when needed was considered important. The participants spoke about previous experiences in which social support, knowledge exchange, shared activities, and friendly competition had a positive impact on their weight loss:

Furthermore, many participants asked for more feedback and notifications from the system. Although the TK provided individualized feedback in 2 arms, it was not widely discussed in the focus groups. One reason for this might be insufficient data to provide personalized feedback owing to the participants’ low level of engagement. The prompting emails sent from the TK seemed to be the main trigger for use and once the reminder messages stopped, use declined. However, the users wished they had received periodic personalized feedback emails with advice on how to reach the individual goals and believed that they would have helped them sustain their engagement:

In addition, some participants expressed confusion as the content on the website was not updated during the trial period. Their use of other commercial services created the expectation of regular updates:

In addition, there was some confusion among the participants regarding what data would be accessible to researchers. Although this issue was not explicitly discussed, the participants often thought that the researchers had access to all their data, which may have impacted the participants’ engagement with and use of the TK, as not receiving feedback from researchers was seen as unfair: