Published on in Vol 22, No 12 (2020): December

Preprints (earlier versions) of this paper are available at, first published .
Engagement With a Behavior Change App for Alcohol Reduction: Data Visualization for Longitudinal Observational Study

Engagement With a Behavior Change App for Alcohol Reduction: Data Visualization for Longitudinal Observational Study

Engagement With a Behavior Change App for Alcohol Reduction: Data Visualization for Longitudinal Observational Study

Original Paper

1Department of Medical Statistics, London School of Hygiene and Tropical Medicine, London, United Kingdom

2Research Department of Behavioural Science and Health, University College London, London, United Kingdom

3Department of Statistics, University of California Irvine, Irvine, CA, United States

4Health Data Research UK, London, United Kingdom

5Institute of Health Informatics, University College London, London, United Kingdom

Corresponding Author:

Lauren Bell, MSc

Department of Medical Statistics

London School of Hygiene and Tropical Medicine

Keppel Street

London, WC1E 7HT

United Kingdom

Phone: 44 02076368636


Background: Behavior change apps can develop iteratively, where the app evolves into a complex, dynamic, or personalized intervention through cycles of research, development, and implementation. Understanding how existing users engage with an app (eg, frequency, amount, depth, and duration of use) can help guide further incremental improvements. We aim to explore how simple visualizations can provide a good understanding of temporal patterns of engagement, as usage data are often longitudinal and rich.

Objective: This study aims to visualize behavioral engagement with Drink Less, a behavior change app to help reduce hazardous and harmful alcohol consumption in the general adult population of the United Kingdom.

Methods: We explored behavioral engagement among 19,233 existing users of Drink Less. Users were included in the sample if they were from the United Kingdom; were 18 years or older; were interested in reducing their alcohol consumption; had a baseline Alcohol Use Disorders Identification Test score of 8 or above, indicative of excessive drinking; and had downloaded the app between May 17, 2017, and January 22, 2019 (615 days). Measures of when sessions begin, length of sessions, time to disengagement, and patterns of use were visualized with heat maps, timeline plots, k-modes clustering analyses, and Kaplan-Meier plots.

Results: The daily 11 AM notification is strongly associated with a change in engagement in the following hour; reduction in behavioral engagement over time, with 50.00% (9617/19,233) of users disengaging (defined as no use for 7 or more consecutive days) 22 days after download; identification of 3 distinct trajectories of use, namely engagers (4651/19,233, 24.18% of users), slow disengagers (3679/19,233, 19.13% of users), and fast disengagers (10,903/19,233, 56.68% of users); and limited depth of engagement with 85.076% (7,095,348/8,340,005) of screen views occurring within the Self-monitoring and Feedback module. In addition, a peak of both frequency and amount of time spent per session was observed in the evenings.

Conclusions: Visualizations play an important role in understanding engagement with behavior change apps. Here, we discuss how simple visualizations helped identify important patterns of engagement with Drink Less. Our visualizations of behavioral engagement suggest that the daily notification substantially impacts engagement. Furthermore, the visualizations suggest that a fixed notification policy can be effective for maintaining engagement for some users but ineffective for others. We conclude that optimizing the notification policy to target both effectiveness and engagement is a worthwhile investment. Our future goal is to both understand the causal effect of the notification on engagement and further optimize the notification policy within Drink Less by tailoring to contextual circumstances of individuals over time. Such tailoring will be informed from the findings of our micro-randomized trial (MRT), and these visualizations were useful in both gaining a better understanding of engagement and designing the MRT.

J Med Internet Res 2020;22(12):e23369




Maintaining alcohol consumption within recommended guidance is widely known to reduce one’s risk of illness or injuries. Such guidance includes the recommendations of the Chief Medical Officer of the United Kingdom to limit alcohol consumption to 14 units a week and to have frequent alcohol-free days [1]. However, anyone in the general adult population who wants to reduce their hazardous or harmful alcohol consumption may face certain challenges to follow such guidance [2]. Challenges include the ease of access to alcohol and alcohol being an addictive substance. This can lead to individuals developing chronic or cyclical patterns of excessive drinking, with the personal behaviors of drinking influenced by internal or external factors [3-5]. Internal factors refer to feeling states or events in an individual’s recent drinking history, such as previous drinking episodes, moods, motives, or cravings that may modify future patterns of drinking [6]. External factors are influential events that occur independently of an individual’s drinking history; for example, how the risk of hazardous drinking of the general population increases during holiday periods or weekends [7,8].

Behavior change apps, sensors, and wearables offer a way of reducing hazardous alcohol consumption through real-time data capture and interventions [9-12]. Benefits of behavior change apps, that can be synchronized with sensors and wearables, include capturing an individual’s dynamic history of alcohol consumption and state of mind while providing around the clock access to support, particularly in moments when an individual’s vulnerability to hazardous drinking may increase [13].

However, a key challenge for the majority of behavior change apps is that levels of engagement remain low [14-16]. Engagement, often a mediator of effectiveness [14], is considered a multifaceted construct composed of behavioral and experiential aspects [17]. Usage data from a behavior change app provides an understanding of behavioral engagement (hereafter referred to as engagement) with the app [18]. Multiple indicators of engagement are thought to convey important information about how users interact with a given intervention, including the frequency (eg, number of log-ins), depth (eg, proportion of available modules accessed), amount (eg, time spent per log-in), and duration (eg, total number of days) of use [19].

Drink Less is a behavior change app that aims to help its users reduce hazardous and harmful alcohol consumption. The app was developed following the multiphase optimization strategy framework (comprising a preparation phase, an optimization phase, and an evaluation phase) [20-23] and the UK Medical Research Council’s guidance on developing complex interventions [24-26]. The app includes 6 different theory and evidence-informed modules: normative feedback, goal setting, cognitive bias training, self-monitoring and feedback, action planning, and identity change. These modules are described in detail by Garnett et al [27]. The app sends a local daily push notification at 11 AM that asks users to “Please complete your drink diaries,” to encourage self-monitoring of drinking behavior. The default 11 AM timepoint was set so as not to disturb late risers and to allow participants time to complete their morning routine; however, the notification timing could be changed by the user.

Owing to the agile nature of app development, optimization of engagement can be done through cycles of research and implementation [28]. Identifying important patterns of engagement for such optimization purposes presents various analytical challenges that visualizations can address. Visualizations have previously been helpful for analyzing a wide variety of rich data streams within public health research [29-33]. Simple visualizations, especially when complemented with clear textual descriptions, are generally recommended for identifying and comparing trends [32]. In previous digital health research, visualizations have delivered at a glance insight from mass volume and time-varying data, including more sophisticated displays of spatiotemporal, contextual, and event-centric outcomes [34-38]. Importantly, visualizations can provide insights into optimization that include (1) patterns of use that may boost or hinder behavior change, (2) a better understanding of temporal engagement with various components of the intervention, and (3) pathways toward personalization of the intervention.


The aim of this paper is to explore the usefulness of simple visualizations in uncovering important temporal patterns of engagement and facilitating decision making for further intervention development. This study presents 2 key contributions to improving engagement with Drink Less. The first contribution, provided in the Results section, is to showcase a number of visualizations that helped us understand temporal patterns of engagement with Drink Less. The second contribution, provided in the Discussion section, explains how insights obtained from these visualizations informed the next stages of intervention optimization.

Data Transformation

Each visualization involved transformation of the data. Original usage data involved merging, by an anonymous user ID, a data set of baseline characteristics (age, sex, employment type, and Alcohol Use Disorders Identification Test [AUDIT] score) to a data set of time stamps of start time of use, screen views, and length (in microseconds) of use. Along with use, the actions of entering an alcohol-free day or recording units of alcohol consumed were measured.


Data set 1 included 19,233 users who downloaded Drink Less between May 17, 2017, and January 22, 2019 (615 days). The inclusion criteria for users included having a baseline AUDIT score of 8 or above, which is indicative of excessive drinking [39]; being from the United Kingdom; being aged 18 years or above; being interested in reducing their alcohol consumption; using app versions 1.0.11 to 1.0.16; and having consented to the Privacy Notice (Multimedia Appendix 1). Screen views data are recorded automatically and downloaded via Panda scripts from Nodechef (a web-based platform for hosting mobile apps) using a secure https protocol. Sessions were derived from screen views using the Pandas script.

Users who downloaded the app on August 21, 2018 (n=5830), were excluded as an article on BBC News was published on this date, which endorsed the app (Garnett et al, unpublished data, 2020); thus, these users were likely to have different characteristics and engagement behavior.

Data set 2 included time stamps of 829,001 sessions and 8,169,005 screen views of the 19,233 users in data set 1. This includes 122,332 entries of alcohol-free days and 123,704 entries of alcohol drinks consumed. All use was recorded from May 17, 2017, to April 16, 2019 (699 days). As such, users had a minimum of 84 days of use measured.

To explore various engagement aspects, we developed sets of data from data sets 1 and 2 with varying engagement measures.

Set A

All use was measured from May 17, 2017, to January 22, 2019 (615 days), including date of download and time stamps of all use. This period was chosen as it reflects a time in which the content of the app was relatively stable.

Set B

Set B included all users whose use was measured in Set A, with data only over the first 30 days from download, with the measure “Did use occur on this day?” (binary, yes or no) for each user.


Log-in Sessions and Frequency of Log-Ins

A session was defined as a continuous series of screen views, with a new session defined as a new screen view after 30 min of inactivity [40]. Clearing or swiping away the daily notification did not register as use and was not considered as either a session or a module view. All time stamps were appropriately adjusted from Coordinated Universal Time to British Summer Time. The amount of use per log-in session was operationalized as time spent (in seconds) per session. Daily use was captured by the measure “Did use occur on this day?” (binary, yes or no) for each user for 30 days (Set B).

Drinking Diary Entry

In the self-monitoring and feedback module, users enter an alcohol-free day and the date of its occurrence, the number of alcoholic units consumed, and the date of consumption. The time stamps in which records were made was measured.


We defined disengagement as the first day of 7 or more consecutive days of no use after download [41]. The days between download and disengagement were derived for each user. Users who did not disengage after downloading the app were censored.

Data Visualization Methods and Analytical Techniques

We used heat maps, timeline plots, k-modes clustering, generalized estimation equations, and Kaplan-Meier plots to explore and visualize patterns of engagement with Drink Less. Analyses were carried out in R [42] and Stata [43]. We used the following R library packages to create the visualizations: ggplot2 for heat maps and timeline plots [44], rayshader to create the 3D animations [45], viridis for color palettes sensitive to readers with color blindness [46], Klar to perform the k-modes clustering [47], survminer for the Kaplan-Meier survival curves and number at risk table [48], gganimate to create animated plots of use over time [49], and patchwork to place graphs side by side [50]. The data visualization methods, data set and engagement measures are shown in Textbox 1.

K-modes clustering is an extension of the k-means algorithm for partitioning categorical data, which uses a general dissimilarity measure [51,52]. Within each cluster, we visualized the probability of opening the app during the day over time with 95% CI. The appropriate number of clusters was explored through the elbow method and silhouette method [53]. The elbow method explains the variance of the data in relation to the number of clusters and shows by how much the addition of another cluster would reduce the dissimilarity measure. The silhouette method shows how well each user fits into their respective cluster through 2 distance measures: separation (ie, the average distance to the closest other cluster) and compactness (ie, the average within-cluster distance) [54,55]. Kaplan-Meier plots show the estimated cumulative proportion of users engaged and the time scale is days after download [56,57].

Data visualization methods, data, and engagement measures.

Set A:

  • Heat maps: Total count of sessions and total amount of time spent on Drink Less, by hour and day of the week
  • Timeline plots: Frequency and median amount of time per session
  • Kaplan-Meier plots: Time to disengagement (defined as days after download followed by 7 or more consecutive days of nonuse)

Set B:

  • K-modes clustering: Was the app used or not each day, over 30 days after download
Textbox 1. Data visualization methods, data, and engagement measures.

To explore the association between the delivery of the notification and subsequent near-term engagement of opening the app (ie, engagement in the hour after the notification is delivered), we compared opening the app (yes or no) between the exposed time period (11 AM to noon) and an unexposed time period (10 AM to 11 AM). We estimated the association between exposure to the notification and opening of the app, which was quantified using a risk ratio. We fitted a marginal model for the outcome of opening the app by using a generalized estimating equation [58] with robust standard errors and an independent working correlation matrix. We fitted an unadjusted model and a model adjusted for the baseline covariates of the continuous variables age, days after download and baseline AUDIT score, which were all included as linear terms, and the categorical variables employment type and gender. Further models explored effect moderation by adding an interaction between exposure to the notification and (1) days after download and (2) cluster (as identified by the k-modes analysis). In the final model, we additionally allowed the association between cluster and exposure to the notification to vary linearly by day after download. Estimated risk ratios with 95% CIs and Wald test P values are presented. For models with interaction terms, we present risk ratios for exposure to the notification estimated at days 1, 7, and 30 after download, estimated separately for each cluster.


The user characteristics are reported in Table 1. Approximately half (49.5%) of the sample were male. The mean age of users was 44 (SD 11.2) years, and the majority worked in nonmanual employment (71.7%). Just under half (46.6%) had a baseline AUDIT score indicating hazardous alcohol consumption (8 to 15, inclusive).

Table 1. User characteristics (N=19,233).
User characteristicsParticipants
Sex, n (%)

Male9540 (49.60)
Age (years), mean (SD)44 (11.2)
Employment type, n (%)

Nonmanual employment13,792 (71.71)
AUDITa risk zone, n (%)

Hazardous (8-15)8958 (46.58)

Harmful (16-19)3949 (20.53)

At risk of alcohol dependence (20-40)6326 (32.89)

aAUDIT: Alcohol Use Disorders Identification Test.

Summative tables of use (screen views and time on app) by module are provided in Multimedia Appendix 2. It was observed that 85% of screen views occurred in the module Self-Monitoring and Feedback. The number of users who reported at least one alcohol-free day or at least one alcohol drink record was 61.86% (11,898/19,233) and 49.11% (9445/19,233), respectively. Over the first 30 days of use after download (derived for Set B data), the median number of sessions per user was 9, with an IQR of 2 to 28 sessions, and the median time spent per user was 24 min, with an IQR of 9 to 55 min.


Patterns of Frequency of Use, Length of Use, Entries of Alcohol-Free Days, and Alcohol Units Consumed

In Figure 1, both heat maps show days of the week along the x-axis and hour of the day along the y-axis. Plot A in Figure 1 shows the frequency of opening the app by hour of the day and day of the week. This shows that there is a strong association between delivery of the notification and opening of the app in the following hour, and this is consistent throughout the week. Plot B in Figure 1 shows the amount of use by hour of the day and day of the week. This shows that the notification is also associated with the distribution of the total time spent on the app. In plot B, hotspots are observed across the evenings and on Saturday, Sunday, and Monday mornings, which are not evident in plot A. A heat map of when Drink Less was downloaded (Multimedia Appendix 3) shows hotspots of downloads on Sunday and Monday evenings. Rotating 3D heat map films of Figure 1, which show the variations more clearly, are provided in Multimedia Appendices 4 and 5.

In Figure 2, plot C shows the median time spent on the app along the y-axis and plot D shows the total number of sessions starting in the hour along the y-axis. Timeline plots show the hour of the day on the x-axis. Plot D shows that the frequency of sessions sharply peaks in the hour after the notification is sent at 11 AM. A second natural peak of frequency occurred in the evenings and a third smaller peak in the mornings. Plot C shows that the median length of time drastically dropped from 11 AM onward, with a slow and steady recovery as the day progressed. An animation of plot D over time is provided in Multimedia Appendix 6, showing that the shape of the distribution over 30 days remains consistent.

Figure 1. Heat maps of total frequency of use (sessions) and total time on app (hours).
View this figure
Figure 2. Median time spent on the Drink Less app per session and frequency distribution of sessions.
View this figure

In Figure 3, plot E shows the frequency distribution of entering an alcohol-free day and plot F shows the frequency of entering a drink record. Timeline plots show the hour of the day on the x-axis. There are more alcohol-free days entered between 11 AM to 12 PM than drink records made, which suggests that the notification is more strongly associated with entering alcohol-free days than entering alcohol units consumed. Both outcomes see similar prominent, natural peaks in the evenings, with an additional smaller peak in the mornings.

Figure 3. Frequency distributions of when alcohol-free days and alcohol units are recorded during the day.
View this figure
Visualization of Engagement Clusters

A total of 3 clusters emerged from the k-modes clustering. This was based on the measure did the user open the app? (binary, yes or no) for the first 30 days after download.

Figure 4 plots the probability of use of the app, stratified by cluster, over time (number of days after download). The 3 ribbons represent the probability of use of the app for each engagement cluster, with 95% CI. On the basis of the observed pattern of engagement, we named the 3 clusters as fast disengagers (10,903/19,233, 56.68%), slow disengagers (3679/19,233, 19.12%), and engagers (4651/19,233, 24.18%). The optimal number of clusters was determined by the elbow method and silhouette method (Multimedia Appendix 7). The silhouette method suggested that the optimal number of clusters was 2, whereas the elbow method suggested 3 clusters. Comparing the results under 2 and 3 clusters showed that the slow disengagers and engagers groups identified under 3 clusters were essentially a subdivision of 1 cluster in the 2-cluster model. We chose to retain 3 clusters based on observed differences in the trajectory of engagement over time between the 2 groups—the engagers and slow disengagers.

The probability of using the app 30 days after download for engagers was 0.69 (95% CI 0.67-0.70), slow disengagers was 0.10 (95% CI 0.10-0.11), and fast disengagers was 0.01 (95% CI 0.01-0.02).

Figure 4. Probability of use on day after download by cluster group with 95% CIs.
View this figure

Table 2 shows the distribution of user characteristics across the engagement clusters. The median number of sessions was lowest for the fast disengagers and much higher for the engagers. Engagers are, on average, more likely to be older, male, working in nonmanual employment, and more likely to report a lower AUDIT risk zone, compared with users within the fast disengagers and slow disengagers clusters.

Table 2. User characteristics by cluster group.
User characteristicsFast disengagers (n=10,903)Slow disengagers (n=3679)Engagers (n=4651)
Male, n (%)4991 (45.78)1920 (52.49)2629 (56.53)
Age (years), mean (SD)43.7 (11.57)43.2 (10.91)45.4 (10.57)
Employment type (nonmanual), n (%)7567 (69.40)2659 (72.28)3566 (76.67)
AUDITa risk zone, n (%)

Hazardous (8-15)5016 (46.01)1577 (42.86)2365 (50.85)

Harmful (16-19)2155 (19.77)798 (21.69)996 (21.41)

At risk of alcohol dependence (20+)3732 (34.23)1304 (35.44)1290 (27.74)
Number of sessions per user, median (25th-75th percentile)3 (1-6)18 (12-28)88 (51-175)

aAUDIT: Alcohol Use Disorders Identification Test.

Table 3 provides the estimated associations between exposure to the notification and app use, based on Set B data. Over the first 30 days after day of download, the probability of using the app in the hour after the delivery of the notification was approximately 4 times higher than the probability of using the app in the hour before. All models of the estimated associations between exposure to the notification and app use are adjusted for the continuous variables of age, days after download, baseline AUDIT score, and the categorical variables of employment type and sex. The cluster-specific effects included an effect moderation of the exposure to the notification by cluster group, and the days after download effects included an effect moderation of the exposure by days after download. The adjusted estimated risk ratio was 4.21 (95% CI 4.07-4.36), and the estimated risk ratio was higher among engagers (Wald test P value: fast disengagers vs engagers P=.001 slow disengagers vs engagers P<.001, slow disengagers vs fast disengagers P=.44).

Table 4 shows the estimated association between exposure to the notification and opening of the app in the 3 clusters at 3 different time points (days 1, 7, and 30).

Table 3. Estimated associations between exposure to the notification and app use.
ModelExposure to notification, estimated relative risk ratio (95% CI)
Unadjusted model4.22 (4.13-4.31)
Adjusted modela4.21 (4.07-4.36)
Days after downloadb

Day 13.93 (3.77-4.10)

Day 74.07 (3.93-4.22)

Day 304.67 (4.38-4.98)

Fast disengagers3.97 (3.70-4.25)

Slow disengagers3.82 (3.60-4.03)

Engagers4.38 (4.18-4.59)

aAdjusted for days after download, employment type, sex, age, and baseline Alcohol Use Disorders Identification Test (AUDIT) score.

bEstimated from the model including the interaction effect of exposure to the notification by days after download, adjusted for employment type, sex, age, and baseline AUDIT score.

cEstimated from the model including the interaction effect of exposure to the notification by cluster, adjusted for days after download, employment type, sex, age, and baseline AUDIT score.

Table 4. Estimated risk ratio with 95% CI for the associations between exposure to the notification and app use within each cluster, at 3 time points (days 1, 7, and 30).
ClustersRisk ratio at day 1 (95% CIa)Risk ratio at day 7 (95% CIa)Risk ratio at day 30 (95% CIa)
Fast disengagers3.66 (3.33-4.02)3.83 (3.57-4.11)4.58 (3.86-5.43)
Slow disengagers4.18 (3.85-4.54)3.87 (3.64-4.12)2.89 (2.43-3.43)
Engagers4.05 (3.82-4.30)4.22 (4.01-4.43)4.90 (4.56-5.26)

aInteraction effect of exposure to the notification and days after download, an interaction effect of exposure to the notification and cluster, and a three-way interaction effect of exposure to the notification, cluster, and days after download, adjusted for employment type, sex, age, and baseline Alcohol Use Disorders Identification Test score.

Visualization of Time to Disengagement

Kaplan-Meier plots, both overall and stratified by clusters, were plotted to show days to disengagement, defined as 7 or more consecutive days of no use, for the first 365 days after downloading Drink Less.

In Figures 5 and 6, the x-axis depicts the number of days after download, ranging from 0 to 365, and the y-axis depicts the survival probability, which is the proportion of users who have not disengaged. The dashed lines at the 0.5 survival probability mark shows the time (days) up to when 50%of each cluster has disengaged. Each hash in the plot represents a right-censored user. The number at risk represents the users in the clusters who remain engaged over the year. In Figure 5, we see that 50.00% (9617/19,233) of users have disengaged at 22 days from download, and Figure 6 shows the divergence of longer-term engagement between clusters. The median number of days to disengagement for engagers was 132 days (95% CI 128-137), slow disengagers was 26 days (95% CI 24-29), and fast disengagers was 3 days (95% CI 2-3).

Figure 5. Time to disengagement (defined as the first day of 7 or more consecutive days of no use) for all users.
View this figure
Figure 6. Time to disengagement (defined as the first day of 7 or more consecutive days of no use) by the engagement cluster.
View this figure

Principal Findings

Visualizations provided important insights into how users engage with the behavior change app Drink Less. They revealed a strong association between delivery of the daily push notification (sent at 11 AM) and use in the next hour, suggesting that the push notification strongly influences how users engage with Drink Less in the immediate hour after the notification is sent. Push notifications (sometimes known as ecological momentary interventions) are programmed messages sent to a user by the app and are commonly employed within behavior change apps to both monitor and provide support to people at risk of harmful alcohol consumption [3,10,59]. Push notifications are a time-varying component of Drink Less that can be further optimized to become just-in-time adaptive interventions that rely on decision rules in the provision of real-time support and can learn and adapt to the contextual and psychological circumstances of individuals over time [59]. Previous research has found that notifications are important components that influence engagement with behavior change apps [60-62]. This includes an ecological momentary assessment study with Drink Less, which found that establishing a daily routine is important for maintaining engagement and that the daily push notification supports such routines [63]. This study also found that time-varying, endogenous factors of motivation and perceived usefulness of the app were the most consistent predictors of engagement.

Our analysis suggested an approximate adjusted four-fold increase in the probability of using the app in the immediate hour following the notification (11 AM to noon) compared with the preceding hour (10 AM to 11 AM). For the 1 in 5 users belonging to the slow disengagers group, two interesting findings emerged. Firstly, the association between the notification and opening Drink Less in the subsequent hour decreased over time, and secondly, patterns of engagement for this group show, on average, a high probability of use over the first week but low probability after the second week. A possible reason for this decline in probability of use and association of the notification and use could be habituation to the daily notification, or turning the notification off. Importantly, we hypothesize that optimizing the notification policy may generate higher rates of engagement for this group.

Future Research to Understand and Optimize the Notification Policy

To carefully create decision rules for the policy to evolve from an ecological momentary intervention to a just-in-time adaptive intervention, we will undertake a micro-randomized trial (MRT). The aim of the MRT is to further develop the push notification policy to improve engagement by targeting internal or external contextual circumstances that either influence excessive drinking (states of vulnerability) or events of engagement with the app (states of acceptability and opportunity) [64]. Visualization of engagement data helped inform the design of our MRT.

Table 5 summarizes how the visualizations from this exploratory research informed the design of our forthcoming MRT.

Primarily, this research guided our decision to shift the delivery time from 11 AM to 8 PM to exploit the potential increase in vulnerability to excess drinking, in an opportune and acceptable moment to engage with Drink Less [64]. To avoid the risk of an underpowered MRT, the expected effect size used in the sample size calculation of our MRT is based on a more conservative model, with the control defined as use between 9 AM and 11 AM and the treatment defined as use between 11 AM and 1 PM. This means that the MRT is powered to detect a marginal effect, quantified as a risk ratio, of sending a notification (compared with sending no notification) of 2.16 on user engagement rather than 4.22. We added 2 parallel arms to the MRT to provide an assessment of how engagement with Drink Less evolves over time when no notifications are provided and an exchangeable sample to compare the current policy of delivering a fixed notification daily, to a random notification policy, varying the content and sequence of notifications.

Table 5. Linking visualization to the design of a micro-randomized trial.
What we learnt from these analysesWhich visualization or analyses showed us thisHow this informed the design of our randomized trial
The present notification appears to be a key driver of engagement
  • Figure 1, plot A: heat map of total sessions.
  • Figure 1, plot B: heat map of total time on app (hours)
  • Table 4: estimated risk ratio with 95% CI for the associations between exposure to the notification and app use within each cluster, at 3 time points (days 1, 7, and 30)
We chose to undertake a micro-randomized trial to both understand the causal effect of the notification on engagement, and to further optimize the delivery of notifications with respect to time-varying covariates, notifications, and outcomes
The impact of the notification seems to be strongest in the hour preceding deliveryFigure 1, plot A: heat map of total sessionsWe set the time window to measure the proximal (ie, near-term) effect as 1 hour after delivery
Evenings seem to be an opportune and acceptable moment to engage with Drink Less. It is also a time of increased vulnerability to excess drinkingFigure 1, plot B: heat map of total time on app (hours)We moved the delivery time of the notification to 8 PM
The notification may encourage the reporting of alcohol-free days more than drink consumed. This may be due to competing pressures for time at 11 AMFigure 3: frequency distributions of when alcohol-free days and alcohol units are recordedWe intervened in the evenings to see if this is a more acceptable and opportune time to report drinks consumed
The notification may reduce the median time per session during the reminder of the dayFigure 2: line plot of median time spent on app (seconds)We included a no-notification arm in our trial to capture a momentary assessment of engagement when no notifications are sent
The depth of engagement with Drink Less is lowMultimedia Appendix 2: summaries of use by module for all usersWe trialed new notifications which target the perceived usefulness of Drink Less to encourage broader engagement
Slow disengagers (3679/19,233, 19.13%) have a high probability of engagement during the first week, but by day 30, this group has a low probability, suggesting a loss of motivationFigure 4: probability of use on day after download by cluster groupWe tested 30 new messages to increase novelty and motivation to remain engaged with Drink Less (Multimedia Appendix 8)
Exogenous impacts, such as public health campaigns, are likely to influence the cohort of users over timeFigure 6: time to disengagement (defined as the first day of 7 or more consecutive days of no use) by the engagement clusterWe included a standard app version arm in the trial, to provide an exchangeable sample to compare the fixed and random notification policies


This paper details exploratory research. Our estimates of the association between the notification and opening Drink Less do not represent a causal effect on engagement, as we are unable to account for systematic differences in use between the 2 periods that are unrelated to the notification. A randomized trial will allow for the causal effect of the notification to be understood. We also found that simple, accessible visualizations achieved our goal of understanding important patterns of engagement; however, when managing denser streams of data, more complex visualizations may be required.

An additional limitation is that disengagement is defined as a period of no use for 7 or more consecutive days and is considered as a one-time event instead of a repeated event; hence, the Kaplan-Meier plots are interpreted for the survival event disengagement for the first time. However, a proportion of users repeatedly disengage and then re-engage with Drink Less. It is not uncommon that even after disengaging a number of times, users re-engage for long, continuous spells of use with Drink Less. We aim to explore this in future research by visualizing the nature of repeated reengagement with accessible graphical applications and available shared toolsets [34]. An additional limitation is that we did not track whether users subsequently turned off or altered the delivery time of their notifications.


Identifying patterns of engagement from voluminous, temporally dense data presents various challenges for researchers and practitioners. The summarization of such data with heat maps, timeline plots, and Kaplan-Meier plots can provide a clear picture of daily, weekly, and long-term patterns of use over time with a behavior change app. Optimizing engagement is a priority for many behavior change apps, and these visualizations provide a way to identify the key features of how this version of a behavior change app is engaged with.

For Drink Less, we have demonstrated the important role of visualizations by showing how these clearly identified how behavioral engagement varies over the day of the week and hour of the day, along with when users first disengage. The visualizations revealed that the daily notification is likely to strongly influence engagement with Drink Less. Both the average probability of use over 30 days and the association between use and the notification remained high for users in the engagers cluster yet steadily declined over time for users in the slow disengagers cluster. This suggests that a fixed notification policy can be effective for maintaining engagement for some users but ineffective for others. It is now our priority to understand the causal effect of the notification on engagement and to consider further optimizing the push notification policy to contextual circumstances of individuals over time to inform the development of a just-in-time adaptive intervention. The MRT aims to inform the development of decision rules to tailor the notification policy to individuals over time, with details found in our protocol [65].


The development of the Drink Less app was funded by the National Institute for Health Research (NIHR) School for Public Health Research, Society for the Study of Addiction, Cancer Research UK, and the UK Centre for Tobacco and Alcohol Studies (UKCTAS). The NIHR School for Public Health Research is a partnership between the Universities of Sheffield, Bristol, Cambridge, and Imperial; University College London; The London School for Hygiene and Tropical Medicine; LiLaC: a collaboration between the Universities of Liverpool and Lancaster; and Fuse—The Centre for Translational Research in Public Health: a collaboration between Newcastle, Durham, Northumbria, Sunderland, and Teesside Universities. The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care. UKCTAS is part of UK Clinical Research Collaboration, a Public Health Research Centre of Excellence. Funding from the Medical Research Council, British Heart Foundation, Cancer Research UK, Economic and Social Research Council, and the NIHR under the auspices of the UK Clinical Research Collaboration is gratefully acknowledged.

The authors would like to thank Dr Dave Crane for his important role in the development and factorial screening trial of the Drink Less app and the National University of Singapore’s Institute of Mathematical Sciences for funding LB and TQ’s visit to the program on Statistical Methods for Developing Personalized Mobile Health Interventions. The authors would also like to thank the UCL Tobacco and Alcohol Research Group for their helpful feedback on the draft of this paper.

LB is supported by a PhD studentship funded by the MRC Network of Hubs for Trials Methodology Research (MR/L004933/2-R18).

CG is funded by the NIHR, the NIHR School for Public Health Research and Cancer Research UK (CRUK: C1417/A22962).

TQ is supported by the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award number R01AA23187; National Institute on Drug Abuse of the National Institutes of Health under award number P50DA039838; National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award number U54EB020404; and National Cancer Institute of the National Institutes of Health under award number U01CA229437. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

OP is funded by Cancer Research UK (CRUK: C1417/A22962).

EW is supported by the MRC (MR/S01442X/1) and Health Data Research UK, which is funded by the UK Medical Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Department of Health and Social Care (England), Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Health and Social Care Research and Development Division (Welsh Government), Public Health Agency (Northern Ireland), and British Heart Foundation and Wellcome. HP receives funding from Public Health England, the NIHR and NHS England.

Conflicts of Interest

CG is involved in the development and testing of the Drink Less app, and OP is an unpaid member of the scientific committee for the Smoke Free app. HP has received consultancy fees from Babylon Health. All other authors declare no conflict of interests.

Multimedia Appendix 1

Privacy notice.

DOCX File , 14 KB

Multimedia Appendix 2

Summaries of Use by Module for all users (n=19,233).

DOCX File , 13 KB

Multimedia Appendix 3

3D rotating heatmap of when sessions begin.

MP4 File (MP4 Video), 2999 KB

Multimedia Appendix 4

3D rotating graph total time on app per session.

MP4 File (MP4 Video), 3167 KB

Multimedia Appendix 5

Plot D animated over time.

MP4 File (MP4 Video), 3436 KB

Multimedia Appendix 6

Heat map of when downloads occur.

DOCX File , 29 KB

Multimedia Appendix 7

Plots of the elbow method and silhouette method.

DOCX File , 38 KB

Multimedia Appendix 8

Notification content of new message bank to be trialled in the micro-randomized trial.

DOCX File , 14 KB


  1. UK Chief Medical Officers' Low Risk Drinking Guidelines. Department of Health. 2016.   URL: https:/​/assets.​​government/​uploads/​system/​uploads/​attachment_data/​file/​545937/​UK_CMOs__report.​pdf [accessed 2020-06-09]
  2. Waters AJ, Schoenmakers TM, Snelleman M, Szeto EH, Franken IH, Hendriks VM, et al. Affect, motivation, temptation, and drinking among alcohol-dependent outpatients trying to maintain abstinence: An Ecological Momentary Assessment study. Drug Alcohol Depend 2020 Jan 1;206:107626. [CrossRef] [Medline]
  3. Beckjord E, Shiffman S. Background for real-time monitoring and intervention related to alcohol use. Alcohol Res 2014;36(1):9-18 [FREE Full text] [Medline]
  4. Dvorak RD, Pearson MR, Day AM. Ecological momentary assessment of acute alcohol use disorder symptoms: associations with mood, motives, and use on planned drinking days. Exp Clin Psychopharmacol 2014 Aug;22(4):285-297 [FREE Full text] [CrossRef] [Medline]
  5. Cook MA, Newins AR, Dvorak RD, Stevenson BL. What about this time? Within- and between-person associations between drinking motives and alcohol outcomes. Exp Clin Psychopharmacol 2020 Oct;28(5):567-575. [CrossRef] [Medline]
  6. Chakravorty S, Kuna ST, Zaharakis N, O'Brien CP, Kampman KM, Oslin D. Covariates of craving in actively drinking alcoholics. Am J Addict 2010;19(5):450-457 [FREE Full text] [CrossRef] [Medline]
  7. Lac A, Handren L, Crano WD. Conceptualizing and measuring weekend versus weekday alcohol use: item response theory and confirmatory factor analysis. Prev Sci 2016 Oct;17(7):872-881 [FREE Full text] [CrossRef] [Medline]
  8. Lac A, Luk JW. Pathways from positive, negative, and specific alcohol expectancies to weekday and weekend drinking to alcohol problems. Prev Sci 2019 Jul;20(5):800-809 [FREE Full text] [CrossRef] [Medline]
  9. Cohn A, Hunter-Reel D, Hagman BT, Mitchell J. Promoting behavior change from alcohol use through mobile technology: the future of ecological momentary assessment. Alcohol Clin Exp Res 2011 Dec;35(12):2209-2215 [FREE Full text] [CrossRef] [Medline]
  10. Morgenstern J, Kuerbis A, Muench F. Ecological momentary assessment and alcohol use disorder treatment. Alcohol Res 2014;36(1):101-109 [FREE Full text] [Medline]
  11. Petry N, Meredith S, Alessi S. Smartphone applications to reduce alcohol consumption and help patients with alcohol use disorder: a state-of-the-art review. Adv Health Care Technol 2015 Dec:47. [CrossRef]
  12. Bae S, Chung T, Ferreira D, Dey AK, Suffoletto B. Mobile phone sensors and supervised machine learning to identify alcohol use events in young adults: Implications for just-in-time adaptive interventions. Addict Behav 2018 Aug;83:42-47 [FREE Full text] [CrossRef] [Medline]
  13. Freisthler B, Lipperman-Kreda S, Bersamin M, Gruenewald PJ. Tracking the when, where, and with whom of alcohol use: integrating ecological momentary assessment and geospatial data to examine risk for alcohol-related problems. Alcohol Res 2014;36(1):29-38 [FREE Full text] [Medline]
  14. Torous J, Nicholas J, Larsen ME, Firth J, Christensen H. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Ment Health 2018 Aug;21(3):116-119. [CrossRef] [Medline]
  15. Forbes G, Newton S, Cantalapiedra Calvete C, Birch J, Dodds J, Steed L, et al. MEMPHIS: a smartphone app using psychological approaches for women with chronic pelvic pain presenting to gynaecology clinics: a randomised feasibility trial. BMJ Open 2020 Mar 12;10(3):e030164 [FREE Full text] [CrossRef] [Medline]
  16. Torous J, Michalak EE, O'Brien HL. Digital health and engagement-looking behind the measures and methods. JAMA Netw Open 2020 Jul 1;3(7):e2010918 [FREE Full text] [CrossRef] [Medline]
  17. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med 2017 Jun 13;7(2):254-267 [FREE Full text] [CrossRef] [Medline]
  18. Sieverink F, Kelders S, Poel M, van Gemert-Pijnen L. Opening the black box of electronic health: collecting, analyzing, and interpreting log data. JMIR Res Protoc 2017 Aug 7;6(8):e156 [FREE Full text] [CrossRef] [Medline]
  19. Pham Q, Graham G, Carrion C, Morita PP, Seto E, Stinson JN, et al. A library of analytic indicators to evaluate effective engagement with consumer mhealth apps for chronic conditions: scoping review. JMIR Mhealth Uhealth 2019 Jan 18;7(1):e11941 [FREE Full text] [CrossRef] [Medline]
  20. Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med 2007 May;32(5 Suppl):S112-S118 [FREE Full text] [CrossRef] [Medline]
  21. Collins LM, Dziak JJ, Li R. Design of experiments with multiple independent variables: a resource management perspective on complete and reduced factorial designs. Psychol Methods 2009 Sep;14(3):202-224 [FREE Full text] [CrossRef] [Medline]
  22. Collins LM, Chakraborty B, Murphy SA, Strecher V. Comparison of a phased experimental approach and a single randomized clinical trial for developing multicomponent behavioral interventions. Clin Trials 2009 Feb;6(1):5-15 [FREE Full text] [CrossRef] [Medline]
  23. Collins LM, Baker TB, Mermelstein RJ, Piper ME, Jorenby DE, Smith SS, et al. The multiphase optimization strategy for engineering effective tobacco use interventions. Ann Behav Med 2011 Apr;41(2):208-226 [FREE Full text] [CrossRef] [Medline]
  24. Anderson R. New MRC guidance on evaluating complex interventions. Br Med J 2008 Oct 22;337(oct22 1):a1937. [CrossRef]
  25. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. Br Med J 2000 Sep 16;321(7262):694-696 [FREE Full text] [CrossRef] [Medline]
  26. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. Br Med J 2008 Sep 29;337:a1655 [FREE Full text] [CrossRef] [Medline]
  27. Garnett C, Crane D, West R, Brown J, Michie S. The development of Drink Less: an alcohol reduction smartphone app for excessive drinkers. Transl Behav Med 2019 Mar 1;9(2):296-307 [FREE Full text] [CrossRef] [Medline]
  28. Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res 2017 Jun 29;19(6):e232 [FREE Full text] [CrossRef] [Medline]
  29. Ola O, Sedig K. The challenge of big data in public health: an opportunity for visual analytics. Online J Public Health Inform 2014;5(3):223 [FREE Full text] [CrossRef] [Medline]
  30. Chui KK, Wenger JB, Cohen SA, Naumova EN. Visual analytics for epidemiologists: understanding the interactions between age, time, and disease with multi-panel graphs. PLoS One 2011 Feb 15;6(2):e14683 [FREE Full text] [CrossRef] [Medline]
  31. Ola O, Sedig K. Beyond simple charts: design of visualizations for big health data. Online J Public Health Inform 2016;8(3):e195 [FREE Full text] [CrossRef] [Medline]
  32. Gelman A, Pasarica C, Dodhia R. Let's practice what we preach. Am Stat 2002 May;56(2):121-130. [CrossRef]
  33. Ballarini NM, Chiu Y, König F, Posch M, Jaki T. A critical review of graphics for subgroup analyses in clinical trials. Pharm Stat 2020 Sep;19(5):541-560. [CrossRef] [Medline]
  34. Morrison C, Doherty G. Analyzing engagement in a web-based intervention platform through visualizing log-data. J Med Internet Res 2014 Nov 13;16(11):e252 [FREE Full text] [CrossRef] [Medline]
  35. Arden-Close EJ, Smith E, Bradbury K, Morrison L, Dennison L, Michaelides D, et al. A visualization tool to analyse usage of web-based interventions: the example of positive online weight reduction (power). JMIR Hum Factors 2015 May 19;2(1):e8 [FREE Full text] [CrossRef] [Medline]
  36. Sharmin M, Raij A, Epstien D, Nahum-Shani I, Beck JG, Vhaduri S, et al. Visualization of time-series sensor data to inform the design of just-in-time adaptive stress interventions. Proc ACM Int Conf Ubiquitous Comput 2015 Sep;2015:505-516 [FREE Full text] [CrossRef] [Medline]
  37. Chen AT, Wu S, Tomasino KN, Lattie EG, Mohr DC. A multi-faceted approach to characterizing user behavior and experience in a digital mental health intervention. J Biomed Inform 2019 Jun;94:103187 [FREE Full text] [CrossRef] [Medline]
  38. Chien I, Enrique A, Palacios J, Regan T, Keegan D, Carter D, et al. A machine learning approach to understanding patterns of engagement with internet-delivered mental health interventions. JAMA Netw Open 2020 Jul 1;3(7):e2010791 [FREE Full text] [CrossRef] [Medline]
  39. Bohn MJ, Babor TF, Kranzler HR. The alcohol use disorders identification test (AUDIT): validation of a screening instrument for use in medical settings. J Stud Alcohol 1995 Jul;56(4):423-432. [CrossRef] [Medline]
  40. How a Web Session is Defined in Analytics. Google Analytics. 2017.   URL: [accessed 2020-04-01]
  41. Du H, Venkatakrishnan A, Youngblood GM, Ram A, Pirolli P. A group-based mobile application to increase adherence in exercise and nutrition programs: a factorial design feasibility study. JMIR Mhealth Uhealth 2016 Jan 15;4(1):e4 [FREE Full text] [CrossRef] [Medline]
  42. R: A Language and Environment for Statistical Computing. R Core Team. 2020.   URL: [accessed 2020-11-25]
  43. StataCorp. Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC; 2017.
  44. Wickham H. ggplot2: Elegant Graphics for Data Analysis. New York, USA: Springer-Verlag; 2016.
  45. Morgan-Wall T. Create Maps and Visualize Data in 2D and 3D. CRAN. 2020.   URL: [accessed 2020-04-01]
  46. Garnier S. Default Color Maps From 'matplotlib'. CRAN. 2018.   URL:, [accessed 2020-04-01]
  47. Weihs C, Ligges U, Karsten L, Nils R. klaR analyzing German business cycles. In: Baier D, Decker R, Schmidt-Thieme L, editors. Data Analysis and Decision Support. Berlin, Germany: Springer-Verlag; 2005:335-343.
  48. Kassambara A, Kosinski M, Przemyslaw B. Drawing Survival Curves Using 'ggplot2'. CRAN.; 2020.   URL:, [accessed 2020-04-01]
  49. Lin Pedersen T, Robinson D. A Grammar of Animated Graphics. CRAN. 2020.   URL: [accessed 2020-11-25]
  50. Lin Pederson T. The Composer of Plots. CRAN. 2019.   URL: [accessed 2020-04-01]
  51. Huang Z. Extensions to the k-means algorithm for clustering large data sets with categorical values. Data Mining Knowl Discovery 1998;2(3):283-304. [CrossRef]
  52. Bishop CM. In: Jordan M, Kleinberg J, Scholkopf B, editors. Pattern Recognition and Machine Learning. New York, USA: Springer; 2006.
  53. Yock AD, Kim G. Technical Note: Using k-means clustering to determine the number and position of isocenters in MLC-based multiple target intracranial radiosurgery. J Appl Clin Med Phys 2017 Sep;18(5):351-357 [FREE Full text] [CrossRef] [Medline]
  54. Handl J, Knowles J, Kell DB. Computational cluster validation in post-genomic data analysis. Bioinformatics 2005 Aug 1;21(15):3201-3212. [CrossRef] [Medline]
  55. Lengyel A, Botta-Dukát Z. Silhouette width using generalized mean-a flexible method for assessing clustering efficiency. Ecol Evol 2019 Dec;9(23):13231-13243 [FREE Full text] [CrossRef] [Medline]
  56. Morris TP, Jarvis CI, Cragg W, Phillips PP, Choodari-Oskooei B, Sydes MR. Proposals on Kaplan-Meier plots in medical research and a survey of stakeholder views: KMunicate. BMJ Open 2019 Sep 30;9(9):e030215 [FREE Full text] [CrossRef] [Medline]
  57. Royston P, Parmar MK, Altman DG. Visualizing length of survival in time-to-event studies: a complement to Kaplan-Meier plots. J Natl Cancer Inst 2008 Jan 16;100(2):92-97. [CrossRef] [Medline]
  58. LIANG K, ZEGER SL. Longitudinal data analysis using generalized linear models. Biometrika 1986;73(1):13-22. [CrossRef]
  59. Bidargaddi N, Schrader G, Klasnja P, Licinio J, Murphy S. Designing m-Health interventions for precision mental health support. Transl Psychiatry 2020 Jul 7;10(1):222 [FREE Full text] [CrossRef] [Medline]
  60. Morrison LG, Hargood C, Pejovic V, Geraghty AW, Lloyd S, Goodman N, et al. The effect of timing and frequency of push notifications on usage of a smartphone-based stress management intervention: an exploratory trial. PLoS ONE 2017 Jan 3;12(1):e0169162. [CrossRef]
  61. Bidargaddi N, Almirall D, Murphy S, Nahum-Shani I, Kovalcik M, Pituch T, et al. To prompt or not to prompt? A microrandomized trial of time-varying push notifications to increase proximal engagement with a mobile health app. JMIR Mhealth Uhealth 2018 Nov 29;6(11):e10123 [FREE Full text] [CrossRef] [Medline]
  62. Alkhaldi G, Hamilton FL, Lau R, Webster R, Michie S, Murray E. The effectiveness of prompts to promote engagement with digital interventions: a systematic review. J Med Internet Res 2016 Jan 8;18(1):e6 [FREE Full text] [CrossRef] [Medline]
  63. Perski O, Naughton F, Garnett C, Blandford A, Beard E, West R, et al. Do daily fluctuations in psychological and app-related variables predict engagement with an alcohol reduction app? A series of n-of-1 studies. JMIR Mhealth Uhealth 2019 Oct 2;7(10):e14098 [FREE Full text] [CrossRef] [Medline]
  64. Nahum-Shani I, Hekler EB, Spruijt-Metz D. Building health behavior models to guide the development of just-in-time adaptive interventions: A pragmatic framework. Health Psychol 2015 Dec;34S:1209-1219 [FREE Full text] [CrossRef] [Medline]
  65. Bell L, Garnett C, Qian T, Perski O, Potts HW, Williamson E. Notifications to improve engagement with an alcohol reduction app: protocol for a micro-randomized trial. JMIR Res Protoc 2020 Aug 7;9(8):e18690 [FREE Full text] [CrossRef] [Medline]

AUDIT: Alcohol Use Disorders Identification Test
MRT: micro-randomized trial
NIHR: National Institute for Health Research
UKCTAS: UK Centre for Tobacco and Alcohol Studies

Edited by G Eysenbach; submitted 10.08.20; peer-reviewed by C Morrison, D Leightley, R Krukowski; comments to author 26.08.20; revised version received 08.10.20; accepted 28.10.20; published 11.12.20


©Lauren Bell, Claire Garnett, Tianchen Qian, Olga Perski, Elizabeth Williamson, Henry WW Potts. Originally published in the Journal of Medical Internet Research (, 11.12.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.