Information visualization can be defined as “the use of computer-supported, interactive, visual representations of abstract data to amplify cognition” [29]. Information visualization can be used either to explore or communicate a set of data [30]. We apply exploratory sequential data analysis [31] to highlight sequential (or temporal) aspects of the data, such as time or module number, that can provide insights into engagement. While this approach has been applied to log-data analysis for session events grouped into a small number of distinct categories [32], there are no precedents for representation of continuous data.

When designing visualizations, it can be helpful to consider Schneiderman’s information seeking mantra: “Overview first, zoom and filter, then details-on-demand” [33]. This can be applied to a single visualization or to linking a series of visualizations. For example, a visualization that depicts navigation paths for an individual could show an overview of all sessions, enable zooming on a particular session, or selecting details of a specific event, or user action. Alternatively, a visualization of usage over time for an entire cohort could support filtering (eg, by gender or initial depression score), and details-on-demand linking to a visualization of an individual’s temporal navigation patterns.

Data used in the visualizations are drawn from two studies using the SilverCloud platform, which is described below. The first dataset is from a small pilot trial of the SilverCloud platform in a primary-care mental health setting in the United Kingdom [34]. Outcome data, log-data, and interview data were captured for 19 participants. The combination of data allows us to explore engagement patterns of individuals in depth. To complement this small, detailed dataset, we used a larger dataset drawn from the usage of the same intervention in a university setting [35]. The second dataset contains 326 participants, and over 44,838 logged events, which allows us to explore the issues surrounding visualization of a larger cohort.

Exploratory sequential data analysis involves the successive transformation of raw data sequences until the product enables statements to be made that answer research or design questions [31]. In this case, the original log-dataset included user ID, time-stamp, and page URL (uniform resource locator) of every action completed, such as reading a content page, saving a journal entry, or updating an activity. The first data processing step was to transform the URLs into meaningful labels. Content page URLs were assigned labels that indicated sequential order. For example, a URL ending with “content/9/54/93/” can be mapped to “Module 2 Subsection 1” or more briefly “2.1”. URLs for features, such as “Journal” or “Mindfulness”, were renamed as such.

The next data processing step is to identify sessions, defined as repeated interaction with the system over an interval of time. To avoid counting periods when the user was not actively engaged with the system, any period of inactivity longer than a threshold starts the count on a new session. The threshold is configurable, and for the examples presented here, a value of 60 minutes is used. The threshold has been set to make allowance for users watching videos repeatedly or taking time to compose longer textual entries for certain exercises. As session time is not a feature in these visualizations, the impact is only in the segmentation of the data into different sessions. A shorter threshold may be appropriate for some interventions or for visualizations where session time is the focus to avoid overestimation of session time.

Each of the individual visualizations entail further transformations of this data, building on these first two steps. These included extracting usage days and first and last content-viewing events within each session for example.

The programs used to produce the visualizations in this paper were developed in the processing language and have been made freely available. Applying these visualizations to another dataset requires two elements: (1) a data file that includes user ID, time-stamp, and event identifier (eg, URL), for all actions, and (2) a key file that maps event identifiers to meaningful labels and their intended sequential use, and differentiates between content and other features. The event identifiers will commonly be in the form of URLs, but other formats might be used. Identifiers do not need to be unique—several URLs might map to the same piece of content.

One program processes the data and outputs a number of spreadsheets that can be checked for correctness as detailed in the readme file. The other programs produce the visualizations presented below. Figure 1 is the data model underlying the visualizations considered in this paper. The full code and a small amount of test data are available in Multimedia Appendix 1.

The data used to develop these visualizations are drawn from studies that used the SilverCloud platform. SilverCloud is a media-rich, Web 2.0 platform that can be used to quickly build interactive Web-based interventions for common mental health problems. It is specifically designed to improve engagement through the following design strategies:

Each module is structured in an identical way and incorporates introductory quizzes, videos, informational content, interactive activities, as well as homework suggestions and summaries. In addition, personal stories and accounts from other clients are incorporated into the presentation of the material.

SilverCloud is an interesting source of data as there are a variety of types of user interaction with the system to explore. As a platform, the focus is on the technology elements (design strategies) that structure the intervention, rather than the content. The sample platform can be used to deliver a range of programs (eg, depression [36] and anxiety [37]). A more detailed description of the implementation for depression (MindBalance) is available in Sharry et al [35], including a video overview.

There are three aspects of the SilverCloud platform that feature prominently in the visualizations we discuss in this paper. First, user navigation possibilities are multiple. It is possible to take a linear approach, clicking the forward arrow to get to the next page or activity planned by the content designer. It is also possible to choose one’s own path through the intervention, selecting which module and submodule to start with. The latter approach is similar to navigating a webpage.

Second, apps are used to support the engagement with content. These include an interactive journal, recordings of mindfulness activities, as well as interactive activities, such as creating a “Thoughts, Feelings, Behaviors” cycle. Some of the apps contain material that can be downloaded and used offline. Third, support is provided by weekly reviews. The client’s supporter, generally a health professional, reviews the content that the client has shared and provides encouragement and guidance. A supporter might recognize and articulate the efforts a client has made, or suggest a particular content page or activity.

Figures 2-4 illustrate the three main aspects of the user interaction with SilverCloud just described that will be discussed in the example visualizations: (1) user navigation, (2) apps, and (3) reviews.

Four visualizations are presented in this section. The first, Navigation Graph Visualization, illustrates the temporal process of engagement by an individual. The second, third, and fourth visualizations aggregate specific aspects of this first one over an entire cohort. Stripe Graph Visualization illustrates the temporal pattern of use over the intervention period for individual users but is compact enough to be stacked, allowing usage across a cohort to be examined and compared. Start–Finish Visualization depicts the shape of users’ sessions across a cohort, showing the starting and finishing content page of each session. Finally, Next Action Heat Map Visualization aggregates the navigation path through the intervention.

We have presented four visualizations of user engagement with a Web-based intervention: Navigation Graph, Stripe Graph, Start–Finish Graph, and Next Action Heat Map. These four visualizations provide a novel way to interrogate the patterns of use through log-data of individual usage and specific aspects of cohort usage. We offer these as an alternative view on user engagement with a Web-based intervention than more commonly used summative measures. In this discussion, we articulate how understanding the process of engagement through visualizations can support the design and evaluation of Web-based interventions.

The visualizations provide examples of how those authoring interventions can inspect the data at different levels of granularity to improve engagement. The Navigation Graph draws attention to strategies that people employed to revisit material. Authors can inspect numerous examples to give them understanding of different ways that material is viewed with in the temporal relationship to the overall intervention. These patterns can then be inspected in the aggregate using the Heat Map Visualization. Alternatively, the intuition afforded by inspecting the data visually can provide the basis for constructing statistical queries on the log-data.

The visualizations of individual patterns of use afford the opportunity to consider personalization in the design process. It is striking how diverse individual patterns of use are in Figures 5-7. This finding indicates that a search for the “best” interactivity may be the wrong line of inquiry. Rather, Web-based interventions need to be designed to enable personalized usage without overwhelming the user with too many options or a lack of direction. It may also be possible to identify different styles of use that could support the tailoring of specific aspects of the intervention. As in the previous example, these visualizations have helped articulate appropriate research questions for pursing better engagement.

The visualization can also help detect usability issues that may mask the underlying effectiveness of the intervention. Both the Start–Finish Graph and the Next Action Heat Map quickly highlight large trends in cohort navigation. If many people are stopping mid-way through a module in the Start–Finish Graph, it suggests that people are losing interest at this point. It may be that the intervention designers need to consider strategies for engagement at this point. Similarly, large breaks in the Next Action Heat Map indicate that many people are following routes other than proposed, highlighting a potential usability or content issue.

Log-data visualization is likely to be most useful in conjunction with other types of data. For example, it can offer a representation to support the elicitation of experience through qualitative interviewing [38] to understand why people used the intervention a particular way. It could also be used as a method of exploratory analysis before the creation and calculation of engagement statistics, a technique currently being developed by Kelders & Gemert-Pijnen [39]. Log-data visualization can, with other methods, add to the richness of understanding engagement needed to support the iterative design approach for complex interventions [40].

Log-data visualization offers a way to inspect the assumptions of usage embodied in evaluation metrics and criteria. The visualizations presented here question the implicit assumption of linear usage that underpins summative adherence metrics such as intervention completion or module completion. For example, the Stripes Graph shows the adherence is often not consistent but comes in bursts, suggesting that measures of adherence calculated based on weekly usage should not be considered in isolation. The Navigation Graphs and the Start–Finish Graphs both suggest that module completion may not be an accurate reflection of engagement either.

Usage visualizations also offer a different perspective on interactivity. The Navigation Graph, for example, illustrates the back and forth nature of content intake and interactive activities. Comparing different users, we can also see preferences for certain types of interactivity are highly specific to the individual. This view of the data enables a more nuanced conceptualization of engagement, shifting from what content people see, to what content people interact with. This is an important distinction for researchers concerned with the quality of engagement. It also provides a view of interactivity that goes beyond its reduction to individual feature usage.

Log-data visualization also provides new opportunities for evaluation. Summative clinical trials alone are unlikely to provide the flexibility needed to address the continuing ongoing refinement of Web-based interventions. Approaches are required that can adjust to the speed of technology evolution and provide outcome data appropriate to the varied settings and configurations in which a Web-based intervention may be used in practice. Engagement visualizations can allow continual assessment of an intervention as it is changed to incorporate new technologies or adapted to new settings. Visualizations could potentially underpin a more dynamic model of evaluation. Further work would be needed to determine the best visualizations for this purpose.

The science of engagement is in its infancy. There is much that could be done in the sphere of visualization depending on the size of dataset, granularity of analysis, and motivation for looking at the log-data. Understanding the experience of an individual user (perhaps correlating with qualitative feedback) is at one end of the spectrum, whereas understanding patterns of engagement for a particular demographic would be at the other end. Other uses, such as by clients themselves to reflect upon their own patterns in relationship to the intervention are also possible.

We offer these visualizations as a demonstration of some of the benefits of understanding engagement through log-data. Specifically, we show how visualizations can (1) allow inspection of content or feature usage in a temporal relationship to the overall program at different levels of granularity, (2) detect different patterns of use to consider personalization in the design process, (3) detect usability or content issues, (4) enable exploratory analysis to support the design of statistical queries to summarize the data, (5) provide new opportunities for real-time evaluation, and (6) examine assumptions about interactivity that underlie many summative measures in this field.

We do not suggest that these visualizations are the ultimate set. Indeed, visualization will need to some extent be specific to the design intent of the Web-based intervention as it is best suited to exploratory analysis. Therefore, it is unlikely for the research community to settle on a single set of visualizations for all interventions, but we felt that having a starting point would be useful. We have made the programs freely available to encourage others to explore these visualizations with their own data.

A shared toolset within the community, which allows exploratory and not just confirmatory analysis of data, will require some degree of standardization, for example through a common format for log-data from online interventions. Common fields such as event identifier (URL), timestamp, and user identifier are clearly required, other data points might also be considered, such as self-report measures and treatment events (eg, support sessions for clients receiving blended treatment). The model presented in Figure 1 illustrates one possible arrangement of such data. This would allow researchers developing different interventions to share a common set of tools for visualizing and analyzing log-data.

Future research in this area could help to provide the rigor of development for engagement currently sought to address the issues of adherence that stymie the regular use of Web-based interventions.