The messaging database included 261 messages that were developed in our previous RCT and included both expert-written messages and peer-written messages [17]. Messages written by experts (study designers, behaviorists, physicians, nurses) were developed through an iterative expert group review process. The creation of these messages was informed by current guidelines [18] and Social Cognitive Theory (SCT) [19]. The current guidelines provided evidence-based content on successful cessation strategies. The SCT, which incorporates vicarious learning and verbal persuasion, informed the content of the expert messages [17]. Messages reflected theoretical determinants of quitting, such as positive outcome expectations and self-efficacy enhancing small goals [19]. Peer-written messages were written by current and former smokers responding to an online survey that presented four scenarios tailored by gender, age, and readiness to quit, and solicited their responses. These messages were then reviewed for use in our system. More details of our methodology to generate peer-written messages have been previously published [17]. Peer-written messages included the more “social” and “real-life” aspects of smoking cessation and represented the day-to-day issues associated with smoking cessation and the social and interpersonal influences on quitting. Such messages align with the concepts of SCT in which the physical and social environment influences individual behavior change [17].

As noted, our comparison standard CTHC was a rule-based system that tailored messages based on a smoker’s readiness to quit. We had previously demonstrated this system to be effective in a large, nationwide RCT (N=900) compared to a robust website control without tailored messages. This website control included such functions as risk, decisional balance, cessation barrier calculators, games linking the chemicals in smoking with their other uses (eg, formaldehyde is used in both cigarettes and in embalming), and a library of informational resources about smoking [10]. In the RCT, two emails were sent in the first 2 weeks, followed by one email every week until 6 months postregistration. Using a 6-month, 7-day point prevalence cessation outcome, smokers who received the motivational messages were assessed to be more likely to quit than those smokers who received the control website (OR 1.69, 95% CI 1.03-2.80) [10]. For this study, we again used this standard CTHC and messages were sent daily to smokers.

We selected this system as our comparison for multiple reasons. Firstly, it allowed isolating the effect of the message selection because the motivational messages’ content was the same for both systems. If we compared it to another system with different motivational messages content, estimating whether the differences between the two groups were due to message selection or the content of the two systems would have been challenging. Moreover, using an effective system provided a rigorous comparison for our system. At the time of the study design, there was no other online motivational messaging system with this level of effectiveness data.

The only difference between the comparison and intervention conditions was that the intervention smokers received motivational messages tailored by the PERSPeCT recommender system. Recommender systems can be implemented using either a content-based [20], collaborative filtering [21] or a hybrid approach [22]. PERSPeCT was implemented as a hybrid recommender system. Given a sample of rating data, content-based recommender systems can learn a function and match users to items based on the provided user profile information (ie, age, gender) and the metadata description of the item or message. Metadata is defined as data about data; it describes the structure or content of a particular resource, object, or entity [23]. Our coding of the messages by the readiness to quit categories in the comparison standard CTHC system is an example of the type of metadata that can be used by content-based recommender systems. Content-based recommender systems work similarly to standard CTHC systems, but the matching function can be optimized based on rating data instead of specified by experts.

In contrast to content-based recommender systems, collaborative filtering recommender systems match users to items based on past rating history. The simplest examples of this approach are nearest-neighbor methods [21]. These methods match a target user with other users that have given similar ratings to the items the users have rated in common. The set of users matched to the target user are referred to as the target user’s nearest neighbors. The method then recommends items to the target user that their neighboring users have rated highly. The assumption behind these methods is that if two users are observed to have close agreement on the ratings of a sufficiently large number of items, they will likely agree closely on the ratings for the remainder of the items.

For PERSPeCT, we chose a hybrid approach because they merge the strengths of content-based and collaborative filtering recommender systems [22]. Thus, they can potentially benefit from expert-driven rules (content-based) and the recommender algorithms. We used the following data sources to develop the models for our algorithm: (1) metadata description of the messages, (2) implicit, and (3) explicit user feedback data (Figure 1). As explained previously, our coding of the messages by the readiness to quit categories is an example of metadata. In preparation for PERSPeCT, we expanded this metadata to include constructs from multiple behavioral theories, such as the SCT, the Transtheoretical Model, and the Theory of Reasoned Action [24]. We also coded the messages for content that may be pertinent to a specific user, including health and lifestyle status, health issues, and treatment options. In total, 40% (102/261) of messages had motivational content, such as reasons to quit, and 53% (139/261) of messages had information about behavioral treatments, such as substitution and distraction.

Implicit feedback data are derived from user actions (ie, website view patterns of each individual accessing the system). As our implicit feedback data, we used the website return data of 900 smokers that participated in our prior RCT [10]. When an email was sent to these smokers, we tracked their website usage in the days following the email. Thus, we had data on the frequency at which each message promoted engagement on the website and the characteristics of the smokers that received these messages.

Explicit feedback data consists of self-reported item ratings (ie, ratings provided by users for items like books or movies, often on a five-star scale). For companies such as Netflix, these are likely to be user ratings of movies. As previously published, two pilot studies were used to generate the explicit feedback data for PERSPeCT [16]. We first recruited 100 current or former smokers to determine appropriate questions for collecting explicit ratings. Each participant was asked to provide ratings using a five-point Likert scale of four different aspects of messages: influence, emotional response, relevance, and preference. Each participant provided ratings for five different randomly selected messages. Per-message analysis showed a positive correlation between the means and variances of the ratings for each question, suggesting that all questions provided similar information. Thus, we decided to use only one question for our data collection pilot, balancing the need to obtain multiple ratings per user and the resulting cognitive load. We chose the influence question stated in the data collection section because this single influence question had strong predictive validity in a previous RCT [25].

A second pilot test was performed to collect a larger rating dataset to bootstrap the learning and evaluation of collaborative filtering models for PERSPeCT [16]. We recruited 846 current or former smokers from online and local sources to provide perspectives on smoking, quitting, and sociocultural contextual information and ratings of the influential aspect of the 261 smoking cessation messages. Each smoker was asked to rate 20 messages, resulting in 16,920 ratings.

We tested a number of classical algorithms to identify one that provided maximal prediction accuracy (ie, we evaluated the ability of the algorithms to generalize ratings to nontraining users). These included the following algorithms: K-Nearest Neighbor (K-NN), probabilistic matrix factorization, Bayesian probabilistic matrix factorization (BPMF), collective matrix factorization, and Bayesian collective matrix factorization. We used a strong-generalization protocol that involved completely separating test users from train users, learning a model using all the train users’ ratings, freezing all nonuser-specific parameters, and finally training the user-specific parameters on a subset of each test user’s observed ratings. To implement this protocol, we first divided the users randomly into five folds and then generated three random train and validation sets for each test fold. We further divided each test user’s ratings into five folds. To evaluate each method’s performance given varying levels of information about a test user, we evaluated all methods with five, 10, and 16 of each test user’s ratings available for inference and learning of user-specific parameters. Each test user has a constant set of four test ratings per test fold. The validation sets were used to set the hyperparameters of each method (eg, K in K-NN). Exhaustive grid search was used and the hyperparameter ranges were iteratively extended to ensure that no selected hyperparameter values occurred at the end-points of the search intervals.

In evaluating rating prediction methods, we used a range of standard performance metrics including root mean squared error (RMSE), Kendall tau-b, and normalized discounted cumulative gain. In all these tests, BPMF was identified as the best single model in our evaluation and was used in the development of PERSPeCT. For example, comparing the RMSE metric between the different algorithms, there was a small but statistically significant gap (P=.01) between the BPMF and other algorithms as determined by a paired t test with Bonferroni correction. The BPMF model estimates a probability distribution over a joint embedding of users and items into complementary latent spaces. The rating a given user supplies for a given item is approximated by the expected value of the product of the latent user and item factor vectors representing the user-item pair, with the expectation taken over the uncertainty in embeddings. Since the algorithm that only included explicit ratings of the 846 smokers performed as well as the model that included all the data sources, for simplicity we chose to develop the model with only this explicit rating. The algorithm was also programmed to choose only from among those messages that matched the participant’s readiness to quit status. Further details regarding our algorithm selection methodology are described in previously published work [16].

We dichotomized the responses to each question that assessed the perceived influence of the intervention and used the chi-square statistic to test for differences. We conducted an additional sensitivity analysis of the perceived influence of the system stratified by the demographic characteristics (eg, age, gender, education, and readiness).

At 30 days, we evaluated change in smoking status compared to baseline. By randomization, we assessed change in smoking status (baseline to follow-up) using the chi-square statistic. Additionally, we assessed differences between the intervention and comparison groups of 30-day cessation using the chi-square statistic.