A literature search was conducted in July 2016 using the databases PsycINFO, Web of Science, PubMed, ABI/INFORM, Communication Source, and the journal Tobacco Regulatory Science. Searches included a Twitter term and a tobacco term: (Twitter OR tweet) AND (tobacco OR nicotine OR...) (Table 1). Tobacco terms were selected based on an article on noncigarette tobacco products [12] and the US Food and Drug Administration’s (FDA) article, Recognize Tobacco in Its Many Forms [13]. Searches were limited to peer-reviewed journals in English published from January 2006 to those available in July 2016. The beginning date was selected because Twitter was launched in 2006. The initial search produced 274 nonduplicate articles (Figure 1).

One coder independently reviewed all titles and abstracts and selected 43 articles from the initial 274 that referenced any of the tobacco products and Twitter. Full text of the articles was then reviewed. Explicit inclusion criteria were determined a priori to reduce coder bias when selecting articles. Articles were included in the review if they met the following criteria: (1) original research in a peer-reviewed journal, (2) focused on tobacco, (3) analyzed Twitter data, and (4) coded data categorically (eg, sentiment, theme). If an article was excluded for failing to meet multiple inclusion criteria, it was counted in the higher-order criteria for exclusion. A total of 16 articles were excluded, so 27 articles were included in the review (Figure 1). We excluded 1 article because it was a narrative review rather than original research, 2 articles because they did not address a tobacco product, 1 article because it analyzed Web search results and tobacco control websites rather than Twitter data, and 12 articles because they did not categorically code tobacco-related Twitter data. For example, a study may have tracked changes in the number of tweets over time but not coded the tweets categorically. The 27 included articles ranged from 2011 to 2016. Although there is risk for bias in studies, this review considered all studies that met the inclusion criteria to evaluate the strengths and weakness of all methodological approaches within this domain. The primary focus of this review is the methodology of qualitative studies rather than the outcomes of quantitative studies. Therefore, the risk of publication bias of studies with significant results and selective reporting of significant results is minimal for this type of review compared with a meta-analysis of quantitative outcomes.

One author read each article to extract methodological information on data collection (data sources, date collected, tobacco topics, search keyword selection), coding methods (type of coding method, number of coders, number of tweets coded, coded retweets, number of Twitter accounts, followed URLs, coding agreement), and coded categories. This information is presented in detail in Tables 2, 3, and 4 and is summarized in the results.

Coding methods for the studies included hand-coding, machine learning, or a combination of the two. Hand-coding involves one or more human coders categorizing data. When 2 or more coders independently code data, a coding agreement score (eg, kappa) between the coders can be calculated. In contrast, machine learning uses an algorithm for a computer to learn how to code data. However, human-coding is used for an initial subset of data to help refine the algorithm to improve its accuracy. Coding categories may be determined a priori based on prior research or they may be developed inductively through the process of coding. Studies used hand-coding only (n=17), machine learning + hand-coding (n=8), and machine learning only (n=2; Table 3). Of the studies using hand-coding, data were coded by the reviewed studies’ researchers in 16 studies, while 1 study used crowdsourcing (ie, many Web workers) [30]. All the studies that used machine learning also used initial hand-coding for a subset of the data, except for 2 studies that used topic modeling [31,32]. Topic modeling produces thematically related word clusters from the text [31].

The data collected in the hand-coded studies ranged from a collection period of 1 day to 1 year, while the machine learning studies ranged from 1 month to 5 years of Twitter data. Across the articles that used hand-coding, the number of coders per tweet ranged from 1 to 6. Coding agreement was reported by 20 articles: coding agreement percentage (n=5; 72% to 95.7%), kappa scores (n=13; kappa=.64 to 1.00), Cronbach alpha (n=1; alpha=.61 to 1.00), and both coding agreement percentage and kappa scores (n=1). Kappa values can be interpreted as poor (<.20), fair (.21-.40), moderate (.41-.60), good (.61-.80), and very good (.81-1.00) [51]. Cronbach alpha can be interpreted as unacceptable (<.50), poor (.50-.59), questionable (.60-.69), acceptable (.70-.79), good, (.80-.89), and excellent (>.90) [52]. However, it is important to note that these thresholds are not derived statistically but instead rely on intuitive judgments. The number of tweets coded per study ranged from 171 to 17,098 for hand-coding and from 7362 to 1,669,123 for machine learning. A total of 14 articles included retweets in their total number of tweets. A total of 15 articles reported the number of unique Twitter accounts, which ranged from 2 to 3804 for hand-coding studies and from 23,700 to 166,857 for machine learning studies (see Table 3).

URLs in tweets can provide information that changes the context or meaning of a tweet. Following URLs to their respective webpages can be time-consuming, but it can increase coding accuracy. Machine learning algorithms can analyze the text within URLs but may require human coders to follow them to their respective webpages. A total of 15 articles reported whether they followed URLs (followed: n=10, did not follow: n=5; Table 3). One article provided the most common URLs [26]. The studies tended to show that advertising or commercial tweets were significantly more likely to contain URLs than other types of tweets [25,33,34].

All the studies developed categories for content. These content areas included one or more of the following: sentiment, theme, location of use, user description, profile photo, or location of user (Table 4).

Studies analyzing tobacco-related Twitter data have grown in number in recent years. Although we searched for articles published from 2006 to 2016, articles meeting inclusion criteria for this review were published from 2011 to 2016, with 85% (23/27) of the publications occurring in 2014-2016. Widely used among adolescents and black and Hispanic individuals, Twitter could provide a rich source of tobacco-related data among these groups. One of the benefits of Twitter research is the ability to focus on emerging issues and products that are not yet addressed in surveillance or epidemiological research. Collecting tobacco data in real time via Twitter could be a useful tool for tobacco surveillance, which could help inform tobacco control policies and social media campaigns.

Twitter data can change rapidly as they are being posted daily [37]. If a study only collects data from one point in time, it may not be reflective of data at any other point in time. Twitter studies should consider collecting data at multiple time points or over longer periods of time to decrease the likelihood that results are idiosyncratic to that point in time. However, there may be some research questions that only require sampling one time or sampling directly before and after an event to gauge short-term responses.

The primary sources of data were Gnip and Twitter’s Streaming API, which offer different strengths and weaknesses. If a study wants access to all tobacco-related tweets, then Gnip may be more effective. However, if a study is interested in tweets about a specific tobacco campaign or has a constrained budget, then Twitter’s Streaming API may be a better data source. Alternatively, if a study is focusing on multiple social media sites (eg, Twitter, Facebook), then using Radian6, Simply Measured, or Sysomos Heartbeat may be appealing because of their cross-platform analysis.

The results of this review suggest that there are some gaps in the types of tobacco products studied by Twitter analyses. E-cigarettes were the product that was addressed by most studies, while none of the studies focused on smokeless tobacco, snus, bidis, or kreteks. The focus on e-cigarettes over other products could be due in part to their recent rise in popularity and recent debates about policies. From 2011 to 2014, e-cigarette use among high school students in the United States significantly increased from 1.5% to 13.4% [53]. Similarly, there was a significant increase in hookah use from 4.1% to 9.4%. However, there were significant decreases for cigarette (15.8% to 9.2%), snus (2.9% to 1.9%), cigar (11.6% to 8.2%), pipe (4.0% to 1.5%), and bidi (2.0% to 0.9%) use [53]. It could be useful to conduct studies comparing tweets about cigarettes, e-cigarettes, and hookah among adolescents to help understand the changing rates of use and Twitter postings that discuss issues related to use. The rates of e-cigarette, hookah, and cigar use among high school students may be affected by the FDA regulations of these products that went into effect on August 8, 2016. One of the provisions of these regulations is that it will be illegal to sell e-cigarettes, cigars, and hookah tobacco to persons younger than 18 years [54]. Researchers may want to analyze e-cigarette, cigar, and hookah tweets before and after August 8, 2016, to gain real-time insight into adolescents’ reactions to the new regulations.

Future Twitter studies could benefit from a standard of reporting data collection methods. Only 4 of the articles in this review reported such rigorous methods of selecting their search keywords as suggested by Kim and colleagues’ [11] search filter framework, while 15 articles reported retrieval precision and 4 reported retrieval recall. The terminology around newer products such as e-cigarettes is growing, so it may be difficult to capture all relevant Twitter conversations with one’s keywords [25]. Future studies will need to continue to refine and expand search keywords.

The methods of coding were hand-coding, machine learning, or a combination of the two. Machine learning can code larger quantities of data at a quicker rate than hand-coding, but human coders may have greater discretion at coding for the complexities and subtlety of language such as humor, irony, or sarcasm. For example, algorithms developed to detect irony only retrieved 54%-57% of tweets coded as irony by multiple independent human coders [55]. Hand-coding can be subject to bias, but creating coding schemes based on prior literature and working to achieve acceptable levels of interrater reliability can help attenuate individual bias. Hand-coding allows researchers to follow URLs, which can change the meaning of the tweet. Viewing the webpage may provide additional information that may not be discernable from the URL. Studies that require determining subtle differences in context may be better suited to hand-coding a small sample of Twitter data, while studies that rely less on context could code large samples with machine learning.

The sentiment of tweets could help evaluate whether the responses to pro- and antitobacco efforts are positive or negative as a way of understanding social norms about these products. Clarity and comparability across studies could be improved if a distinction is made between attitude and emotion when coding for sentiment. For example, a tweet could be pro-vaping but have a negative emotional tone or it could be anti-vaping but have a positive tone. Only 1 article clarified the meaning by making an explicit distinction between coding for emotional sentiment and message attitude [28]. This is reflective of coding for stance (in favor of, against, or neutral) versus sentiment (emotional tone) [56].

The three most common themes used for coding were advertisements or marketing; health, safety, harms; and use. Surveillance of these themes could be beneficial to understand whether tobacco advertisements are being circulated on Twitter with the potential of reaching underage individuals, whether the content is making unproven claims about the health and safety of their products, or promoting the use of their products to vulnerable populations such as youth or ethnic minorities.

The most common user account descriptions that the studies coded for were personal accounts, industry, news, unclassified, and bots. Determining the user description and demographic information for the accounts that tweet about tobacco could help determine whether tobacco companies, pro-vaping advocacy groups, or antitobacco efforts are circulating with a potential to reach certain groups (eg, adolescents). Researchers could also monitor how news organizations are presenting tobacco-related information to the public.

It may be difficult to determine the demographics of the person tweeting or of the audience exposed to the tweets, which could be especially problematic when studies want to focus on vulnerable populations (eg, adolescents). A Twitter account could be run by an individual, multiple people (eg, vape shop employees), or bots. To reduce bias, bots should be identified and the tweets from these accounts removed from analysis or identified as tweets originating from automated accounts [57]. Only 4 studies coded for bots with a range from 6.9% to 80.7% of tweets classified as bots. Even if an account is run by an individual, a Twitter profile provides little information. It may or may not include a photograph, profile description, location, website, and birthday. It does not include gender or ethnicity and race, so this information needs to be estimated. Although none of the studies reported the percentage of tweets with accounts missing profile information, 5 studies coded for an unclassified category with 0.2%-38% of tweets being unable to be classified based on the account profile. If profile information is included, it could be used in combination with natural language processing to infer information about the individual from his or her tweets [29]. For example, algorithms have estimated gender with a 75.5% accuracy based on tweets and a 92.0% accuracy based on tweets, screen name, full name, and profile description [58]. A study that combined analysis of text and image processing predicted gender with an accuracy of 85.1% [59]. Additionally, algorithms based on tweets were capable of predicting the exact age of the user within a margin of 4 years, while accuracy for age categories were 93.0% for <20 years, 67.4% for 20-40 years, and 81.6 for >40 years [60].

It is recommended that tobacco Twitter studies adopt methodological standards of reporting and data quality assessment. Important information to consider reporting include data sources (eg, Gnip, Twitter’s Streaming API), the date range of tweets collected, the number of tweets coded, whether retweets were coded, whether coders followed URLs, whether images were coded, the categories coded, the decision criteria for each category, the number of unique Twitter accounts, and the types of Twitter accounts. Studies that use hand-coding should also consider reporting the number of independent coders, the number of tweets coded across coders, and their coding agreement, while it is important for machine learning studies to detail the development and refinement of their algorithms. Providing this information is likely to increase comparability across studies and the ability to replicate results.

Depending on research goals, studies may want to sample Twitter’s databases multiple times, especially if they collect from Twitter’s Search API or Streaming API, which put limitations on the amount of data that can be collected per sample. Sampling at multiple times is also important for studies that have access to Twitter’s Firehose through Gnip, considering that tweets are in real time, which could lead to fluctuations based on real-world events (eg, the FDA’s new regulations for e-cigarettes). Following messages over time could also help establish trends in the content of posts. However, some studies may be interested in short-term reactions to an event, which requires different sampling methods than following trends. For example, if a study is interested in short-term reactions to the FDA’s new regulations for e-cigarettes, then a single sample before and after the implementation of the new regulations could be sufficient.

When coding for sentiment, researchers could improve clarity by making a distinction between whether they are coding for message attitude or for emotional tone [28]. A clear distinction between the two could improve comparability of sentiment ratings across studies.

If relevant to the research questions, Twitter studies may want to code images and URLs. This added step can be time-consuming, but doing so could change the context or meaning of a tweet. For example, an image could help determine if a tweet should be coded as humor or sarcasm, while following a URL could help determine if the tweet is an advertisement. Failing to code images and URLs could result in missing significant content that could affect coding accuracy and skew results.

Analyzing Twitter user profiles could provide context for tweets. The same pro–e-cigarette tweet could hold very different significance for a study’s results if it is from a vape shop versus an adolescent. However, the limitations of analyzing user profiles should be recognized and steps taken to improve accuracy. It should be noted that user profiles may be misleading (eg, a tobacco industry representative posing as an unaffiliated citizen) or profiles of bots. Bots should be identified and potentially removed to reduce bias and improve the quality of data [57]. We cannot assume that the analysis is describing individuals but must instead consider the poster to be an “account” rather than a person.

A limitation of Twitter data is that it does not provide much information on the effects of tweets on behavior. For example, tweeting about tobacco use does not necessarily mean that the person tweeting uses tobacco [40]. Additionally, an individual may tweet about a quit smoking campaign with positive sentiment, but that does not mean that the campaign has influenced his or her smoking behavior. Follow-up studies with the individuals tweeting about the campaign would need to be conducted. None of the studies in the review included follow-up survey studies with individuals from their Twitter sample. Although recruitment of individuals through Twitter may come with its own set of obstacles, this could be an avenue for future Twitter research. Following specific individuals over time could allow for the analysis of changes in message content.

There are several limitations of this systematic review. First, the results are limited to the databases and search keywords selected, which could have resulted in incomplete retrieval of identified research. Second, this review is limited to its inclusion criteria and the decision rules of the single, independent coder who selected the articles and extracted the data to be included in the review. Individual bias was limited by explicit inclusion, exclusion, and data extraction criteria. However, some studies that were included or excluded for this review may have varied given different inclusion criteria or a different coder. Finally, this review is limited to methodology of categorically coded tobacco Twitter data. Different methodological results and recommendations may have been made if the topic of Twitter data had been different (eg, marijuana) or if the review had focused on different outcomes (eg, popularity of tweets or diffusion of tweets).

Categorically coded Twitter research can be used for certain insights that other survey research does not provide: emerging issues, popular content in real time, changes over time, how tobacco companies and pro-vaping advocacy groups use social media to increase message exposure in the population (eg, youth who might otherwise be protected from tobacco marketing), how tobacco control policies and campaigns can most effectively use social media, arguments by groups that may be incorporated into media message design, and quick reactions to antitobacco media campaigns and regulations. There are several approaches that researchers are taking to this end, each having its own set of strengths and weaknesses. Standards for data collection and coding should be developed to more easily compare and replicate tobacco-related Twitter results. Additional recommendations, dependent on one’s research goals, include the following: sample Twitter’s databases multiple times, make a distinction between message attitude and emotional tone for sentiment, code images and URLs, analyze user profiles, and identify and remove bots.