Background: Although hundreds of millions of dollars are spent each year on public health advertising, the advertisement content, design, and placement are usually developed by intuition rather than research.
Objective: The objective of our study was to develop a methodology for testing Web-based advertisements to promote smoking cessation.
Methods: We developed 10 advertisements that varied by their content (those that empower viewers to quit, help viewers to quit, or discuss the effects of smoking). We then conducted a series of Web-based randomized controlled trials that explored the effects of exposing users of Microsoft’s Bing search engine to antismoking advertisements that differed by content, placement, or other characteristics. Finally, we followed users to explore whether they conducted subsequent searches for smoking cessation products or services.
Results: The advertisements were shown 710,106 times and clicked on 1167 times. In general, empowering advertisements had the greatest impact (hazard ratio [HR] 2.6, standard error [SE] 0.09 relative to nonempowering advertisements), but we observed significant variations by gender. For instance, we found that men exposed to smoking cessation advertisements were less likely than women to subsequently conduct smoking cessation searches (HR 0.2, SE 0.07), but that this likelihood increased 3.5 times in men exposed to advertisements containing empowering content. Women were more influenced by advertisements that emphasized the health effects of smoking. We also found that appearing at the top right of the page (HR 2.1, SE 0.07) or at the bottom rather than the top of a list (HR 1.1, SE 0.02) can improve smoking cessation advertisements’ effectiveness in prompting future searches related to smoking cessation.
Conclusions: Advertising should be targeted to different demographic groups in ways that are not always intuitive. Our study provides a method for testing the effectiveness of Web-based antismoking advertisements and demonstrates the importance of advertisements that are tailored according to specific demographics.
In the United States alone, tens of millions of public health dollars are spent annually on advertisement campaigns in the belief that it is possible to change smoking behaviors . For example, the Centers for Disease Control and Prevention’s Tips From Former Smokers campaign, which ran in 2013, cost roughly US $48 million [ ]. However, it is difficult to know which advertisements are effective and which are not. Research into the effectiveness of advertisements often relies on weaker scientific evaluation methods, such as before-and-after designs [ ] or designs that measure very long-term population-level effects.
As public health advertisements move onto the Web, however, it has become possible to conduct randomized controlled trials (RCTs)—the gold standard in research design—of smoking advertisement content and characteristics. This can be done by exposing users to advertisements, and then following these users on the Web to measure the influence of these advertisements on the users’ health risk behaviors as observed in their searches [- ]. For example, investigators can measure subsequent searches, key words used in online posts or emails, shopping behavior [ ], or exercise behavior (eg, as measured by global positioning systems), providing a robust picture of the outcomes associated with Web-based advertisements. This can be done quickly and at a fraction of the cost of a real-world trial.
The private sector has long used such inexpensive experiments to test products [- ]. For instance, Google and Microsoft often test the impact of website design or user interfaces by randomly exposing participants to different design concepts and then observing participants’ responses. For example, the New York Times tested the effects of various font types on readers’ perceptions of the validity of the same block of text by randomly changing the font and then surveying the readers about their perceptions of the text [ ]. Likewise, researchers have experimentally manipulated the number of positive or negative posts that users saw on Facebook, and then examined the emotional content of users’ subsequent posts [ ].
Search engine queries are known to reflect real-world behaviors . As such, researchers have used search engine queries to infer behaviors in the real world, both health related and otherwise [ ], and their effect on future health outcomes. The use of Web-based advertising to measure health outcomes was pioneered by Eysenbach [ ]. Web-based advertisements have also been used to measure sentiment to behaviors on the Internet [ ] and to increase consumer demand for smoking cessation interventions [ ]. However, the use of Web-based advertising to induce a behavioral change and its measure through search engine queries has not, to our knowledge, been attempted thus far.
In theory, a mix of Web-based advertisements can be developed and targeted to specific populations. Users can then be randomly shown advertisements within this mix to determine whether they had the intended effect on the intended audience. Using these data, it should be possible to continuously refine advertisements such that they can have ever-increasing effectiveness in changing individual behavior.
Therefore, the goal of this study was to take the first step toward proving this concept by developing a methodology for testing Web-based advertisements to promote smoking cessation. We conducted an RCT showing how 10 advertisements developed by a public health practitioner could influence the likelihood that Internet search engine users would subsequently conduct searches related to smoking cessation. We then asked whether some advertisements are more effective than others, and explored variation by demographic group.
We conducted an RCT using the Bing Ads system (Microsoft Corporation, Redmond, WA, USA) to randomly display 10 different advertisements, and then followed exposed or unexposed participants to explore whether they were more likely to subsequently search for ways to quit smoking. The control intervention was “usual care,” meaning users were served whatever advertisements the Bing Ads system would have otherwise served.
We asked a public health professional to design textual advertisements. These advertisements conformed to basic public health communications campaign theory. They were designed without the assistance of an advertising firm to better reflect real-life public health advertisements, which are usually not designed with such expertise. We then categorized the advertisements into 1 or more of 3 categories according to whether the text (1) empowered participants to quit (“empowering”), (2) suggested ways to quit (“helping”), or (3) discussed the effects of smoking on one’s health (“effects”). We then linked these advertisements to the most commonly visited smoking cessation websites (as determined by Microsoft’s search engine, Bing). These antismoking sites were operated by a government body, a nongovernmental organization, or a private entity, and thus were chosen because of the perceived difference in authority of each organization type.
The advertisements contained text and took the form of a title, a body, and a link to a URL (). We used a full factorial design such that we tested all combinations of title/body and URL. The advertisements were only shown to people who searched from computers located in the United States, and who performed searches on the Bing search engine (Microsoft Corporation). shows a sample advertisement. We note that the market share of Bing in the United States is around 19%, according to recent estimates [ ]. The correlation between the number of Bing users per county in the United States and the number of people in that county according to the 2010 US Census is R2=.83 (P<.001). Thus, it is estimated that Bing users are a representative sample of the US population.
|Stop It!||Smoking lowers quality of life and causes illness and death. Quit now.||Effects|
|Be a Hero. Quit Smoking.||Protect the health of you and your loved ones.||Empowering|
|Smoking Makes You Ugly||See the effects of smoking on your looks and health.||Effects|
|Quit Smoking Your Way||Find a quit method that will work for you!||Helping|
|Free Help to Quit Smoking!||Tips and tools to help you quit smoking. |
Choose what’s best for you!
|Stop While You Still Can||Tobacco use is responsible for 1 in 5 deaths in the US. Quit smoking.||Effects|
|Who Says Quitting is Bad!||Learn about the immediate benefits of quitting smoking.||Helping|
|Want to Ruin Your Life?||Smoking harms nearly every organ of the body. |
Smoking also kills.
|Quitting Smoking Is Hard||Having a plan to quit smoking makes it easier. |
Get help to prepare one.
|Need Motivation to Quit?||Learn from former smokers who have smoking-induced illness and disability.||Helping|
List of terms matched in users searches on “from smoking” on the Bing search engine.
1. cancer from smoking
2. birth defects from smoking
3. yellow teeth from smoking
4. black lungs from smoking
5. hairy tongue from smoking
6. hole in throat from smoking
7. diseases from smoking
8. hole in neck from smoking
9. damaged lungs from smoking
10. wrinkles from smoking
The advertisements were shown when users’ searches were matched by (1) broad terms (eg, those searches containing the words “smoking” or “cigarettes”) or (2) specific terms (eg, “smoking causes black lungs”). These specific terms were identified by finding the 10 most common queries submitted during November 2014 to the Bing search engine that contained the phrase “from smoking.”shows the list of terms.
The advertisements were randomly shown either directly underneath the search text (“top of the page”) or to the right of the search results (“right of the page”). The advertising system sometimes presented our advertisements in addition to other advertisements paid for by sponsors. In those cases, a placement at “location 1” meant that the advertisement was shown as the first advertisement in the list, and lower locations implied a lower order within the set of advertisements shown.
The Bing Ads system chose which of the 10 advertisements to show randomly using a random number generator. Each of the 10 advertisements had the same probability of being displayed. While it is likely that most users saw the advertisements, only some users actually clicked on it. Upon clicking, they were led to antismoking sites that we had preselected from government websites, nongovernmental organization websites, or commercial websites. We selected these websites according to their search rankings. Importantly, the sites did not contain detailed smoking cessation information. Therefore, we used future searches to test the effect of the advertisements on seeking cessation information, as detailed below.
The advertisements were shown between June 10, 2015 and September 10, 2015.
We extracted all searches conducted on Bing by people who were shown 1 or more of the advertisements we presented from 1 month before the beginning of the advertisement campaign (ie, May 10, 2015) and until 1 month after its completion (October 10, 2015). This was done to obtain a sense of the baseline number of searches and the advertisement types that these searches generated. This way, we could more easily detect a media event (eg, a celebrity death attributable to smoking) in the middle of a campaign and account for variations from baseline that could have occurred as a result.
Searches comprised the text of searches, date and time, and an anonymous user identifier. Where users had a Microsoft account and were signed in, their age and gender, as well as the zip code location of users, were also recorded.
We categorized searches according to their text into a 3-level scale: we assigned a score of 0 to searches that only mentioned “smoking” or “cigarettes,” without additional context. We gave a score of 1 to searches that mentioned symptoms associated with smoking, such as “wrinkles from smoking.” We gave searches a score of 2 if they mentioned specific diseases associated with smoking, such as “cancer from smoking.” We refer to this scale as the query damage scale.
Our primary outcome of interest was whether the user conducted at least one query, filtered to include only those queries indicative of an intention to quit smoking search (IQSS). Such queries were those that included 1 or more of the following terms: (1) direct reference: quit, cessation, stop smoking, smoking withdrawal; (2) smoking cessation medications: nicotine, bupropion, carenicline, Chantix, Buproban, Aplenzin, Wellbutrin, Budeprion, Zyban, clonidine, Catapres, Kapvay, Nexiclon, Clophelin, Nicoderm, Nicorette; (3) electronic cigarettes: e-cigarettes, electronic cigarette, vaporizer, vape, smokeless, vapor; and (4) support: support group, Smokefree, smoke free, American Cancer Society, American Lung Association.
We constructed a Cox proportional hazards model to assess the relation between the likelihood of conducting subsequent IQSS and the independent attributes shown in.
1. Attributes of the advertisements:
a. Location on the page: either above the search results or to their right.
b. Whether the advertisement was clicked on.
c. Type of match between the query and the ad keywords: advertisements can be exact (the exact phrase used for triggering the advertisements is the query) or approximate (the words triggering the advertisement appear in query, as well as other words).
d. The websites they referred to (government, nongovernmental organization, or other).
2. The search words that triggered the advertisements.
3. User demographics:
The assumption of proportionality was tested and met. To measure advertisement (and advertisement parameter) success, we further computed a conversion fraction, defined as the fraction of advertisement displays that resulted in a future IQSS. We then defined the conversion fraction ratio (CR) as the ratio of conversion fractions when an attribute was present to when it was not. Because the number of participants was very large, we only considered differences of 10% to be clinically meaningful irrespective of the statistical significance of the finding.
Institutional Review Board Approval
Our study was approved by the Microsoft Institutional Review Board and was declared exempt by the Columbia University Institutional Review Board under the understanding that the Columbia University researchers would not have access to the data and would not seek funding for the study.
The advertisements were shown 710,106 times and clicked on 1167 times. Of these showings, 171,297 were linked to 3086 users, who also conducted 663,493 searches on Bing.shows statistically significant advertisement and user attributes relative to the hazards of IQSS for those who were exposed to a smoking cessation advertisement of any type relative to those who were not exposed to an advertisement. shows that advertisements placed on the top right of the page were twice as likely (hazard ratio [HR] 2.11, standard error [SE] 0.067) to induce subsequent searches indicating IQSS (a search for some kind of smoking cessation product or advice). Additionally, advertisements placed below those that did not have smoking cessation content were more effective than those placed at the top of a list (HR 1.13, SE 0.020 relative to the bottom of a list). Participants were only 1% more likely to subsequently conduct IQSS for each time that they were exposed to an advertisement (HR 1.01, SE 0.001). This suggests that 10 advertisements would be needed for a 10% increase in near-term smoking cessation-related searches.
Men were less likely than women to have future IQSS (HR 0.24, SE 0.073). Older participants who were exposed to smoking cessation advertisements were 2% more likely to subsequently search for IQSS for each additional year of the participant’s age than were those who were not exposed to such advertisements in a linear fashion (SE 0.002).
Advertisements that had empowering text, those that had text that suggested help, and those that discussed the effects of smoking tended to induce more IQSS. Websites produced by nongovernmental and commercial websites were more likely to induce IQSS than were governmental websites overall.
|Variable||Hazard ratio||SEb||P value|
|Ad shown on top of page (to the right)||2.11||0.067||<.001|
|Ad rank (top to bottom of list)||1.13||0.020||<.001|
|Number of previous exposures to ad||1.01||0.001||<.001|
|Empowering ad? (Not empowering)||2.57||0.092||<.001|
|Helping ad? (Not helping)||2.46||0.115||<.001|
|Ad discussing effects? (Not effects)||1.36||0.138||0.03|
|Government website (Not government)||0.55||0.106||<.001|
|NGOc website (Not NGO)||0.63||0.094||<.001|
|Query damage scale||0.55||0.091||<.001|
aHazards reflect exposure to the treatment (smoking cessation advertisement) relative to the control (no advertisement) by selected advertisement characteristics (reference characteristic). Only statistically significant variables are shown.
bSE: standard error.
cNGO: nongovernmental organization.
Recall that the CR refers to the likelihood of IQSS when an attribute is present relative to when it is missing. A higher CR indicates that more users shown the advertisement subsequently conducted an IQSS. Asshows, men were 3.5 times more likely to respond to empowering advertisements than to advertisements in other categories. Men were also more than twice as likely to respond to advertisements provided on government websites relative to other websites. Women were roughly 1.3 times more likely to respond to advertisements that stressed the effects of smoking relative to those that did not.
We further modeled the data to predict which users would likely perform smoking cessation searches in the 2 days following exposure to the advertisement, given their personal attributes, as well as those of their search and of the advertisements. To do this, we built a linear regression model with interactions, estimating its performance as measured by the area under the receiver operating characteristic curve (AUC) using 5-fold cross-validation . The AUC of this classifier was 0.892.
After performing sequential forward feature selection , we found that by using 7 of the 13 attributes, we could obtain an AUC of 0.870. The 7 variables were advertisement shown on top, advertisement position, age, gender, empowering advertisement, URL referring to a government website, and the query damage scale.
shows statistically significant (P<.05, with Bonferroni correction) attributes. The interactions in the model reveal that people of different ages responded differently to advertisement placement. Similarly, different advertising content affected people of different genders and ages in a dissimilar manner. Thus, tailoring the advertisements to specific audiences requires the selection of very specific advertisement wording, placements, and URLs.
|Ad shown on top||.0208||0.0039|
|URL designed by government||.0383||0.0041|
|Age × ad position||.0001||0.00001|
|Age × empowering||.0009||0.0001|
|Age × ad shown on top||–.0010||0.00005|
|Age × government website||–.0008||0.0001|
|Ad position × damage scale||–.0023||0.0005|
|Ad position × empowering||.0089||0.0009|
|Ad position × ad shown on top||.0109||0.0008|
|Ad position × government website||–.0050||0.0006|
|Damage scale × ad shown on top||–.0104||0.0020|
|Empowering × gender||.0202||0.0027|
|Empowering × ad shown on top||.0172||0.0032|
|Empowering × URL is government||–.0994||0.0062|
aSE: standard error.
In this study, we explored the effect of various advertisement characteristics on participants’ likelihood of conducting future searches related to smoking cessation. Our results show that men were more likely to subsequently conduct smoking cessation searches when exposed to advertisements containing empowering content, but women were more influenced by advertisements that emphasized the health effects of smoking. We also found that women and older people were generally more likely to be influenced by antismoking advertisements, and that placement of smoking cessation advertisements in the middle of a list of unrelated advertisements can improve their effectiveness in prompting future searches related to smoking cessation.
Thus, our results indicate a good deal of variation in the likelihood of future smoking cessation searches that is explained by the characteristics of the advertisements, the characteristics of the participants, and the type of entity that produced the website. Foremost, we found that smoking cessation advertisements that empowered individuals to quit were more effective among men, whereas those that suggested ways to quit were more effective among women. Further, we found that the landing page had implications for users’ likelihoods of subsequent smoking cessation searches.
Overall, our findings suggest that targeting Web-based advertising may improve the effectiveness of those advertisements. Therefore, it is possible that relatively simple alterations in ad-serving algorithms can improve public health. Likewise, the landing page can also differ depending on the characteristics of the user. Based on our results, we would want to rely more on government websites as landing pages when the searcher is young, for instance. That women are more susceptible to Web-based advertisements (irrespective of their placement or content) is encouraging, as this is the group for which mortality has been on an unprecedented rise due to smoking .
There were some important limitations to this work. Mainly, our study was limited by a lack of comprehensive data. First, it is unclear that subsequent searches actually translated into changes in smoking behavior—and without such data, our understanding of the health consequences of advertising is limited. Second, our demographic targeting was less specific than we would have liked. In the ideal, it would be possible to use information stored on users’ computers to devise Bayesian prediction algorithms. Such algorithms can be used to devise typologies for individual users (not just age and gender), and then to design much more targeted advertisements for such users.
However, there remains a clear challenge in understanding the influence of Web-based advertising on public health. Although private corporations conduct tens of thousands of such experiments each year, academics are required to obtain the consent of all participants in human subjects research. Even with informed consent, the privacy challenges associated with working with potentially identifiable data are daunting. For these reasons, academics who would be naturally positioned to better understand the public health consequences of Web-based advertising targeting cannot directly engage in such research. In this way, institutional review standards that have not pivoted to accommodate the volume and availability of data in the Internet age limit the role of those trained and experienced in public health research.
Our findings—that the characteristics of the advertisement’s content, the placement of the advertisements, and the entity producing the advertisement should vary by the demographic characteristics of the searcher—are an important first step in demonstrating the power of Web-based advertising. Future research should seek to refine advertisements based on demographic data, as well as to improve their design and delivery with the support of advertising professionals. Ultimately, this work has the potential to improve the effectiveness and efficiency of public health advertising to promote healthy behavior and mitigate chronic disease.
All funding for this research was provided by Microsoft Research. The investigators performed this work as part of their salaried research.
Conflicts of Interest
The authors report no conflicts of interest.
- American Lung Association. State funding for tobacco prevention & cessation programs. 2013 Jul. URL: http://www.lungusa2.org/slati/reports/funding-for-tobacco-prevention-and-cessation-programs-fact-sheet-fy13.pdf [accessed 2016-09-25] [WebCite Cache]
- Xu X, Alexander RL, Simpson SA, Goates S, Nonnemaker JM, Davis KC, et al. A cost-effectiveness analysis of the first federally funded antismoking campaign. Am J Prev Med 2015 Mar;48(3):318-325. [CrossRef] [Medline]
- Zaza S, Briss P, Harris K, Agency for Health Research Quality. The Guide to Community Preventive Services: What Works to Promote Health?. New York, NY: Oxford University Press; 2005.
- Kohavi R, Round M. Emetrics Summit 2004: front line Internet analytics at Amazon.com. 2004. URL: http://ai.stanford.edu/~ronnyk/emetricsAmazon.pdf [accessed 2016-09-25] [WebCite Cache]
- Kohavi R, Crook T, Longbotham R. Online experimentation at Microsoft. Presented at: Third International Workshop on Data Mining Case Studies; 2009 June 28-July 1; Paris, France URL: http://www.exp-platform.com/Documents/ExP_DMCaseStudies.pdf
- Tang D, Agarwal A, O'Brien D, Meyer M. Overlapping experiment infrastructure: more, better, faster experimentation. Presented at: ACM SIGKDD; July 25-28, 2010; Washington, DC p. 17-26 URL: https://pdfs.semanticscholar.org/181d/97d1f292074252ffee62a0bd8de284a06138.pdf
- Lewis RA, Reiley DH. Online ads and offline sales: measuring the effect of retail advertising via a controlled experiment on Yahoo!. Quant Mark Econ 2014 May 28;12(3):235-266. [CrossRef]
- Morris E. 08. Hear, all ye people; hearken, O earth. New York Times. 2012 Aug 08. URL: http://opinionator.blogs.nytimes.com/2012/08/08/hear-all-ye-people-hearken-o-earth/?_r=0 [accessed 2015-12-09] [WebCite Cache]
- Kramer AD, Guillory JE, Hancock JT. Experimental evidence of massive-scale emotional contagion through social networks. Proc Natl Acad Sci U S A 2014 Jun 17;111(24):8788-8790 [FREE Full text] [CrossRef] [Medline]
- Yom-Tov E. Crowdsourced Health: How What You Do on the Internet Will Improve Medicine. Cambridge, MA: MIT Press; 2016.
- Yom-Tov E, Borsa D, Hayward AC, McKendry RA, Cox IJ. Automatic identification of Web-based risk markers for health events. J Med Internet Res 2015 Jan 27;17(1):e29 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G. Infodemiology and infoveillance: framework for an emerging set of public health informatics methods to analyze search, communication and publication behavior on the Internet. J Med Internet Res 2009;11(1):e11 [FREE Full text] [CrossRef] [Medline]
- Yom-Tov E, Fernandez-Luque L. Information is in the eye of the beholder: seeking information on the MMR vaccine through an internet search engine. In: Proceedings of the 2014 conference of the American Medical Informatics Association. Presented at: AMIA Annual Meeting; 2014; Washington, DC.
- Graham AL, Milner P, Saul JE, Pfaff L. Online advertising as a public health and recruitment tool: comparison of different media campaigns to increase demand for smoking cessation interventions. J Med Internet Res 2008;10(5):e50 [FREE Full text] [CrossRef] [Medline]
- Schwartz E. Search Engine Land. Is Google's search market share actually dropping? Redding, CT: Third Door Media, Inc; 2015 Dec 10. URL: http://searchengineland.com/googles-search-market-share-actually-dropping-237045 [accessed 2016-11-14] [WebCite Cache]
- Duda R, Hart P, Stork D. Pattern Classification. New York, NY: John Wiley & Sons; 2001.
- Wang H, Preston SH. Forecasting United States mortality using cohort smoking histories. Proc Natl Acad Sci U S A 2009 Jan 13;106(2):393-398 [FREE Full text] [CrossRef] [Medline]
|AUC: area under the receiver operating characteristic curve|
|CR: conversion fraction ratio|
|HR: hazard ratio|
|IQSS: intention to quit smoking search|
|RCT: randomized controlled trial|
|SE: standard error|
Edited by CL Parra-Calderón; submitted 28.08.16; peer-reviewed by V Agarwal, D Lee; comments to author 20.09.16; revised version received 25.09.16; accepted 28.10.16; published 21.11.16Copyright
©Elad Yom-Tov, Peter Muennig, Abdulrahman M El-Sayed. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 21.11.2016.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.