This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Mass media campaigns for public health are increasingly using digital media platforms, such as web-based advertising and social media; however, there is a lack of evidence on how to best use these digital platforms for public health campaigns. To generate this evidence, appropriate campaign evaluations are needed, but with the proliferation of digital media–related metrics, there is no clear consensus on which evaluation metrics should be used. Public health campaigns are diverse in nature, so to facilitate analysis, this review has selected tobacco control campaigns as the scope of the study.
This literature review aimed to examine how tobacco control campaigns that use traditional and digital media platforms have been evaluated.
Medicine and science databases (Medical Literature Analysis and Retrieval System Online [MEDLINE], EMBASE, PsycINFO, Cumulative Index to Nursing and Allied Health Literature [CINAHL], and Scopus), and a marketing case study database (World Advertising Research Center) were searched for articles published between 2013 and 2018. Two authors established the eligibility criteria and reviewed articles for inclusion. Individual campaigns were identified from the articles, and information on campaigns and their evaluations were supplemented with searches on Google, Google Scholar, and social media platforms. Data about campaign evaluations were tabulated and mapped to a conceptual framework.
In total, 17 campaigns were included in this review, with evaluations reported on by 51 articles, 17 marketing reports, and 4 grey literature reports. Most campaigns were from English-speaking countries, with behavioral change as the primary objective. In the process evaluations, a wide range of metrics were used to assess the reach of digital campaign activities, making comparison between campaigns difficult. Every campaign in the review, except one, reported some type of engagement impact measure, with website visits being the most commonly reported metric (11 of the 17 campaigns). Other commonly reported evaluation measures identified in this review include engagement on social media, changes in attitudes, and number of people contacting smoking cessation services. Of note, only 7 of the 17 campaigns attempted to measure media platform attribution, for example, by asking participants where they recalled seeing the campaign or using unique website tracking codes for ads on different media platforms.
One of the key findings of this review is the numerous and diverse range of measures and metrics used in tobacco control campaign evaluations. To address this issue, we propose principles to guide the selection of digital media–related metrics for campaign evaluations, and also outline a conceptual framework to provide a coherent organization to the diverse range of metrics. Future research is needed to specifically investigate whether engagement metrics are associated with desired campaign outcomes, to determine whether reporting of engagement metrics is meaningful in campaign evaluations.
By 2019, advertising on the internet made up over half of all media spending in 8 countries, including the United Kingdom, China, the United States, and Australia [
Despite the increasing popularity of digital media use, there is a lack of robust evidence on how best to use digital platforms for public health campaigns, including questions around which platforms, or combinations of platforms, are most effective for driving behavioral change [
To generate this evidence, appropriate evaluations of campaigns are needed. With the proliferation of digital media platforms, metrics such as likes, engagements, impressions, and click-through rates have become commonplace in evaluations [
Given varied objectives, strategies, and activities of public health campaigns, this review focuses on campaigns relating only to tobacco control to facilitate comparison. Today, some tobacco control campaigns are among the most advanced public health campaigns in terms of funding, strategy, and evaluation, and have a large underpinning evidence base that describes effective campaigns [
This paper examines how tobacco control campaigns that use traditional and digital media platforms have been evaluated in the published literature. A better understanding of how to evaluate these campaigns will enable practitioners and researchers to develop greater insight into how to effectively use digital media platforms for tobacco control campaigns, and more widely, for public health campaigns.
Data were collected through 3 search approaches: (1) in medicine and science journal databases, (2) in a marketing case studies database, and (3) through internet searches for grey literature, campaign websites, and social media sites.
For medicine and science journals, a search was conducted using the Medical Literature Analysis and Retrieval System Online (MEDLINE) via OvidSP (Wolters Kluwer Health), EMBASE via OvidSP, PsycINFO via OvidSP, and Cumulative Index to Nursing and Allied Health Literature (CINAHL; EBSCO) and Scopus (Elsevier). The search strategy used the following terms: (smok*.mp OR tobacco/) AND (campaign.mp OR mass media.mp) AND (digital.mp OR online.mp). Search results were limited to articles in English and published in the last 5 years (2013-2018). This timeframe was selected to ensure the relevance of this review because of the fast-changing nature of digital platforms and their usage patterns.
The review was supplemented with a search of the marketing database WARC (World Advertising Research Center). For this search, the keyword terms were smoking OR tobacco, with results limited to the last 5 years, within the
Subsequently, the reference lists of included articles and systematic reviews identified in the literature search were reviewed for additional relevant references.
The first stage of this review involved 2 authors (LC and BH) independently reviewing the same subset (25%) of all identified database search results to establish and test the eligibility criteria (see
The second stage of the literature review involved the identification of individual campaigns from the included articles (see
Flowchart of search strategy and campaign selection.
All articles identified throughout the data collection process were recorded using Endnote (Version X8, Clarivate Analytics). Information from multiple sources was then tabulated by campaign to provide a complete picture of the evaluation measures and methods used by each campaign. To provide context for the evaluations, data on each campaign’s objectives, target audience, and details of media usage (both paid and unpaid) were also collected.
To summarize evaluation measures used by different campaigns, data were mapped to a conceptual framework (
Conceptual framework of campaign evaluation metrics and measures
Process evaluation | Impact evaluation | Outcome evaluation | |||
|
Awareness | Proximal impact I: Engagement | Proximal impact II: Priming steps | Distal impact: Trialing behaviors |
|
|
|
|
|
|
|
|
Seen the campaign and perception of the campaign | Showing interest in the campaign or message by taking an action | Priming steps of behavioral change | Initial trialing behaviors and antecedents of behaviors | Desired behavioral change |
Delivery of Television ads (Target Audience Rating Points [TARPs] or Gross Rating Points [GRPs]) Other |
Campaign recall (including frequency)
|
|
Knowledge and beliefs Attitudes: about smoking, tobacco industry, etc Attitude: intention to quit Information-seeking action offline (spoke with health care provider) |
Contact smoking cessation service or registrations to service Quit attempts |
Sustained quit attempts Population smoking prevalence rates (For nonsmokers): Conversation with family or friend about smoking cessation |
aAll italics indicate metrics and measures that relate to digital media platforms.
bIn this review, media channel attribution and campaign responses were measured through both digital platform evaluation methods and traditional evaluation methods.
The medicine or science database searches identified 336 articles. After removal of duplicates, 208 articles were screened. This identified 49 articles for full-text review, and subsequently 24 articles were included in this review. The marketing database search identified 73 reports, and after review, 26 were included. From hand-searching references of the included articles, 30 additional articles were identified for this review (see
After further searches for more information about the identified campaigns in grey literature reports, campaign websites and social media pages, 6 campaigns were excluded for the following reasons: insufficient information about the campaign, insufficient information about the digital aspects of the campaign, lack of evaluation data, campaign related to e-cigarettes, and intervention assessed as not primarily a campaign. As a result, 17 campaigns were included in this review, reported on by 51 peer-reviewed articles and 22 marketing reports. However, 5 of the marketing reports provided contextual campaign information but did not contain unique evaluation data. Therefore, the analysis of evaluations of the 17 campaigns was based on 51 peer-reviewed articles, 17 marketing reports, and 4 grey literature evaluation reports.
Of the 17 identified campaigns, 7 were only located in marketing reports and grey literature, highlighting the benefit of using these additional sources of information for this review. Of the 51 peer-reviewed articles included in this review, 29 reported on the
Most campaigns were from high-income, English-speaking countries, with 6 from the United States, 4 from Canada, 3 from Australia, and 2 from the United Kingdom. In all, 13 of the 17 campaigns had a primary objective of behavioral change, 2 were awareness-raising campaigns, and 2 were campaigns aimed at changing social norms.
The types of evaluation measures used for campaigns are summarized in
Reported evaluation measures in behavioral change campaigns.
Campaign | Process | Awareness | Proximal impact: engagement | Proximal impact: priming steps | Distal impact | Outcome |
Tips from Former Smokers | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Stop before the suffering starts | ✓ | ✓ | ✓ | ✓ | ✓ | —a |
Stoptober | — | ✓ | ✓ | ✓ | ✓ | ✓ |
The Real Cost | ✓ | ✓ | ✓ | ✓ | — | ✓ |
Be a Failure | ✓ | ✓ | ✓ | ✓ | — | — |
16 cancers | — | ✓ | ✓ | — | ✓ | — |
SmokeFree Teen | ✓ | — | ✓ | — | ✓ | — |
Fingerband campaign | ✓ | ✓ | ✓ | — | — | ✓ |
Break it Off | — | — | ✓ | ✓ | ✓ | ✓ |
Keep Trying | — | — | ✓ | — | ✓ | — |
No judgments. Just help | — | — | ✓ | — | ✓ | ✓ |
Personal Testimonies | — | — | ✓ | — | ✓ | — |
The Smoking Kid | ✓ | — | — | — | ✓ | — |
aNo data was available on these evaluation measures.
Reported evaluation measures in awareness raising and social norm change campaigns.
Campaign | Process | Awareness | Proximal impact: engagement | Proximal impact: priming steps | Distal impact or outcomes |
Truth FinishIt | ✓ | ✓ | ✓ | ✓ | ✓ |
The Facts Now | ✓ | —a | ✓ | — | ✓ |
Take it right outside | — | ✓ | ✓ | ✓ | ✓ |
Quit the Denial | ✓ | ✓ | ✓ | ✓ | — |
aNo data was available on these evaluation measures.
The conceptual framework as described in
The majority of campaigns (8/10) using digital videos reported a metric about the reach of the digital video [
The reach of web banner ads was reported as impressions or exposures by 2 campaigns [
In all, 7 campaigns evaluated whether people recalled (ie, without prompting with campaign material) or recognized (after being shown campaign material) the campaign, which was primarily measured through sampled surveys or interviews [
Proximal impact measures of engagement, such as the number of visits to a website or ad click-through rates (the percentage of times an ad is clicked) [
All but one campaign in this review reported at least one proximal impact measure of engagement. Of all the evaluation measures identified in this review, campaign website visits was the most commonly reported measure (11/17 campaigns) [
The number of times an ad was clicked or the click-through rate were only reported in 2 of the 11 campaigns that used web static banner ads (
In all, 5 campaigns reported on whether people exposed to the campaign took an intermediary action of seeking more information about the issue on the internet [
A total of 5 campaigns used other digital media–based measures as part of the evaluation of proximal impact. These included measuring mobile phone app downloads [
In all, 3 of the 17 campaigns measured knowledge-related outcomes, such as about the health-related harms of smoking or of second-hand smoke [
The number of people contacting smoking cessation services was reported in 9 of the 13 behavioral change campaigns [
Finally, 4 campaigns evaluated the number of people with sustained quit attempts [
In all, 7 campaigns attempted to measure media platform attribution, that is, where the audience was exposed to the campaign [
This review found that there is a wide range of metrics used in tobacco control campaign evaluations, as a consequence of the diversity of media platforms and activities employed by campaigns (see
Another group of metrics identified in this review were engagement metrics, which result from digital media activities, and were not present in traditional broadcast media. Examples of these metrics included likes, comments, and retweets. The sheer number of these engagement metrics is overwhelming, and it is challenging to know which are meaningful [
There are currently moves to try to create more uniform digital metrics across the board [
Metrics should be consistent with the objectives of the campaign [
Reported metrics should be the simplest metric available for reporting the intended concept, that is, the metric understood by most people. While complex metrics may help practitioners understand how campaigns are performing at the time, they are usually not widely understood. Furthermore, combined metrics, such as “the campaign produced XXX impressions in total,” should be avoided, as they are ambiguous about how the number is calculated across different media.
The conceptual framework in
Through the use of this conceptual framework to review the range of metrics, we identified strengths and gaps in the evaluations in this review. A large proportion of campaigns reported proximal impact engagement measures, such as website visits, whereas a smaller proportion evaluated proximal impact priming step measures of health-related knowledge and attitudes. The review also identified that marketing reports generally focused more on process evaluation measures and proximal impact engagement measures, whereas peer-reviewed articles focused more on priming step measures. This distinction has practical implications, as campaigns with smaller evaluation budgets often rely on marketing reports to evaluate the effectiveness of a campaign. Conversely, researchers may only look at peer-reviewed articles to identify best practice in campaign development. As all levels of evaluation are of value, it is important that the full spectrum of evaluation measures is reported to understand the effectiveness of a campaign.
Many mass media campaigns are based on behavioral change theories that have priming steps of changes in knowledge, attitudes, or beliefs as intermediary stages before the behavioral change outcome [
One of the major challenges facing practitioners is knowing where to invest resources given the diverse media landscape. The number of platforms is overwhelming, and without evidence of which are more useful at achieving campaign objectives, decisions are sometimes made based on opinions or trends. Therefore, this review examined whether campaign evaluations measured attribution, that is, how activity on each media platform used by the campaign contributed to the campaign’s outcomes. Despite this being important information, only a low proportion of campaigns (7/17) measured attribution. The methods used to measure attribution included survey self-report, using unique website tracking codes for different media format ads, and using an ecological study approach of correlating exposure of different media use combinations with reported campaign awareness and outcomes.
The majority of mass media campaigns use more than one media platform, as reflected in the campaigns included in this review. Previous research has shown that advertising campaigns on multiple platforms produces higher return-on-investment, and campaigns in sectors that are higher-involvement, such as pharmaceuticals, benefit most from synergistic campaigns using both traditional and digital media [
In all, 4 of the campaigns in this review used surveys or interviews to determine where people had encountered the campaign. However, this method has widely been found to be inaccurate, particularly where different media interact with one another or are viewed at the same time, making it difficult for people to recall where they encountered the campaign [
Given the absence of practical methods for campaign evaluators to accurately measure attribution for individual campaigns, there needs to be guidance provided to practitioners on what are generally the most effective combinations of media use. To develop such best practice guidelines, more studies examining the synergistic effects of different combinations of media platforms for public health mass media campaigns are required. The study design used by Allom et al [
One of the key strengths of this review is the use of peer-reviewed literature, marketing reports, grey literature, campaign websites, and social media sites to collect data for the campaigns. The triangulation of data provides a more comprehensive and practical view of how campaigns are currently evaluated.
This review included a wide range of campaigns in terms of scale, making comparison between campaigns difficult. However, the challenges in campaign evaluation identified in this review are common to all health-related campaigns, regardless of size and resourcing. The inclusion of English-only articles and the high representation of campaigns from English-speaking countries may limit the generalizability of this review’s findings and miss potential advances in non-English speaking countries. In addition, the large number of evaluation studies emanating from one campaign
This review examined how recent tobacco control campaigns that used traditional and digital media platforms were evaluated. It found that in today’s fragmented and rapidly evolving media environment, a wide and diverse range of measures and metrics were used in campaign evaluations, particularly for campaign activities relating to digital media use. Purposeful selection of metrics, and utilization of a conceptual framework can help practitioners and researchers make sense of the multitude of metrics and conduct evaluations that further our understanding of how best to use traditional and digital media to communicate health messages to target audiences.
Eligibility criteria for literature review.
Tobacco control campaigns including a digital media component and their evaluation methods.
gross rating points
Medical Literature Analysis and Retrieval System Online
target audience rating points
LC received support for this research through a PhD scholarship under the Prevention Research Support Program, funded by the New South Wales Ministry of Health.
None declared.