School-based alcohol, tobacco, and other drug (ATOD) surveys are internationally recognized as one of the most important ways of understanding ATOD use and related risk behaviors among young people [1]. In the United States, such surveys often include questions on attitudes (eg, perceived harm of substance use) and behaviors (eg, bullying); they are also used to monitor national trends [2] and provide aggregated information to school districts, states, and communities [3]. Administration of these surveys continues to be important, especially in the midst of a national addictions crisis that affects youth and adolescents [4] and the continually changing patterns of substance use and risk behaviors in this population, such as the recent surge in e-cigarette use [5]. Survey data and reports are also valuable in supporting continued efforts to design effective state and local education standards for substance use prevention [6] and evidence-based prevention strategies [7].

Though these surveys are designed for public health surveillance and prevention, there is a risk that they will be misunderstood as experimental research being conducted on youth, which occupies a more controversial space in public discourse [8]. Such misunderstandings, in conjunction with potentially contentious questionnaire topics like sexual health and substance use, have led to federal lawsuits against schools for administering surveys; however, two notable federal appellate cases had been resolved prior to 2007 in favor of the school districts [9]. Nonetheless, public health surveys continue to be attacked in traditional and online media [10-12], although such instances are rarely documented in academic literature. At times, clusters of incidents have also been captured in blog posts advocating for a specific position on school-based surveillance surveys [13]. School responses to such events, documented in these stories and posts [10-13], have ranged from a simple apology to cessation of survey administration.

Although little research has focused on the outcomes of public controversy on school-based surveys, such negative exposure is known to have a “chilling effect” on public health scholars in general, leading to self-censorship, modification of agendas, and even career changes [14]. Controversy can also serve as an informal social control on the types of information that is acceptable to collect, even for the purposes of harm reduction [15]. When criticisms emerge from outside of academia, they tend to involve social media, blog posts, and occasional extreme reactions, such as threats of violence or demands that a particular scholar be terminated [16]. In fact, the emergence of social media has fundamentally changed how such controversies play out, with new potential for “huge waves of outrage” to emerge rapidly in an “online firestorm” [17].

Rather than stemming from fundamentally irrational behavior, online firestorms often manifest as aggressive forms of sousveillance [18] that “use the available masses of weak ties in social media to publicly enforce social-political norms” [19]. While similarities have been drawn between firestorms and moral panics, a recent study suggested that “social appropriateness of attacking the denounced actor” was an important antecedent of firestorm participation, as opposed to a predetermined moral stance [20]. In fact, the piling on behavior observed in firestorms is a fundamental component of moral grandstanding—wanting to be perceived as moral by others [21]. Although firestorms may not promote long-standing change in public discourse [22], they can result in significant short-term consequences. This can be observed, for example, in the case of Justine Sacco, a communications executive who was publicly terminated after a firestorm erupted in response to a tweet she sent in 2014 [23]. These consequences can even accrue when the firestorm is based on misleading or false information, as with Dominique Moran who was fired in response to social media outrage accusing her of racism, though she was offered her job back after the full spectrum of evidence was presented [24].

In 2015, an annual ATOD survey was administered by local school officials to students in grades 6 through 12 in several hundred schools in a midwestern state. Participation in the survey by any school was voluntary. Schools that elected to participate were told that they had the choice of either electronic or paper administration formats; they were informed of their responsibility to seek either active or passive consent from parents and guardians for their children to participate in the survey. More than 25% of all schools in the state participated, and the aggregate student body of participating schools was predominantly white (approximately 70%), generally reflecting the state population.

As part of this process, the finalized survey instrument was made fully and openly available on a website for parental review for several months prior to survey administration, and schools were instructed to direct parents to that website. Schools were also instructed that at the time of survey administration, students should be told that completing the survey, as well as responding to individual survey items, was voluntary, even if their parent or guardian consented to their participation. At the beginning of the survey instrument, participants were prompted to indicate their date of birth, the initials of their first and last names, and the first letter of the school they attended at the beginning of the first grade. These data were to have been used to generate a confidential survey ID for the purposes of matching data between years, but were not designed to identify individuals, nor did the survey administrators have the ability to do so. This was the first year that this long-standing survey attempted to ask these questions, the purpose of which was to support confidential longitudinal data analyses. This information would have been used, for example, in identifying a particular grade or cohort with unique risks, which is commonly done in many areas of public health, for example, in Willets, 2004 [25]. Once the pilot test was fielded, the item asking for date of birth and initials raised parental curiosity. One of the families contacted a traditional media outlet, which aired a local television story about the survey.

The focus of the news story was not the new questions, though that was what had sparked interest. Instead, parents were mostly upset with several standardized risk and protective factor questions from the family domain of the Communities that Care (CTC) Youth Survey [26] having to do with “arguing about the same things in my family over and over,” “having serious arguments,” and “often insulting or yelling at each other.” Immediately following this story, the new questions, but not the CTC items, were removed from the final survey tool. Despite this, a minor firestorm erupted on social media in the comments section of the local news station’s page and a vitriolic debate ensued.

School-based surveys support harm reduction and substance use prevention efforts. The concomitant risk of public controversy and, more recently, of a digital firestorm may make school officials, epidemiologists, and public health scholars wary of advancing this important work. To better understand the nature of digital firestorms targeting school-based ATOD surveys, this study analyzes the qualitative comments made on social media during a small firestorm targeting a school-based ATOD survey administered by an independent organization and funded by a state-level government agency. The study also uses that information to make recommendations for how those who administer school-based surveys might proactively set in place protective structures that minimize the likelihood of controversy.

Data were captured by archiving all public comments posted to social media in response to the news story described above (ie, the firestorm) about a school-based public health survey. Local media featured this story on television then published the story—the only story covering this topic—to their website and two social media outlets. On one of the social media platforms, the post had no active engagement of any type. Thus, study data were drawn exclusively from the other social media platform. The thread was active (ie, the time from posting the story to social media until the final comment was posted) for approximately 35 hours. Because of the nature of online debate, at the point of archiving, several of the more incendiary posts had either been made private or had been redacted prior to capture and were, therefore, not included in the dataset. Although all data were posted in a digital public forum, to protect individuals’ privacy, this study did not identify the social media platform in question; quotes that might potentially be used to identify individuals were paraphrased or edited for publication, without changing the tone or meaning, to remove colloquialisms that would render the strings unique.

This work was granted Exempt status by the Indiana University Institutional Review Board (number: 1511842027).

More than a decade ago, researchers identified a “crisis of trust” in how the public at-large views science [27]. As lay exploration of science related to all topics, including public health, has moved to the Internet, it is probable that the level and type of interactivity matters in how content is understood and the ways that the public reacts to the content [28]. Further, comment sections in online science articles often veer from the initial story content, particularly in the direction of moral and ethical claims and community-driven narrative [29].

Because these comments were made in a public digital forum, they offered a naturalistic opportunity to explore and categorize the components of a digital firestorm targeting a public health survey tool. For this reason, we used a general inductive approach [30] informed by theoretical thematic analysis [31] to extract individual and overlapping categories from individual comments.

Data were transcribed into Dedoose and cleaned by the lead author (RAG), who then segmented comments into two broad categories: oppositional and supportive. For comments that were brief or unclear (eg, only the word “Agreed”), opposition or support was determined by context. Comments by the same person that were broken into multiple posts on the thread were counted as single comments when they focused on the same message, though this occurred very infrequently since this was a small firestorm.

A second reviewer validated comment sorting before the study moved forward. Then, within each of the two categories, the second researcher (1) completed multiple deep reads of the dataset, (2) generated preliminary themes and prepared a codebook, (3) assigned one or more codes to each social media comment, (4) revised the themes, and (5) recoded each individual social media comment. The lead author then (6) received the coded data and codebook, (7) reviewed each individual comment, (8) suggested revisions to the codebook, and (9) recoded each comment, highlighting any potential areas of disagreement. Finally, the two researchers met in person to discuss the data and continued shared analysis until 100% agreement on coding for each comment was achieved.

This study reviewed the full spectrum of social media comments posted as part of a localized social media firestorm surrounding administration of a school-based ATOD survey. As one might expect, such discourse is primed for a certain amount of controversy [33]. Analyses found that the responses were split at the broadest levels of opposition or support, though nearly two-thirds (133/207, 64.3%) opposed or criticized the survey. Further, most comments focused on the survey itself, the perceived context in which it was offered, or broader claims about society; few responses displayed individually directed flaming behavior (eg, “negative violations of...interactional norms”) [34].

Notably, only 7.2% (15/207) of the total responses contained factual information about the ATOD survey, and many of the other comments specifically relied on inaccurate assumptions about the survey. It is objectively true that the survey responses could not have been linked to individual students by the survey administrators; however, many commenters who wrote oppositional messages, especially those related to child protective services and privacy, either were unaware of this fact or did not believe it to be true. Likewise, many conspiratorial or seemingly irrational comments presupposed a broader, sinister agenda, components of which would require the ability to link survey data to individual students. At the same time, it is important to avoid “white hat bias” [35] by noting that some comments supportive of the survey were also advanced based on the same incorrect premises; in particular, supportive statements suggested that the ATOD survey could prevent individual cases of abuse or maltreatment or that parents only disliked the CTC items because they might be caught hiding harmful behavior; however, neither of those scenarios were possible.

In total, many (99/207, 47.8%) coded comments were in the following categories: parental autonomy and privacy, child protective services and police, conspiratorial or irrational, prevent abuse and neglect, or hiding something. Although there was no overarching code for misinformation, comments in each of these categories, both positive and negative, logically presupposed inaccurate information about the purpose of the survey or the survey methods. In the context of digital moral outrage [36], one interpretation of the high volume of this type of comment is that the negative comments were indicating that the survey’s perceived invasions of privacy or question content violated social or moral norms in varying ways. On the other hand, the positive comments were presenting oppositional norms in which child safety was more highly valued than privacy or preferences about the kinds of questions that should be posed to youth.

Consistent with prior research [29], some of the comments not directly related to privacy, both positive and negative, also relied on a priori personal or community-based normative beliefs. This was most commonly observed in the government related and holistic health categories, manifesting as general mistrust or disdain for the government in the former and an underlying belief or overt assertion that home life is interlinked with school life in the latter. Many comments in these categories could have been posted as template responses to any news story focusing on government activity or the role of the school in raising youth.

One important general finding from this study is that while this social media firestorm occurred in the context of a school-based ATOD survey, much of the online activity was driven by factors external to the survey itself, though the consequences to the survey administration process could easily have been substantial. In fact, organizations administering school-based ATOD surveys have a vested interest in avoiding controversy, especially in online formats. Based on the data from this study, it is tempting simply to suggest that more specific, directed information provision about the nature of the survey and the purpose of administration would serve as firestorm protection for school-based ATOD surveys. This is an especially compelling argument given that this particular firestorm included a high density of conflict (ie, prosurvey and antisurvey) stemming from norms unrelated to the actual survey itself. At the same time, such an approach likely must be nuanced and should not minimize or trivialize these norms. For example, if there is widespread pre-existing concern about privacy, in general, in a segment of the population, then it may be beneficial to avoid the appearance of violating privacy, even when such a thing could not occur in practice.

The data in this study cannot determine whether misunderstandings about the survey stemmed from a lack of information availability or from a lack of belief in available information. It is plausible that the reality is a mixture of the two. Researchers have long acknowledged that “public understanding of science, and of public risk perceptions, are not so much about public capabilities...but about the trust and credibility they are prepared to invest...” [37]. This may extend beyond conspiratorial or government-related concerns to the schools themselves, which may not be trusted [38] or which may, as seen in this study, be viewed solely as an adversarial government apparatus. Thus, while it may be necessary to thoroughly outline survey procedures and protocols for public consumption, it is likely not sufficient. This study also suggests the importance of providing clarity about the survey’s purpose, how the data will be used, how they will be safeguarded, and who oversees the survey at each level. Further, there may be value in facilitating some level of community member coconstruction of the messaging, both to engender trust and to identify potential pain points that might spark outrage [39].

There is also a documented disconnection between public outrage and objective level of risk, especially when risks are perceived as involuntary or unfamiliar [40]. It is important to acknowledge the community’s perceived risks associated with data collection, even when they may objectively be minimal, as meaningful concerns [41] and to take additional steps to mitigate or address them. The most obvious step is to avoid collecting anything that might appear to link data to participants, even when safeguards are in place to prevent that from occurring. Other strategies might also include additional or redundant layers of student protection, for example, messaging written on the survey itself indicating, “even if you have been instructed by your teacher that you must take this survey, you are not required to do so.”

Ultimately, however, prior research [15,18-21,29] suggests that online firestorms are complex and often emerge as a result of many factors outside of the context of the specific target. Data from this study do nothing to attenuate that claim, with a significant digital presence in the firestorm allocated to a priori beliefs about the government, child welfare, and society. Thus, it is likely impossible to be 100% protected against the possibility of digital outrage when conducting a school-based survey. At the same time, depending on the scope or intent of a given public health project (eg, global immunization), it may be possible to generate targeted social media messaging to assuage some concerns (eg, the World Health Organization’s syndicated, organized, social media campaigns) [42].

This study is subject to several limitations. First, although all data points were captured and reviewed, this study was intended to deeply understand this particular incident. One cannot assume that all digital firestorms will include similar content, though we suggest applicability of these findings specifically to school-based ATOD surveys administered for epidemiological purposes. Second, the study team did not compute interrater reliability for the coding, choosing instead to complete iterative review until 100% concordance was reached. Interrater reliability for qualitative research has long been a subject of discussion among scholars, who have noted that agreement on basic themes can easily be achieved, but interpretation of those themes can diverge [43]. Such divergence has been observed even in highly rigorous qualitative studies and tends to result from complex information (ie, multiple themes and subthemes) within a single piece of data [44]. Because of the nature of the data in this study—generally short, written statements rather than transcripts of interviews or focus groups and the a priori determination of the research interest (ie, the survey)—it was feasible to achieve full agreement on themes and interpretation relative to the research topic among the coders. Finally, this study did not have the ability to determine whether comments originated from people in the state where the firestorm occurred, nor was it able to determine whether comments were from online trolls or social bots intending to influence the discussion, for example, in Ferrara et al, 2016 [45].

To the authors’ knowledge, there are no extant studies that have investigated digital outrage relative to public health ATOD surveys in schools and few studies that have researched this in public health in general. Much of the writing about online firestorms has examined marketing and branding, including stakeholder relations and online paracrisis (ie, “a publicly visible crisis threat that charges an organization with irresponsible or unethical behavior”) [46]. This work has included, for example, experimental assessment of a proposed model of paracrisis development in response to a visual ad [47] and nonacademic books designed to help businesses (eg, How to Protect [or Destroy] Your Reputation Online) [48]. This study advanced understanding of the online firestorm phenomenon specifically as it pertains to school-based epidemiological surveys, which was heretofore mostly unexplored. At the same time, as noted above, it is limited in scope to the specific incident described and does not address, nor was it intended to address, the more general research literature conceptualizing digital outrage.