To assess coverage of COVID-19 and the sentiment it evoked, we analyzed the landing pages from major online news sources in countries with robust media presence. We selected the major online news sources from 11 countries: the United States, the United Kingdom, Canada, Australia, New Zealand, Ireland, Germany, France, Italy, Spain, and Russia. We included an additional international category to better reflect the global focus of certain online news sources.

For each country, the major online news sources were identified by reference to profiles in BBC Media, which is an authority in curating global news source information and lists of news sites with the most traffic and is curated by SimilarWeb. Focusing on major national online news sources, as defined by online visibility, captures some of the main sources in shaping societal knowledge and opinions [29]. It should be noted that the focus on online news sources excludes the impact of social media, epistemic communities, and other influences on public perception. However, due to their depth of penetration, political heterogeneity, and reliability, major online news sources provide an excellent proxy of overall public perception.

For each online news source, we collected the archived front-page snapshots dating back to 2015 via Internet Archive [30], cutting off coverage in 2020 at October 15. We used the pre-2020 articles to fine-tune the accuracy of article collection and provide statistics on reporting of certain topics pre–COVID-19. Each front page was sourced for potential news items using a custom-based pipeline we developed (Section 1 in Multimedia Appendix 1). We defined each article as the combination of the metadata elements title and description—sometimes referred to as headline and subheading—that are broadly akin to titles and abstracts in scientific publications [31]. Such metadata are reasonably standardized among online news sources and they offer a headline-like summary of the article, typically designed for sharing on social media, making them suitable for topic detection and sentiment analysis. In total, we collected 26,077,939 articles from front pages of 172 online news sources (Table 1), with the full list of contributing sources in Table S1 in Multimedia Appendix 1.

Contemporary news sites rapidly change their content, which is spread through multiple sections, so it is difficult to gauge the level of attention received by particular articles. Front pages of news sites should be reliable reflections of the information many users are exposed to, since they are the main points of entry. This is opposed to other article collection strategies, such as RSS (Really Simple Syndication) or downloading the entire website content, that can provide limited control over assessing how many people have actually read any given article [16,32]. Focusing our efforts on articles from landing pages of major online news sources enabled us to assess the number of COVID-19–related articles that a large proportion of online news consumers were exposed to.

For each article we extracted, we analyzed the text content of the metadata title and description to determine whether the article could be associated with one of the following topics: cat, sport, Merkel, Putin, Johnson, Biden, Trump, cancer, climate, or COVID-19. Non–COVID-19 topics were selected to provide a reference with large expected volumes of coverage (ie, politicians) that cover a wide range of sentiments (еg, cat as nonnegative and cancer as negative). Each topic was identified on the basis of the keywords presented in Table 2. The only normalization we applied to the words for topic identification was case folding, otherwise the words were not stemmed nor lemmatized. Topics that were used solely for sentiment analysis—cat, sport, climate, and cancer—were not identified for non-English online news sources.

We chose keywords for each topic in order to maximize the precision of topic identification. Because we focused on titles and descriptions, mentions of specific keywords here made it unlikely that they were only tangentially relevant to the article at hand (eg, explicit mentions of politicians’ names). In the case of COVID-19, we tested the extent to which our topic detection misclassified the topic by identifying COVID-19 articles in the pre–COVID-19 era of 2015 to 2019. Out of 21,693,591 articles, only 7375 (0.03%) were erroneously identified as related to COVID-19, demonstrating high precision of the selected keywords. In the majority of cases, misclassifications stemmed from mentions of lockdowns, which were chiefly gun related, but in the case of the United Kingdom, they were even related to a seagull attack on a school. Subsequent subtopic identification stratified the different threads of COVID-19 coverage, providing a wider set of keywords.

We further identified subtopics within COVID-19 news coverage by a similar keyword-based approach. Since many subtopic words can have several forms (eg, dead, died, and dies), we stemmed the words associated with each subtopic (eg, healthy and healthier are both stemmed to health); these are presented in Table 3. An article was identified as pertaining to a subtopic if, after stemming its title and description, a token corresponding to a stemmed keyword in Table 3 was identified.

We used a well-established sentiment analysis tool, Vader [33], to identify emotionally polarized content, which has previously been applied to news media. It is suitable for short snippets of text, such as the titles and descriptions in our metadata. For a given piece of text, Vader provides a compound score between –1 and 1, with –1 being entirely negative, 0 being neutral, and 1 being entirely positive. For instance, the phrase “I find your lack of faith disturbing” offers a Vader score of –0.42, whereas the phrase “I find your lack of faith encouraging” gives a score of 0.5994. In our case, a sentiment score for a single article consists of a Vader compound score for the concatenation of the article title and description.

Novel topics are associated with many subject-specific keywords and phrases (eg “social distancing” and “lockdown” for COVID-19). Applying sentiment analysis to text with novel keywords can result in software being unable to correctly annotate polarization. We assessed Vader sentiment annotations on articles identified as one of the subtopics in Table 3. This revealed an artifact of the tool, wherein “positive coronavirus test” was labeled as emotionally polarized in the positive direction by virtue of the word “positive.” In order to mitigate the effect of this subject-specific misclassification, the words “positive” and “negative,” for symmetry, were removed from articles related to coronavirus testing prior to applying Vader annotation.

We examined whether coverage of a given topic was more emotionally polarized than another topic by contrasting their sentiment distributions. Directly comparing topic sentiment distributions between different online news sources is not sound. Different online news sources can be more sensational and negative or toned-down and neutral, which gives radically different sentiment distributions. To address this issue, we calculated whether specific topic coverage was more negative, positive, or neutral relative to other articles within a particular online news source.

For each article “a” (ie, that article’s title and description metadata) in English-language online news sources, we calculated the Vader compound sentiment score sent(a). For all 2020 articles and topics, in a given online news source, we calculated the mean of the Vader compound sentiment scores, denoted as μ_ONS,TOPIC (online news source [ONS]; equation 1). As a reference statistic for the distribution of sentiment scores not relating to a topic, we calculated the mean Vader score of the articles from a given online news source that were not identified as a given topic, denoted as μ_ONS,TOPIC (equation 2):

We estimated the extent of COVID-19 coverage in online news media by identifying articles relating to the pandemic and comparing this number to the total number of articles between January and October 2020.

For each online news source, we performed topic detection, categorizing each article title and description as relating to the coronavirus—topic called COVID-19—if the title and description contained any keyword of a specific set, given in Table 2. Keywords for this simplified topic detection model were chosen to maximize the precision and accuracy of content identification in order to avoid cross-contamination with other topics. In English, these keywords included covid, coronavirus, lockdown, quarantine, and pandemic, but not, for example, hospital and death. The keywords were adjusted for the six languages that we used in this study: English, German, French, Spanish, Italian, and Russian (Table 2).

COVID-19 featured in 25.3% of the news articles (1,135,561/4,477,867) across the online news sources (Figure 1). While this proportion varied between countries, it was consistently large, with the lower bound at 20% and the upper bound at 30%. Thus, even using our relatively simple topic detection model, we were able to demonstrate that 2020 online news coverage was dominated by COVID-19.

To provide a point of reference to coverage of COVID-19, we identified other global topics with regular and topical media presence (Table 2). We selected topics such as Donald Trump (Trump), Joe Biden (Biden), Boris Johnson (Johnson), Angela Merkel (Merkel), and Vladimir Putin (Putin). Unsurprisingly, mentions of politicians are most common in their home countries (Figures S1-S5 in Section 2 of Multimedia Appendix 1) (eg, Vladimir Putin in Russia). Nonetheless, each politician received an order of magnitude less media attention than COVID-19 in 2020. Specifically, mean coverage of COVID-19 across 11 countries was 25.66%, whereas politicians received 2.50% for Trump, 0.45% for Biden, 0.18% for Putin, 0.17% for Johnson, and 0.09% for Merkel. Higher coverage of COVID-19 was also reflected at the national level. Although 2020 was an election year in the United States, Trump received a mean of 15.29% of 2020 media mentions in US online news sources, as opposed to 25.91% of mean mentions for COVID-19. Furthermore, US articles in 2020 mentioning both Trump and COVID-19 accounted for a mean coverage of 3.82%.

To provide temporal perspective on the media attention of COVID-19, we plotted the proportions of global coverage in the period from January to October 2020 (Figure S6 in Section 2 of Multimedia Appendix 1). COVID-19 attention in 2020 spiked between March and May, coinciding with many countries following the Chinese strategy of lockdown and other distancing measures. Proportion coverage leading up to the second COVID-19 wave in Europe and the United States did not reach levels seen from March to May but did stay above 20% in the regions we considered. These results provide a quantification of the extent of media attention received by COVID-19 across countries and languages, reflecting the global and protracted impact of the pandemic.

The emotion that news coverage of COVID-19 evokes is an important factor in a society’s response to the pandemic [25]. We addressed this issue by sentiment analysis contrasting emotional polarization of COVID-19 news with that of certain reference topics and all non–COVID-19 articles for each online news source. Such contrast allows us to determine whether sentiment distribution of COVID-19 coverage was polarized compared to reference topics and other online news for a given online news source.

To quantify the emotional content of news article text, we employed Vader, which has been previously applied to news article analysis [33,34] (see Methods section). For each online news source whose primary language was English (91/172, 52.9%), we created Vader annotations for both the title and description of each of its articles. We grouped articles by their annotated topics (Table 2) to provide a comparison with COVID-19 sentiments. Politicians—Merkel, Trump, Johnson, Biden, and Putin—were used as reference points for subjects with frequent coverage. We selected four additional topics to offer intuitive reference points on the positive and negative sentiment spectrum: cat, sport, climate, and cancer. The topics cat and sport were used, as they would not necessarily be associated with negative sentiments. Likewise, the topics of climate—identified by the key phrases global warming, climate crisis, and climate change—and cancer were used as references that we would expect to be associated with negative emotions. Altogether, individual sentiment annotations for each online news source were grouped by one of the topics: cat, sport, Merkel, Johnson, Biden, Trump, COVID-19, and cancer.

For each online news source and topic, we calculated the relative sentiment skew statistic (rsskew_ONS,TOPIC; see Methods section), which measures the polarization of a given topic within an online news source. We noted how many topics had a negative or positive relative sentiment skew value (Table 4). For the nonpolitician non–COVID-19 topics (ie, cat, sport, and cancer), skew was in the expected direction, suggesting that they are appropriate references for assessing sentiment of COVID-19 articles. We noted negative relative sentiment skew values for 74 out of 91 (81%) English-language online news sources. Nonetheless, this observation cannot be taken as evidence of significant negative polarization, as these relative sentiment skew values are not substantially different than rsskew=0, which indicates no polarization. The mean relative sentiment skew for COVID was –0.04 (SD 0.07) (Table 4). Since cancer and COVID-19 are both diseases and exert a large burden on public health, one might expect their sentiment distributions from online news to be similar. However, COVID-19 sentiment distribution was not as extreme as that of cancer, which was 100% negative per online news source, and had a mean relative sentiment skew of –0.53 (SD 0.12). Perhaps surprisingly, the sentiment distribution for COVID-19 articles was more akin to coverage of climate, which was a priori expected to be negative, akin to cancer, or to subjects covering heterogenous topics by virtue of their wide-ranging implications for society, such as politicians (Figure 2). This, however, can be an indicator of topics being intertwined: since heads of state are responsible for the pandemic response, they can be expected to be mentioned in relation to COVID-19.

These results suggest that the sentiment of COVID-19 coverage in online news media is heterogeneous and is certainly not as clearly polarized as cancer, though the volume of coverage might play a role (Figure S7 in Section 3 in Multimedia Appendix 1). One explanation could be the all-permeating nature of the pandemic, where it becomes background to most reporting. Therefore, COVID-19 articles cannot all be categorized as fully negative, contrary to the expectation of pandemic subject matter. In fact, on average, they appear to not be polarized in either the positive or negative direction, especially when compared to reference topics. This suggests that coverage of COVID-19 was highly heterogeneous, with many themes contributing to the totality of messaging.

We studied the text content of COVID-19–related title and description metadata to reveal the leading themes associated with heterogeneous pandemic reporting.

We investigated article subtopics and calculated the most commonly used words and bigrams (ie, consecutive combinations of two words) to demonstrate the most frequent mentions in COVID-19 coverage. For each of the 91 English-language online news sources, we calculated the ranks of single words and bigrams in articles that pertained to COVID-19. The articles were further subdivided within each online news source as negative (Vader score <–0.2; 247,542 articles), positive (Vader score >0.2; 192,643 articles), or all (any Vader score; 589,709 articles). This subdivision aimed to reveal whether certain keywords or bigrams were more frequently associated with differently polarized text. For each subdivision, we averaged the individual online news source ranks of each word and bigram found across all 91 online news sources. In Table 5 we present the top 20 highest-ranking words and bigrams across all 91 online news sources. Words and bigrams in Table 5 reveal many themes that are intuitively associated with the coronavirus, such as testing, vaccine, death, etc. In particular, negative articles had unique top words and bigrams that are intuitively associated with negative emotions. In the singletons, these were death, crisi, and fear, while in bigrams these were covid_crisi, covid_death, coronavirus_death, and death_toll—note that words are stemmed.

To calculate the news coverage proportion related to these top themes, we created a constrained set of COVID-19 subtopics, based on the highest-ranking words and bigrams from Table 5. We removed terms that pointed to nonspecific coronavirus coverage, such as COVID, coronavirus, pandemic, or news. We extended the list of subtopics to include those we did not find in Table 5 but considered as strongly related to COVID-19 coverage, such as hospital, quarantine, symptom, or isolation, with a full list of subtopics in Table 3. For each subtopic, we calculated the proportion of COVID-19 coverage per online news source (Figure S8 in Section 4 of Multimedia Appendix 1) and relative sentiment skew per online news source for the subtopic (Figure S9 in Section 4 of Multimedia Appendix 1). The means of coverage and sentiments per online news source are plotted against each other in Figure 3. The subtopics in Table 3 account for a mean of 67.14% of all COVID-19 articles across English-language online news sources. Of these, the top three are case, lockdown, and death, which account for a mean of 9.29%, 8.56%, and 8.08% of COVID-19 articles, respectively. Figure 3 and Figure S9 in Multimedia Appendix 1 suggest that out of case, death, and lockdown, only death carried a firmly polarized sentiment, with case and lockdown not being significantly skewed in either positive or negative directions. The words fear, crisis, and death unsurprisingly indicated substantial negative polarization (Figure 3).

We analyzed the impact on news sentiment in 2020 of the three most negative subtopics, fear, crisis, and death (Figure 3). For each online news source, we calculated the mean sentiment of all the 2020 articles after removing articles that mentioned one of those three topics. For all 91 online news sources, removing articles mentioning one of the top three negative topics resulted in a statistically significant—at the level of .05 with Bonferroni correction—shift toward mean positive sentiment (Section 5 in Multimedia Appendix 1). By contrast, removal of all the sentiment-heterogeneous (Figure 2) articles from all 2020 articles resulted in a significant shift toward mean positive sentiment in 40 out of 91 (44%) online news sources, a significant shift toward mean negative sentiment for 11 out of 91 (12%) online news sources, and no statistically significant result for 39 out of 91 (43%) online news sources. Altogether, articles mentioning fear, crisis, and death accounted for a mean of 16% of COVID-19 articles across 91 online news sources; due to their highly polarizing nature, they may play a significant role in shaping societal perception of the pandemic.

Of the three most negative topics, fear, crisis, or death, the latter was the most frequently mentioned for COVID-19 (Figure 3). In total, death was mentioned in the context of COVID-19 in 2.33% of all coverage in 91 English-language online news sources. All death mentions in 2020 in 91 English-language online news sources accounted for a mean coverage of 5.74%, whereas in the pre–COVID-19 period of 2015 to 2019, they accounted for 4.07%. Therefore, we can identify that death in the context of COVID-19 constituted a significant proportion of negatively associated coverage that appears to have contributed to overall death reporting in the news.

These results demonstrate that even though the overall coverage of COVID-19 in 2020 was not significantly polarized by sentiment, there was a nontrivial proportion of negative news that contributed to overall reporting negativity in 2020.