Published on in Vol 25 (2023)

Preprints (earlier versions) of this paper are available at, first published .
Global COVID-19 Policy Engagement With Scientific Research Information: Altmetric Data Study

Global COVID-19 Policy Engagement With Scientific Research Information: Altmetric Data Study

Global COVID-19 Policy Engagement With Scientific Research Information: Altmetric Data Study

Authors of this article:

Han Woo Park1, 2, 3, 4 Author Orcid Image ;   Ho Young Yoon5 Author Orcid Image

Original Paper

1Department of Media & Communication, YeungNam University, Gyeongsan-si, Republic of Korea

2Graduate Department of Digital Convergence Business and East Asian Cultural Studies, YeungNam University, Gyeongsan-si, Republic of Korea

3Cyber Emotions Research Center, YeungNam University, Gyeongsan-si, Republic of Korea

4Big Local Big Pulse Lab, YeungNam University, Gyeongsan-si, Republic of Korea

5Division of Communication & Media, Ewha Womans University, Seoul, Republic of Korea

Corresponding Author:

Ho Young Yoon, PhD

Division of Communication & Media

Ewha Womans University

Posco 620, Ewhayeodae Gil 52


Seoul, 03760

Republic of Korea

Phone: 82 2 3277 4491


Background: Previous studies on COVID-19 scholarly articles have primarily focused on bibliometric characteristics, neglecting the identification of institutional actors that cite recent scientific contributions related to COVID-19 in the policy domain, and their locations.

Objective: The purpose of this study was to assess the online citation network and knowledge structure of COVID-19 research across policy domains over 2 years from January 2020 to January 2022, with a particular emphasis on geographical frequency. Two research questions were addressed. The first question was related to who has been the most active in policy engagement with science and research information sharing during the COVID-19 pandemic, particularly in terms of countries and organization types. The second question was related to whether there are significant differences in the types of coronavirus research shared among countries and continents.

Methods: The Altmetric database was used to collect policy report citations of scientific articles for 3 topic terms (COVID-19, COVID-19 vaccine, and COVID-19 variants). Altmetric provides the URLs of policy agencies that have cited COVID-19 research. The scientific articles used for Altmetric citations are extracted from journals indexed by PubMed. The numbers of COVID-19, COVID-19 vaccine, and COVID-19 variant research outputs between January 1, 2020, and January 31, 2022, were 216,787, 16,748, and 2777, respectively. The study examined the frequency of citations based on policy institutional domains, such as intergovernmental organizations, national and domestic governmental organizations, and nongovernmental organizations (think tanks and academic institutions).

Results: The World Health Organization (WHO) stood out as the most notable institution citing COVID-19–related research outputs. The WHO actively sought and disseminated information regarding the COVID-19 pandemic. The COVID-19 vaccine citation network exhibited the most extensive connections in terms of degree centrality, 2-local eigenvector centrality, and eigenvector centrality among the 3 key terms. The Netherlands, the United States, the United Kingdom, and Australia were the countries that sought and shared the most information on COVID-19 vaccines, likely due to their high numbers of COVID-19 cases. Developing nations, although gaining quicker access to COVID-19 vaccine information, appeared to be relatively isolated from the enriched COVID-19 pandemic content in the global network.

Conclusions: The global scientific network ecology during the COVID-19 pandemic revealed distinct types of links primarily centered around the WHO. Western countries demonstrated effective networking practices in constructing these networks. The prominent position of the key term “COVID-19 vaccine” demonstrates that nation-states align with global authority regardless of their national contexts. In summary, the citation networking practices of policy agencies have the potential to uncover the global knowledge distribution structure as a proxy for the networking strategy employed during a pandemic.

J Med Internet Res 2023;25:e46328



Research Background

Given the expanding international scientific linkages, it is advantageous to view this expansion as a communications network and recognize the network as a new form of global governance. This view has been made possible by the increasing worldwide scientific connections. Over the past 2 decades, the global network has grown denser and more concentrated, indicating the presence of numerous subconnections that can form exclusive cliques [1]. The expansion has occurred despite the overall reduction in the size of the network, suggesting the presence of power dynamics in global health research and development governance.

Previously, ecological studies of COVID-19 research articles were typically reviewed without considering the contextual and spatial features of the scientific system. Studies typically concentrated on the bibliometric characteristics of COVID-19 research articles [2,3], neglecting the identification of institutional actors who cite recent scientific contributions related to COVID-19 in the policy domain, and their locations. In other words, the global collaboration involved in COVID-19 research has not been exhaustively investigated in terms of the policy-related proportion of overall scientific information exchange, which is assessed by the number of citations to peer-reviewed published works.

Although scientific research publications were not the study focus, a recent study examined the institutional responses of about 300 different intergovernmental organizations to the COVID-19 epidemic [4]. This study discovered that 52.8% of intergovernmental organizations shared third-party knowledge or information, whereas only 32.5% developed their own expertise. The study was able to quantify intergovernmental organization responses to pandemics by searching the websites of global and regional organizations for references to COVID-related phrases. The study aimed to establish the government’s skills in information intervention for risk management, identify influential agents in a global information network, and discover socioeconomic elements that influence the government’s information-sharing behaviors across regions. The researchers of the study examined the online presence of various government health agencies using network analysis. They found that European agencies had the biggest web impact in response to COVID-19, and that income inequality and gross domestic product per capita were associated with the high online visibility of government health agencies. This indicates that socioeconomic factors may play a role in predicting the government’s distribution of COVID-19 information during the pandemic. The data source used in this study was not particularly exhaustive of citations found in policy documents and websites maintained by intergovernmental organizations and national government agencies.

Thus, taking sociological institutionalist and space-related scientometrics into account [5], the goal of this study was to conduct an in-depth investigation of the institutional actors who cite the most recent scientific contributions related to topics covered by COVID-19, using novel big data analytics, namely the altmetric search and data curation technique [6]. Altmetrics examine, among other things, the influence of scientific research findings on media, documents, and other types of writings. Traditional citation tracking and evaluation rely on peer-reviewed publications since citations are regarded as a sign of research impact, whereas altmetrics involve nonacademic public behavior, such as social media. As social media make research citations more efficient than traditional venues [7], altmetrics use not only social media but also scholarly databases, such as Scopus, to evaluate research output across fields. Owing to their statistical relationship with actual research impact, altmetrics are considered as a credible alternative to standard research evaluation [8], similar to Web of Science, PubMed, and Google Scholar [9]. Quantitative analyses of research articles produced by numerous authors from various countries revealed that global research collaboration networks are continuously expanding [10] and traditional measures alone would not be sufficient.

Indeed, policy papers released by internationally famous organizations as well as nationally recognized public authorities can benefit and supplement a knowledge-sharing infrastructure reference system [4]. Consequently, it would be interesting to investigate whether developing and emerging nations have expanded their engagement in global COVID-19 science as a result of the visibility provided by an innovative approach to big data analytics. Not only COVID-19 research and development spending but also policy responses in impoverished countries could not match those in affluent countries. It is apparent that the backdrop of developing countries contributes to the difficulties of responding quickly to the pandemic. Developing countries lack sufficient fiscal capacity to assist their populations for a longer period of time when social distancing is effective to prevent the spread of COVID-19 [11]. The large number of younger populations in developing countries require economic activity to support their families, making it difficult to maintain social distancing [12]. The tradition of living together as a large family, partly due to poverty, creates intergenerational living conditions that contribute to the relatively high level of COVID-19 virus exposure among older people, while the insufficient capacity of the health care system makes treating COVID-19 patients difficult [12,13]. Indeed, developing countries lack testing kits as well as infrastructure, such as health care facilities, including equipment and personnel [13,14]. All of this contributes to the greater difficulty of applying known policy responses to COVID-19 in developing countries compared with developed countries.

In this regard, an examination of the interconnected networks involved in the geopolitical features of the scientific footprint, particularly the national distribution of citation effect, would benefit the health science community while also contributing to a better understanding of the global network of policy responses, particularly for developing countries. As a result, we investigated the knowledge-sharing reference system and addressed the possibilities of establishing a new global information order through network representation of space-related research data, as well as some future research directions.

Related Work and Research Questions

The primary purpose of big data analytics, such as altmetrics, is to monitor rapidly expanding scientific issues in the wake of social, political, and cultural events and infectious disease outbreaks [15]. When research results become a new global focal point, the function of big data metrics is significant because it enables the measurement of social media mentions, newspaper coverage, and policy document citations [16]. Numerous large-scale data analyses have been conducted amidst the COVID-19 epidemic, for example, a study analyzed the scientific information transmission networks and news-sharing behaviors regarding COVID-19 on Twitter [17]. However, relatively little systematic research has been conducted on how global and national policymakers approach COVID-19 research topics [18].

In infodemiology investigations, policy documents are regarded as an authoritative, credible, and esteemed source that serves as a valuable information resource during pandemics. Specifically, the aggregation of online citation data can serve as a starting point for evaluating strategic approaches for health administration and public campaigns, and for this reason, network analysis and big data have been integrated into the field of research [19]. As these methods can be used to improve public health, it would be beneficial to illustrate a network pattern. This would also contribute to the creation of an efficient communication system. That said, our 2 research questions are as follows:

  1. Who has been the most active in terms of policy engagement with science and research information sharing during the COVID-19 pandemic? Which policy sources, in terms of website domains and countries, provide the greatest number of research references from scholarly articles on COVID-19 subjects?
  2. Are there substantial differences in the types of coronavirus research shared in each country, according to citation networks? More specifically, how is the networked structure of research information sharing organized between nations and major research topics?


Figure 1 depicts the data collection and analysis process. The first procedure of data collection was to identify keywords that accurately represent policy contexts of the scientific research citation network. After a series of pilot studies using a range of keywords, including COVID-19 booster injection, 3 queries were ultimately selected. COVID-19, COVID-19 vaccine, and COVID-19 variants were the 3 keywords that were used.

Figure 1. Study flowchart.

Data Collection and Preprocessing

The data collection procedure was as follows. First, the 3 keywords were used to extract data from the Altmetric citation database. There were 216,787 research outputs for COVID-19, 16,748 for COVID-19 vaccine, and 2772 for COVID-19 variants. These results were acquired by limiting the search to research articles published in journals indexed by PubMed between January 1, 2020, and January 31, 2022. Note that the numbers reflect the deletion of duplicate database records. As some articles appeared in both the “COVID-19” search and the “COVID-19 vaccine” or “COVID-19 variants” search, duplicate articles were removed by manually comparing the titles and author names with the Excel (Microsoft Corp) function for filters and removing duplicate records. is a database of citations for both research articles and policy reports. The database includes policy sources, such as government directives, reports, and white papers, as well as publications from independent policy institutes, expert advisory committees, research institutions, and international development organizations [15]. Altmetric intends to collect global policy sources on climate change, health, transportation, and economics [19]. Typically, policy sources are updated on the websites of organizations. Altmetric derives research citations from policy documents by tracking the interest in policy sources and collecting data from organization websites. The URLs of policy agencies that cited COVID-19 studies are included in the calculation of citation attention from policy documents and texts. In general, citation frequency is used to evaluate the significance of research. Policy mentions are essential for calculating Altmetric impact scores, as citations reflect current public interest. The evaluation of policy documents relies on sources. By employing link searching, identifier analysis, text extraction, and policy mentions, Altmetric combines PDFs, metadata, and research outcomes. Through text extraction, Altmetric establishes a connection between policy document mentions and journal research. Altmetric operationalizes the national origin of policy papers that cite COVID-19 study articles by using a variety of information categories, including country code top-level domain (ccTLD), language, location, etc. Previous studies have also employed a similar strategy [20]. Altmetric provides additional information regarding the algorithm for more details.

Second, after the data set was extracted, URL information was used to determine the country of origin for policy documents. A domain is the namespace in which an organization operates, which serves as a proxy indicator of its institutional attributes and identity., for example, refers to the World Health Organization (WHO), which is an example of an intergovernmental organization. Website visitors can determine the physical location of a domain by inspecting its ccTLD. As an illustration, the domain name is clearly associated with the United Kingdom. Similarly, each URL referring to COVID-19 research publications was accompanied by website domains and the countries where they were hosted.

Third, a network data set for visualization was made. A network data set is composed of nodes and links that connect the nodes. For the citation network, nodes are research or policy documents and the links are citations between them. In other words, the governance of the COVID-19 pandemic consists of nodes (research information and policy documents) and links (citations between intergovernmental health agencies and scientific articles). By constructing the network data set, the examination of the connectedness between hosting countries and research topics resulted in the discovery of insights in the information flow.

Analysis Methods

The process of analyzing citation networks involves quantifying the intricate and intertwined connections between the various components. This study presents an analytic approach for evaluating the competence of information intervention against COVID-19 by intergovernmental health organizations. The act of citing and disseminating research findings may be interpreted as an endeavor by public authorities to mitigate risk. The effectiveness of risk reduction strategies is evaluated based on the content of policy reports from a variety of international organizations and nations, which are referenced as URLs.

This research analyzed the data with descriptive statistics and network analysis in order to comprehend the strategies. The citing institutions were divided into 3 categories for descriptive purposes: intergovernmental organizations, national and domestic governmental organizations, and nongovernmental organizations (think tanks and academic institutions). A few basic indexes and visualization were used for network analysis. Network analysis employs statistical methods and visual mapping to investigate the configuration of nodes and their interactive connections along cited and citing dimensions [21,22]. Network analysis indicators can evaluate the structural positions and roles of health agencies and their host countries, as well as the statistical links between scientific and policy texts. Several centrality measures were used to determine the frequency of direct immediate connections in the network data sets: degree, 2-local, and eigenvector values. High centralities usually indicate how quickly nodes spread information. Central nodes transfer data to other nodes, and degree centrality counts a node’s direct links. The amount and pattern of ties are used to calculate a node’s geodesic distance and indirect fraction of direct ties. Even if a node has moderate centralities, its 2-local and eigenvector centralities may be higher because of the shortest path. Nodal centralities were measured and compared using a network diagram. UCINET-NetDraw was used to calculate and draw centralities and visualizations [23]. The network analysis results can show efficient information flow, mutual communication, or interorganizational collaboration in health policy-making institutions. Cross-national mobility of research findings can provide suggestions for public health development and promote critical junctures in cross-functional networks, as cited research articles on policy documents have become scientific intellectual settings for knowledge dissemination. Hierarchical or geographic clustering based on the interacting relationships between countries in the 3 data sets can ensure regional integration during the COVID-19 pandemic.

Policy Engagement With Scholarly Articles

According to Table 1, the WHO was the most influential reporter in terms of citations to scientific papers. Intergovernmental organizations represented 73.5% (119/162) of citations. National and domestic governmental organizations represented the second most influential reporter. Despite the fact that COVID-19 swiftly became a pandemic, the identified institutions were not diverse. Citations are intended to serve as a guide for decision-makers, instructing them to refer to the characteristics of COVID-19 and to investigate the impact of vaccines on the effectiveness of treatment. That said, it is important to note that governmental organizations in Europe drove citations, whereas think tanks and academic institutions in the United States drove citations. Table 2 displays the distribution of countries with regard to COVID-19 citations.

Table 1. Website domain distribution for COVID-19 citations.
Category and websiteValue (N=162), n (%)
Intergovernmental organizations119 (73.5)

who.int108 (66.7)

worldbank.org4 (2.5)

wmo.int3 (1.9)

europa.eu2 (1.2)

oecd-ilibrary.org1 (0.6)

fao.org1 (0.6)
National and domestic governmental organizations32 (19.8)

folkhalsomyndighete.se11 (6.8) (4.3)

parliament.uk7 (4.3)

rivm.nl4 (2.5)

gov.scot2 (1.2)

sencanada.ca1 (0.6)
Nongovernmental organizations (think tanks and academic institutions)11 (6.8)

rand.org5 (3.1)

nber.org2 (1.2)

csis.org2 (1.2)

interacademies.org1 (0.6)

awmf.org1 (0.6)
Table 2. Country distribution for COVID-19 citations.
Continent and countryaValue (N=162), n (%)

Switzerland111 (68.5)

Sweden11 (6.8)

United Kingdom9 (5.6)

Netherlands4 (2.5)

Belgium2 (1.2)

France1 (0.6)

Germany1 (0.6)

Italy1 (0.6)
North America

United States14 (8.6)

Canada1 (0.6)

Australia7 (4.3)

aThe World Health Organization and World Meteorological Organization are in Switzerland, the World Bank is in the United States, the Food and Agriculture Organization is in Italy, 7 institutions of the European Union are in Belgium, and the Organization for Economic Co-operation and Development is in France.

The headquarters of the WHO is in Switzerland, and Switzerland represented 68.5% (111/162) of citations. Moreover, the United States represented 8.6% (14/162) of citations, Sweden represented 6.4% (11/162) of citations, Australia represented 4.3% (7/162) of citations, and the United Kingdom represented 5.6% (9/162) of citations. The proportion of countries other than European countries was 13.5% (22/162). Since the majority of intergovernmental bodies are located in Europe, we recalculated the values without them, and the number of citations for Europe was 25 and that for all other countries was 18. European citations were 1.40 times greater than those of other nations.

Table 3 lists the policy groups in the results on COVID-19 vaccines. The overall number was 1900, which is 11.7 times greater than the COVID-19 result of 162, suggesting that there are vested interests in vaccine effects and policy adaptation.

Regarding COVID-19 vaccines, intergovernmental organizations represented 76.5% (1453/1900) of citations, national and domestic governmental organizations represented 17.5% (333/1900) of citations, and nongovernmental organizations (think tanks and academic institutions) represented 6.0% (114/1900) of citations, similar to the findings for COVID-19 citations. For COVID-19 vaccines, the WHO represented 67.5% (1282/1900) of citations, similar to the finding for COVID-19 citations. The second most influential reporter was the Dutch government’s National Institute for Public Health and Environment (73/1900, 3.8%).

Table 4 shows the country distribution for COVID-19 vaccine citations. For COVID-19 vaccine citations, when intergovernmental organizations were included, Switzerland ranked first, followed by the Netherlands, the United States, and the United Kingdom. On the other hand, when intergovernmental organizations were excluded, the Netherlands ranked first, followed by the United States, the United Kingdom, and Australia. Ethiopia represented a noticeable number of citations (n=18), reflecting the interest of this African country in COVID-19 vaccination.

Interestingly, the Netherlands, the United States, the United Kingdom, and Australia experienced unprecedented increases in COVID-19 cases between December 2020 and January 2021. In Multimedia Appendix 1, we have presented the number of new COVID-19 cases for these 4 countries from March 2020 to March 2022. In all these countries, the number of new cases peaked around January 2021. Thus, they experienced an urgency of policy measures related to COVID-19 vaccination. Indeed, the scientific citations reflect the reality.

The number of COVID-19 variant citations was 521 (Table 5). Regarding COVID-19 variants, the WHO represented the highest number of citations (446/521, 85.6%), followed by (19/521, 3.6%). Organizations other than the WHO did not actively use scientific articles.

Table 6 shows the country distribution for COVID-19 variant citations. For COVID-19 variant citations, when intergovernmental organizations were included, Switzerland ranked first, followed by the Netherlands, the United Kingdom, and the United States. On the other hand, when intergovernmental organizations were excluded, the Netherlands ranked first, followed by the United Kingdom, Australia, and the United States. As mentioned above, the rapid increase in new COVID-19 cases affected these countries. However, the number of citations was not as high as those for COVID-19 vaccines, and the appearance of European countries was greatly reduced.

Table 3. Website domain distribution for COVID-19 vaccine citations.
Category and websiteValue (N=1900), n (%)
Intergovernmental organizations1453 (76.5)

who.int1282 (67.5)

worldbank.org54 (2.8)

europa.eu45 (2.4)

fao.org33 (1.7)

au.int18 (0.9)

oecd-ilibrary.org10 (0.5)

iadb.org6 (0.3)

amnesty.org2 (0.1)

wmo.int2 (0.1)

unicef.org1 (0.1)
National and domestic governmental organizations333 (17.5)

rivm.nl73 (3.8)

folkhalsomyndighete.nl50 (2.6) (1.8)

gov.scot30 (1.6)

files.parliament.uk29 (1.5)

org.au23 (1.2)

cdc.gov20 (1.1)

rijksoverheid.nl19 (1.0) (0.9)

congress.gov12 (0.6)

regjeringen.no8 (0.4) (0.3)

awmf.org5 (0.3)

idsa.in4 (0.2)

officielebekendmaki.nl3 (0.2)
Nongovernmental organizations (think tanks and academic institutions)114 (6.0)

urban.org20 (1.1)

interacademies.org18 (0.9)

nber.org15 (0.8) (0.8)

brookings.edu14 (0.7)

africaportal.org8 (0.4)

rand.org8 (0.4)

americanactionforum.org6 (0.3)

americalatinagenera.org2 (0.1)

bruegel.org2 (0.1)

csis.org2 (0.1)

realinstitutoelcano.org2 (0.1)

georgetown.edu1 (0.1)

mdpi.com1 (0.1)
Table 4. Country distribution for COVID-19 vaccine citations.
Intergovernmental organization status and regionValue, n (%)
Intergovernmental organizations included (n=1900)

United States and Europe1811 (95.3)

Switzerland 1285 (67.6)

Netherlands190 (10.0)

United States161 (8.5)

United Kingdom97 (5.1)

Italy51 (2.7)

France10 (0.5)

Norway8 (0.4)

Germany5 (0.3)

Brussel2 (0.1)

Spain2 (0.1)

Others89 (4.7)

Australia57 (3.0)

Ethiopia18 (0.9)

South Africa8 (0.4)

India4 (0.2)

Panama2 (0.1)
Intergovernmental organizations excluded (n=447)

Netherlands145 (32.4)

United States98 (21.9)

United Kingdom97 (21.7)

Australia57 (12.8)

Italy18 (4.0)

Norway8 (1.8)

South Africa8 (1.8)

Germany5 (1.1)

India4 (0.9)

Panama2 (0.4)

Belgium2 (0.4)

Spain2 (0.4)

Switzerland1 (0.2)
Table 5. Website and domain distribution for COVID-19 variant citations.
Category and websiteValue (N=521), n (%)
Intergovernmental organizations460 (88.3)

who.int446 (85.6) (1.2)

worldbank.org3 (0.6)

au.int2 (0.4)

oecd-ilibrary.org2 (0.4)

iadb.org1 (0.2)
National and domestic governmental organizations48 (9.2)

folkhalsomyndighete.nl19 (3.6)

rivm.nl15 (2.9) (1.2) (0.8)

parliament.uk2 (0.4)

gov.scot1 (0.2)

rijksoverheid.nl1 (0.2)
Nongovernmental organizations (think tanks and academic institutions)13 (2.5)

interacademies.org3 (0.6)

rand.org3 (0.6)

brookings.edu2 (0.4)

nber.org2 (0.4) (0.2)

awmf.org1 (0.2)

urban.org1 (0.2)
Table 6. Country distribution for COVID-19 variant citations.
Intergovernmental organization status and countryValue, n (%)
Intergovernmental organizations included (n=515)

Switzerland446 (85.6)

Netherlands35 (6.7)

United Kingdom16 (3.1)

United States11 (2.1)

Australia5 (1.0)

Italy3 (0.6)

Ethiopia2 (0.4)

France2 (0.4)

Germany1 (0.2)
Intergovernmental organizations excluded (n=64)

Netherlands35 (54.7)

United Kingdom10 (15.6)

Australia8 (12.5)

United States7 (10.9)

Italy3 (4.7)

Germany1 (1.6)

Network of COVID-Related Research Citations

Figure 2 depicts a global network map created with the help of a graphical theoretic layout made available through UCINET-NetDraw [21]. The thickness of the lines connecting scientific themes and countries increased as the number of scientific topics per policy report issued in the countries increased.

Figure 2. Global network map of COVID-19–related research topics. AfricanU: African Union; EU: European Union; IADBank: Inter-American Development Bank; OECD: Organization for Economic Co-operation and Development; UN: United Nations; WHO: World Health Organization.

COVID-19 vaccine was the scientific topic that appeared in most policy papers, followed by COVID-19 variants. In policy documents from early 2022, COVID-19 was the least commonly cited research topic. The size of the squares representing each topic grew in proportion to the number of times it appeared. The link to the WHO is easily distinguishable in Figure 2. Countries, such as Australia, the Netherlands, the United States, and the United Kingdom, showed strong ties to the key term COVID-19 vaccine, along with other intergovernmental organizations. In terms of country diversity, COVID-19 vaccine was the most famous topic among many countries other than those in North America and Europe. In particular, the vested interest of African countries appeared to be strong.

Tables 7 and 8 summarize the multiple network centrality measures of scientific topics and countries. In terms of these measures, it was noted that the topic of COVID-19 vaccine was more central than the other 2 topics of COVID-19 variants and COVID-19 itself.

Through the network measures, the use of scientific information by intergovernmental organizations was clearly visible. Moreover, the absence of Asian countries was noticeable. This may be due to the way government reports are stored, for example, use of JavaScript, inability to use a simple web scrapper, or language issues. Further research would help in understanding this imbalance.

Table 7. Country and organization centralities in the COVID-19 research information network (normalized).
NumberCountry/organizationDegree2-local eigenvectorEigenvectorCoreness
1World Health Organization70.61526893960.9880.115
2United States5.5000.5310.0870.115
4United Kingdom4.2690.5310.0660.115
7World Bank2.3461050630.0380.115
8European Union1.769841440.0300.077
9United Nations1.615730380.0260.077
10African Union0.808372250.0130.077
11Organization for Economic Co-operation and Development0.500202420.0070.115
14Inter-American Development Bank0.269119450.0040.077
18South Africa0.1150.4730.0020.038
Table 8. Topic centralities in the COVID-19 research information network.
Research topicDegree2-local eigenvectorEigenvector
COVID-19 vaccine73.26924080330.943
COVID-19 variants19.8088253990.323

Principal Findings

According to the findings, the global scientific network ecology surrounding the COVID-19 pandemic was found to contain distinctive kinds of links that were concentrated around the WHO. In articulating global health policy and coordinating global governance in the early stages of disease diffusion [24,25], the WHO was indeed at the center of the global response system. In addition, based on an analysis of the various kinds of citation ties, Western countries were found to operate effectively in terms of network construction. The preeminent standing of COVID-19 vaccines demonstrates that nation-states align with global authority regardless of the national context in which they operate.

As citation analysis is a promising area of assessing scientific influence, the selection of the networking practices of citations by policy agencies can serve as a proxy signal for the networking strategy that can be used for the next possible pandemic [26]. In other words, a top-down process directed by a global health authority and a catch up from each country could result in a structural change, which may be substantially modified by the altmetrics-supplied research collaboration network.

The findings of this study have a number of significant ramifications. Analyzing policy document publication citation practices is a new research field, which is currently being explored [27]. For instance, the typical number of citations to scholarly works found in COVID-19 policy documents has not been thoroughly investigated. Furthermore, present research indicates that policy document publishers used academic references in a unique way during the pandemic period. The findings, for instance, identified the scientific articles and journals most frequently cited in WHO policy documents. Hence, the organization of citation networks serves as a stand-in for the policy knowledge base, which is an essential component in the process of mitigating potential health hazards. It is likely that the widespread dissemination of COVID-19 might have been accelerated as a result of the delayed recognition of scientific and research information in the public health and government sectors. The increase in the use of preprint scholarly articles highlights the urgent need for COVID-19 scientific information [28]. This is evinced by the fact that each international organization and nation state has its own system for referencing scholarly sources within policy contexts. This is particularly important for developing countries, such as Iran, where policy resources are insufficient for public health management [29]. The increased need for evidence-based policy decisions shows a growing interest in the results of scientific studies and research, particularly when applied to situations involving risk reduction.

Citations from policy websites may also serve as crucial agenda-setting signals for mass media sources that aggregate news and develop country-level trends related to COVID-19 prevention. Although there may be a convergence of research, policy, and the media, the cooperation between these 3 institutional entities has not been sufficiently studied [30]. In other words, this condition signifies a decentralized information transfer toward a more unified health governance in order to establish a solid knowledge base.


From a methodological standpoint, this research has some issues and limitations. This study was not intended to assess the success of scientific research on pandemics in academic settings or in the broader community. Therefore, the number of citations from authors of policy texts should not be viewed as a measure of the impact of the research. In a similar vein, the disparity in centrality measures between policy authors in citation networks does not imply that these authors are ranked differently. The variance, on the other hand, illuminates the extent to which policy document authors are connected to the scientific and research communities during the COVID-19 pandemic. In addition, it should be noted that articles and journals with a high number of citations should not be considered the most credible sources for research findings or publication outlets. This is due to the fact that a large number of citations is not necessarily indicative of a reliable source. The establishment of systematic citation links between scientific research and health policy texts signifies the development of a communication network in addition to the dissemination of information.


The goal of this study was to find out how institutions view the quality of research information, as well as whether they are aware of COVID-19 and the emergency effect that its vaccine has on the prevention of the global pandemic. We found that was the source of majority of the websites and domains containing scientific documents on COVID-19 variants. This demonstrates that is actively seeking and sharing information in the COVID-19 research community. Furthermore, the “COVID-19 vaccine” key term had the most extensive connections in terms of degree centrality, as well as 2-local and eigenvector centralities. The countries that were most actively sought after and tied with one another were the Netherlands, the United States, the United Kingdom, and Australia. Although developing countries obtained COVID-19 vaccine information more quickly and with greater ease, their marginal position in the global network indicated that they were isolated from enriched COVID-19 pandemic content. As developing countries are predominantly consumers of pandemic content as opposed to active producers of information, they are isolated from the research and development activities associated with COVID-19 [31], which in turn can impact their connections to other information sources from established countries. Additionally, this may be due to the fact that the prioritization of global research may not be particularly suited to developing nations. The dissemination of enriched COVID-19 content may prioritize regions with enhanced connectivity and more significant resources, resulting in the concentration of data within particular networks. This can lead to developing nations receiving less attention and representation in the global influx of pandemic-enriched content.

The entire structure of the COVID-19 research information network regarding website distribution and centrality included in the study demonstrated the need for emergency services for scientific information in global public health governance. A number of methods that facilitate this could be suggested, especially for developing countries. Among them, one approach is to expand the network of partnerships and knowledge exchange with developed nations. Similar to COVAX (COVID-19 Vaccines Global Access), the alliance for the exchange of knowledge and information can induce and position developing countries in the center of a global network with enriched pandemic content. Another approach is to increase alliances and networks between developing nations, which can increase the significance of their responsibilities in the global information flow network.

This study is notable because it is one of the few altmetric studies that looked at how international policy institutions in the health sector used the internet for information seeking and sharing from the perspective of actor-document networks.

We were unable to determine whether the quarantine and lockdown stances on COVID-19 of specific nations eventually influenced their network positions or the manner in which research information was provided to them. Additional research should be conducted using qualitative methodologies to analyze the citation circumstances of policy research. For example, it must be evaluated whether policy agencies established their own autonomous cluster in order to pool resources and collaborate. Furthermore, we did not investigate the extent to which policy institutions working with smaller or developing nations aided in enhancing citizen involvement with the COVID-19 issues at hand. Furthermore, we were unable to detect structural movement in the online altmetric network between several time points that were analyzed.


HWP would like to thank Jung-min Lee for assisting with data curation in the early stage. At the start of this research, discussions were held with Dr Jung-won Yoon on the selection of keywords. We also thank for providing useful data for academic research. This is part of a larger project involving COVID-19 and altmetrics research.

Conflicts of Interest

None declared.

Multimedia Appendix 1

COVID-19 new case trends for the Netherlands, the United Kingdom, the United States, and Australia.

PNG File , 118 KB

  1. Barnett G, Ruiz J, Xu W, Park J, Park H. The world is not flat: Evaluating the inequality in global information gatekeeping through website co-mentions. Technological Forecasting and Social Change. Apr 2017;117:38-45. [CrossRef]
  2. Deng Z, Chen J, Wang T. Bibliometric and visualization analysis of human coronaviruses: Prospects and implications for COVID-19 research. Front Cell Infect Microbiol. 2020;10:581404. [FREE Full text] [CrossRef] [Medline]
  3. Mahi M, Mobin M, Habib M, Akter S. A bibliometric analysis of pandemic and epidemic studies in economics: Future agenda for COVID-19 research. Soc Sci Humanit Open. 2021;4(1):100165. [FREE Full text] [CrossRef] [Medline]
  4. Ulybina O, Ferrer L, Alasuutari P. Intergovernmental organizations in the face of the Covid-19 pandemic: Organizational behaviour in crises and under uncertainty. International Sociology. Jun 25, 2022;37(4):415-438. [CrossRef]
  5. Frenken K, Hoekman J. Spatial Scientometrics and Scholarly Impact: A Review of Recent Studies, Tools, and Methods. In: Ding Y, Rousseau R, Wolfram D, editors. Measuring Scholarly Impact. Cham. Springer; 2014;127-146.
  6. Holmberg K, Park H. An altmetric investigation of the online visibility of South Korea-based scientific journals. Scientometrics. Jul 28, 2018;117(1):603-613. [CrossRef]
  7. Park H, Biddix P, Park H. Discussion, news information, and research sharing on social media at the onset of Covid-19. Profesional De La información. Jul 13, 2021;30(4) [CrossRef]
  8. Bornmann L. Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime. Journal of Informetrics. Oct 2014;8(4):935-950. [CrossRef]
  9. Martin-Martin A, Orduna-Malea E, Harzing A, Delgado López-Cózar E. Can we use Google Scholar to identify highly-cited documents? Journal of Informetrics. Feb 1, 2017;11(1):152-163. [CrossRef]
  10. Wagner C, Park H, Leydesdorff L. The continuing growth of global cooperation networks in research: A conundrum for national governments. PLoS One. 2015;10(7):e0131816. [FREE Full text] [CrossRef] [Medline]
  11. Alon T, Kim M, Lagakos D, VanVuren M. How Should Policy Responses to the COVID-19 Pandemic Differ in the Developing World? National Bureau of Economic Research. 2020. URL: [accessed 2023-06-10]
  12. Barnett-Howell Z, Watson O, Mobarak A. The benefits and costs of social distancing in high- and low-income countries. Trans R Soc Trop Med Hyg. Jul 01, 2021;115(7):807-819. [FREE Full text] [CrossRef] [Medline]
  13. Díez F, Duval R, Maggi C, Ji Y, Shibata I, Tavares MM. Options to Support the Incomes of Informal Workers During COVID-19. IMF Research Special Series on COVID-19. 2020. URL: https:/​/www.​​-/​media/​Files/​Publications/​covid19-special-notes/​en-special-series-on-covid-options-to-support-the-incomes-of-informal-workers-during-covid-19.​ashx [accessed 2023-06-10]
  14. Giri A, Rana D. Charting the challenges behind the testing of COVID-19 in developing countries: Nepal as a case study. Biosafety and Health. Jun 2020;2(2):53-56. [CrossRef]
  15. Park H, Park H. Research evaluation of Asian countries using altmetrics: Comparing South Korea, Japan, Taiwan, Singapore, and China. Scientometrics. Aug 3, 2018;117(2):771-788. [CrossRef]
  16. Park H, Youn J, Park H. Global mapping of scientific information exchange using altmetric data. Qual Quant. Mar 15, 2019;53(2):935-955. [CrossRef]
  17. Park H, Park H. Global-level relationships of international student mobility and research mentions on social media. Profesional De La información. Mar 20, 2021;30(2) [CrossRef]
  18. Chong M, Park H. COVID-19 in the Twitterverse, from epidemic to pandemic: information-sharing behavior and Twitter as an information carrier. Scientometrics. 2021;126(8):6479-6503. [FREE Full text] [CrossRef] [Medline]
  19. What are Sltmetrics? Altmetric. URL: [accessed 2023-06-10]
  20. Yin Y, Gao J, Jones B, Wang D. Coevolution of policy and science during the pandemic. Science. Jan 08, 2021;371(6525):128-130. [FREE Full text] [CrossRef] [Medline]
  21. Lee M, Yoon H, Smith M, Park H, Park H. Mapping a Twitter scholarly communication network: A case of the association of internet researchers’ conference. Scientometrics. Aug 2017;112(2):767-797. [CrossRef]
  22. Sena V, Arranz N, Lucas P, Park H, de Arroyabe J. “Editorial: Big data and network analysis in national innovation systems (NIS)”. Technological Forecasting and Social Change. Jul 2021;168:120790. [CrossRef]
  23. Borgatti S, Everett M, Freeman L. UCINET 6 for Windows: Software for Social Network Analysis. Harvard, MA. Analytic Technologies; 2002.
  24. Gostin L. COVID-19 reveals urgent need to strengthen the World Health Organization. JAMA Health Forum. Apr 01, 2020;1(4):e200559. [FREE Full text] [CrossRef] [Medline]
  25. Hassan I, Mukaigawara M, King L, Fernandes G, Sridhar D. Hindsight is 2020? Lessons in global health governance one year into the pandemic. Nat Med. Mar 2021;27(3):396-400. [FREE Full text] [CrossRef] [Medline]
  26. Wade N. Citation analysis: A new tool for science administrators. Science. May 02, 1975;188(4187):429-432. [CrossRef]
  27. Fortunato S, Bergstrom C, Börner K, Evans J, Helbing D, Milojević S, et al. Science of science. Science. Mar 02, 2018;359(6379):eaao0185. [FREE Full text] [CrossRef] [Medline]
  28. Fraser N, Brierley L, Dey G, Polka J, Pálfy M, Nanni F, et al. The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLoS Biol. Apr 2021;19(4):e3000959. [FREE Full text] [CrossRef] [Medline]
  29. Raoofi A, Takian A, Akbari Sari A, Olyaeemanesh A, Haghighi H, Aarabi M. COVID-19 pandemic and comparative health policy learning in Iran. Arch Iran Med. Apr 01, 2020;23(4):220-234. [CrossRef] [Medline]
  30. Mahfouz B, Mulgan G, Capra L. Case Study: Mapping the impact of science on education policy during the COVID-19 pandemic. In: Proceedings of the 26th International Conference on Science and Technology Indicators. Presented at: 26th International Conference on Science and Technology Indicators; September 7-9, 2022, 2022; Granada, Spain. [CrossRef]
  31. Fry C, Cai X, Zhang Y, Wagner C. Consolidation in a crisis: Patterns of international collaboration in early COVID-19 research. PLoS One. 2020;15(7):e0236307. [FREE Full text] [CrossRef] [Medline]

ccTLD: country code top-level domain
WHO: World Health Organization

Edited by A Mavragani; submitted 07.02.23; peer-reviewed by E Ferrara, R Sauvayre; comments to author 22.03.23; revised version received 17.05.23; accepted 30.05.23; published 29.06.23.


©Han Woo Park, Ho Young Yoon. Originally published in the Journal of Medical Internet Research (, 29.06.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.