This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
Journal impact factor (IF) is the leading method of scholarly assessment in today’s research world, influencing where scholars submit their research and where funders distribute their resources. COVID-19, one of the most serious health crises, resulted in an unprecedented surge of publications across all areas of knowledge. An important question is whether COVID-19 affected the
In this paper, we aimed to comprehensively compare the productivity trends of COVID-19 and non–COVID-19 literature as well as track their evolution and scholarly impact across 3 consecutive calendar years.
We took as an example 6 high-impact medical journals (Annals of Internal Medicine [Annals], The British Medical Journal [The BMJ], Journal of the American Medical Association [JAMA], The Lancet, Nature Medicine [NatMed], and The New England Journal of Medicine [NEJM]) and searched the literature using the Web of Science database for manuscripts published between January 1, 2019, and December 31, 2021. To assess the effect of COVID-19 and non–COVID-19 literature in their scholarly impact, we calculated their annual IFs and percentage changes. Thereafter, we estimated the citation probability of COVID-19 and non–COVID-19 publications along with their rates of publication and citation by journal.
A significant increase in IF change for manuscripts including COVID-19 published from 2019 to 2020 (
The rapid surge of COVID-19 publications emphasized the capacity of scientific communities to respond against a global health emergency, yet inflated IFs create ambiguity as benchmark tools for assessing scholarly impact. The immediate implication is a loss in value of and trust in journal IFs as metrics of research and scientific rigor perceived by academia and society. Loss of confidence toward procedures employed by highly reputable publishers may incentivize authors to exploit the publication process by monopolizing their research on COVID-19 and encourage them to publish in journals of
The COVID-19 pandemic has presented unique challenges to modern societies. The spread of SARS-CoV-2 worldwide during the early 2020 posed unprecedented disruption in human lives and significant stresses to public health structures and socioeconomic systems [
From an early stage, it was evident that the rate of publications was the most of any disease published thus far; however, very few constituted high-level original research including meta-analyses, systematic reviews, or control trials while preprints, opinion pieces, editorials or commentaries filled the void [
These steps of disproportionately shorter-than-usual processes taken by academic journals reflect the particular urgency for information-sharing and altmetric dissemination across scientific fields [
Critiques of such approaches have been described in the literature. Palayew et al [
A discussion on the real repercussions of an overwhelming focus on COVID-19 research for scientometrics, such as that of the journal impact factor (IF), lacks in the existing literature. Given the role of these measures as benchmarks of research productivity and scholarly impact, editorial practices in favor of fast-track COVID-19 research output have fueled critiques, which view these as attempts to artificially inflate metrics at the potential expense of research quality.
Journal IF constitutes the principal approach to assess scholarly impact in modern research. This appraisal often guides scholars to select where to submit their research and funding bodies to decide where to allocate their resources. Considering the surge of COVID-19 research from the start of the pandemic, a crucial question arises on its influence upon the
To fulfil the purpose of our study, we selected 6 high-impact medical journals, namely Annals, The BMJ, JAMA, The Lancet, NatMed, and NEJM. We conducted a comprehensive search of the literature using the Web of Science database for manuscripts published between January 1, 2019, and December 31, 2021. To distinguish between COVID-19 and non–COVID-19 publications, we filtered manuscripts based on their title, abstract, or keywords using the following terms: “COVID-19” OR “SARS-COV2” OR “Coronavirus” OR “2019-nCoV.” Citation counts for each manuscript were retrieved using the Clarivate report function. The search of the literature was performed on a single day to reduce daily updates of the database. Manuscripts were restricted to peer-reviewed original research and review articles. No further exclusion criteria were applied to our search.
Calculation of a journal’s IF in our study was based on the ratio between the number of citations and manuscripts published in a given journal over a single year for that journal.
This approach was employed to enhance the temporal resolution of the analysis of scholarly influence from journals in publishing. To assess the effect of COVID-19 and non–COVID-19 literature on scholarly impact of these journals, we initially tracked the evolution of their IFs yearly from 2019 to 2021. We then calculated the percentage change in IFs year on year. Thereafter, we estimated the citation probability of any given COVID-19 and non–COVID-19 publication by journal amid the whole duration. These were expressed as normal distributions and calculated using the normal distribution function (NORMDIST) in Microsoft Excel 2016. On a more granular level, we estimated the publication and citation rate of COVID-19 and non–COVID-19 manuscripts on a monthly basis. Statistical significance was established as
The IFs of all 6 high-impact medical journals significantly increased for manuscripts including COVID-19 published from 2019 to 2020 (
Annual impact factor of 6 high-impact medical journals (Annals of Internal Medicine [Annals], The British Medical Journal [BMJ], Journal of the American Medical Association [JAMA], The Lancet, Nature Medicine [NatMed], and The New England Journal of Medicine [NEJM]) based on (A) manuscripts with and (B) without COVID-19 publications between 2019 and 2021. Changes in annual impact factor comparing manuscripts (C-E) with and without COVID-19 publications between 2019 and 2021. ns: not significant; **
The probability of highly cited manuscripts published between 2019 and 2021 across all journals was significantly increased for COVID-19 manuscripts compared to non–COVID-19 ones (Annals:
Probability distributions of time-adjusted citation count for 6 high-impact medical journals—(A) Annals of Internal Medicine, (B) The British Medical Journal, (C) Journal of the American Medical Association, (D) The Lancet, (E) Nature Medicine, and (F) The New England Journal of Medicine—based on non–COVID-19, COVID-19–only, and combined publications between 2019 and 2021. ***
Probability distributions of (A) time-adjusted citation count across 6 high-impact medical journals (Annals of Internal Medicine [Annals], The British Medical Journal [BMJ], Journal of the American Medical Association [JAMA], The Lancet, Nature Medicine [NatMed], and The New England Journal of Medicine [NEJM]) based on COVID-19–only publications between 2019 and 2021. Publication and citation rates based on (B and D) COVID-19–only and (C and E) non–COVID-19 publications between 2020 and 2021.
The publication rate of COVID-19 manuscripts across all journals saw an increase between 2020 and 2021 with a peak by the second quarter of 2021 (
Our study showed a significant increase in IF change across 6 high-impact medical journals (Annals, The BMJ, JAMA, The Lancet, NatMed, and NEJM) based on publications including COVID-19 manuscripts from 2019 to 2020 and to 2021, when compared to non–COVID-19 ones. The probability of highly cited manuscripts was significantly increased in COVID-19 manuscripts across most journals and throughout the entire duration, when compared to non–COVID-19 ones. The citation rate for COVID-19 publications peaked by the second quarter of 2020 and that of the publication rate approximately a year later.
Our results reflect the capacity of the scientific community to respond against a global health emergency with high-impact publications on COVID-19 at an exponentially expanding rate. With high hopes for a breakthrough, scientists have indeed rushed to publish positive results on the disease [
Fast-track publications practices have frequently been scrutinized for the rigidity of the research output. This underlies concerns about the quality control of the external peer review and internal editorial evaluation, thorough revision by authors, and journal editing of the manuscript. Most notably, The Lancet and NEJM came under intense fire during the second half of 2020 due to the publishing of false data in a highly influential study regarding the benefit of hydroxychloroquine or chloroquine for the treatment of COVID-19. This information found its way under the public spotlight causing a controversial deluge, leading to its retraction and a barrage of criticism at the integrity and quality of the research and its peer reviewing [
The growing concern that editorial practices can be as much responsible for the influx in publications as the heightened popularity of the topic among the academic community becomes evident. The attributed responsibility on editorial processes is mainly based on the asymmetrical treatment of COVID-19 research and the consequential encouragement of scientists to focus on COVID-19 by journals. These two acts invite certain types of research by making the route to publication more certain and less time intensive.
There are bearing implications to the potential inflation of journal metrics of research productivity and scholarly impact, such as IF. Journal IFs are commonly used by educational or research groups and various funding bodies to make decisions on the promotion of research proposals, grant applications, but also the awarding of positions to individuals and even salary considerations. In a sense, they provide a way to gauge a scientist or research group’s academic value, or an academic journal’s scientific rigor. Journal IFs, parallel to money, constitute a value system of scholarly influence. To maintain their value across time, they need to rely on stable and transparent processes that remain intact and are always faithfully followed. For academic journals, the main mechanism that controls publication rates and incentivizes research quality is a well-established and thorough peer-reviewing process. Similar to how currency manipulation works, when peer reviewing is altered, there is a risk of distorting the value of and trust in journal IFs as perceived by academia and society.
Apart from the obvious loss of confidence toward the procedures employed by highly reputable publishers, academic journals also face the risk of losing the interest of researchers in publishing to other competitors, and this might be damaging from a business perspective. COVID-19 articles can be a contributing factor to this phenomenon which is exacerbated when fast-track reviewing is made a priority. The lack of transparency and information on how and where exactly fast-track reviewing was implemented during the pandemic magnifies this issue as there is no real way for external parties to assess how much of this is artificially driven. This sows confusion among scholars on how to evaluate the quality of published research and may encourage authors to publish in journals of
Another disservice that journal IF inflation and sudden changes in standards might cause is putting honest and hard-working researchers at a disadvantage. Shortened and sometimes less rigorous peer-review processes, combined with the observed surge in preprints, opinion pieces, and commentaries, while by no means unimportant, increase scientific noise and can waste resources that could be used in a lengthier but more impactful research. The rearrangement of peer reviewing might also benefit authors who are willing to exploit the system in order to inflate their productivity metrics and get an edge over colleagues who are less inclined to take advantage of the
Our study was prone to various inherent limitations. Assessment of IF by year can provide an enhanced temporal resolution of the scholarly influence presented by journals from their research output. However, overtime citations become inflated, and calculating year-specific IFs becomes challenging for a retrospective analysis. To overcome this, we applied a time adjustment on the citations count based on the time elapsed from the start of the search up to date. However, we were not able to account for any traction cycles or short-term events that articles might have experienced over time. Although the IF of all included journals in our study was affected symmetrically by this inherent pitfall, it is likely that the derived yearly IFs were underestimated, especially in articles published at later years. Nevertheless, this phenomenon portrays the crudeness and imperfect abstraction of IF in gaining a more granular investigation. Similarly, COVID-19 manuscripts were restricted to article and review types without taking into consideration related editorials, opinions, or commentaries that constituted a significant portion of the surge in COVID-19 publications from the start. In the same manner, the protocol of data acquisition employed to collect manuscript count was limited to a single database (ie, Web of Science), which could have consequently magnified the quantity of eligible publications. Lastly, a manual screening of the derived publications was not possible, which led to an automatic filtering based on title, abstract, or keywords that best describe COVID-19 terminology. Hence, we could not establish in full whether the retrieved manuscripts indeed focused on COVID-19 and not on other research domains related to it. Taken together, our results and the conclusions derived may be considered more conservative and should be interpreted with caution.
The rise of COVID-19 has resulted in a surge of scientific production across all areas of knowledge globally. Our findings ultimately demonstrated that the IF, likelihood of being highly cited, and publication and citation rates of manuscripts published across 6 high impact medical journals (Annals, The BMJ, JAMA, The Lancet, NatMed, and NEJM), between 2019 and 2021, were positively skewed by COVID-19 manuscripts. The eruption of COVID-19 publications reinforced the capacity of the scientific community to step up to the challenge, but casted doubt on the reliability of highly susceptible IFs—as shown here—in evaluating scholarly impact. The loss of trust on journal IFs as measures of scientific rigor and confidence in the procedures employed by highly influential publishers may incentivize a culture of exploitation by researchers and journals against the scientific process.
Supplementary tables.
Annals of Internal Medicine
impact factor
Journal of the American Medical Association
Nature Medicine
The New England Journal of Medicine
The British Medical Journal
None declared.