Published on in Vol 27 (2025)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/59598, first published .
Evaluation and Comparison of the Academic Quality of Open-Access Mega Journals and Authoritative Journals: Disruptive Innovation Evaluation

Evaluation and Comparison of the Academic Quality of Open-Access Mega Journals and Authoritative Journals: Disruptive Innovation Evaluation

Evaluation and Comparison of the Academic Quality of Open-Access Mega Journals and Authoritative Journals: Disruptive Innovation Evaluation

Authors of this article:

Yuyan Jiang1 Author Orcid Image ;   Xue-li Liu1, 2 Author Orcid Image ;   Liyun Wang1 Author Orcid Image

Original Paper

1Henan Research Center for Science Journals, Xinxiang Medical University, Xinxiang, China

2Faculty of Humanities & Social Sciences, Xinxiang Medical University, Xinxiang, China

Corresponding Author:

Xue-li Liu, BM

Faculty of Humanities & Social Sciences

Xinxiang Medical University

Library and Information Building, 2nd floor

No. 601 Jinsui Avenue, Hongqi District

Xinxiang,

China

Phone: 86 1 383 736 0965

Email: liueditor03@163.com


Background: Some scholars who are skeptical about open-access mega journals (OAMJs) have argued that low-quality papers are often difficult to publish in more prestigious and authoritative journals, and OAMJs may be their main destination.

Objective: This study aims to evaluate the academic quality of OAMJs and highlight their important role in clinical medicine. To achieve this aim, authoritative journals and representative OAMJs in this field were selected as research objects. The differences between the two were compared and analyzed in terms of their level of disruptive innovation. Additionally, this paper explored the countries and research directions for which OAMJs serve as publication channels for disruptive innovations.

Methods: In this study, the journal information, literature data, and open citation relationship data were sourced from Journal Citation Reports (JCR), Web of Science (WoS), InCites, and the OpenCitations Index of PubMed Open PMID-to-PMID citations (POCI). Then, we calculated the disruptive innovation level of the focus paper based on the local POCI database.

Results: The mean Journal Disruption Index (JDI) values for the selected authoritative journals and OAMJs were 0.5866 (SD 0.26933) and 0.0255 (SD 0.01689), respectively, showing a significant difference. Only 1.48% (861/58,181) of the OAMJ papers reached the median level of disruptive innovation of authoritative journal papers (MDAJ). However, the absolute number was roughly equal to that of authoritative journals. OAMJs surpassed authoritative journals in publishing innovative papers in 24 research directions (eg, Allergy), accounting for 40.68% of all research directions in clinical medicine. Among research topics with at least 10 authoritative papers, OAMJs matched or exceeded MDAJ in 35.71% of cases. The number of papers published in authoritative journals and the average level of disruptive innovation in each country showed a linear relationship after logarithmic treatment, with a correlation coefficient of –0.891 (P<.001). However, the number of papers published in OAMJs in each country and the average level of disruptive innovation did not show a linear relationship after logarithmic treatment.

Conclusions: While the average disruptive innovation level of papers published by OAMJs is significantly lower than that of authoritative journals, OAMJs have become an important publication channel for innovative research in various research directions. They also provide fairer opportunities for the publication of innovative results from limited-income countries. Therefore, the academic community should recognize the contribution and value of OAMJs to advancing scientific research.

J Med Internet Res 2025;27:e59598

doi:10.2196/59598

Keywords



Henry Oldenburg, the founding editor of the journal Philosophical Transactions of the Royal Society, noted that the purpose of scholarly journals is to provide a forum for researchers to transfer knowledge to each other and contribute to perfecting the grand design of all the philosophical arts and sciences. However, due to technical limitations, printed journals have been the only form of journal publication since the first journal was introduced in the 17th century. Therefore, researchers must usually wait for 10 to 12 months or more before publishing a paper [1]. The appearance of web-based publications has solved the limitations of page size and publishing cycles. Readers can access the latest research more quickly through online publishing and distribution, which provides a more flexible and efficient way for knowledge dissemination and sharing. This transformation significantly enhanced content deliverability, but accessibility has not kept pace. The urgent need to address this issue has fueled the rise of the open-access (OA) movement and the development of OA journals. Simultaneously, the rapid development of science and technology, combined with the pressure of “publish or perish,” has created an increasing demand for efficient, timely, and cost-effective publication methods. Furthermore, the rapid increase in the number of publications has imposed a growing financial burden on academic institutions and researchers seeking access to scholarly literature. Providing a platform that can accommodate the rapid publication and huge volume of emerging research while facilitating academic exchange and reducing the economic strain of accessing academic resources has become a widespread concern for the scientific community.

Open-access mega journals (OAMJs) may currently be one of the most effective solutions, offering faster publishing speeds, higher acceptance rates [2], and support for the initial exploration of new fields [3]. These advantages align well with the needs of authors [4]. Additionally, many OAMJs provide readers with value-added functions that most traditional journals still lack [5].

Since the establishment of the Public Library of Science (PLoS) in 2001, numerous traditional academic publishers have launched their own OAMJs to expand their product lines [6]. This has positioned OAMJs as increasingly important players in facilitating academic exchanges [7]. However, the evaluation criteria of most OAMJs only pay attention to research rationality [8]. Coupled with the financial incentives of OA publishers to publish widely, concerns have arisen about the quality of research published in OAMJs, which have a higher publishing error rate [9] and possibility of predatory publishing [10].

Most OAMJ papers have received a significant number of citations [11], which often exceed the average citation level [12]. However, the evaluation processes of authoritative journals cannot guarantee the accuracy of the research [13]. Despite this, prejudice still exists among some scholars [14], and the complex attitudes within different academic groups have not significantly improved [15].

It is unfair to view OAMJs solely as a negative development. As Professor John Ioannidis of Stanford University [16] points out, several features of OAMJs align with desirable scientific practices. For instance, OAMJs enhance the visibility of academic papers and their broader academic impact. Furthermore, OAMJs reduce selective publication biases by accommodating results that are considered undesirable in traditional professional journals, thus fostering the diversity of perspectives and challenging orthodoxy.

Since 1972, the mainstream evaluation system has relied on citation metrics [17], which means indicators such as the Journal Impact Factor (JIF) have been widely used in academic review, promotion, and tenure evaluation. However, it is important to note that citation metrics primarily reflect the impact of journals rather than their academic innovativeness. Related research has shown that relying on these impact indicators introduces biases against innovation [18], often resulting in flawed decisions in research assessment and evaluation [19].

Disruptive innovation is defined as the process by which a new product, service, or technology transforms an existing market by creating a new market or significantly altering an existing one. This is one of the core driving forces for developing productivity. Therefore, conducting a scientific and reasonable evaluation of the innovation levels in OAMJ papers is crucial for determining their value. Given that clinical medicine is the primary focus of OAMJ papers [5], this paper selects the authoritative journals and the representative OAMJs related to this field as its research subjects. This study compares and analyzes the differences in disruptive innovation levels between these 2 groups and examines the countries and research directions for which OAMJs serve as publication channels for disruptive innovations.


Research Objects

The objects of this study include 2 groups of journals, representing authoritative journals in the field of clinical medicine and OAMJs. The selection criteria for authoritative journals were as follows: (1) JIF ranks high in the field; (2) selected journals are not review journals; and (3) academic authority is widely recognized. The selection criteria for representative OAMJ journals were as follows: (1) OA peer-reviewed journals that charge publication processing fees; (2) an annual average of more than 2000 OA research papers published on clinical medicine topics; (3) a 3-year impact factor within Q1 or Q2.

Through a search of relevant journals in the field of clinical medicine, we discovered that the impact factors of the British Medical Journal (BMJ), the New England Journal of Medicine, The Lancet, and the Journal of the American Medical Association (JAMA) ranked among the highest in impact factors between 2017 and 2019, with widely recognized academic authority [20-25]. We also found that BMJ Open, PLoS ONE, and Scientific Reports published more than 2000 open-access research papers on clinical medicine topics annually, establishing a certain degree of academic influence.

Based on these criteria, this study selected BMJ, New England Journal of Medicine, The Lancet, and JAMA as the authoritative journals and BMJ Open, PLoS ONE, and Scientific Reports as the representative OAMJs.

Because there is no disruptive innovation in review articles, and because Bornmann et al [26] suggest that the disruption index should only be calculated for research papers with at least 10 references, this study exclusively focused on clinical medical research papers with more than 10 references from the aforementioned journals.

Data Sources

The data required for this study included journal information, literature information, and citation relationship information, obtained through Journal Citation Reports (JCR), Web of Science (WoS), InCites, and the OpenCitations Index of PubMed open PMID-to-PMID citations (POCI), respectively. POCI, a data set sourced from the National Institutes of Health, comprises over 700 million citations and 29 million bibliographic resources. In POCI, citations are regarded as first-class data entities with 7 attribute fields, including the citation time span. To evaluate disruptive innovation, we selected focus papers and used their PMIDs along with POCI data identifiers to calculate the disruptive innovation level for each paper.

Data Acquisition and Processing

First, we logged into JCR, WoS, and InCites to retrieve the journal information and literature information of the selected journals and save them. Then, we logged into opencitations.net, entered the download page, selected the POCI data set, and downloaded the dump data in CSV format via the Figshare platform. Next, we used Navicat software (PremiumSoft CyberTech Ltd) to import all data into the local SQLite database and process the downloaded POCI data. Finally, in the local database based on the POCI data transformation, we used the PMID numbers extracted from the full records of the focus literature to identify the citation relationship associated with the focus papers. We established relevant data tables for the subsequent calculations.

The disruption index calculation was highly dependent on the time window chosen. The findings of Liu et al [27] confirm that the stabilization time window for the disruption index varies across different disciplines. We believe this is partly due to the different stages of development in various disciplines. Bornmann and Tekles [28] suggest that a time window of 3 years or more is an important prerequisite for ensuring the validity of the calculation results. Given the complexity of the disruption index calculation, the large volume of data [29], and our lack of access to extensive commercial citation data resources, we used the open citation data provided by OpenCitations for the calculation. To guarantee the accuracy of the results of the paper's innovation measurement while minimizing the computational workload, the citation time window in this study was set to 3 years.

Evaluation Indicators

Since Christensen [30] of Harvard Business School posited the theory of disruptive innovation in 1996, it has become an important paradigm in the field of innovation research. Based on this theory, Wu et al [31] put forward the disruption index (often referred to as the D Index for short), which can be used to measure disruption by calculating the citation substitution of focus papers in the citation network. Based on the disruption index, Bornmann et al [26] discussed the effectiveness of the convergence of this index and the variants that may improve the measurement effect. Ruan et al [32] analyzed the application limitations of the disruption index for measuring progress in the fields of science and technology.

Considering the process of metric calculation, F-type citations reflect that the focus paper disrupts the reference, B-type citations indicate that the focus paper is a development of the reference, and R-type citations reveal that the focus paper inherits the reference. Liu et al [33] argued that the disruption Index is a “relative” concept because it measures disruption based on the relative size of NF (F-type citations) and NB (B-type citations). They defined the disruption index, which is only reflected by F-type citations, as the “absolute disruption index.” Additionally, Liu et al [27,34] empirically studied the stabilization time window for the disruption index in different subject areas and resolved mathematical inconsistencies in its traditional formation (eg, NR not satisfying monotonicity).

Based on this disruption index, Bornmann et al [35], Horen et al [36], Sullivan et al [37], Meyer et al [38], Jiang and Liu [39], and others have mined disruptive papers in the fields of scientometrics, craniofacial surgery, pediatric surgery, synthetic biology, and energy security respectively. On this basis, Jiang and Liu [40] proposed a method for calculating the Disruption index using open citation data during previous research. They carried out evaluations at both the literature and journal levels [41-45], contributing to a mature framework for evaluating disruptive innovation.

In equations (1) to (3), NF refers to the literature that only cites the focus paper (FP); NB refers to the literature that cites both the focus paper and at least one reference (R) of the focus paper; NR refers to literature that only cites at least one reference (R) of the focus paper but not the focus paper; n is the number of articles contained in the journal; and DZi is the absolute disruption index (DZ) of the ith article in the journal.


Comparative Analysis of the Journals’ Disruptive Innovation Levels

Annual changes in JIF, JDI, and the number of published articles for the 7 journals selected in this study between 2017 and 2019 are shown in Table 1. The key findings are as follows. First, the average JDI of the selected authoritative journals is 0.5866, while the average JDI of OAMJs is 0.0255, indicating a significant difference. Second, the JDI values of the authoritative journals are higher than those of OAMJs. Third, JDI does not necessarily increase with a higher JIF.

Additionally, to better measure the difference in disruptive innovation level between OAMJ papers and authoritative journal papers, this study defined 2 parameters: the median DZ of authoritative journals (MDAJ) and the median DZ of OAMJs (MDOAMJ), which were 0.289 and 0.0019, respectively. The findings indicated that (1) the disruptive innovation level (94.60 %) of authoritative journal papers was higher than MDOAMJ and (2) only 1.48% (861/58151) of OAMJ papers had a disruptive innovation level higher than MDAJ. However, the number of OAMJ papers (n=861, 1.48%) was close to half of the number of authoritative journals (n=1128 of 2222).

Table 1. Annual changes in JIFa, JDIb, and number of articles for the 7 selected journals.
JournalJIFJDINumber of articlesNumber of COCIc articles

201720182019201720182019201720182019All201720182019All
BMJd23.25927.60430.2230.26910.18970.2580140147140427123126124373
NEJMe31.39836.21638.6370.52760.38110.4584275276277828258270265793
The Lancet53.25459.10260.3920.74590.71320.6682242191192625233183187603
JAMAf79.25870.6774.6990.92681.03250.8684179147157483177136140453
BMJ Open2.4132.3762.4960.07300.02110.021021602112351277842055202533457425
PLoSg ONE2.7662.7762.7400.01800.02110.019019,97617,46114,88052,31710,4449153789227,489
Scientific Reports4.1224.0113.9880.01610.02230.018024,80617,11819,84361,76790716543765323,267

aJIF: Journal Impact Factor.

bJDI: Journal Disruption Index.

cCOCI: OpenCitations Index of CrossRef Open DOI-to-DOI Citations

dBMJ: British Medical Journal.

eNEJM: New England Journal of Medicine.

fJAMA: Journal of the American Medical Association.

gPLoS: Public Library of Science.

Comparative Analysis of Disruptive Innovation Level Across Fields of Paper

Due to the complexity and diversity of clinical medical research, this study summarized statistics based on the classification results of InCites to assess the disruption of OAMJ papers and authoritative journal papers across various research fields. The findings, shown in Table 2, revealed several key insights.

There were differences in the average levels of disruptive innovation among OAMJ papers published in different research fields. The highest level was observed in the research field of immunology (0.1965), while neuroimaging had the lowest level (0.0049).

Similarly, there were differences in the average levels of disruptive innovation in authoritative journal papers, with reproductive biology showing the highest level (15.7902) and sport sciences (0.0031) showing the lowest level.

Across all fields, authoritative journal papers outperformed OAMJ papers in terms of disruptive innovation at the overall level. In 24 research directions such as allergy—accounting for 40.68% of all clinical medicine research areas—the number of OAMJ papers with disruptive innovation levels exceeding MDAJ surpassed half the number of authoritative journal papers. Among the topics with no less than 10 authoritative papers published, the topics in which the number of papers published by OAMJ with a level of disruptive innovation exceeding MDAJ's was no less than half of the number of papers published in authoritative journals accounted for 35.71% of the total.

Table 2. Comparative analysis of disruptive innovation level across fields of paper.
FieldAuthoritative journalsOAMJa

AmountAverage
(Dzb)
MIDc
(Dz)
Amount
(Dz>MDOAMJd), n (%)
AmountAverage
(Dz)
MID
(Dz)
Amount (Dz>MDAJe), n (%)
Allergy40.22680.04813 (75)320.02640.00282 (6.25)
Andrology0f20.04080.0027
Anesthesiology160.84930.428016 (100)2800.02470.00152 (0.71)
Audiology and speech-language pathology0630.01360
Behavioral sciences02560.02180.0017
Cardiac and cardiovascular systems2173.18480.2502211 (97.24)25100.01410.001515 (0.60)
Clinical neurology1463.29950.3190138 (94.52)18340.01490.001210 (0.55)
Critical care medicine571.64190.225357 (100)5630.01640.001210 (1.78)
Dentistry, oral surgery and medicine22.17441.21662 (100)6070.02880.00390 (0)
Dermatology182.42970.465818 (100)2990.04000.00384 (1.34)
Emergency medicine81.32460.12138 (100)1960.05080.002412 (6.12)
Endocrinology and metabolism1342.84540.2190128 (95.52)25830.02150.002445 (1.74)
Engineering, biomedical0­—4670.04180.0052
Gastroenterology and hepatology652.29360.315162 (95.38)17690.02450.002520 (1.13)
Genetics and heredity310.87520.172028 (90.32)22470.00620.000211 (0.49)
Geriatrics and gerontology101.17170.315710 (100)3210.03270.00185 (1.56)
Health care sciences and services214.03060.242821 (100)7320.05380.002729 (3.96)
Health policy and services54.36982.98454 (80)920.06460.00540 (0)
Hematology722.93230.721365 (90.28)6890.01580.00152 (0.29)
Immunology442.67760.386340 (90.91)34320.19650.002218 (0.52)
Infectious diseases787.16370.369870 (89.74)19690.04600.004349 (2.49)
Integrative and complementary medicine0250.09960.0043

Materials science, biomaterials01830.07160.0152

Medical ethics010.01310.0131
Medical informatics10.22450.22451 (100)300.11210.00693 (10)
Medical laboratory technology10.21200.21201 (100)250.01150.00130 (0)
Medicine, general and internal3576.23890.3111335 (93.84)14150.14110.002868 (4.81)
Medicine, legal0570.01630.0013
Medicine, research and experimental48.53060.17414 (100)630.06380.00673 (4.76)
Neuroimaging030.00490.0025
Neurosciences91.05620.49699 (100)77940.00940.00089 (0.12)
Nursing01590.06380.0047
Nutrition and dietetics291.07170.160528 (96.55)7640.03000.003128 (3.66)
Obstetrics and gynecology891.68460.126479 (88.76)10190.03490.002854 (5.30)
Oncology2078.94471.3285199 (96.14)60320.03370.002513 (0.22)
Ophthalmology194.70310.107318 (94.74)17760.02480.001971 (4)
Orthopedics220.78360.099822 (100)6580.02060.002121 (3.19)
Otorhinolaryngology60.93140.07476 (100)2440.02490.002715 (6.15)
Pathology20.057501 (50)570.01390.004557 (100)
Pediatrics660.64160.156963 (95.45)6670.03160.002026 (3.90)
Peripheral vascular disease251.20930.184423 (92)6160.01750.00259 (1.46)
Pharmacology and pharmacy41.58331.41074 (100)11410.06120.00656 (0.53)
Primary health care0600.16770.0099
Psychiatry500.24030.050943 (86)15110.03780.001396 (6.35)
Psychology, clinical30.46010.33003 (100)1620.00750.00050 (0)
Public, environmental, and occupational health1393.81250.3887132 (94.96)38610.05070.004492 (2.38)
Radiology, nuclear medicine, and medical imaging81.40560.27147 (87.50)14180.03030.002122 (1.55)
Rehabilitation10.31390.31391 (100)2560.03400.00104 (1.56)
Reproductive biology115.790215.79021 (100)3630.01590.00220 (0)
Respiratory system761.40900.232273 (96.05)9340.02480.001914 (1.50)
Rheumatology501.87200.117444 (88)7380.01910.002118 (2.44)
Sport sciences20.003101 (50)7800.02900.0016780 (100)
Substance abuse119.74140.148210 (90.91)1910.02680.00407 (3.66)
Surgery471.00350.310745 (95.74)7970.03100.002515 (1.88)
Toxicology10.00810.00811 (100)3120.03080.0029111 (35.58)
Transplantation50.04640.03034 (80)820.01340.00199 (10.98)
Tropical medicine71.41920.69696 (85.71)2680.03110.00391 (0.37)
Urology and nephrology562.73700.147652 (92.86)15730.01750.001937 (2.35)
Virology26.04731.68382 (100)12120.01070.00060 (0)

aOAMJ: open-access mega journal.

bDZ: absolute disruption index.

cMID: median.

dMDOAMJ: median DZ of the OAMJ papers.

eMDAJ: median DZ of the authoritative journal papers.

fNot applicable.

Comparative Analysis of Disruptive Innovation Level of Papers Across Countries

The disruptive innovation levels of OAMJ papers and authoritative journal papers published by different countries are shown in Tables 3 and 4, which include data for countries with records in both types of journals. To better describe the relevant results, this study used the following criteria. First, the number of authoritative journal papers is used as a measure of national scientific research strength. A higher number of published authoritative papers reflects stronger national scientific research capabilities. Second, the difference in the disruptive innovation levels between published OAMJ papers and authoritative journal papers serves as a measure of the recognition of OAMJs by scholars in different countries. A smaller difference indicates a higher degree of recognition. Third, the ratio of OAMJ papers to authoritative journal papers measures the degree of favoritism scholars from different countries show toward OAMJs. A larger ratio suggests a higher degree of favoritism toward OAMJs by scholars from a particular country.

From Tables 3 and 4, we observed several key insights. First, there was a difference between the average disruptive innovation level of OAMJ papers across countries, with South Korea ranking the highest (0.5476) and Serbia ranking the lowest (0.0043). Similarly, the average disruptive innovation level of authoritative journal papers also varied across countries, with Peru ranking the highest (35.6125) and Serbia ranking the lowest (4.7261).

Table 3. Comparison of papers published in different countries.
CountryAvga (DZbAJc)Amount of AJ articles, n (%)

Amount
MDOAMJd, n (%)
Avg (DZOAMJ)Amount of OAMJ articles, n (%)Amount
DzMDAJe, n (%)
United States4.72611537 (76.01)1435 (93.36)0.05881499 (2.58)60 (4)0.01240.9753
United Kingdom5.4159833 (41.20)818 (98.20)0.11132967 (5.10)178 (6)0.02053.5618
Canada7.5419479 (23.69)471 (98.33)0.0607845 (1.45)27 (3.20)0.00801.7641
Germany8.5920399 (19.73)396 (99.25)0.0538517 (0.89)12 (2.32)0.00631.2957
Australia9.7657355 (17.56)350 (98.59)0.07971336 (2.30)33 (2.47)0.00823.7634
France11.2194340 (16.82)338 (99.41)0.0757371 (0.64)13 (3.50)0.00681.0912
Netherlands8.2619296 (14.64)291 (98.31)0.0563648 (1.11)19 (2.93)0.00682.1892
Italy9.9492275 (13.60)274 (99.64)0.1065255 (0.44)12 (4.71)0.01070.9273
Spain12.6857227 (11.23)225 (99.12)0.0380297 (0.51)6 (2.02)0.00301.3084
Switzerland10.9062216 (10.68)215 (99.54)0.1126265 (0.46)13 (4.91)0.01031.2269
Sweden8.9276191 (9.45)185 (96.86)0.0346490 (0.84)13 (2.65)0.00392.5654
Denmark8.9724169 (8.36)165 (97.63)0.0761381 (0.65)12 (3.15)0.00852.2544
China Mainland13.1854167 (8.26)166 (99.40)0.0738945 (1.62)47 (4.97)0.00565.6587
Japan17.6698166 (8.21)161 (96.99)0.0439319 (0.55)7 (2.19)0.00251.9217
Belgium13.5063160 (7.91)158 (98.75)0.1137202 (0.35)9 (4.46)0.00841.2625
Brazil14.4105139 (6.87)135 (97.12)0.0448147 (0.25)7 (4.76)0.00311.0576
Poland12.6655128 (6.33)128 (100)0.025480 (0.14)2 (2.50)0.00200.6250
South Korea17.2768117 (5.79)117 (100)0.5476181 (0.31)4 (2.21)0.03171.5470
New Zealand12.9764103 (5.09)100 (97.09)0.1546231 (0.40)4 (1.73)0.01192.2427
South Africa15.547898 (4.85)97 (98.98)0.0410163 (0.28)6 (3.68)0.00261.6633
India15.846895 (4.70)94 (98.95)0.0913154 (0.26)10 (6.49)0.00581.6211
Norway15.719092 (4.55)88 (95.65)0.0238334 (0.57)8 (2.40)0.00153.6304
Israel11.671487 (4.30)86 (98.85)0.018943 (0.07)1 (2.33)0.00160.4943
Finland13.946284 (4.15)83 (98.81)0.1585161 (0.28)7 (4.35)0.01141.9167
Austria16.616283 (4.10)83 (100)0.253765 (0.11)2 (3.08)0.01530.7831
Russia17.603577 (3.81)77 (100)0.025912 (0.02)0 (0)0.00150.1558
Ireland12.572676 (3.76)76 (100)0.0607189 (0.32)8 (4.23)0.00482.4868
Taiwan19.815368 (3.36)68 (100)0.0245213 (0.37)4 (1.88)0.00123.1324
Czech Republic10.152265 (3.21)65 (100)0.046421 (0.04)1 (4.76)0.00460.3231
Singapore17.407765 (3.21)65 (100)0.0808106 (0.18)4 (3.77)0.00461.6308
Argentina15.202161 (3.02)60 (98.36)0.117818 (0.03)3 (16.67)0.00770.2951
Turkey19.707361 (3.02)61 (100)0.098321 (0.04)4 (19.05)0.00500.3443
Hong Kong18.619358 (2.87)57 (98.28)0.020893 (0.16)2 (2.15)0.00111.6034
Mexico25.828058 (2.87)57 (98.28)0.013233 (0.06)0 (0)0.00050.5690
Chile19.792652 (2.57)52 (100)0.158737 (0.06)6 (16.22)0.00800.7115
Colombia25.513952 (2.57)52 (100)0.075217 (0.03)2 (11.76)0.00290.3269
Greece16.543852 (2.57)52 (100)0.068140 (0.07)2 (5)0.00410.7692
Portugal26.489052 (2.57)52 (100)0.298257 (0.10)4 (7.02)0.01131.0962
Hungary15.070751 (2.52)51 (100)0.009920 (0.03)0 (0)0.00070.3922
Malaysia18.808647 (2.32)47 (100)0.052373 (0.13)4 (5.48)0.00281.5532
Kenya22.986243 (2.13)40 (93.02)0.062960 (0.10)3 (5)0.00271.3953
Pakistan25.570541 (2.03)40 (97.56)0.118147 (0.08)7 (14.89)0.00461.1463
Ukraine27.070637 (1.83)37 (100)0.00794 (0.01)0 (0)0.00030.1081
Iran29.160236 (1.78)36 (100)0.061149 (0.08)3 (6.12)0.00211.3611
Romania31.136736 (1.78)36 (100)0.021013 (0.02)0 (0)0.00070.3611
Peru35.612535 (1.73)35 (100)0.011811 (0.02)0 (0)0.00030.3143
Nigeria33.445534 (1.68)34 (100)0.041024 (0.04)2 (8.33)0.00120.7059
The Philippines25.493532 (1.58)32 (100)0.261917 (0.03)3 (17.65)0.01030.5313
Bangladesh30.126931 (1.53)31 (100)0.081578 (0.13)9 (11.54)0.00272.5161
Serbia29.495031 (1.53)31 (100)0.00436 (0.01)0 (0)0.00010.1935

aAvg: average.

bDZ: absolute disruption index.

cAJ: authoritative journal.

dMDOAMJ: median DZ of the open-access mega journal (OAMJ) papers.

eMDAJ: median DZ of the authoritative journal papers.

Table 4. Indicator correlation analysis.
CategoryAJaOAMJb

Avgc (Dzd)AmountAmount
(Dz>MDOAMJe)
Proportion
(Dz>MDOAMJ)
Avg
(Dz)
AmountAmount
(Dz>MDAJf)
Proportion
(Dz>MDAJ)


AJ

Avg
(Dz)

1–0.1494–0.14520.359**–0.1118–0.265**–0.256**–0.0809–0.461**–0.322**

Amount
11**–0.514**0.518**0.895**0.744**0.459**0.513**0.448**

Amount
(Dz>MDOAMJ)



1–0.503**0.518**0.894**0.743**0.459**0.512**0.444**

Proportion
(Dz>MDOAMJ)




1–0.196*–0.537**–0.497**–0.197*–0.338**–0.448**
OAMJ

Avg
(Dz)




10.579**0.732**0.827**0.878**0.463**

Amount




10.806**0.523**0.615**0.764**

Amount
(Dz>MDAJ)






10.821**0.711**0.616**

Proportion
(Dz>MDAJ)







10.689**0.398**








10.553**









1

aAJ: authoritative journal.

bOAMJ: open-access mega journal.

cAvg: average.

dDZ: absolute disruption index.

eMDOAMJ: median DZ of the OAMJ papers.

fMDAJ: median DZ of the authoritative journal papers.

* P<.01

**P<.001

There was a linear relationship between the number of authoritative journal articles and the average disruptive innovation level of each country after logarithmization (as shown in Figure 1), with a correlation coefficient of –0.891, which indicated that authoritative journals did not consistently recognize the work of scholars from different countries. However, for OAMJ papers issued by each country, the average level of disruptive innovation did not show a linear relationship after logarithmic processing (as shown in Figure 2).

Scholars from different countries also had varying levels of recognition for OAMJs and tendencies to publish them. A higher recognition of OAMJs correlated with a higher likelihood of publishing papers in these journals (0.553). This was significantly correlated with the disruptive innovation levels of both authoritative journal papers (–0.461) and OAMJ papers (0.878). Moreover, the proportion of OAMJ papers meeting the MDAJ threshold was significantly and positively correlated with the tendency to publish papers in OAMJs (0.689).

Among all countries, those that published more than 10% of OAMJ papers with a level of disruptive innovation above MDAJ, while maintaining an average annual publication count of at least 1 in the selected journals, included Argentina, Turkey, Chile, Colombia, Pakistan, Philippines, Bangladesh, and Egypt.

Figure 1. Distribution of the number of publications in authority journals with average disruptive innovation levels among different countries.
Figure 2. Distribution of the number of publications in open-access mega journals (OAMJs) with the average disruptive innovation levels among different countries.

Principal Results

Differences Between the Disruptive Innovation Levels of OAMJ and Authoritative Journals

In this study, the disruptive innovation levels of OAMJs were lower than those of authoritative journals. A plausible explanation is that OAMJs require more rationality than innovation during the review process. Therefore, OAMJ papers were more focused on incremental innovation and proximity innovation, while authoritative journals were more likely to publish papers with breakthrough innovation.

OAMJs As Important Publishing Platforms for Innovative Research Across Fields

In this study, the overall disruptive innovation levels of authoritative journal papers were higher than those of OAMJ papers across all clinical medical research fields. However, the number of OAMJ papers with a higher level of disruptive innovation than MDAJ was close to half of the number of authoritative journal papers. Moreover, in almost half of the research directions examined, the number of OAMJ papers with a higher level of disruptive innovation than MDAJ exceeded half of the count for authoritative journal papers.

This phenomenon indicates that the contribution of OAMJs to the overall development of science rivals that of authoritative journals. While OAMJs must continue to emphasize overall quality control, their contribution to scientific development should not be overlooked [46].

Additionally, OAMJ papers involve some research directions that are not covered by authoritative journal papers. Therefore, it is unreasonable to judge the level of disruptive innovation of a paper based solely on the source journal without distinguishing the research direction. Thus, we must evaluate the contributions of OAMJs fairly and scientifically [47].

OAMJs Provide a Fair Publishing Channel for Innovative Achievements from Limited-Income Countries

Research has confirmed that national bias exists in the peer-review process [48], and specific journals have preferences for their authors’ countries of origin [49]. Therefore, authoritative journals demonstrate varying degrees of recognition for the research results of scholars from different countries. This phenomenon was also evident in this study, where notable differences were observed in the disruptive innovation levels of authoritative journal papers published by scholars from different countries. Furthermore, the disruptive innovation levels in authoritative journals from each country were linearly and negatively correlated with the volume of publications.

A reasonable explanation for this is the first-mover advantage of high-income countries in the field of science and technology. These nations benefit from a group of senior scholars in various fields whose work often receives recognition that exceeds its quality in the current peer-review process [50]. This recognition also extends to the young scholars they collaborate with [51]. On the other hand, this study also found that OAMJs play an important role in promoting innovation diffusion in limited-income countries. Therefore, OAMJs may provide fairer publishing opportunities, and in tandem with postpublication peer review [52], may contribute to the accelerated development of science and technology.

Limitations

This study has some limitations that must be acknowledged. First, participating authors with different ranks were treated as equal contributors, disregarding variations. Second, all focus papers were categorized under a single subject classification, without considering their inherent interdisciplinary properties. Third, only the knowledge diffusion of focus papers within the scope of periodical literature was considered, ignoring other venues. These limitations must be explored in future research based on natural language processing technology, and more accurate results can be obtained by jointly using multiple types of citation data sources.

Conclusions

The disruptive innovation levels of OAMJs were different from those of authoritative journals. However, OAMJs have become an important publication channel for innovative results in several research directions, providing fairer opportunities for researchers from limited-income countries to publish their work. Therefore, the academic community should acknowledge and appreciate the contributions of OAMJs to scientific research.

Acknowledgments

Author XL was supported by the National Social Science Foundation of China (23BTQ085).

Authors' Contributions

XL reviewed and edited the manuscript, supervised the study, and acquired funding. YJ conceptualized the study, was responsible for the methodology, provided the software and other resources, performed the formal analysis, and wrote the original manuscript draft. YJ and LW curated the data.

Conflicts of Interest

None declared.

  1. Björk B, Solomon D. The publishing delay in scholarly peer-reviewed journals. J Informetr. Oct 2013;7(4):914-923. [CrossRef]
  2. Björk B. Publishing speed and acceptance rates of open access megajournals. Online Inf Rev. Nov 12, 2018;45(2):270-277. [CrossRef]
  3. Solomon D. A survey of authors publishing in four megajournals. PeerJ. 2014;2:e365. [FREE Full text] [CrossRef] [Medline]
  4. Björk B-C. Evolution of the scholarly mega-journal, 2006-2017. PeerJ. 2018;6:e4357. [FREE Full text] [CrossRef] [Medline]
  5. Björk B-C. Have the "mega-journals" reached the limits to growth? PeerJ. 2015;3:e981. [FREE Full text] [CrossRef] [Medline]
  6. Khelfaoui M, Gingras Y. Expanding Nature: Product line and brand extensions of a scientific journal. Learn Publ. Oct 05, 2021;35(2):187-197. [CrossRef]
  7. Wakeling S, Spezi V, Fry J, Creaser C, Pinfield S, Willett P. Academic communities: The role of journals and open-access mega-journals in scholarly communication. J Doc. Jan 14, 2019;75(1):120-139. [CrossRef]
  8. Björk B, Catani P. Peer review in megajournals compared with traditional scholarly journals: Does it make a difference? Learn Publ. Jan 20, 2016;29(1):9-12. [CrossRef]
  9. Erfanmanesh M, Teixeira da Silva JA. Is the soundness-only quality control policy of open access mega journals linked to a higher rate of published errors? Scientometrics. Jul 5, 2019;120(2):917-923. [CrossRef]
  10. Beall J. Predatory publishing is just one of the consequences of gold open access. Learn Publ. Apr 2013;26(2):79-84. [CrossRef]
  11. Nugent K, Edriss H, Ball S, Björk B-C. Mega journals, scientifically sound peer review and medical organizations. Am J Med Sci. Jan 2019;357(1):3-4. [CrossRef] [Medline]
  12. Borrego A. Are mega-journals a publication outlet for lower quality research? A bibliometric analysis of Spanish authors in. Online Inf Rev. Nov 05, 2018;45(2):261-269. [CrossRef]
  13. Retracted: growing up in a recession. Rev Econ Stud. Mar 06, 2023;90(2):1009-1009. [CrossRef]
  14. Pinto AP, Mejdalani G, Mounce R, Silveira LF, Marinoni L, Rafael JA. Are publications on zoological taxonomy under attack? R Soc Open Sci. Feb 10, 2021;8(2):201617. [FREE Full text] [CrossRef] [Medline]
  15. Dalton ED, Tenopir C, Björk B. Attitudes of North American academics toward open access scholarly journals. Portal Libr Acad. 2020;20(1):73-100. [CrossRef]
  16. Ioannidis JPA, Pezzullo AM, Boccia S. The rapid growth of mega-journals: threats and opportunities. JAMA. Apr 18, 2023;329(15):1253-1254. [CrossRef] [Medline]
  17. Garfield E. Citation analysis as a tool in journal evaluation. Science. Nov 03, 1972;178(4060):471-479. [CrossRef] [Medline]
  18. McKiernan E, Schimanski L, Muñoz NC, Matthias L, Niles M, Alperin J. Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations. eLife. Jul 31, 2019:8. [CrossRef]
  19. Wang J, Veugelers R, Stephan P. Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy. Oct 2017;46(8):1416-1436. [CrossRef]
  20. Malmivaara A. Generalizability of findings from randomized controlled trials is limited in the leading general medical journals. J Clin Epidemiol. Mar 2019;107:36-41. [FREE Full text] [CrossRef] [Medline]
  21. Maleck WH, Boldt J, Koetter KP. German publications in leading English language journals. Dtsch Med Wochenschr. Jan 19, 2001;126(3):33-41. [CrossRef] [Medline]
  22. Gehanno J, Ladner J, Rollin L, Dahamna B, Darmoni SJ. How are the different specialties represented in the major journals in general medicine? BMC Med Inform Decis Mak. Jan 21, 2011;11(1):3. [FREE Full text] [CrossRef] [Medline]
  23. Gehanno J, Rollin L, Ladner J, Darmoni SJ. How is occupational medicine represented in the major journals in general medicine? Occup Environ Med. Aug 12, 2012;69(8):603-605. [CrossRef] [Medline]
  24. Wu J, Li H, Huang Y, Tong G, Tian M, Wu Y, et al. Chinese contribution to NEJM, Lancet, JAMA, and BMJ from 2011 to 2020: a 10-year bibliometric study. Ann Transl Med. May 2022;10(9):505. [FREE Full text] [CrossRef] [Medline]
  25. Gehanno J, Grosjean J, Darmoni SJ, Rollin L. Reliability of citations of medRxiv preprints in articles published on COVID-19 in the world leading medical journals. PLoS One. Aug 10, 2022;17(8):e0264661. [FREE Full text] [CrossRef] [Medline]
  26. Bornmann L, Devarakonda S, Tekles A, Chacko G. Are disruption index indicators convergently valid? The comparison of several indicator variants with assessments by peers. Quant Sci Stud. Aug 2020;1(3):1242-1259. [CrossRef]
  27. Liu X, Shen Z, Liao Y, Zhu M, Yang L. Research on the stable time window of disruption index. Lib Inf Serv. Sep 20, 2021;65(18):49-57. [CrossRef]
  28. Bornmann L, Tekles A. Disruption index depends on length of citation window. Eur J Public Health. Mar 25, 2019;28(2). [CrossRef]
  29. Song P, Feng C, Long C, Yang Z, Song Y. Study on discovery of outstanding scientific and technological talents in specific domains based on optimized disruptive index. J Intell. May 10, 2022;41(5):61-65. [CrossRef]
  30. Christensen C. The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail. Rev., updated, and with a new chapter, Nachdr. Boston, MA. Harvard Business School Press; 2008.
  31. Wu L, Wang D, Evans JA. Large teams develop and small teams disrupt science and technology. Nature. Feb 13, 2019;566(7744):378-382. [CrossRef] [Medline]
  32. Ruan X, Lyu D, Gong K, Cheng Y, Li J. Rethinking the disruption index as a measure of scientific and technological advances. Technol Forecast Soc Change. Nov 2021;172:121071. [CrossRef]
  33. Liu X, Liao Y, Zhu M. A preliminary study on the application of disruption index to scientific research evaluation. Inf Stud Theory Appl. Dec 31, 2021;44(12):34-40. [CrossRef]
  34. Liu X, Shen Z, Liao Y, Yang L. The research about the improved disruption index and its influencing factors. Lib Inf Serv. Dec 20, 2020;64(24):84-91. [CrossRef]
  35. Bornmann L, Devarakonda S, Tekles A, Chacko G. Disruptive papers published in Scientometrics: meaningful results by using an improved variant of the disruption index originally proposed by Wu, Wang, and Evans (2019). Scientometrics. Mar 14, 2020;123(2):1149-1155. [CrossRef]
  36. Horen S, Hansdorfer M, Kronshtal R, Dorafshar A, Becerra A. The most disruptive publications in craniofacial surgery (1954-2014). J Craniofac Surg. Oct 01, 2021;32(7):2426-2430. [CrossRef] [Medline]
  37. Sullivan GA, Skertich NJ, Gulack BC, Becerra AZ, Shah AN. Shifting paradigms: The top 100 most disruptive papers in core pediatric surgery journals. J Pediatr Surg. Aug 2021;56(8):1263-1274. [CrossRef] [Medline]
  38. Meyer C, Nakamura Y, Rasor BJ, Karim AS, Jewett MC, Tan C. Analysis of the innovation trend in cell-free synthetic biology. Life (Basel). Jun 11, 2021;11(6):551. [CrossRef] [Medline]
  39. Jiang Y, Liu X. A bibliometric analysis and disruptive innovation evaluation for the field of energy security. Sustainability. Jan 05, 2023;15(2):969. [CrossRef]
  40. Jiang Y, Liu X. A new method of calculating the disruption index based on open citation data. J Inf Sci. Oct 03, 2024. [CrossRef]
  41. Jiang Y, Liu X. An innovative evaluation index of scientific journals—Journal Disruption Index (JDI) and its empirical research. Chin J Sci Technol Period. Dec 2006;33(4):965-972. [CrossRef]
  42. Jiang Y, Liu X. A construction and empirical research of the journal disruption index based on open citation data. Scientometrics. May 18, 2023;128(7):3935-3958. [FREE Full text] [CrossRef] [Medline]
  43. Jiang Y, Wang L, Liu X. Innovative evaluation of Chinese SCIE-indexed scientific journals: An empirical study based on the disruption index. Chin J Sci Technol Period. Aug 15, 2023;34(8):1060-1068. [CrossRef]
  44. Jiang Y, Liu X, Zhang Z, Yang X. Evaluation and comparison of academic impact and disruptive innovation level of medical journals: bibliometric analysis and disruptive evaluation. J Med Internet Res. May 31, 2024;26:e55121. [FREE Full text] [CrossRef] [Medline]
  45. Jiang Y, Liu X. The disruptive innovation evaluation and empirical analysis of Chinese, Japanese, Indian and South Korean scientific journals. J Scholarly Publ (forthcoming). 2025. [CrossRef]
  46. Park M, Leahey E, Funk R. Papers and patents are becoming less disruptive over time. Nature. Jan 2023;613(7942):138-144. [CrossRef] [Medline]
  47. Jiang Y, Liu X. The relationship between absolute disruption index, peer review index and CNCI: a study based on virology papers. Lib Inf Serv. Feb 05, 2023;67(3):96-105. [CrossRef]
  48. Thelwall M, Allen L, Papas E, Nyakoojo Z, Weigert V. Does the use of open, non-anonymous peer review in scholarly publishing introduce bias? Evidence from the F1000Research post-publication open peer review publishing model. J Inf Sci. Jul 05, 2020;47(6):809-820. [CrossRef]
  49. Zhu H. Home country bias in academic publishing: A case study of the he New England Journal of Medicine. Learn Publ. Jun 2021;34(4):578-584. [CrossRef]
  50. Kardam K, Kejriwal A, Sharma K, Kaushal R. Ranking scholarly work based on author reputation. 2016. Presented at: International Conference on Advances in Computing, Communications and Informatics (ICACCI) .?2088; September 21-24:2084-2088; Jaipur, India. [CrossRef]
  51. Bu Y, Murray DS, Xu J, Ding Y, Ai P, Shen J, et al. Analyzing scientific collaboration with “giants” based on the milestones of career. Proc Assoc Info Sci Tech. Feb 2019;55(1):29-38. [CrossRef]
  52. Jiang Y, Liu X. Open peer review: Mode, technology, problems, and countermeasures. Chin J Sci Tech Period. Sep 15, 2022;33(9):1196-1205. [CrossRef]


BMJ: British Medical Journal
DZ: absolute disruption index
JAMA: Journal of the American Medical Association
JCR: Journal Citation Reports
JDI: Journal Disruption Index
JIF: Journal Impact Factor
MDAJ: the median of authoritative journal papers
MDOAMJ: the median of open-access mega journal papers
OA: open access
OAMJ: open-access mega journal
PLoS: Public Library of Science
POCI: OpenCitations Index of PubMed open PMID-to-PMID citations
WoS: Web of Science


Edited by A Schwartz; submitted 17.04.24; peer-reviewed by M Hassan, F Lin, A Borrego; comments to author 15.05.24; revised version received 02.07.24; accepted 13.11.24; published 15.01.25.

Copyright

©Yuyan Jiang, Xue-li Liu, Liyun Wang. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 15.01.2025.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.