The steps taken to acquire the data included the following: log in to JCR; select Medicine, General & Internal in the “Browse categories” page; select JCR Year=2018 and Citation Indexes=SCIE in the filters; export the result to XLS format; according to the acquired journal title, select the publication year as 2018 and the literature type as Article to search the WoS core database; and export the full record of the journal literature in XLS format. Finally, we downloaded the related H1 Connect literature data and POCI data according to the list of journals.

To process the data, we undertook the following steps: use Navicat to import the full record of research papers into the local SQLite database, process the downloaded POCI data, extract the PMID numbers of all the focus papers from the full record, retrieve the references and citations of the focus papers as well as the citations of the references of the focus papers in the local database transformed based on the POCI data, and establish the relevant data tables for the subsequent calculations.

Some researchers have observed, at an early stage, that some technological innovations complete and improve current technologies without replacing them while others outright eliminate technologies that were used in the past. However, for a long time, scholars did not analyze and explain the essence of this phenomenon. It was not until 1986 that Tushman and Anderson [18] summarized the phenomenon as follows: There are 2 types of major technological shifts that disrupt or enhance the capabilities of existing firms in an industry. However, Christensen [19], a professor at Harvard Business School in the United States, argued that disruptive innovations are new technologies that replace existing mainstream technologies in unexpected ways. Building on these views, Funk and Owen-Smith [20] provided deeper and more insightful insights. They argued that the dichotomy between disruptive and augmentative technologies lacks nuance and that the impact of new technologies on the status quo is a matter of degree rather than absolute impact.

In this regard, Govindarajan and Kopalle [21] also pointed out that disruptive innovation lacks reliable and valid measurement standards. Therefore, Funk and Owen-Smith [20] created the consolidation-disruption (CD) index, which aims to quantify the degree of technological change brought about by new patents. The index drew the attention of Wu et al [22], who analogized the basic principle of the CD index to measure disruption by calculating the citation substitution of the focus paper in the citation network and who were the first to apply the evaluation of disruptive innovation to the world of bibliometrics.

As an important carrier of academic results, it is important to evaluate papers quantitatively, rationally, and efficiently in terms of their innovation [23]. The disruption (D) index has received widespread attention after it was proposed by Wu et al [22]. A subset of scholars then explored disruptive papers in specific subject areas, including scientometrics [24], craniofacial surgery [25], pediatric surgery [26], synthetic biology [27], energy security [28], colorectal surgery [29], otolaryngology [30], military trauma [31], breast cancer [32], radiology [33], ophthalmology [34], plastic surgery [35], urology [36], and general surgery [37], based on the D index. Park et al [10] also analyzed the annual dynamics of the disruption level of papers and patents across the subject areas.

Another group of scholars conducted in-depth research on the index itself: Bornmann et al [38] explored the convergent validity of the index and the variants that may enhance the effectiveness of the measure and tested the validity of the D index based on the literature in the field of physics [39]. Ruan et al [40] provided an in-depth reflection on the limitations of the application of the D index as a measure of scientific and technological progress. Liu et al [41,42] empirically investigated the stabilization time window of the D index in different subject areas and addressed the mathematical inconsistency of the traditional D index and proposed an absolute disruption index (Dz; as in Equation 2) [43].

This series of studies has made it possible to evaluate the disruption of research papers based on the D index, which has gradually matured. On this basis, Jiang and Liu [44] proposed the Journal Disruption Index (JDI) to evaluate the disruptive innovation level of journals (as in Equation 3) and validated the evaluation effect of this indicator based on Chinese SCIE journals [45].

In Equations 1, 2, and 3, N_F refers to the literature that only cites the focus paper (FP), N_B refers to the literature that cites both the focus paper and at least one reference (R) of the focus paper, and N_R refers to the literature that only cites at least one reference (R) of the focus paper but not the focus paper. n is the number of “Article” type pieces of literature contained in the journal, and Dz_i is the Dz of the i^th article in the journal.

In this study, Dz and JDI were chosen to evaluate the disruption of the selected studies at the literature and journal levels, respectively. Bornmann and Tekles [46] argued that 3 years is necessary regardless of the measurement for that discipline. Considering that, in the determination of the stabilization time window of the D index for each discipline conducted by Liu et al [42], the stabilization time window for clinical medicine is 4 years after publication. Therefore, in the process of calculating Dz in this paper, the citation time window of the focus papers was set to 2018 to 2022 to ensure the validity of the results. In addition, because the JDI value was too small, we multiplied the JDI by 1000 in all subsequent presentations.

The peer review indicators selected for this study included the peer review score (PScore), weighted peer review stars (PStar_w), and weighted peer review evaluation times (PTime_w). In this case, the weighted indicators refer to the weighting of ratings and number of evaluations by the number of evaluators when an evaluation was completed by more than one reviewer.

The advantage of using peer review indicators is that it can make up for the lag and one-sidedness of relying solely on the citations among the literature to assess the quality of the literature. It also corrects the shortcomings of the traditional JIF to judge the quality of the literature. Compared with a single impact indicator, it is more scientific.

The impact indicators selected for this study included the Cumulative Citation for 5 years (CC₅), JIF, Journal Citation Indicator (JCI), and Journal Cumulative Citation for 5 years (JCC₅). Among these, JIF is the total number of citations for scholarly articles published in the journal in the past 2 years divided by the total number of citable articles. JCI is the average Category Normalized Citation Impact (CNCI) of the citable literature published in the specific journal in the previous 3 years. The JCC₅ is obtained by dividing the sum of the citation frequencies recorded in the POCI repository of the corresponding research papers (focus papers) of the selected journals for the years 2018 to 2022 by the number of research papers published by the journal (as in Equation 4).

In Equation 4, a_iC_t represents the number of citations of the i^th research paper of the journal in year t, and n is the number of research papers published by the journal.

The academic impact and level of disruptive innovation of the selected 114 journals, the results of the correlation analyses of the indicators, and the differences in the rankings are shown in Table 1, Table 2, and Figure 1, respectively. From these, we can see that (1) journals that are at the top of the ranking of impact indicators are usually also ranked at the top in the ranking of disruptive innovation, (2) journals at the bottom of the impact rankings are usually at the bottom of the innovation rankings, and (3) there is little difference in the ranking results of journals under different impact indicators, but there is a big difference in the ranking results of influence and disruptive innovation. Although there are moderate correlations between the JDI of journals and the 3 influence indicators of JIF, JCI, and JCC₅, the average difference between the disruptive innovation ranking and academic influence ranking of the journals included in the study reached 20 places.

In order to better analyze the differences between the academic impact and level of disruptive innovation with peer-reviewed results of journals in the field of general and internal medicine, papers indexed in H1 Connect were referred to as “H-papers,” and the source journals of “H-papers” are referred to as “H-journals.” The evaluation indicators, ranking of H-journals, and the percentage of H-papers are shown in Table 3 and Table 4. We can see that (1) the top 7 journals in terms of both academic impact and disruptive innovation are all H-journals, (2) the average impact ranking of H-journals is higher than the average innovation ranking, and (3) some journals with low impact and innovation also became H-journals.

Ideally, if an article is accepted by a specific journal, it is because its overall quality is similar to other papers previously published in that journal [47]. However, journal-level indicators are, at best, only moderately suggestive of the quality of an article [48], which makes indicators that measure specific articles more popular [49]. Therefore, in this study, research papers in the field of general and internal medicine were also evaluated in terms of their academic impact and level of disruptive innovation, and the results are shown in Figure 2. From the results, we can see that research papers that rank high in the impact ranking usually also rank high in the disruptive innovation ranking. There were 8 (8/15, 53%), 96 (96/150, 64%), and 880 (880/1500, 58.67%) of the Top 0.1%, Top 1%, and Top 10% papers, respectively, selected based on the Dz and CC₅ that were the same. Second, the level of disruptive innovation of research papers with the same level of impact varied greatly, and the impact level of research papers with the same level of disruptive innovation also varied greatly. Third, despite the high correlation (r=0.635, P<.001) between the Dz and CC₅ of the selected research papers, the average difference between their innovation and impact rankings reached about 2700. Fourth, the actual analysis results showed no correlation between the innovation of the selected research paper and the number of references, which indicates that the Dz index is basically unaffected by the difference in the number of references in the actual evaluation process (r=0.006, P=.43).

In order to better analyze the differences between the academic impact, level of disruptive innovation, and peer review results of journal research papers in the field of general and internal medicine, the differences between the academic impact and disruptive innovation level and the peer review results of H-papers were analyzed. The specific results are shown in Table 5 and Figure 3. From the results, we see that there were 8 (8/15, 53%), 65 (65/150, 43.3%), and 187 (187/1500, 12.47%) H-papers among the top 0.1%, top 1%, and top 10% papers, respectively, selected based on Dz. There were 5 (5/15, 33%), 74 (74/150, 49.3%), and 220 (220/1500, 14.67%) H-papers among the top 0.1%, top 1%, and top 10% papers, respectively, selected based on CC₅. Second, there was a significant positive correlation between the peer review indicators, disruptive innovation indicators, and academic impact indicators of H-papers, reflecting the consistency between the quantitative evaluation and peer review at the overall level. Third, the average impact ranking of H-papers was 865 (top 5.68%), and the average disruptive innovation ranking was 1726 (top 11.35%), which means that the average impact ranking of H-papers was higher than the average disruptive innovation ranking. Fourth, some papers with low academic impact and disruptive innovation level also became H-papers. Fifth, compared with the CC₅, Dz has a minor correlation advantage with PTime_w and PStar_w.

In addition to rating and commenting on the included research papers, different labels are added by the reviewers of articles in H1 Connect according to their different internal characteristics. The relevant definitions are shown in Table 6 (refer to Du et al [50]). We categorized and counted the 257 H-papers (in this study, if a paper had more than 1 tag, it was counted separately in the calculation of each tag), and the results are shown in Table 7. From this, we can see that (1) research papers with the tags in the “Novel Drug Target,” “Technical Advance,” and “New Finding” categories had a high academic impact and a high disruptive innovation level; (2) research papers with the “Changes Clinical Practice” tag showed only a moderate academic impact and disruptive innovation level but had high levels of peer-reviewed recognition and attention; (3) experts showed higher recognition and concern for research papers with tags of “Negative/Null Result,” “Controversial,” “Refutation,” and “Interesting Hypothesis,” but their academic impact and disruptive innovation level were lower than others.

From these research results, large differences were seen between the innovation and impact rankings of the journals and research papers included in the study. This is also consistent with the findings of Guo and Zhou [51]. This phenomenon reflects the essential difference between the 2 different evaluation systems. It also proves that the evaluation of the disruptive innovation of research papers and journals based on Dz and JDI is not consistent with the traditional evaluation system of impact.

The essence of disruptive evaluation is to measure the innovation from the substitution level of knowledge structure. This evaluation method brings new ideas to the field of scientific research evaluation, helps relevant institutions and scholars remove the constraints of the traditional evaluation system, and helps to establish a value orientation of encouraging innovation for scientific research and scientific and technological journals, so as to promote the benign development of the academic ecology.

Although, given the consistency of scientific evaluation, there will be some uniformity in the level of disruptive innovation, level of academic impact, and peer review results of journals as well as research papers.

However, from the research results presented in this paper, we can see that (1) the top 7 journals in terms of both academic impact and disruptive innovation were all H-journals, (2) more than one-half of the top papers selected based on Dz and CC₅ were the same, (3) the average H-papers were ranked at the top in terms of impact and innovation, and (4) the results of the different evaluation systems only had high consensus on the authoritative journals and top papers in the field.

These findings are also consistent with those of Goman [52] and Chi et al [53]. A fundamental reason for this phenomenon is that the purposes of the 3 evaluation systems are inherently different. Therefore, in the actual evaluation process, the 3 kinds of indicators are not interchangeable, and the combination of the 3 evaluation systems may be a feasible solution to establish a comprehensive, scientific, and reasonable journal evaluation system.

From the aforementioned findings, we can see that research papers of the “Novel Drug Target,” “Technical Advance,” and “New Finding” types have high academic impact and high levels of innovation, which is in line with their classification definitions. This is also consistent with the findings of Du et al [50], Thelwall et al [54], and Jiang and Liu [55]. Second, research papers of the “Changes Clinical Practice” type showed only moderate levels of innovation and impact but had high levels of peer-reviewed recognition and attention. This reflects the difficulty of evaluating the impact of a particular academic paper on clinical practice, whether the evaluation system is based on academic impact or level of disruptive innovation. Third, peer-reviewed experts show higher recognition and concern for research papers of the types “Negative/Null Result,” “Controversial,” “Refutation,” and “Interesting Hypothesis,” but the level of impact and disruptive innovation of these papers are lower. This partly reflects the current academic community’s excessive focus on positive results; deliberate avoidance of negative results; and overall lack of support for debatable, falsified, and other types of research.

Several scholars have recently suggested that the evaluation system of medical journals should be redesigned for contemporary clinical impact and utility [56]. In this regard, Thelwall and Maflahi [57] advocated that the references of a guideline are an important basis for studying clinical value. However, Traylor and Herrmann-Lingen [58] found only a weak correlation between the number of citations to individual journals in the guidelines and their respective JIF. Therefore, the JIF is not a suitable tool for assessing the clinical relevance of medical research. The results of this study similarly found that research papers of the “Changes Clinical Practice” type showed only moderate levels of disruptive innovation and academic impact, but such research papers received higher recognition and attention from peer review experts. Therefore, combining quantitative evaluation with peer review may be a feasible way to measure the impact of medical research on clinical practice.