Prognostic and diagnostic studies that train and apply machine learning (ML) models on health data often fail to adhere to minimal reporting standards [1,2], with inadequate details on model development and evaluation, and fail to fully cover sources of bias [3-5]. Transparent reporting on the development and application of such models is believed to improve reliability, fairness, and usefulness as well as ethical, legal, and regulatory oversight [6].

There are many reporting guidelines and checklists that have been developed for health research [7]. Although a recent evaluation of the use of reporting guidelines by peer reviewers was not able to reach a conclusion on their use and utility [8], one other study found a positive association between reviewer ratings of adherence to reporting guidelines and favorable editorial decisions [9]. Another study reported a significant positive correlation between adherence to reporting guidelines and citations and between adherence to reporting guidelines and publication in higher impact factor journals [10]. Furthermore, there is evidence that the completeness and quality of reporting of research studies is associated with the use of reporting guidelines [11-17].

However, ML modeling studies do not often use reporting guidelines developed for statistical models [18], and reporting deficiencies are being seen in contemporary ML modeling articles [19]. To address this issue, multiple reporting guidelines specific to ML studies have been developed [1,6,20-27]. In general, reporting guideline “inflation” can lead to confusion among authors and peer reviewers regarding the appropriate ones to use [28]. These ML study reporting guidelines overlap but are not the same, with each covering a subset of what can be considered good reporting practice [20]. They each focus on subsets of a typical analytics workflow without being comprehensive. Some guidelines may be nonspecific to health care (eg, DC-Check [26]), and others may omit important aspects of ML modeling methodology (eg, the absence of guidance on model tuning and optimization [21,27]). The existence of multiple guidelines may hinder the adoption of good reporting guidelines in general, as this makes it difficult for researchers to determine the most suitable set of guidelines to use for a particular study and for journal editors to consistently prescribe reporting requirements for authors [20].

In this paper, we consolidated items from current ML reporting guidelines and checklists into a single set. We limited our scope to in-silico studies and those where an ML model is running in shadow mode, as these are necessary first steps in developing ML models that are useful in practice [27]. Given that our items consolidate material from previously published consensus and expert guidelines, they can be used by authors as a checklist to ensure adequate reporting of their studies, by peer reviewers to confirm that important details are included in manuscripts, and by journal editors to ensure that good reporting practices are applied and applied consistently across articles and journals.

The objective of this study was to identify high-quality ML reporting guidelines and checklists from the literature and consolidate them into a set of reporting items. The approach we followed was informed by recommended practices for conducting scoping reviews [29,30] and developing reporting guidelines [31,32].

The literature search focused on articles that contained a checklist, a flow diagram, or structured text developed to guide authors on the minimum level of detail to include in research papers reporting findings or a specific aspect of research [32].

We executed a very broad search query in January 2023 and updated it in June 2023 on PubMed to maximize the capture of published reporting guidelines. The query searched for all articles that contained the term “reporting guideline” with either one of the terms “machine learning” or “artificial intelligence,” that is, “reporting guidelines” AND (“machine learning” OR “artificial intelligence”), limited to English articles published in or after the year 2000. We retrieved 73 articles from the search.

The EQUATOR Network [33] database was searched for articles on “machine learning” and “artificial intelligence” separately. We also adopted an expert-driven approach by curating recent review articles that presented reporting guidelines, reproducibility guidelines, reviews of guidelines, or critique articles on ML practices in medical or biological studies. Recursively, we also reviewed their respective references that reported informative items of the strengths or weaknesses of ML-based studies in medicine. This enabled us to identify an additional 137 articles.

After removing duplicates, there were 192 articles remaining.

The titles and abstracts of the 192 identified articles were reviewed by one of the authors (WK) and screened according to the inclusion and exclusion criteria presented in Textbox 1.

As illustrated in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram [36] in Figure 1, the selection criteria resulted in 27 articles. These were then subjected to a quality assessment, as described in the following section.

We constructed a checklist to evaluate the quality of reporting guidelines and checklists based on the recommendations in the literature on good practices for guideline development [31,32,37,38]. We only considered criteria that were relevant for our purposes (eg, whether a reporting guideline had a website was not considered a quality criterion that was relevant for our purposes). The quality assessment criteria are listed in Table 1.

Two independent reviewers, one coauthor (WK) and an independent reviewer, separately evaluated each of the identified 27 articles. The κ statistic was used to evaluate the interrater agreement of the score for each quality criterion, and the results are shown in Table 1. The average κ across all criteria was 0.71. Values equal to and above 0.61 are considered to be moderate [42].

The use of a Delphi method to narrow down the reporting items was not used very often (criteria 5 and 6), and the literature review that was performed was not always clearly documented (criterion 3). Many guideline development efforts have used a form of expert meeting to review the reporting items (criterion 7), although we did not require these to be face to face. Pilot testing of reporting guidelines was performed in less than half the time (criterion 8).

We considered an article to be of high quality if at least 61% of the responses were “yes” when each criterion score was averaged across the 2 raters (after reconciliation, where relevant). This threshold also coincides with the median of the quality score. This threshold resulted in 17 articles that were assigned a quality score≥5.5. The scores for each of the 27 articles and whether they scored high or low are shown in Table 2.

As a sensitivity analysis, if a higher threshold score of 7 is used (77% “yes” responses on the quality criteria; this is the mean value), this would have resulted in the exclusion of only 1 item from our final list. Therefore, there was little sensitivity in the 61% to 77% threshold range.

Reporting items were identified in the 17 articles that made it through the quality assessment. Items in these articles were excluded if they met any of the following criteria:

Note that the abovementioned exclusions pertain to items and not articles. For example, guideline articles such as DECIDE-AI (Developmental and Exploratory Clinical Investigations of Decision support systems driven by Artificial Intelligence), CONSORT-AI (Consolidated Standards of Reporting Trials–Artificial Intelligence), and SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence) discuss the use of clinical trial designs to evaluate ML models. The specific reporting items that pertain to the conduct of clinical trials were not considered, but other items that were specific to the modeling task were included in our review.

This set of items was converted into a checklist that can be used to ensure that all relevant information has been reported.

The resultant checklist was sent to 11 members of the JMIR AI editorial board for an independent review. They were asked to comment on the clarity of the definitions and levels of granularity of the reporting items. They were also invited to identify any significant omissions that were not covered by the consolidated list and any challenges that they foresee in its application. The feedback from the editorial board was incorporated into a revision of the reporting items and checklist in this paper.

Six prognostic or diagnostic ML studies that were published in JMIR AI were identified, and the resulting items were used to assess whether the authors reported the specific details. The perspective of this assessment was as a reviewer and was not intended to score the papers on adherence to our reporting items. This was performed by one of the authors (KEE). The result of that effort was used to further revise the item definitions, item descriptions, and checklist to ensure that they can be applied in practice.

During the presentation of the consolidated reporting items and checklist, a number of general principles were followed in their presentation to ensure that they would be practically useful and within our scope:

We identified 37 reporting items from 17 high-quality reporting guideline articles. The articles were published between 2014 and 2023, and most articles were published in the last 3 years.

The percentage of articles that covered each of the categories is shown in Table 3. We see that the median percentage of “data description” articles that appeared in source articles was 75%, but only a median of 33% of the “model explainability” items appeared in a source article. This table indicates the coverage of each category in the ML reporting guidelines literature and illustrates the importance of developing a consolidated set of reporting items across this body of work.

Although the items assume that the observations pertain to patients, they can also pertain to providers or administrators, depending on the study. Furthermore, the following order of categories follows a theoretical workflow sequence for a study; however, this may differ from how the information is actually presented in an article or a report.

The reporting items and checklist are described in terms of the information that needs to be documented in a study report. They are defined assuming a single ML model that is being developed and evaluated, but in practice, multiple models may be part of the same study, and the items should be generalized accordingly.

In subsequent sections, we present reporting items grouped into the 5 categories.

We examined the coverage of the reporting items in each article in each of the reporting categories. Figure 2 presents the proportions of reporting items discussed in each article. The last bar on the right shows the median proportions of proposed items that were discussed in the articles for each category. Although the consolidated items have a higher focus on describing the “study details” (median 70%) and “data description” (median 75%), reporting on model explainability and details of methodology are lacking with a median percentage of 33% and 40%, respectively, being covered in those articles.

We also saw that some source articles, such as A06, A08, A23, and A24, did not cover all 5 categories that we defined. This demonstrates that the coverage of reporting items and categories is not uniform across source articles.

We also examined how often reporting items in this paper were discussed in the 17 high-quality articles. Figure 3 shows the list of reporting items and the number of articles that discuss them. We can see that some reporting items shown toward the top are rarely discussed and, not surprisingly, belong to the methodology category. Handling missing data and model performance evaluation during training are the most common methodology items.

The bottom of the figure with the items appearing in most articles is dominated by data description items. Sample size calculations and providing information on data characteristics are the most common in that category. Internal and external model evaluation items also appeared in almost all the articles. This is not surprising, because that is a basic requirement for in-silico studies.

Table 9 shows the rank correlation between article coverage and article quality score for the 17 articles. We can see that there is a strong and significant positive relationship only for the category “study details.” This suggests that the highest-quality source papers focused more on defining reporting items for the study details compared with articles with quality scores closer to the median. Coverage of the reporting items was not associated with article quality for the other categories, although the negative association for the “methodology” and “model evaluation” categories suggests that the highest-quality source papers focused less on those items.

Multiple guidelines have been developed for ML prognostic and diagnostic modeling studies [3,4,27,45,46,50,52-59,61,62,79]. However, having multiple and overlapping guidelines can be confusing for researchers, journal editors, and reviewers, as it is less obvious which guidelines should be adhered to, and this may create friction in the adoption of good reporting practices. Such a state of affairs is not desirable, as there is evidence that the application of reporting guidance has benefits in higher-quality research and positive editorial decisions and increased citations.

In this paper, we developed a consolidated list of 37 reporting items for ML prognostic and diagnostic modeling studies conducted in silico or in shadow mode. The consolidated items were obtained from 17 high-quality source articles and represented consensus and expert guidance to ensure that these types of ML studies are adequately reported upon. The items were reviewed by independent experts and were applied in a review of a sample of JMIR AI articles, both of which informed further refinements and clarifications.

The results in Figure 2 support the need for our consolidated reporting items and checklist, since none of the source articles covered all 37 items presented. Some of our reporting items were covered in only 2 source articles, with the highest coverage being reporting items that appeared in 16 of the 17 source articles.

It was found that descriptions of the data and reporting study details were the most common categories in the source literature, whereas model explainability and methodological guidance (ie, data preparation descriptions and descriptions of model training) had the least coverage in the ML reporting guidelines literature overall. Using our checklist can help ensure that explainability and methodological considerations are documented appropriately. However, current explainability methods have come under criticism in terms of their value and the guarantees that they can provide [80].

The items can be used as a checklist (see the checklist in Multimedia Appendix 1), where the authors can indicate whether they have reported the relevant information. If it is reported, then the location in the article can be provided. If an item is not applicable, some reasoning can be provided. The use of a checklist will ensure consistency and that authors consider all reporting items.

Satisfying the reporting items does not necessarily mean that prognostic and diagnostic ML modeling study articles will increase in length. Some of the information may already be published elsewhere; therefore, the reporting items can be satisfied by citing other work. Any additional burden on authors will arguably be offset by the noted benefits to authors and readers, as well as by enhancing the transparency and reproducibility of ML studies.

Although we used a broad search of the literature to identify and select papers for our consolidated guidelines and checklist, we acknowledge that some may have been missed. However, any impact from missed papers is likely to be limited. In addition, a recent systematic review of reporting guidelines for prediction models that included ML within its scope did not identify any articles that were not within our scope [20].

Because this was a consolidation effort, whatever omissions exist in the literature, for example, will also be evident in our consolidated items. A design decision in this study was not to add new items that were not directly derived from the 17 source articles to ensure that we do not introduce a new source of bias in the results.

While we accounted for interrater reliability in the assessment of source article quality, the process of consolidating guidelines does involve subjectivity in the application of the inclusion and exclusion criteria. The external review by the JMIR AI editorial board helped ensure face validity to ameliorate some individual subjectivity.

The scope of our reporting items has been on structured data. ML modeling for other data types, such as text and images, was outside the scope of this study. Although this limitation excludes many studies, a large number of studies are still covered by our reporting guidelines and checklist. As noted earlier, approximately one-fifth of the noneditorial papers published in JMIR AI were on structured data. Future work should extend these consolidated reporting items and checklist to other data modalities (such as text and images).

Another scoping decision for the reporting items in this paper is that they focused only on in-silico and shadow mode studies. These are necessary first steps in developing and validating ML models. Future research can extend this work and cover the deployment and monitoring of prognostic and diagnostic ML models into practice.

This paper does not provide an explanation and elaboration for each reporting item. This information and references to examples are available in the source articles, as indicated in the mapping tables, and were therefore not repeated here.

The checklist that is provided in this paper (Multimedia Appendix 1) is intended to be used by authors to ensure that all relevant information for prognostic and diagnostic studies is reported. Extending the reporting checklist to an evaluation instrument would require additional effort to develop scoring items and scoring schemes [81,82]. This would allow the community to assess and track changes in reporting quality over time and evaluate study quality in meta-analysis projects.