Both the use and reuse of electronic medical and patient-related data offers enormous potential for clinical research [1,2]. National programs such as the German Medical Informatics Initiatives support knowledge discovery and data sharing using adequate computational infrastructure and secure processes [3]. In this context, provenance information (Textbox 1) offers access to quality assured, traceable, and credible shared data. These and other advantages of data provenance have been demonstrated, for instance, in the EU Horizon 2020 TRANSFORM project [4] or in the MeDaX Knowledge Graph Prototype [5]. Schröder et al [6] provided an “Electronic Laboratory Notebook” use case in the wet lab to show how provenance supports the understanding and reproducibility of research investigations. At the same time, researchers not considering the origin of data run into the hazard of systematically incomplete or wrong data [7].

Notably, the FAIR (findable, accessible, interoperable, reusable) guiding principles for data stewardship [9] explicitly mention provenance [10,11]. A provenance-oriented approach requires thorough planning, execution, and evaluation of data management processes in the respective application domain [2]. In the scientific context, adherence to criteria such as consistency, interoperability, and confidentiality are generally required across all software tools [1,12,13].

The concept and implementation of provenance are essential for most scientific domains, such as environmental fields (geoprocessing workflows or climate assessments), in nuclear fusion engineering, or material sciences [14,15]. In particular, the biomedical domains demand comprehensive investigation and information about their data management scenarios, including extract, transform, load jobs for data transfer and integration. Reliable data and data pipelines both require provenance data to be embedded in concepts for traceability to understand the relationships between results and source data.

Our work reviews approaches and criteria for provenance tracking in the biomedical domain and discloses current knowledge gaps. This comprises modeling aspects and metadata frameworks for meaningful and usable provenance information during the creation, collection, and processing of scientific biomedical data. The review also covers the examination of quality aspects relating to provenance.

We followed the scoping methodological framework developed by Arksey and O’Malley [16] for conducting a scoping review with the following stages: (1) stage 1—identification of the research questions (RQs); (2) stage 2—identification of relevant studies; (3) stage 3—study selection; (4) stage 4—data extraction and charting; and (5) stage 5—collating, summarizing, and reporting the results.

The protocol of this scoping review has been published in JMIR Research Protocols (international registered report identifier DERR1-10.2196/31750) [17]. In accordance with the original protocol developed for this scoping review, the search period was initially planned to include studies published from January 2006 to March 2021. However, due to the extensive nature of the data extraction and write-up process, it became apparent that additional time was necessary to ensure a comprehensive analysis of the relevant literature. As a result, we extended the search period from 2006 to the end of 2022.

No other changes were made to the original protocol. Thematic analysis methods were applied to analyze the extracted data by organizing themes according to the RQs [18]. In line with the framework developed by Arksey and O’Malley [16], the review does not attempt to assess the quality of studies or the risk of bias. It also does not assess the generalizability of the results.

The main objective of this review was to investigate existing evidence regarding approaches and criteria for provenance tracking and disclosing current knowledge gaps in the biomedical domain. The objective led to the following RQs:

Concepts and matching keywords were categorized into 4 groups (Table 1): target domain refers to the context of the research topic and includes studies with a biomedical, health care, clinical, or scientific background. In this work, scientific background is limited to domain-independent studies and excludes all other domain-specific studies. Provenance concerns the information about the genesis of a given object. Provenance properties cover specific requirements tied to the term provenance; they also describe selected characteristics in this context. Objective includes the purposes or intention of provenance capture. The comprehensive search strategy is recorded in the study protocol [17], and search strings combined with Boolean operators are attached (Multimedia Appendix 1).

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart depicts the selection process. First, we identified all relevant studies in the PubMed and Web of Science databases based on our search strategy. After deduplication, we launched a transparent screening process by importing all relevant studies into Rayyan [19], a systematic review supporting solution. The studies were then reviewed by at least 2 independent researchers. In the case of vote agreement, the study was either included in the next review phase or excluded from the review. A third independent reviewer was consulted to solve the conflict if no consensus could be reached. The study screening phase started with a title and abstract evaluation for eligibility. Included studies were submitted to a full-text screening, while performing a thorough investigation on the study report. Reviewers voted for inclusion or exclusion considering the inclusion and exclusion criteria. Finally, the residing set of qualified studies was moved into the data extraction pipeline. A description of the study selection is provided in the protocol [17].

Studies were included if they (1) were focused on the biomedical domain or were domain independent, (2) described provenance-tracking approaches, and (3) were written in English. Studies were excluded if they (1) were not specific to the biomedical or general domain, (2) were gray literature, and (3) did not focus on provenance-tracking approaches.

We followed a collaborative and iterative process to define a charting table for data extraction. Individual reviewers (KG, FK, FH, SG, AAZ, and DW) then scrutinized all studies and extracted central textual occurrences into the data extraction sheet. The variables in the data extraction sheet correspond with the RQs. General characteristics of the studies, approaches for classification and tracking of provenance, and their associated challenges along with the significance and completeness of provenance information in the given context were part of the investigational charting. The reviewers independently charted the data in a structured and consistent way and discussed the results.

The extracted data were analyzed using summary statistics by calculating the total number and percentages of all studies per category, if applicable. Charts were presented for the distribution of the individual data elements where applicable. The data analysis was partially supported with scripts in Python (version 3.10.0) [20]. Plots were generated with R version 4.0.4 (R Core Team) [21] and version 1.3.0 of the tidyverse package [22].

Further analysis was performed using qualitative evaluation. The reporting of the results and outcomes was structured according to the RQs. On the basis of the analysis of the review results, we have developed a road map for a customized provenance framework that considers the life cycle of the software framework (Provenance-Software Framework Life Cycle [SFL]). Implications for future research, practice, and policy makers were outlined. Our reporting adheres to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) reporting guidelines [23].

The search in the PubMed and Web of Science databases resulted in 764 hits and included papers from January 1, 2006, to December 31, 2022. Afterward, 140 duplicates were removed. The remaining 624 papers were subjected to title-abstract screening in an interactive selection process, leaving 118 eligible papers for the full-text review. The full-text papers were further screened to identify papers eligible for the subsequent step of data charting. During this step, additional 52 papers were excluded (see the Stage 4: Charting the Data section). These papers either did not meet the study design context (31/52, 60%) or they lacked the domain concept (15/52, 29%). Four papers reported the same study or contained parts of it, and 2 were not a full paper. A total of 66 articles were included in the data extraction phase (Table 2). The paper selection followed the PRISMA [24] approach (Figure 1), and the PRISMA-ScR checklist is presented in Multimedia Appendix 2.

All papers (n=66) were first published between 2006 and 2022 (Table 2). More than half of the reviewed studies were published within the past 5 years, which we selected for our scoping review. Predominantly, studies originated from the biomedical or health care domain (45/66, 68%), followed by the domain-independent studies (21/66, 32%). Document characteristics of the study corpus are provided in Multimedia Appendix 3.

All studies in this review were screened with regard to the 5 RQs described in the Methods section. The following subsections describe our findings for RQ1 to RQ5.

Overall, 47 papers reported 74 distinct challenges impeding the implementation of provenance. We categorized these challenges into organizational and technical groups, provided details (Multimedia Appendix 8) and presented a temporal overview of the challenges (Figure 5). In summary, issues are related to data annotation, metadata, and modeling of provenance, as well as performance-related challenges. However, the need for more detailed provenance information; consideration of compliance managing topics (eg, security-related conditions); and adherence to quality and software engineering principles such as exchange, discovery, and interoperability emerged later in the course. Furthermore, usability and scalability questions emerged very early in context with provenance consumption.

More than three-quarters of the reported challenges are technical (64/74, 86%). Thereof, approximately one-fourth is associated with provenance granularity issues (15/64, 23%). Curcin et al [1] pointed out that a granular tracking of relevant human interactions, automated processes, or logging is needed and emphasized the difficulty of choosing a proper level of granularity of provenance and associated with this, the right semantic complexity [4,57]. Beyond that, a balanced trade-off between fast execution and provenance granularity must be found [42]. In fact, a fine-granular provenance level impacts the computing and storage resources [57,58]. Furthermore, managing sensitive data restriction requires the integration of adequate security level granularity into the provenance model [56].

Approximately one-third of the reported challenges (20/64, 31%) either mention the insufficient availability of metadata—which subsequently leads to incomplete provenance models—that claims the terminological heterogeneity in the metadata terms associated with study data sets [84] or does not conform to explicit annotation standards [82]. An improved availability of provenance metadata and FAIR enrichment of the data was demanded [29,38,80]. Furthermore, stakeholders should be involved in the semantic enrichment of provenance data [4,51]. However, during this metadata annotation phase, a lack of semiautomated procedures for ontology selection, semantic modeling, or mapping techniques was reported [2,29,51]. Although the use of existing models is encouraged [59], as it improves semantic interoperability [29], the reuse of vocabularies to represent provenance information remains an extensive task [29]. Cheng et al [70] noted that it was necessary to properly integrate domain-specific demands into the provenance model. Ruiz-Olazar et al [85] claimed that a unified data model for handling metadata is still missing.

One-fifth of the studies (14/64, 22%) reported performance problems during the acquisition of provenance data, such as workflow overhead [54,73] and scalability [13,85] issues. Increasing data volumes hamper the processing of provenance visualization and stream handling [76]. One proposal with respect to the cost-intensive visualization was to reduce the size of large provenance graphs [39]. Other authors reported challenges with quality [26,29,36,41,52] and usability [54,64,69,70,73,75]. According to the literature, data quality and reuse are lacking due to the deficit in provenance deployment, particularly for observational and administrative studies [26]. Furthermore, the lack of information about experimental origins in genomics data and their related systematic quality control assessment reduce the quality of provenance and the level of creditability [52]. In particular, the low uptake of high-quality semantic models [9] and the unavailability of provenance in general [36] cause information loss and data quality issues. A minor concern is the usability because provenance is recognized to be still in infancy [73]. The challenge of applying more software engineering techniques (4/64, 6%) [4,32,42,67] was reported to facilitate provenance implementation across a broad range of software tools in the biomedical domain and beyond [1].

Significantly fewer organizational challenges (10/74, 13%) [1,4,38,56-58,64,69,73] were reported, partly attributable to a basic unawareness of provenance benefits and less exchange between stakeholders. Khan et al [58] stressed that provenance capture must be established as a standard practice, not as an afterthought. McClatchey et al [64] also recommended working toward gaining the stakeholder’s acceptance and confidence in the infrastructure. In the same vein, it is recommended to integrate developers already in the design phase [1]. However, financial challenges were reported due to the necessary investments in provenance-enabled tooling and capabilities [4]. The upcoming relevance of patient-mediated data handling raised new challenges and requirements, especially with respect to policy and governance topics [69] and rigorous validation approaches [74].

Because of the extensive information obtained from RQ1, we extended the RQs to gain more insights into the provenance tracing and classification requirements identified in RQ1.

Interestingly, most of the 15 papers referred to claims relating to quality aspects.

For example, a more robust assessment of data quality is required [36], and clearer and more consistent policies and policy ontologies are requested to prevent disclosure of sensitive data [56] and more trained staff is required [50], including data managers, software architects, or semantic web specialists. User-friendly interfaces should help scientists in the provenance querying process [54]. Developers should not only recognize technologies but also data model recommendations during the design phase [1]. Performance of provenance reasoning needs to be improved [68] and the further development of ontologies needs to be automated [4,49]. The term “intelligible machines” rather than “intelligent machines” was suggested to better respect the specific aspects of big data technologies in medical research [57]. Integrating the Healthcare Enterprise standards, health care legacy protocols, interoperability, and legacy issues are furthermore mentioned [48], and mappings between entities of various provenance models should be completed [65]. Future integration into a recognized ISO standard similar to BioCompute was proposed [60].

The literature predominantly reports on a qualitative evaluation of completeness during the data management processes. However, we found one study describing a data management process dealing with metadata for traceability in clinical studies, which delivered complete provenance in this respect [12]. Curcin [4] proposed to incorporate provenance information in the validation against standards.

One study applied data from 6 clinical research studies and >100 variables to evaluate the coverage of the provenance ontology in the semantic annotation of the study descriptions [49]. Two other documents invoked the need for minimal information elements to ensure sufficient process specification [28] and the existence of rich provenance information for reconstructing and rerunning pipelines [29].

A visualization of provenance data in neuroimaging took a semiqualitative approach for measuring the coverage. They mapped the metrics to use cases for the traceability of results and concluded that there is no absolute measure possible to verify the visualization approach [61]. Arshad et al [61] tested 15% of their workflows for verifiability of results, comparability of workflows, progression of the data for the analysis and origin of results, and evolution to see how data products evolved during an experiment.

Furthermore, Sahoo et al [38] examined the proportion of provenance metadata information across research articles using a qualitative hypothesis method. The method also provides a provenance ranking algorithm for the computation of a reproducibility rank for each article. The outcome of the self-contained DataLad data set presented valid, machine-actionable provenance information for every single result file of the performed data processing [82].

Numerical indication of completeness was not achieved in any of the other papers. However, the papers pointed out the advantages of provenance capture, for example, related to the longevity and accessibility of data after years [40].

On the basis of the insights obtained from the literature review, we developed a road map for the implementation of a tailor-made provenance framework (Provenance-SFL). This approach is based on the SFL for the development, provision, and management of software [87]. The heterogeneous tracking approaches, their artifacts, and varying degrees of fulfillment of the RQs are depicted in Figure 6 and determine our main discussion points.

This scoping review presents various approaches and criteria for provenance tracking as a crucial aspect of research, especially in the biomedical domain. Our holistic view leads to an extensive summary of pooled research results, provides possible answers to the 5 RQs and discloses current knowledge gaps.

Following the previously published scoping review protocol and the described deviation led us to include 66 full-text papers from initially 764 papers found in the PubMed and Web of Science databases. Using a structured and pretested data extraction sheet, contextual, but detailed, results were extracted to answer the 5 RQs defined in the protocol. The results of the scoping review led us to propose a Provenance-SFL road map, which distinguishes between the framework types and model characteristics, the validation status, and the requirement and provenance characteristics (Figure 6).

Most models in this review referred to the W3C PROV and OPM standards. As shown in Figure 2, an increased number of papers relate to the implementation frameworks published between 2016 and 2022. One reason for the increase in implementations might be the substantiation to extend W3C PROV and OPM [58]. Another reason might be the increasing awareness of data management practices. At present, heterogeneous data sources, dynamic infrastructures, data exchange across boundaries, and a lack of standards for quality measures characterize the state of electronic health record data sets [48].

A provenance framework must acknowledge the semantic complexity of the domain and its relevant facets and requirements [58] and the importance of good data management tooling and practices (Figure 2). In addition to requirements analysis, a thorough strategy is necessary to plan the typical data management steps such as collecting, managing, and analyzing data [30]. According to Curcin [4], validation readiness can be achieved by separating modeling and verification of provenance data from the software implementation. We agree that precise requirements analysis, as part of the software life cycle, and the subsequent individual life cycle steps, such as testing and maintenance procedures, support the consequent temporal evolution and hence improve the quality of provenance frameworks and applications.

When incorporated in an inspection, provenance information must be sufficient for a content-related validation against applicable and accepted standards [4]. Therefore, precise validation methods for provenance services regarding usability and performance, scalability, fault tolerance, and functionality are needed [64]. We saw that validation approaches are linked to the evolution of provenance modeling and subsequent implementation attempts. Curcin et al [1] argued that it was necessary to launch more formal software engineering techniques to foster provenance implementation across a broad range of software tools in the biomedical domain and beyond. In that sense, formal validation as part of the software engineering process contributes to increased software and data quality. Formal validation requires testing efforts and testing evidence. Accurate alignment of testing procedures against predefined requirements in the software life cycle could not be identified in the included papers.

Provenance information is of high value for the scientific and biomedical community (eg, researchers); support staff (eg, developers); patients and other third parties (eg, data privacy officer or authority; Figure 4). It is interesting to see that despite the high impact of provenance (Multimedia Appendix 7), only some stakeholders provide sufficient provenance information. Rather, it appears that responsibility for overall provenance management is being shifted to the support staff [88]. We argue that available technology, IT knowledge, and data management skills need to be paired with both domain-specific knowledge and combined with constraints of legal nature or guidance [4,50]. This complexity indeed results in a very time-consuming business. However, automation and metadata collection can support this process [4,6]. As a matter of fact, good provenance information strengthens the credibility of the data and proves that data have not been intentionally or unintentionally changed throughout the data life cycle [89].

We believe that the persisting disagreement on the interpretation of the term provenance hampers the uptake of existing frameworks. A unique understanding of the concepts surrounding provenance should be developed, followed by engineering efforts for modeling, implementation, and validation interventions. The ISO 8000-2:2022 standard [90] defines the term data quality and clearly recommends defining degrees of requirements. This definition should be considered for use in provenance systems.

With regard to the implementation of provenance systems, we observed that increasing legal and scientific demands require research projects to be implemented more transparently. However, the granularity of provenance could not yet be resolved and so-called knowledge bottlenecks [50,65] persist. It is important to understand that appropriate modeling of provenance information and effective provenance management techniques are required to protect sensitive provenance data.

It furthermore remains unclear how to scale provenance systems for high amounts of data [2,58] (eg, how to store and represent provenance information in an aggregated and efficient manner or how to assist users in sophisticated provenance queries [13]). Without doubt, automated and scalable solutions become impelling due to new challenges arising from the disposal and use of permanently increasing computing power [40]. Growing focus is on the usability of the interface, particularly when provenance systems are implemented in the broad medical community, including patients, physicians, and researchers [73].

The lack of mandatory specifications or guidelines for provenance capture might be the reason why other papers only mention partial completeness. We strongly recommend doing more research on completeness checks as part of provenance tracing. The level of completeness and accuracy of provenance information of core data elements, especially in real-world data, could reveal data integrity issues and thus, affect the overall validity of the study results. Furthermore, reproducibility significantly depends on the accuracy of provenance information. For example, Mondelli et al [27] delivered a tool for better scientific and longitudinal data management, which supports users, reproducibility by provenance, and reproduction through docker containers.

Interestingly, the concept of “quality of provenance” is not clearly defined in any of the papers included for this review. We believe that data quality issues need to be addressed to reach completeness, accuracy, and timeliness of the data and to create trust in it.

Another direction reveals the importance of good and systematic data management practices [51] and the coordination with relevant stakeholders through the data life cycle. As such, first approaches toward interdisciplinary collaboration started within the MIRAPIE community project [91] and were also recognized in the CAESAR project [75].

In summary, our review collects and structures the challenges during the accomplishment of provenance capture (Figure 5). Challenges that address missing or lacking organizational and technical capabilities were triangulated into more specific subcategories such as organizational (eg, investment and training and administrative) and technical (eg, granularity, performance and modeling and metadata annotation, delimitation reproducibility, and replicability) challenges.

In the realm of similar studies, a recent systematic literature review by Sembay et al [92] delved into provenance data management in health information systems. This review is a valuable resource if interested in the technical aspects, encompassing various methods, models, methodologies, and technologies associated with provenance data management in health information systems. It followed the guidelines proposed by Kitchenham and Charters [93] for performing systematic literature reviews in software engineering. Another systematic literature review by Ahmed et al [94] studied the impacts of data provenance in health care and General Data Protection Regulation compliance–based data provenance from a technological perspective. Although our RQ1 shares conceptual and domain-related similarities with the compared studies, there are notable differences in scope, RQs, methodological approaches, and thematic analysis focus.

More than a year after we published our protocol and concluded the thematic analysis of our review findings, a scoping review on biomedical data provenance was published [95]. The authors acknowledged and referenced our protocol. As they had anticipated, our review comprised 5 broad RQs, with the primary overlap occurring in RQ1. However, our scoping review offers more comprehensive results concerning the practical application of provenance and the associated challenges, including aspects such as completeness and validation. In addition, we provide in-depth descriptions of thematic areas and their subtopics, supported by tables, figures, and, most notably, a development of a tailor-made provenance framework road map. These aspects can be considered as the added value of our findings.

Our scoping review shares only few references with the other 2 (in detail: 11 out of 66 references are identical with references used in Johns et al [95] and 2 references are identical with those used in Sembay et al [92]). Therefore, all 4 reviews examine different research results and focus on other aspects. Ahmed et al [94] did not provide the list of the selected 59 articles.

This work applied a rigorous scoping review methodology using the framework developed by Arksey and O’Malley [16]. All screening stages were carried out by at least 2 independent reviews of 4 members. A previously published protocol [17] guided our review. The fact that the scoping review includes comprehensive results for the 5 related RQs and a road map for a tailor-made Provenance-SFL framework with many additional results as supplements can be considered a strength of this review. The criteria for categorization of provenance impact, requirements, and challenge classes were identified qualitatively in peer review.

One limitation may be because we excluded gray literature from our analysis.

In this paper, we highlighted various approaches and criteria for provenance tracking together with their referenced artifacts, and we developed a road map for a tailor-made Provenance-SFL framework.

Provenance capture benefits all stakeholders involved in data processing, but it is associated with manifold and individual challenges during design, implementation, and the active use scenario phase.

Sophisticated data management planning, documentation, metadata expression, and automation along the sensitive data-processing pipelines need to be scrutinized and implemented throughout the data life cycle and in adherence to the underlying infrastructure condition. With rising data volumes and the legal and scientific demands, there is an urgent need for greater transparency in implementing provenance systems in research projects, despite the challenges of unresolved granularity and knowledge bottlenecks. In addition, the roles and responsibilities of a data stewardship escorting the data should be expressed in this context and intensive training and education measures should be put in place. Guidance and recommendations are requested to provide the systematic measurement of provenance and calls for defining a minimal or gold standard. Governance for good data management and scale-up of good data management capabilities matter in this regard.

The mentioned artifacts, particularly those related to quality aspects, can be seen as transition points resulting from incomplete preliminary work. Therefore, harmonized engineering efforts are now necessary to overcoming the existing hurdles. Awareness of these challenges can facilitate an easier qualified and accurate provenance construction and auditable consumption while enforcing FAIR principles and interoperability standards for data sharing. The effect of provenance for data quality monitoring and the impact of expressive metadata on provenance quality can be considered as open RQs for future work.