In line with major clinical research questions on BCBM, this study evaluated screening performance using the Preliminary Attainability Assessment of Real-World Data (PAR) framework, which can assist clinicians in assessing data attainability from an RWD source at a preliminary stage of the study design. This study was divided into 2 phases. In the preparation phase, we first identified clinical questions related to current unmet needs according to the perspectives of clinical experts on BCBM for evaluation of the framework. In the evaluation phase, the working group assessed the preliminary data attainability of the listed clinical questions with a specific data source, using the PAR framework presented in the following section.

Data attainability was defined in this study as the availability of data for reconstruction and extraction required from a particular data source to conduct clinical research regarding sample size, data fields, and content.

The Samsung Medical Center Breast Cancer Registry (SMC BCR) is a hospital-based real-time registry implemented in March 2021 that leverages the institution’s anonymized and deidentified CDW platform: the Data Analytics and Research Window for Integrated Knowledge (DARWIN)–C. The inclusion criteria for the SMC BCR comprise the intersection of three patient conditions: (1) visited at least one of the medical oncology, breast surgery, or radiation oncology departments, (2) diagnosed with breast cancer under codes C50 or D05 of the International Classification of Diseases codes, and (3) aged older than 15 years at enrollment [19]. The number of breast cancer patients in SMC BCR was 45,129, and the registry covered the period from June 1995 to December 2021. The SMC BCR consists of 24 base data marts that represent disease-specific breast cancer characteristics and care pathways. The registry provides clinical variables in the form of structured data, including demographics, diagnostic history, treatment information (operation, chemotherapy, and radiation therapy), laboratory test results, and featured data fields extracted from free-text records, such as pathology, radiology, or genomic laboratory reports [19].

For BCBM patient identification, we used a BCBM data mart in the SMC BCR. The BCBM data mart contains predefined and preprocessed data from patients (n=1443 up to December 2021) with at least one BM indicator after a breast cancer diagnosis. The indicators, defined by clinical experts, are as follows: craniotomy record (payment-based), whole-brain radiotherapy treatment (regions for radiation treatment are brain, whole brain, and partial), gamma knife (code for gamma knife), and intrathecal methotrexate treatment. The data quality of the BCBM data mart was evaluated using manual chart reviews performed by 2 clinical nurses with expertise in data management; data validity was greater than 98% via 10% random sampling.

The study protocol was approved by the Institutional Review Board of the SMC (2021-09-036), which waived the need for informed consent as our study data were deidentified, and anonymized data were extracted from the CDW. This study followed the Standards for Quality Improvement Reporting Excellence (SQUIRE) guidelines [20].

In this study, we propose the PAR framework for data attainability screening at the preliminary step and evaluate its utility with clinical questions that reflect the most recent and internationally relevant unmet needs of clinicians in BCBM [21-26]. A survey was conducted to evaluate the clinical significance of the clinical questions. The mean score was 4.37 (range 3.57-5.00). We found that the correlation between scores given by experts was higher for the questions with higher average scores (Multimedia Appendices 1 and 2). RWE generation has received attention in the BCBM therapeutic area owing to limited clinical trial opportunities despite increasing clinical importance. However, incomplete gold standards for RWD study protocols and the unpredictable “hidden labor” of the secondary use of clinical data serve as barriers when clinical researchers attempt to design the most suitable methods to address their research questions [16,17]. We identified particular gaps when SMC built a site-specific CDW platform and developed a BCR as its first implementation case. After the release of the SMC BCR, clinical researchers attempted to generate RWE by using this registry, especially in areas with significant unmet needs, such as BCBM. Nevertheless, clinical research using RWD has not been as successful as expected in our experience despite the fact that well-known technical barriers are addressed; the CDW’s functional user interface provides clinicians with access to nonidentified and anonymous high-quality data sets. The greatest challenge faced by clinical researchers at the next step was securing a sufficient understanding of the data to avoid information distortion during the research data preparation stage. Therefore, we proposed the PAR framework to assess the feasibility of research at the preliminary stage and evaluated this framework with BCBM clinical questions with various entry points. To the best of our knowledge, a systematic framework to explore research feasibility at the conjunction between the data source and clinical questions has yet to be presented. Without such a framework, sample size, data fields, and content have not been properly accounted for. The methodology of this study can contribute to the acceleration of RWE generation by strengthening the transparency and reproducibility of the RWD research process and lowering the entry barrier for clinical researchers.

Clinical researchers with research questions derived from empirical insight in clinical practice have difficulties securing an understanding of accumulated RWD and solidifying their study designs. In contrast to conventional medical research methodologies, securing the reliability and validity of research variables for RWD studies is designed at a post– rather than pre–data collection stage. In addition, RWD is a conceptual collective term encompassing all data obtained through health care activities, and the content varies by data source [27]. Therefore, research using RWD requires significant and iterative effort prior to formal hypothesis testing, including selecting an appropriate data source, curating the data, repurposing it, and preprocessing it [28,29]. As a result, local information system expertise and deep content knowledge are often required to understand the idiosyncratic manner in which data sources are captured and stored [17,30]. Depending on the level of the data structure, data extraction frequently requires a high level of technical training as well. Through the process of matching research variables with data fields, the PAR framework illustrates how the process of clarifying research questions refines the range of data that should be analyzed before addressing a specific research question. Consequently, time and effort could be greatly reduced to ensure understanding of the selected data source.

The advantage of the PAR framework is that the results of stages 1 and 2 are explicitly communicated and can be reused. For example, in clinical question D, the research variable “time from neoadjuvant to BM” can be measured as the time between “neoadjuvant regiment start date” and “BM event date.” This level of operational definition can be reviewed and reused by peers with clinical expertise regardless of the storage structure of the data source. Above all, this reusability-enhancing reproducibility is further extended when applied to data sources that adopt not only institution-specific structures but also common data models, such as the Observational Medical Outcomes Partnership (OMOP) common data model [31] or the Informatics for Integrating Biology & the Bedside (i2b2) data model [32]. As specific data field conditions and query code levels with standardized data models and vocabularies can be reused, the accumulation of operational definitions of data levels can be considered an additional knowledge base. Furthermore, the accumulation of these definitions enhances consistency in the conduct of RWD research by enabling discussion regarding content validity in a more comparative manner. Accordingly, connecting formative efforts in the RWD research process, from data storage to processing, is a promising way to ensure reliability of research outcomes [33].

CDWs contain clinical data from EHRs for retrospective analysis to enable clinical researchers to utilize RWD for research directly, and the scope of RWD use has significantly expanded over the past few decades [31-34]. However, we identified several challenges while conducting RWD studies using the SMC BCR with clinical questions. A deficiency in the exact data variables for the questions recognized by clinical researchers was detected in stage 1. Researchers need to reassess stages 1 and 2 when no directly matching data field exists for a research variable. A surrogate definition could be considered as a substitution. For example, we adopted an indirect variable to represent systemic metastasis using the data for systemic treatment and used the start date of the palliative treatment cycle as the date of metastasis. Since most metastatic breast cancer patients with systemic metastasis receive systemic treatments, data on these indirect variables were readily available from the SMC BCR. However, when using an indirect variable as a surrogate, the validity of the research outcome is lower than when using a direct variable, which could be a primary limitation of RWD studies. Alternatively, if the variables will be frequently used for the generation of RWE, construction of featured data marts based on raw data and local practice rules is recommended. Since the SMC CDW has a well-constructed BCR with a BM mart, it was relatively easy for the investigators to extract the necessary data variables.

Additionally, it was not possible to extract key variables for clinical question E. This result aligned with the experts’ survey scores for the least feasible and least suitable questions using EHR data. Symptoms described by patients were only recorded in EHRs as free text, and there was no specific template or location. It was not possible to preprocess this information using a rule-based semantic engine for the CDW. Despite capturing symptoms of patients’ complaints, integrating the information into the CDW was difficult, because the location and template of the data were not aligned across departments. Moreover, the terminology for neurological symptoms is not standardized and is subject to a relatively high level of cultural dependency. To address this, integrating other types of data, such as prospective cohort studies or patient-generated data from mobile or watch-type devices, can be considered. If investigators continue to track patient-reported outcomes through cohort studies and integrate this information with other variables in CDWs, the use of RWE based on CDW data can be increased.

This study is limited in its generalizability. We assessed a single clinical domain, BCBM, and extracted data from a single data source, the SMC BCR. Further application of the PAR framework in different domains or with different types of RWD will be needed to develop the framework. However, the PAR framework and training case presented in this study could help guide clinical researchers in assessing preliminary attainability for future studies using RWD.

We proposed and evaluated a PAR framework to assess data attainability to answer major clinical research questions in BCBM. The adoption of the PAR framework is associated with improved efficiency in clinical research using RWD in the preliminary stage. This framework could contribute to improving the quality of RWD-based clinical research by enhancing its transparency and reproducibility.