To find relevant archetypes from the open repository, we first had to understand which kinds of terms end users tended to enter. As archetype modeling methodology [23] shows, domain experts identify core clinical concepts and related data elements involved in a particular scenario and organize them into mind maps or design tables. These archetype design specifications are the main source of search keywords. We considered that the input of end users was mainly the names of clinical concepts or related data elements.

Ideally, the user enters the clinical concept and the system feeds back the archetype defining the concept, or the user enters data elements related to a concept and the system feeds back the archetype that covers all the data elements. However, it is difficult to distinguish clinical concepts and data elements from the end user’s input, unless it forces users to input separately. More importantly, data elements defined by end users may be the concept in an archetype repository, or the defined concept is the data element of an archetype. If we match concepts and data elements separately, users may miss some important relevant archetypes.

Based on these considerations, we tried to translate the problem into identifying potentially relevant clinical concepts from the input. We proposed to reorganize the archetype collection with the dependencies between clinical concepts, data elements, and archetypes and used a probabilistic approach to meet the uncertainty of user information needs.

Table 2 shows the distribution of archetypes across different clinical domains.Clinical domain classification refers to the concept schema proposed by Hruby et al [39].

Table 3 shows the distribution of archetypes, concepts, and data elements across different types of archetypes in the collection. In addition, there were 31 specialized archetypes, 11 of whose parent archetypes are no longer in the CKM.

Then, we learned the dependencies between concepts. Table 4 shows the top relevant concepts suggested by four different percentages of values of p(c_j|e(C_i)) for “dosage” and “examination of a lung,” respectively.

After that, we constructed four clinical resource networks, G₁, G₂, G₃, and G₄, according to the top 3%, 5%, 8%, and 10% of values, respectively. Each graph consisted of 5513 nodes, which were 3982 data element nodes, 504 concept nodes, 504 duplicated concept nodes, and 523 archetype nodes, with 6366 edges from T to C and 543 edges from Cʹ to A. For edges C to Cʹ, G₁ had 1590 arcs, G₂ had 2485 arcs, G₃ had 2958 arcs, and G₄ had 3263 arcs.

To compare the performance of different graphs in supporting retrieval, we calculated the average precision (AP) values for the 11 standard recall points of each graph for the test queries and then computed the mean average precision (MAP) values. The results (Table 5) showed that the retrieval method based on G₃ achieved the best MAP (MAP=0.32), with an AP of 0.35, 0.31, and 0.3 for each query, respectively.

Next, we compared the results of our method based on G₃ with baseline methods. To comprehensively validate the performance, we selected the MAP, AP, and precision at 10 (P@10) as evaluation metrics. Archetypes in the CKM are updated regularly, so it is difficult for us to compare the result on the same collection. We searched relevant archetypes in the CKM for the three queries given on December 12, 2018, and evaluated its performance against the ground truth. The result (Table 6) shows that our method outperforms all the baseline methods, achieving the best AP and P@10 across different test queries, as well as the best MAP. For instance, for query 1, our method, CKM, Bayesian network, and BM25F achieved a P@10 of 0.50, 0.40, 0.20, and 0.20, respectively. Furthermore, we can observe that the MAP of BM25F (MAP=0.177) and Bayesian network (MAP=0.127) was lower than that of CKM (MAP=0.227), which means that there are limitations in using clinical concepts and data elements to represent each archetype. Our approach takes into account the semantic associations between concepts and effectively compensates for this deficiency.

The same trend is observed when evaluating precision-recall graphs across all test queries. Figure 5 shows the precision-recall curves evaluated against the ground truth. Here, BM25F falls short in performance. For instance, for a recall of 0.3, our method, CKM, Bayesian network, and BM25F achieved a precision of 0.38, 0.30, 0.05, and 0, respectively. Additionally, the 11-point MAP curve of the Bayesian network is similar to that of our approach, but the performance is much worse than ours. Meanwhile, compared with the curve of the CKM, our curve is smoother and has higher precision when the recall is below 0.6. These results may be explained by the fact that dependencies between concepts could help identify relevant archetypes.

The dual model methodology used by openEHR distinguished the clinical content domain from the technical domain, which enabled reusable CIMs (archetypes) [9]. We were interested in identifying relevant CIMs online to standardize clinical concept representation within EHRs, so we developed a graphical retrieval method based on an extended Bayesian network and validated its feasibility using an online clinical information knowledge source: OpenEHR CKM. We combined a qualitative representation of the retrieval task, by using a graphical representation of relationships among data elements, concepts, and archetypes, with quantitative representation of the uncertainty of information needs, using a probabilistic approach. Compared with three typical retrieval methods (BM25F, Bayesian network, and CKM) in the medication, laboratory test, and diagnosis retrieval tasks, our method achieved the best MAP (MAP=0.32). In the diagnosis retrieval task, CKM and BM25F could not find the relevant archetype “openEHR- EHR-SECTION.problems_and_diagnoses.v1.” Our method could successfully identify the models covering “diagnostic reports,” “problem list,” “patients background,” “clinical decision,” etc, as well as “problems and diagnoses.”

Although end users were mainly concerned about whether an archetype covered the concept name and core data items, we could not obtain satisfied performances without considering any potential knowledge that might be mined from the collection. Here, BM25F and Bayesian network just used clinical concepts and data elements as main features to represent each archetype and performed worse compared with the other models. In the laboratory test retrieval task, the recall of BM25F was 0.158, whereas ours was 1.0 and CKM was 0.895. In the diagnosis retrieval task, the value of precision at 3 of Bayesian network was 0, whereas ours was 1.0 and CKM was 0.333. A possible reason was that we used exact matching instead of fuzzy matching. The most important reason was that they only encoded the dependence relationships between variables and did not take into account the semantic associations between them. Previous studies showed that using the structure of existing knowledge resources and distributional statistics drawn from text corpora could help estimate semantic similarity and relatedness between medical concepts [43]. In the openEHR framework, archetypes should map to clinical terminologies (such as SNOMED CT). However, most archetypes currently in the CKM lacked this kind of mapping, which could have limited the calculation of semantic relatedness. In this study, we learned relationships between concepts by a probabilistic approach and constructed a concept subnetwork with two layers. The results showed that the performance significantly improved, which explained the effectiveness of using prior knowledge to improve retrieval results.

Accordingly, how to find the top n concepts relevant with each concept became crucial. We used e(C_i) as an event representing some type of evidence about the relevance of a concept C_i, and keywords were used as evidence in the experiment. With their help, we could find that the concepts “medication list” and “medication order list” were related, even though their concept name and data elements were totally different. There was also other semantic information that could be used as evidence, such as “purpose” and “use.” How to use them to better support retrieval might need to be further clarified. However, this method could also include in the network some lower relevant concepts, as shown in the column “Top 10%” in Table 4. For better results, we used AP and MAP as evaluation metrics to help select relevant concepts; meanwhile, we noticed that many concepts had the same values of conditional probabilities. This was because of the probabilistic approach we applied, which reminded us that we could not simply select the top n concepts as the relevant ones. Here, we adopted concepts with top n percentages of values of conditional probabilities.

When modeling clinical resources network, we took the relationship of specialization between archetypes into consideration. It helped us find “openEHR-EHR- COMPOSITION.report-result.v1,” a specialized archetype of “openEHR-EHR- COMPOSITION.report.v1,” which BM25F could not find. In addition, we could also find relevant compositional parent archetypes successfully, even though we did not use the relationship of aggregation. For example, in the diagnosis retrieval task, our method could find “openEHR-EHR-SECTION.clinical_decision.v0,” which defined an archetype slot to allow “openEHR-EHR- EVALUATION.problem_diagnosis.v1.” It was because the compositional archetype used the clinical concept of the allowed archetype as its data element. When we linked the data element node to its corresponding concept node, we in fact modeled the relationship of aggregation.

The key idea of our approach lay in identifying potentially relevant clinical concepts from the input. In a two-level model methodology, clinicians were usually the end users. In most scenarios, they were not familiar with openEHR archetypes and did not know what archetype-friendly concepts were. This requires the retrieval method to be as insensitive to the input as possible. For example, take the medication retrieval task. If the user inputs “medication item, order start date/time, dosage, dose unit, comment,” using some frequent words in the archetype library, the CKM performed better than ours. The AP value of CKM was 0.82 (P@10=0.7, recall=1) whereas ours was 0.45 (P@10=0.6, recall=1). However, when the user used uncommon words, such as “medicine name” (Table 1), our method, CKM, Bayesian network, and BM25F achieved an AP of 0.35, 0.26, 0.11, and 0.08, respectively. In addition, as Table 6 shows, our AP was almost equal across different retrieval tasks (0.35, 0.31, and 0.30, respectively), whereas the other retrieval methods were not. From the clinical domain, queries 2 and 3 mainly belonged to the topic of detection/treatment results, whereas query 1 belonged to treatment, which indicated that our performance was relatively stable across different clinical domains. All these showed that our method was more robust than the others.

Additionally, better retrieval results could help users to identify reusable archetypes quickly, promote reuse of archetypes, and improve standardization of CIMs, thereby enhancing interoperability of EHRs. Archetype modeling methodology [15,23] showed that clinicians and domain experts should compare archetype design specifications with retrieved archetypes to decide whether new archetypes need to be developed or whether an existing one could be adapted. Our method could successfully identify relevant archetypes that the CKM could not find, such as “openEHR-EHR- SECTION.problems_and_diagnoses.v1” in the diagnosis retrieval task. If this archetype was the case need, domain experts might create a new one as they thought it did not exist in the CKM. Our method achieved the best recall (recall=1) in different retrieval tasks, which could help reuse archetypes and promote the semantic interoperability of EHRs.

Our study has important limitations. First, it is a feasibility study based on openEHR archetypes. Whether our method can be applied to other CIMs, such as HL7 templates, and to what extent it needs to be localized still need to be clarified and validated. In fact, the key features used in our method are data elements, clinical concepts, CIMs (archetypes), and their relationships. It indicates that our method has potential feasibility if these features are available for other CIMs. Which results are potentially possible will be discussed in future work.

Second, our method presented in this study lacks the calculation of the semantic relevance of synonyms or homonyms, both for queries and network modeling. However, relevant semantic computing methods [43] can be applied to our retrieval method. With their help, we may be able to identify that “medication item” and “medicine item” referred to the same term, and the results would be improved. In the future, we will validate its feasibility and effectiveness.

Third, we did not validate the impact of our method on interoperability. In fact, the basic problem of semantic interoperability in EHRs must be solved from the perspective of the business domains the concepts originally belong to. Our approach only addresses specific technical issues in the CIM modeling process.

Furthermore, there are other limitations. First, the relevant archetypes in the three retrieval tasks that we manually annotated may be controversial, according to different experts. Second, we compared our performances with the CKM on different archetype collections, which may lead to inaccurate results.

In this paper, we proposed an extended Bayesian network retrieval method for finding relevant CIMs. We graphically represented openEHR archetypes using an extended Bayesian network with two concept layers. The results show that it is an effective approach to meet the uncertainty of retrieval tasks, and the key step in modeling this network is to learn the dependencies between concepts. Our better retrieval results could encourage clinicians and domain experts to reuse existing CIMs to represent EHR data in a standard manner, thereby enhancing the interoperability of EHRs. Furthermore, our study provided how the inference process was carried out. Comparing the results of our method with baseline methods, we had the best performance. To optimize the method, further research should focus on the potential feasibility for other CIMs and the calculation of semantic relevance of synonyms or homonyms.