The delivery of knowledge-based care depends on patient engagement, where patients take an active role in their care [1-3]. Patient portals are considered a health information technology that promotes patient engagement [4-6] by providing patients with online tools to take an active role in their care [7-9]. Since patient portals are more than just static repositories for patient data [10,11], they support the new vision of health services that enable patient-provider information sharing [12,13], thus contributing to empowering patients [14,15], supporting shared decision making [16], and engaging patients actively in their care [6,17]. A patient portal is an internet-based application combining knowledge and software tools [18,19] that allow patients to have autonomous access to their electronic health record (EHR) anywhere at any time [20,21]. Besides its core function of providing EHR access [22], the features of a patient portal range from viewing visit notes, requesting medication refills, appointment scheduling, access to test and lab results, secure messaging with the health provider, e-visits, or reporting patient-generated health data [17,23-26]. Patient portals are used in different organizational settings such as independent physician practices and hospitals, group practices, or large, integrated health care delivery organizations [27-29].

The widespread use of the internet, rise of mobile computing, and progress in patients’ technical aptness have led to an increase in the use of patient portals in various countries such as the United States, Denmark, and Australia [12,28,30]. However, there are countries where patient portals are still not widely used [31,32]. A benchmarking study presented by Ammenwerth et al [33] in 2020 analyzed the eHealth progress of 14 countries worldwide with different health systems and different levels of economic development. The study showed that the use of patient portals and the provided patient portal functionalities vary significantly between the countries. While Finland and South Korea, for example, allow patients the best access to their health record data, 6 of the 14 analyzed countries do not offer their population any access to their health data online [33].

The low use of patient portals in both developing and high-income countries [31,34] creates a problematic situation for the health informatics professionals who are responsible for strategic and tactical information technology management in their health care institution or department; we will call these professionals “health information managers.” On the one hand, there already exists a very heterogeneous landscape and a broad diversity of patient portals [12,27,35,36] regarding their intended deployment and functionalities [25,30,37]. In contrast, many health information managers still feel they have too little knowledge of patient portals [32]. They admit having difficulties understanding the portals’ various application areas and scopes, defining their general requirements, and selecting a patient portal for their specific context or problem [32]. Health information managers are responsible for planning, organizing, and following up on all activities related to health information technology [38,39]. This also involves selecting, introducing, and managing patient portals for their health care institution [32].

To support health information managers with a tool for comparing patient portals and defining which general type and functionalities of patient portals they need, we developed the TOPCOP taxonomy (Taxonomy of Patient Portals based on Characteristics of Patient Engagement) [32]. The need for such a patient portal taxonomy had already been stressed in a recently published Cochrane Review on the impact of patient portals [40]. The TOPCOP taxonomy is shown in Figure 1. A comprehensive description of the dimensions is published elsewhere [32].

A taxonomy is a classification system to classify similar objects of a domain into groups based on distinct characteristics and offers a set of decision rules [41-44]. The reduction of complexity, identifying similarities and differences among objects [45,46], and the understanding of interrelationships are major advantages of taxonomies [44,47,48]. Taxonomies thus support researchers and practitioners in better understanding a domain and distinguishing among its objects [49,50]. The TOPCOP taxonomy was built by applying the formal taxonomy-building method proposed by Nickerson et al [51]. This method specifies the necessary steps and integrates 2 optional, iterative development approaches to conceptionally and empirically build and empirically evaluate a taxonomy [51,52]. The TOPCOP taxonomy was created conceptionally based on a literature review to assess the characteristics and functionalities of patient portals [32]. It was evaluated empirically by classifying patient portals offered on the market and health providers’ portals online [32].

A taxonomy is determined by the user’s intended purpose, which guides the taxonomy’s development by focusing on the specific phenomenon of interest [50,53]. Different users or different purposes may therefore lead to a different taxonomy [51,54]. We created the TOPCOP taxonomy for health information managers to classify and compare patient portals [32]. Further, the taxonomy should serve the health information managers in defining the general type and functionalities of patient portals and should help them select the most suitable solution offered on the market. The dimensions were built to distinguish among patient portals with the scope on patient engagement.

Since there is no objective metric to define the usefulness or quality of a taxonomy [44,51,55], the method by Nickerson et al [51] provides a set of conditions to determine usefulness. Applying these conditions during the building process, the taxonomy’s usefulness was empirically validated by classifying all patient portals of interest with the taxonomy [32].

According to design science research, taxonomies are fundamental design artifacts to provide knowledge and understanding of a problem domain [49]. In design science research, the design process for an artifact is divided first into building and then into evaluating [56,57]. Following the design science research paradigm, one suitable criterion to evaluate a taxonomy is by having users assess its usefulness in achieving its intended purpose [58,59]. Therefore, we now wanted to evaluate and further improve the TOPCOP taxonomy together with the projected users, guided by the evaluation criteria of the taxonomy’s usefulness related to its intended purpose.

The panelists were asked to evaluate the TOPCOP taxonomy by assessing every single dimension related to its appropriateness for classifying and comparing patient portals. In Figure 4, we present the assessment for the 20 dimensions of the initial TOPCOP taxonomy (Figure 1) indicating the median for each dimension. All 13 panelists evaluated all 20 dimensions. Since each dimension’s median ranged between 7 and 9 without disagreement, group consensus on the dimensions’ appropriateness was achieved for all 20 dimensions [99].

Further, the panelists were asked to propose unsuitable and missing characteristics to refine the existing dimensions and to suggest new dimensions to improve the taxonomy.

Two panelists proposed refining dimension D6 Prescription Renewal by merging the characteristics “basic renewal” and “advanced renewal” to create the characteristic “with renewal” instead. They argued that differentiating the dimension into the initial 2 characteristics is confusing rather than strengthening the distinguishability of patient portals. Seven panelists proposed 8 new characteristics to improve the existing dimensions D1, D2, D5, D6, D8, D9, D11, and D12 (Figure 5 shows the content of each dimension). Five panelists proposed the new dimensions Account Protection, App Expandability, Medical Specialty, Medication Summary, Portal Type, and Web Accessibility to enhance the taxonomy. We present all the proposals for improvement of round 1 in Figure 5.

In round 2, the panelists had to assess the proposals from round 1 (Figure 5). They were asked to evaluate the merger of the characteristics of dimension D6. Further, they assessed the appropriateness of the suggested characteristics of dimensions D1, D2, D5, D8, D9, D11, and D12.

All 13 panelists assessed the proposed 8 characteristics and justified their rating whenever it fell between 1 and 6. As demonstrated in Figure 6, the median assessment for the appropriateness of all changes ranged between 7 and 9 without disagreement. Therefore, all 8 characteristics were appropriate for improving the taxonomy and became part of the taxonomy [99].

The panelists were further requested to evaluate the appropriateness of the proposed dimensions D21 Account Protection, D22 App Expandability, D23 Medical Specialty, D24 Medication Summary, D25 Portal Type, and D26 Web Accessibility.

In Figure 7, we demonstrate that for all these dimensions, the median ranged between 7 and 8. However, the condition for disagreement [99] was fulfilled for dimensions D21 Account Protection and D26 Web Accessibility. Therefore, only dimensions D22, D23, D24, and D25 were considered appropriate for improving the taxonomy and became part of the taxonomy. Since no panelist made any proposal for changing an existing dimension or for a new dimension in round 2, only dimensions D21 and D26 became subject to re-evaluation in round 3.

In Figure 8, we present the taxonomy in progress after round 2 showing for which characteristics group consensus was achieved and for which dimensions no group consensus was achieved.

In round 3, the panelists were asked to again assess the new dimensions D21 Account Protection and D26 Web Accessibility as group consensus was not achieved in round 2. All 13 panelists assessed both dimensions.

As demonstrated in Figure 9, a median of 8 without disagreement [99] was achieved for dimension D26. Dimension D26 was therefore appropriate and became part of the taxonomy. For dimension D21, a median of 7 was achieved. However, as at least one rating is a 1 and at least one rating is a 9, disagreement existed among the panelists [99]. All 5 panelists who assessed dimension D21 with values of 1 and 3 argued consistently that a patient portal must provide the highest data protection due to legal or patient requirements. Therefore, as strong account protection is a mandatory requirement, dimension D21 is not appropriate for distinguishing among patient portals. Comparing the ratings of dimension D21 for rounds 2 and 3 showed that, besides the fact that disagreement was re-confirmed in round 3, 3 ratings deteriorated (Figure 9), which means that group consensus converged even more strongly towards disapproval of dimension D21. To avoid the known risk of fatiguing the panelists with too many evaluation rounds [92], we did not launch another evaluation round. Since no group consensus was achieved, dimension D21 was not integrated into the taxonomy.

After assigning the new dimensions to suitable aspects and organizing and numbering the dimensions accordingly, the final TOPCOP taxonomy resulted in 25 dimensions based on 65 characteristics assigned to 7 aspects and is presented in Figure 10. In Multimedia Appendix 4, we provide a detailed description of the dimensions and characteristics.

In round 4, the panelists were asked to assess the general need for a taxonomy, the appropriateness of patient engagement as a guiding concept, and the TOPCOP taxonomy’s usefulness related to its intended purpose. All 13 panelists participated in round 4.

In Figure 11, we present the evaluations’ results for the research questions Q1, Q2, and Q4–Q9. Since group consensus was achieved without disagreement [99] for all research questions, no further interview round was launched.

The results shown in Figure 11 can be interpreted as follows. The panelists clearly agreed that there is a need for a taxonomy to distinguish among patient portals (median of 8 for Q1). Twelve panelists considered patient engagement to be an appropriate distinguishing concept for comparing patient portals (median of 8 for Q2). Only the panelist who assessed question Q2 with a rating of 3 proposed “System Architecture, Data Types, and Interoperability” as a more appropriate concept for comparing patient portals. Further, the panelists were asked to propose alternative distinguishing concepts appropriate for comparing patient portals (Q3). Since research question Q3 was an open-ended question, it is not part of Figure 11. Therefore, we present the proposed alternative concepts in Table 3.

We continued interpreting the results presented in Figure 11, which relate to the case study and the assessment of the taxonomy’s usefulness. With a median of 8, the panelists considered the form and structure of the final taxonomy to be understandable (Q4). However, 1 panelist who assessed Q4 with a rating of 4 argued that the taxonomy contains too many dimensions while, on the contrary, 1 panelist who assessed Q4 with a rating of 7 proposed refining the taxonomy with additional subcharacteristics to achieve a more accurate comparison of patient portals. All panelists considered the final taxonomy to be appropriate for classifying patient portals, giving ratings between 7 and 9 with a median of 8 (Q5). Further, they considered, with a median of 8, the taxonomy to be appropriate for comparing patient portals (Q6) and appropriate for better understanding of patient portals based on characteristics supporting patient engagement (median 7 for Q7).

Since the TOPCOP taxonomy is also intended to help health information managers select patient portals offered on the market, the panelists were requested to assess its usefulness in this regard. With a median of 8, the group consensus was achieved on both the taxonomy’s usefulness for creating an initial requirement profile for patient portals (Q8) and selecting patient portals offered on the market based on characteristics supporting patient engagement (Q9). To sum up, by applying the TOPCOP taxonomy to compare 2 patient portals, we could indeed show that it is useful in contrasting and comparing patient portals from different vendors. In Figure 12, we show an example of a panelist’s comparison. By marking each patient portal’s characteristics, the differences and similarities of the 2 patient portals could be easily recognized.

Evaluation is a challenging, essential, and crucial component of the research process [110,115]. One criterion for assessing artifacts such as taxonomies is by evaluating their usefulness related to their intended purpose [111,116]. However, there are only very few taxonomy-specific evaluation guidelines [117], but multiple evaluation approaches can be applied in health informatics [118-120]. Szopinski et al [49] analyzed the various approaches researchers applied to evaluate taxonomies in the information system’s domain and demonstrated that the Delphi technique was hardly used. Analyzing 61 evaluation approaches, they found just 1 study where the Delphi technique was used to evaluate the taxonomy [49]. In this study, panelists were asked to classify items into a deductively developed taxonomy and assess if the items were classified correctly [121]. In contrast, our modified Delphi approach aims to (1) first improve a conceptually and empirically created taxonomy [32] in multiple rounds by the users and (2) then evaluate the jointly created taxonomy by achieving users’ consensus on the usefulness of the taxonomy. Therefore, the health information managers first compared the real-world patient portals with each other by performing a use case and then assessed the taxonomy related to its intended use. In Table 4, we describe the differences between the classic Delphi technique and our approach. The modified Delphi approach we used is, to the best of our knowledge, a new Delphi approach in health informatics for evaluating a taxonomy. We, therefore, call this approach the Taxonomy-Evaluation-Delphi (TED) approach.

With this study, we were able to demonstrate that the need for a taxonomy to compare and classify patient portals exists among health information managers and that the concept of patient engagement to compare and select patient portals is considered appropriate.

Applying a modified Delphi approach, we improved the TOPCOP taxonomy based on the specific needs of the users. The final TOPCOP taxonomy consists of 25 dimensions with 65 characteristics, compared to 20 dimensions and 49 characteristics of the initial TOPCOP taxonomy.

We were able to demonstrate that the health information managers considered the final taxonomy to be useful in classifying and comparing patient portals. Further, we demonstrated that the final TOPCOP taxonomy supports the users in better understanding patient portals and assists them in selecting patient portals offered on the market. We were able to collect 4 alternative ideas on distinguishing concepts to compare patient portals that may serve for future research. As an additional outcome of our study, we created, to the best of our knowledge, a new Delphi approach in health informatics for evaluating a taxonomy.

At present, there exists only a limited number of publications related to patient portal taxonomies. Ammenwerth et al [20] developed a taxonomy that aims to distinguish patient portals in a systematic review dealing with their effect on patient empowerment and health-related outcomes. Roehrs et al [124] developed a taxonomy that aims to identify open questions related to personal health record (PHR) data types, features, and architecture types. A PHR provides patients with web-based access to their health data that is under the control of the patient [124], while an EHR typically is under the control of the provider [7,19,22]. Fernández-Alemán et al [125] analyzed free web-based PHRs to identify their features and functions to better understand the PHR market. They created a framework of 4 dimensions intended to support patients in selecting a PHR that best fits their needs [125]. Scheplitz et al [126] created a framework for patient portal functionalities to record all possible functions to identify specification gaps related to software development. Walker et al [8] developed a framework to evaluate how well health information technology can support patient engagement by applying 5 engagement scoring levels.

These attempts only provide part of a potential patient portal taxonomy and are developed for different users and purposes. Since the user determines the intended purpose of a taxonomy and the purpose guides the development by focusing on a specific phenomenon of interest, different users or purposes may lead to different taxonomies [46,51,55]. Further, a useful taxonomy must yield utility for a specific problem domain [58]. To sum up, the found taxonomies are not suitable to yield utility for health information managers for classifying and comparing patient portals based on characteristics appropriate for promoting patient engagement and understanding the differences and similarities. Therefore, the TOPCOP taxonomy was specifically developed for health information managers to compare and select patient portals offered on the market.

Our approach to evaluating the TOPCOP taxonomy has some potential limitations.

First, panelists were selected from Germany, Austria, and Switzerland while the scope of the TOPCOP taxonomy is to support health information managers from any country. Since we were not able to attract participants from other countries, we aimed to assemble the panel as heterogeneously as possible, with different educational backgrounds and work experience, to achieve the best possible understanding of different viewpoints. Further, by selecting participants from 3 countries, we were able to map expertise from 3 different health care systems and integrate experiences with different levels of health care systems’ digitization.

The second limitation relates to the panelists’ experience with patient portals. Patient portals are not widely used in Germany, Austria, or Switzerland [32], and the experience with patient portals varied from little experience to much experience among the selected health information managers. Including participants with little experience with patient portals may lead to different results than if the participants had a deep understanding. As the taxonomy is not intended to only serve highly experienced but also inexperienced users, a composition of the panel that considers different levels of experience and understanding may increase the variety of viewpoints and the range of user needs related to the taxonomy. This variety may make the taxonomy even more useful [127].

The third limitation relates to the risk that the panelists may misunderstand what to evaluate. During the initial phone calls with potential participants, we noticed that some users assumed that the taxonomy’s evaluation related to the suitability of functionalities for patient portals. However, the evaluation related to a dimension’s appropriateness for classifying and comparing patient portals based on patient engagement. To ensure that there was no confusion, we explained the difference in individual introductory video conferences. Further, in the survey’s introductory part, we outlined the scope of the evaluation and formulated the questionnaire’s questions with unambiguous wording.

The fourth limitation is related to the Delphi technique itself. Delphi aims to obtain group consensus on opinions [68], but the achievement of consensus does not necessarily mean that the correct answer was found [84]. Besides, the composition of the panel may influence the research outcome [88,128]. To address these problems, we assembled the panel as heterogeneously as possible to integrate the broadest possible viewpoints and experience with patient portals. The selection of the panel was guided by the goal of achieving the best expertise available.

To determine the achievement of group consensus, we applied the RAND/UCLA concept appropriate for evaluation [101] and widely used to assess the appropriateness of health care services [100,129]. The survey was conducted anonymously to avoid the problem of dominant panelists possibly influencing individual opinions [102]. By applying all these measures, we believe that we were able to reduce any inherent bias in a possible method in the best way.

The scope of the TOPCOP taxonomy is to serve health information managers with different degrees of knowledge related to patient portals and for various areas of application. The taxonomy may thus serve health information managers as a starting point to better understand the complex domain of patient portals since it describes the various aspects of patient portals. Further, 2 or more patient portals can be described by marking the respective characteristics. This shows the differences and similarities of the patient portals (Figure 12) and so supports the health information managers in classifying and comparing patient portals.

Since each health care institution may have different requirements related to a patient portal, the TOPCOP taxonomy can serve to create a requirement profile. By marking those characteristics in the taxonomy that best meet the needs of a health care institution, health information managers can create an initial requirement profile. This profile can then be used for a targeted search and selection of suitable portals offered on the market.

By providing a standardized terminology to describe various aspects of patient portals independent of clinical setting or country, the TOPCOP taxonomy is also useful for advancing research and evaluation of patient portals. It can, for example, be used to systematically describe patient portals as part of systematic reviews on their impact. The need for a taxonomy in this context has already been stressed in patient portal reviews [40].

The TOPCOP taxonomy aims to support health information managers in comparing and selecting patient portals. By providing a standardized terminology to describe various aspects of patient portals independent of clinical setting or country, the taxonomy will also be useful for advancing research and evaluation of patient portals. Since the health information managers contributed to the taxonomy’s development, we were able to improve the taxonomy’s quality and usefulness based on the users’ needs.

The taxonomy consists of a manageable number of characteristics and dimensions and is therefore flexible for future changes. If needed, new dimensions can be added or removed according to future technological development. Further, due to its flexible form, the users can adjust the taxonomy to their personal needs. The initial TOPCOP taxonomy was developed by analyzing patient portals from 15 countries worldwide. It was then improved by health information managers with various degrees of patient portal experience from 3 countries with different levels of health care digitization. We, therefore, consider our taxonomy suitable to compare and classify patient portals from any country. The taxonomy may also contribute to the progress of health care digitization as it may enhance human resources capacity and effectiveness.