This was a mixed methods study that included a pre-post observational (patient experience survey) component, semistructured interviews, usability survey, and scenario-based usability evaluation. A mixed method approach was used to leverage the strengths of both quantitative and qualitative data and produce a more complete understanding of the phenomenon of interest. Quantitative methods provided insights into whether there were usability problems and for whom, while qualitative methods provided insights into what the problems were.

The study was conducted at a publicly funded oral health service in New South Wales, Australia. The oral health service includes a tertiary dental hospital, a major teaching facility, and community-based (outpatient) oral health clinics across 4 locations. It provides general and specialist oral health services to children and eligible adult patients. Eligible adults are those who qualify for government subsidies for health care specifically aimed at individuals and families who are socially and economically disadvantaged. The oral health service has 160 dental chairs and sees, on average, 12,000 patients a month. The facility is operated by administrative staff and dental clinicians, including registered dentists, specialists, oral health therapists, postgraduate students, dental technicians, nurses, and allied health. This study evaluated the implementation of a patient portal at a single clinic within the oral health service.

This study focused on a web-based application for managing patient appointments in outpatient clinics known as Florence v1.0, which went live at the clinic on December 6, 2023. The web application allows patients to view upcoming appointments, send messages to or receive messages from clinics, confirm upcoming appointments, request cancellation, or request a reschedule of their appointments. Florence was only able to be delivered in English when this study was conducted.

For the PREM data, we compared the clinic that implemented Florence to other clinics at the dental hospital. We performed an ordered logistic regression to evaluate the degree to which respondents felt it was easy for them to change their appointments (5 rating levels: strongly agree [1], agree [2], undecided [3], disagree [4], and strongly disagree [5]). The covariates were age, pre- versus postimplementation of Florence, and clinics with Florence versus clinics without Florence in the same dental hospital.

The usability survey and scenario-based usability evaluation data were analyzed descriptively. In line with previous research [27], SUS scores for each question were summed and then multiplied by 2.5 to convert the original scores ranging from 0 to 40 to a range of 0 to 100. SUS scores above 68 indicate average usability and a score below 68 is considered to be below average [28]. As the age distribution (Table 1) was predominantly (>90%) individuals in the 2 age brackets, 35‐64 years and 65‐84 years, we collapsed age into 2 groups, namely adults (18–64) and older adults (≥65). A general linear model ANOVA was used to test whether CALD status, age (as a binary, with 18‐64 compared to ≥65), or the interaction between CALD status and age predicted the SUS scores. Two Human Factors researchers reviewed each usability problem identified during usability testing and those that emerged during interviews and developed a set of design recommendations. These design recommendations were validated with the IT staff, including a digital product analyst and the designers of the portal, in two 30-minute meetings.

Deidentified transcripts and field notes from usability testing were thematically analyzed independently by ABA, who first identified and coded the transcribed data through inductive thematic analysis, using a data-driven approach to understand participants’ experiences and perceptions [29,30]. Our analytical approach included data familiarization, coding, generating initial themes, reviewing potential themes, defining and naming themes, and producing the report [31]. Codes were later reviewed by MB. Disagreements in themes were discussed in meetings to clarify the rationale and how a theme emerged until a consensus was reached. Findings from the usability tests were later mapped to an existing guideline for designing for older adults [16].

Data from the usability survey provided insights into whether there were usability problems and for whom. Data integration first occurred at this point, as data from the usability survey confirmed the need to focus on people for subsequent interviews and usability tests. Data from the interviews confirmed the demographic of interest and provided in-depth insights into what the problems were within and beyond the portal, while data from the usability test focused on the demographic of interest and provided richer insights into the portal’s interface. Findings from the survey, interview, and usability test were integrated to form a rich description of patients’ experiences.

Ethics approval was obtained from the hospital’s Human Research Ethics Committee (X23-0274 & 2023/ETH01612). Consent was obtained from all participants, who were informed of their right to withdraw from the study at any time. All identifying characteristics have been removed to protect anonymity.

In total, 4571 respondents completed the PREMs surveys in the designated study period. Descriptive statistics for the sample are shown in Table 1. A total of 2116 surveys were completed in the preimplementation period (Sept 6 to Dec 5, 2023) and 2455 in the postimplementation period (Mar 6 to Jun 5, 2024).

When asked how easy it was for participants to change their appointment when they needed to, fewer participants agreed that this task was easy at the Florence clinic compared with the non-Florence (or other) clinics (Figure 1). However, there was no significant change from preimplementation to postimplementation for either the Florence clinic (Wilcoxon z=−1.62, P=.052) or non-Florence clinics (z=−1.54, P=.06). Patients aged 65 years and older were slightly less likely to indicate that changing appointments was easy, compared with younger patients (odds ratio 1.13, 95% CI 1.00‐1.30).

In addition to the routine PREMs survey, another online survey was disseminated by text message to the mobile phone numbers of all 2309 patients who were eligible. Initially, 442 patients followed the link, although 104 did not consent to complete the survey. A further 103 respondents consented but did not complete enough of the survey items to provide usable data, leaving a final sample of 235 respondents (10% of those invited).

Most of the respondents were patients (n=219, 93.2%), with 13 carers (5.5%) and 3 parents (1.3%). There were more males (n=124, 52.8%) than females (n=107, 45.5%), and a minority were nonbinary (n=4, 1.7%). The majority of the respondents spoke English (n=191, 81.3%), while others preferred to speak other languages (n=44, 18.7%) and were identified as CALD. Of these, Chinese or Mandarin was the most frequently preferred language (n=21, 50%). A total of 127 (54%) respondents were less than 65 years, and 108 (46%) were 65 years and above.

The participants’ overall mean SUS score was 63.3 (95% CI 60.9‐65.7). Unadjusted SUS responses for all participants are aggregated in Figure 2, revealing moderately high scores for ease of use (3.57), ease of learning (3.63), and confidence using the portal (3.57).

The ANOVA results showed that age significantly predicted SUS scores (F=10.4; P=.001), with younger participants reporting the portal to be more usable than older participants (Table 2). CALD status did not predict SUS scores, and the interaction between age and CALD status was also not statistically significant.

In total, 18 patients participated in semistructured interviews. There were more females (n=10, 56%) than males (n=8, 44%) and the mean age of participants was 56 (range 27‐75) years. With respect to devices used at home, half of the participants reported using a combination of smartphone, tablet, and computer, 4 (22%) reported using 2 of those devices, and 5 (28%) reported using one device in their daily activities. The majority (n=16, 89%) of the participants reported that they use these devices regularly, but 2 stated that they ask friends or relatives to help them. While all participants spoke English, some (n=4) reported a preference for other languages including Mandarin, Cantonese, Arabic, and Filipino.

Interviews ran for an average of 15 minutes. The findings were grouped into 4 themes: uses of the portal, positive views on the portal, barriers to portal uptake, and recommendations for improving the portal.

How the portal was used emerged as a key theme from the interviews. Most participants explained that they used the portal to organize and keep track of their appointment dates and times. The portal was seen as an appointment diary that made participants feel like they were in control of their appointments:

In addition, the portal was reportedly helpful for cancelling appointments:

Similarly, the portal was used to confirm patient appointments, to let the clinic know whether the patient would attend or not:

Similarly, it was stated that the portal was used by participants to leave messages for the clinic. Participants indicated that they preferred using Florence to calling the clinic by phone.

There was positive feedback about the portal. Specifically, many participants reported that the portal was very easy to use, simple, and straightforward. For example, a patient said,

Others reported that the layout was reasonable, logical, and easy to learn as the instructions could be followed in a step-wise fashion.

In addition, some participants mentioned that the portal was easy to access, particularly because there was no need to remember or input any password:

Another participant said,

In addition to the positive feedback, one reported benefit of the portal was its ability to save time for patients. For example, when compared with the usual practice of making phone calls to the clinic, participants stated that the portal was preferred:

Another stated:

Some participants described improvements they would like to see in the portal. These include translating the portal to other languages to improve inclusivity for CALD people, providing accessibility support/functionalities to improve inclusivity for people with visual or hearing impairments, providing clarity on the send message function and others. A full list of recommendations is listed in Appendix C in Multimedia Appendix 1.

In total, 17 patients participated in the scenario-based usability evaluation. There were more females (n=11, 65%) than males (n=6, 35%), and the mean age of participants was 70 (range 62‐84) years. Two participants were eligible by age only, while 15 were eligible by CALD status only. One of the 2 English-speaking participants was 62 years old. Of the 15 CALD participants, 7 spoke Mandarin, 5 Cantonese, 2 Tagalog, and 1 Vietnamese.

All participants were able to log into the portal, but for many (65%), this required multiple attempts, and most (82%) required assistance. Most (94%) participants were able to confirm their appointments, with only 24% requiring assistance. Only 3 participants (18%) were able to successfully send a message to the clinic, despite the majority (n=11, 65%) attempting this task multiple times, as can be found in Table 3.

Table 4 below shows problem areas identified during usability testing. These problems were mapped to an existing guideline for designing for older adults that was violated in the portal evaluated [16].

Textbox 1 shows the set of recommendations that emerged following review and discussion of the findings, to improve the patient portal design.

In this study, we found that the usability of a patient portal and the approach to its implementation in a real-world setting could be improved, particularly for CALD and older people. In terms of system usability scores, older adults recorded significantly lower scores, although CALD people did not. Nonetheless, in the qualitative interviews, CALD participants reported language as a factor contributing to poor usability and the need for translation assistance for CALD people was noted in the usability testing, which represented a barrier to uptake. Participants noted a range of benefits of using the portal, specifically relating to the ease of managing their appointments. Ease of use and the simplicity of the portal were key factors commonly reported among younger users. However, design problems, including inadequate incorporation of features that maximize inclusivity, were found to limit the usability of the portal for some participants in our study. Also, we found that the purpose of the portal was unclear to some patients and had no impact on the ease of changing appointments. In addition, some relevant staff members were not aware of, or trained on, the portal and how it might impact their work with patients. Taken together, our study elucidates the importance of careful implementation and inclusive design approaches to ensure usability and engagement with patient portals in real-world settings.

Seminal works have established a range of models (eg, Inclusive Design Model) [32], guidelines (eg, Web Content Accessibility Guidelines) [16,33], toolkits (eg, Inclusive Design Toolkit) [34], and key user capabilities (eg, vision, hearing, thinking, mobility, reach, and dexterity) [35] that must be considered to produce an inclusive design. A design is inclusive when it is produced through thorough consideration of the skills and capabilities of the end-users [36]. However, our study revealed that the portal design was not fully aligned with existing inclusive design guidelines, which is likely to result in problems for many groups of users. For example, some participants reported that the “agree box” was too small, a particular concern for people with visual impairment. Others described challenges with using the OTP, a concern that may adversely affect people with memory limitations. And some participants reported on their inability to use the portal due to language barriers, implying limited consideration of people who identify as CALD. These findings add to existing literature suggesting that inclusive design of patient portals is not yet normalized [37-39]. We therefore call for better adherence to existing design guidelines for older adults when designing patient portals, such as the ones identified as violated in this study (Table 4 [16]).

Despite an increased number of usability evaluation studies on patient portals being published in the past few decades, specific design guidance on how to improve inclusivity, particularly for CALD and older people, has not had corresponding growth [12]. In a review of 85 articles on patient portals on usage behavior and usability, 29 studies reported on usability, with the majority (18, 62% studies) using self-report surveys [40]. Another review of 120 articles on patient portals identified 20 usability studies and revealed that the majority (14, 70% studies) assessed usability with subjective or self-report methods (eg, surveys and/or interviews) [13]. The authors found that in-depth objective usability studies were more effective at uncovering a variety of design problems but were rarely used [13]. Whilst useful, survey methods only report on whether there are usability concerns, and which groups are most impacted, rather than the specific design problems. In our study, the PREMs and postimplementation survey results indicated that there were usability problems, but the combination of interviews and objective usability evaluation uncovered the specific design problems contributing to poor usability. Therefore, we stress the importance of incorporating different perspectives, as was done in our study, to better identify design problems and provide guidance on how to improve usability.

Our study also revealed that the purpose of, and the functionalities in, the portal were unclear to some patients, preventing uptake. This indicates some potential gaps in the implementation approach. A systematic review of 18 studies on interventions to improve the uptake of patient portals in vulnerable populations found that most interventions were focused on the individual, rather than the organization [23]. To increase portal uptake, the authors further identified organization-based strategies such as the use of various promotion and education techniques to increase patients’ exposure to the portal, aligning with our research findings [23]. Therefore, we recommend that health care organizations aiming to implement patient portals should use a suite of pre- and postimplementation strategies to better engage patients and to facilitate uptake.

Based on the findings of this study, we provide a range of broad recommendations on patient portals for health service organizations and designers. Health service organizations should engage with patients early on in the process of co-designing the portal; carefully target under-served populations for early feedback; craft strategies to create awareness before and after roll-out of the technology; and train staff members whose roles and responsibilities may be impacted by the implementation of the portal. Furthermore, designers should use familiar language, terms, and cues to ensure inclusion of different patient demographics; minimize the number of tasks and actions users need to remember and act on; carefully design accessibility functionalities to cater to the needs of older adults; and ensure clarity of the portal’s purpose.

To our knowledge, this was the first study to use a mixed methods approach, including pre-post PREMs evaluation, usability survey, semistructured interviews, and scenario-based usability testing to evaluate patients’ experience of using a web portal in a dental outpatient setting, particularly from the perspective of older adults and those who identify as CALD. Our approach allowed us to identify design problems and how these may be optimized in future portal designs. Limitations of our study include that data were collected in a single health care organization, which may limit the generalizability of our results to dissimilar organizations. Also, as the pre-post PREMs evaluations were anonymous, we were unable to match the data between pre- and postimplementation patients, and the sample may have been different, although Table 1 indicates good balance between the demographic characteristics of the pre- and postimplementation samples. We also could not track participant involvement across all the data collection methods used due to ethical restrictions. Although self-report methods (eg, interviews and surveys) were used, which are prone to bias, our scenario-based usability testing was based on participant observations. Our study revealed emergent findings for the CALD group, but we did not include a comparator group in our usability testing, so we recommend further evaluation of patient portals for this subgroup beyond language barriers. Future work will involve evaluating the impact of our design recommendations to determine the extent to which they enhance the usability of the portal.

A benefit of patient portals is their potential to improve efficiency in the use of health service resources in outpatient settings, but this can only be realized if the portal is usable and used. Our study found no impact of the portal on the ease of changing appointments and revealed areas where the usability of the patient portal and the approach to its implementation could be improved. There is a need to better leverage inclusive design principles when developing patient portals, and objective usability evaluation methods when assessing the portals to ensure they are usable and can be used by all patients.