We considered the CHERRIES checklist when reporting this study [38]. In early 2019, we conducted a large cross-sectional interned-based survey among the general adult population of Hungary, which explored eHealth literacy [39]. In addition, shared decision making [40] and PREMs [30,41] were investigated among those respondents, who used ambulatory care over the past 12 months due to health problems. We recruited 1000 online respondents. Quotas were used to ensure the representativeness of the sample according to the 2011 Population Census [42] data by gender, age, educational level, type of settlement, and NUTS 1 (Nomenclature of Territorial Units for Statistics) region of residence, including a fair representation of people aged 65 and over. Recruitment and data collection were carried out from a commercial online panel by a survey company (Big Data Scientist Kft); reports on dropout rates and the sampling frame were not available. All materials were in Hungarian and all participants spoke the same language (Hungarian). Participation was voluntary and anonymous. No incentives were offered for answering the survey. Participants gave their informed consent prior to the study. Ethical approval was obtained from the Hungarian Medical Research Council (ID: 47654-2/2018/EKU). The electronic questionnaire was piloted by the authors. Respondents could revise and change their answers for completed items, and full completion was required unless the “do not know/do not want to answer” option was offered.

Our sample included those respondents who had a face-to-face appointment with an HCP in the previous 12 months due to their own health problems and answered whether or not the visit had happened at their usual HCP.

eHealth literacy was measured with the Hungarian version of the self-reported eHealth Literacy Scale (eHEALS) [39,43]. This instrument has been used internationally both as a descriptive tool [22,39] and as a patient-reported outcome measure of digital health interventions [19,23,43]. The eHEALS consists of 8 items, each scored on a 5-point Likert scale. Items 1 and 2 are related to the awareness of health resources (“I know what health resources are available on the Internet”; “I know where to find helpful health resources on the Internet”); items 3 and 4 are related to searching for health resources (“I know how to find helpful health resources on the Internet”; “I know how to use the Internet to answer my questions about health”); items 5 and 8 are related to the utilization of health resources (“I know how to use the health information I find on the Internet to help me”; “I feel confident in using information from the Internet to make health decisions”); and items 6 and 7 are related to the appraisal of health resources (“I have the skills I need to evaluate the health resources I find on the Internet”; “I can tell high quality health resources from low quality health resources on the Internet”). Item levels are added for a total score ranging from 8 to 40. Higher scores indicate greater eHealth literacy [43]. The psychometric properties of the Hungarian eHEALS as well as its association with health outcomes and behavioral health risk factors in the general population have been shown in the validation study [39]. Because eHEALS showed convex relationship with self-rated health in the validation study, we decided to group respondents into 4 quartiles to explore potential nonlinear associations with PREMs in an easily interpretable manner [39,44,45]. We also performed sensitivity analysis using alternative eHEALS category boundaries. The eHEALS questionnaire is included in Multimedia Appendix 1.

Respondents’ experiences with ambulatory care were assessed by the set of questions recommended by the OECD’s Health Care Quality Indicators Project [26]. The questionnaire consists of 2 sections: (1) access to care, including questions on unmet medical needs and waiting times; and (2) patient experiences. The access to care section consists of 4 binary questions about unmet medical needs over the past 12 months (missed medical visits, interventions, or medications due to travel difficulties, or cost burden) and 4 questions on waiting times concerning the last medical appointment: waiting time to get the appointment (appointment waiting time) and on the day of consultation (office waiting time), and whether waiting was a problem in either case. In the patient experiences section, respondents were asked if the HCP (1) spent enough time with them; (2) provided easy-to-understand explanations; (3) provided the opportunity to ask questions or raise concerns about recommended treatment; and (4) involved in decisions about care and treatment as much as the respondent wanted to be. Answers were recorded on a 4-point Likert scale, with higher scores indicating more perceived problems. In the final item, respondents rated their perception of the overall quality of the appointment on a 5-point Likert scale ranging from poor (0) to excellent (4). Additional items inquired the HCP type, setting and time of the last visit, and whether the respondent visited his/her usual HCP. Questions related to unmet medical needs were posed if respondents had had health problems over 12 months preceding the survey, while waiting times and patient experiences were inquired only if the respondent had participated in ambulatory consultation with an HCP. The full PREM questionnaire is included in Multimedia Appendix 2.

Following the practice of countries using PREMs for monitoring health system performance, we created composite scores from PREM items [26]. The Unmet Medical Needs Score (range 0-4) reflected the number of areas where respondents experienced an unmet need (missed visit due to travel burden; missed visit due to cost burden; missed intervention due to cost burden; and missed medication due to cost burden). The binary Any Unmet Medical Need variable indicated if patients experienced unmet need in any of the 4 items. Waiting times were transformed to continuous variables by considering the midpoint of the respective waiting time answer option. Log-waiting times were used in regression analyses, assuming a 0.5-day waiting time for respondents with an appointment on the same day. We used the binary variable Any Waiting Problem to indicate if the office waiting time or appointment waiting time was a problem to the respondent. The 4 PREMs were used to create a composite Problem Score, which was the sum of the individual answer options of each PREM (1=yes, definitely; 2=yes, to some extent; 3=no, not really; and 4=definitely not). Hence, the composite score ranged from 4 to 16, where higher values represent more problems experienced during the visit. We also constructed a Negative Experiences Score (range 0-4) by counting the PREM items that did not receive a “yes, definitely” answer. Finally, we created the binary variable Any Negative Experience, which indicated if the response to a PREM item was other than “yes, definitely.”

We recorded respondents’ sociodemographic variables, such as age, gender (female or male), education (primary, secondary, or tertiary), family status (married or not married), employment status (with a paid job or without a paid job including students, pensioners, unemployed, etc), and place of residence (capital, other cities, or village). Age groups were formed according to main Medical Subject Heading (MeSH) categories, adding 18-year olds to the young adult category (young adults: 18–24-year olds; adults: 25–44-year olds; middle aged: 45–64-year-olds; aged > 80: 65+ year-olds) [46]. Net monthly household income was queried in 11 range categories, and per-capita household income was calculated by dividing the category midrange values by the number of household members, without adjustment for the number of children. The midrange value of the upper open category was calculated by fitting the Pareto curve as proposed by Parker and Fenwick [47]. We generated income groups according to quintiles of per-capita monthly net household income, with lower limits of €203 (US $241), €285 (US $338), €365 (US $432), and €463 (US $549), respectively, for the second, third, fourth, and fifth quintiles calculated from the third to eighth national decile group means by linear interpolation [48]. We also recorded respondents’ health status using the Minimal European Health Module (MEHM) [49,50]. The MEHM included an item on self-perceived health (very bad, bad, fair, good, or very good); an item on whether the respondent had long-standing health problems (1=chronic morbidity present); and the Global Activity Limitation Indicator, which assessed for activity limitations due to health problems (not limited at all, limited but not severely, or severely limited) [50].

Descriptive methods were used when analyzing the sociodemographic characteristics of the sample as well as the PREM items. To test the basic psychometric properties of the PREM scores constructed from multiple items, we assessed their distributional properties, performed exploratory factor analysis (EFA), and calculated internal consistency (Cronbach α) [51]. The normality assumption was tested via the Shapiro–Wilk test [52] and the Kaiser–Meyer–Olkin (KMO) test was used to check the suitability of data for EFA [53]. Pairwise biserial or polychoric correlations were calculated between PREM items and eHEALS scores. Polychoric correlation assumes bivariate normally distributed latent variables behind ordinal response items and provides the correlation coefficients for those latent variables using a maximum-likelihood estimation [54]. The bivariate associations between eHEALS quartiles and PREM scores as well as demographic variables were tested via ANOVA, the Kruskal–Wallis test, and the chi-square test of independence [55-57].

We performed multivariate regression analyses to explore the relationship between eHEALS quartiles and PREM items, as well as the composite PREM scores, after controlling for sociodemographic variables, respondents’ health status (MEHM), the setting of the visit (GP, public specialist, or private specialist), and type of HCP (GP, specialist, or other allied health professional). The following models were conducted: (1) logistic regression for binary PREM items or constructed binary variables, (2) ordered logit models for polytomous PREM items, and (3) ordinary least squares (OLS) models for waiting times and composite PREM scores. We tested the joint significance of eHEALS quartiles as a single predictor variable using the Wald test. OLS models were tested for heteroskedasticity via the Breusch–Pagan test and for specification error via the Ramsey regression equation specification error test (RESET). We applied robust regression if heteroskedasticity was detected [58]. In case we detected model functional misspecification error, log-transformation or square-root transformation was performed on the dependent variable [59]. Goodness of fit of logistic and ordered logit models were tested, respectively, by the binary and ordinal versions of the Hosmer–Lemeshow test [60]. Unmet medical needs were also explored in an extended sample of those respondents who experienced health problems over the past 12 months, regardless of whether they had ambulatory consultation with an HCP. All calculations were performed using the Stata version 14.2 statistical software package (StataCorp) [61]. The level of significance was set at P<.05, and we applied no more than 15 observations per predictor variable when running multiple regression models [62]. Analyses were carried out without applying weights on the sample.

From the 1000 survey respondents, 736 had ambulatory HCP consultation within 12 months, out of which 5 happened over telephone. In 118 cases the respondent did not have a health problem, and 25 respondents could not tell if the visit happened at the regular HCP. After applying all criteria in sequence, 666 individuals were included in the sample (Table 1). Respondents with tertiary education and from the highest income quintile were slightly over-represented, whereas rural citizens were slightly under-represented compared with the general population. Mean age of our sample was 48.9 (SD 17.6) years. The demographic characteristics of the sample, all survey respondents, and the general population are summarized in Table 1. Responses on the income, chronic morbidity, and activity limitation items were provided by 86.5% (576/666), 89.0% (593/666), and 96.4% (642/666) of the respondents, respectively. The first, second, third, fourth, and fifth income quintiles of the sample corresponded to €115 (US $136), €247 (US $293), €332 (US $393), €397 (US $470), and €669 (US $786) mean per-capita household income levels, using the April 2020 12-month-average exchange rate of 330.73 HUF/€ (279.14 HUF/US $) [63]. The responses by PREM items are summarized in Table 2.

Mean eHEALS score was 29.3 (SD 4.9). eHEALS quartile mean scores were as follows: first quartile 23.5 (range 12-26; 191/666, 28.7%), second quartile 28.2 (range 27-29; 151/666, 22.7%), third quartile 31.2 (range 30-32; 182/666, 27.3%), and fourth quartile 36.0 (range 33-40; 142/666, 21.3%). Mean age of individuals in the fourth eHEALS quartile was 44.5 years (SD 17.1), which was lower than that of individuals in the first (49.9 years [SD 17.4]), second (51.2 [SD 17.5]), and third quartiles (49.3 [SD 17.7; F_3,662=4.07, P=.007). Mean eHEALS scores did not differ between male and female respondents (t₆₆₄=1.27, P=.21). However, while the percentage of female respondents decreased evenly from the first (101/364, 27.8%), second (92/356, 25.8%), third (86/364, 23.6%), and fourth eHEALS quartiles (85/364, 23.4%), male respondents were concentrated in the first (90/302, 29.8%) and third (96/302, 31.8%) quartiles (χ²₃=8.2, P=.04). The difference in terms of education (χ²₆=5.6, P=.47) and income (χ²₁₂=7.9.6, P=.79) was not significant between the four eHEALS groups.

A majority of respondents (380/631, 60.2%) did not report unmet medical needs in any areas. One unmet need was reported by 18.5% (117/631), 2 unmet needs by 8.9% (56/631), 3 unmet needs by 7.4% (47/631), and 4 unmet needs by 4.9% (31/631) of respondents. The Unmet Medical Needs Score had a single-factor structure with a KMO value of 0.73, suggesting moderately adequate sampling for EFA. The Cronbach α of .73 suggested acceptable internal consistency of this score constructed by adding the PREM items of the Unmet Medical Needs section.

Mean office waiting times were 63.3 (SD 71.0) minutes; 23.0% (152/661) of respondents waited less than 15 minutes, while waiting time was longer than 2 hours for 14.2% (94/661) of the sample. Long office waiting time was a problem for 26.9% (179/666) of all respondents, and for 34.8% (179/514) of those who waited longer than 15 minutes. Mean appointment waiting time was 16.8 (SD 27.8) days. Whereas 37.6% (242/643) of the sample could get an appointment on the same day, 18.2% (117/643) of respondents had to wait longer than 30 days. Long appointment waiting time was a problem for 15.3% (102/666) of all respondents, and for 24.1% (102/424) of those who did not get appointment on the same day. Any waiting problem either at the HCP office or before getting an appointment was reported by 33.5% (223/666) of the sample.

The Problem Score showed strong right skew (mean 2.0, SD 2.5; median 1; kurtosis 4.5; skewness 1.4; Shapiro–Wilk test P<.001). EFA suggested a single-factor structure with adequate sampling (KMO statistic=0.82) and good internal consistency (Cronbach α=.87). Whereas 0.5% (3/623) of respondents indicated the worst experience in all domains (definitely not answers for all 4 items; score 16), the experience was flawless (definitely yes answers for all 4 items; score 4) for 40.9% (255/623). The Negative Experiences Score had bimodal distribution (mean 1.5, SD 1.6; kurtosis 1.7; skewness 0.5; Shapiro–Wilk test P<.001) and a single-factor structure with adequate sampling (KMO statistic=.80) and good internal consistency (Cronbach α=.83). Problems were reported in 0, 1, 2, 3, and 4 domains by 40.9% (255/623), 18.0% (112/623), 11.2% (70/623), 11.4% (71/623), and 18.5% (115/623) of respondents, respectively. The strong correlation between the Problem Score and Negative Experiences Score (polyserial ρ=0.95) suggested that counting the domains with answers other than definitely yes accounted for most of the information within the patient experiences section.

The polychoric correlation matrix of PREM items and eHEALS score is shown in Table 3. The patient experience measures were strongly intercorrelated, whereas the correlation between those PREMs and waiting times and unmet medical need measures were moderate or weak. The corresponding waiting times and waiting problems were strongly correlated. The overall quality of the visit showed a strong negative correlation with items of the patient experiences section of the survey; the correlation was moderate or weak with remaining items. The eHEALS score showed a weak negative correlation with any of the PREMs. The correlation between PREM items and the time of the last visit was minimal.

The association between eHEALS quartiles and the problem score was not significant (Kruskal–Wallis test with ties, χ²₃=4.9, P=.18; Figure 1); conversely, the association was significant (χ²₁₂=24.4, P=.01) when the negative experiences score was considered (Figure 2). While we found no relationship between eHEALS quartiles and the time spent with the patient (χ²₉=14.5, P=.11) and opportunity to ask questions (χ²₉=9.2, P=.42), the association was significant (χ²₉=24.2, P=.002) between how easy it was to understand the HCP’s explanations (Figure 3) and the extent to which the HCP involved the respondent in decisions (χ²₉=18.2, P=.03; Figure 4). The association between eHEALS quartiles and the overall quality score was significant (Kruskal–Wallis test with ties χ²₃=10.1, P=.02; Figure 5). By contrast, the association between eHEALS quartiles and the share of respondents by overall quality categories was not significant (χ²₁₂=20.6, P=.07; Figure 6). Although the differences were small, the results suggest that respondents in the lowest eHEALS quartile had the least positive experience with HCP communication. Positive experiences were most frequently reported in the third and fourth eHEALS quartiles, whereas the subgroup with the highest eHEALS scores reported somewhat more negative experiences, when compared with respondents in the third eHEALS quartile. We found no association between eHEALS quartiles and either unmet medical needs or waiting times.

After controlling for sociodemographic variables, respondents’ health status, the setting of the visit, and type of HCP in ordered logit models (Table 4), the association was significant between eHEALS quartiles and the “Easy to understand explanations” item (Wald test χ²₃=11.8, P=.008). Although eHEALS quartiles jointly were not significant predictors of the “Opportunity to ask questions” item (Wald test χ²₃=4.7, P=.19), the differences between the first and third eHEALS quartiles were significant in both of those items (P<.001 and P=.04, respectively). The association was not significant between eHEALS quartiles and either the time spent with the patient (Wald test χ²₃=1.8, P=.61) or the involvement of the patient in decisions about care and treatment (Wald test χ²₃=3.4, P=.33).

After controlling for covariates, the overall quality also differed between respondents in the first and third eHEALS quartiles (Table 5), although the joint effect of eHEALS quartiles on the overall quality was not significant (Wald test χ²₃=6.0, P=.11). Unmet medical needs (Multimedia Appendix 3) and waiting times (Multimedia Appendix 4) were not associated with the eHEALS quartiles. The sensitivity analysis has shown similar findings in the majority of models using alternative eHEALS group boundaries (Multimedia Appendix 6). In half of the alternative scenarios the probability of missed interventions was also significantly lower (P=.05 to P=.02) in the third (moderately high) than in the first (lowest) and eHEALS score group, but not in the predefined quartiles (P=.05). We did not find significant (P=.16 to P=.88) association between eHEALS quartiles and any unmet needs variables in the extended sample involving those respondents who had a health problem over the past 12 months, regardless their participation in ambulatory HCP consultation (Multimedia Appendix 7). The detailed analysis of the effect of covariates on the PREM modules was out of scope for this paper, and has been provided elsewhere [30,41,64].

The specification of robust linear regression models was acceptable for the log-problem score and the negative experience score (Table 5). Findings show that the difference was significant between the first and third eHEALS quartiles in the log-problem score (P=.02) and negative experience score models (P=.02). The joint Wald test of eHEALS quartiles was not significant in either model (log-problem score F_3,430=2.28, P=.08; negative experience score F_3,430=2.17, P=.09; any negative experience χ²₃=0.8, P=.84). In addition, logistic regression models for any unmet medical needs and any waiting problems had an acceptable fit (Multimedia Appendix 5); eHEALS was not a significant predictor in any of these models (P=.05 to P=.42). In several scenarios of the sensitivity analysis, the unmet medical needs score and any unmet medical needs suggested less unmet needs in the third (moderately high) than in the first (lowest) eHEALS score groups (Multimedia Appendix 6).