At the outset of the COVID-19 pandemic, the VCES was adapted from a previously validated telepsychiatry survey developed and used within the TeleMental Health program at the Centre for Addiction and Mental Health (CAMH), a mental health and addiction hospital located in Toronto, Ontario [6]. This telepsychiatry survey was updated to respond to the pressing need to evaluate virtual care both within our hospital and externally. Several adaptations were made to extend the use of the tool beyond its original intended use with clients accessing psychiatric consultation and assessment through videoconference [6].

The approach to developing and validating the original telepsychiatry survey is outlined in the paper by Serhal et al [6]. Our team of clinicians, researchers, and health care leaders reviewed the telepsychiatry survey and adapted it for use in the current context. All items were reviewed and updated, in particular, with the need to ensure relevance that is accorded with the expansion of virtual care provision beyond physicians and psychiatrists. For example, items on the newly developed VCES ask about the client’s experience with the virtual appointment instead of the telepsychiatry appointment. Another item asks whether the health care provider explained the risks and benefits of treatments or interventions rather than the risks and benefits of medications. After consultation with clinicians across multiple programs and services at the hospital, the survey was also updated to ensure applicability not only to clients but to family members as well. This was an important shift given the key role that families play in mental health and addiction care and recovery [13,14] and in supporting access to care through videoconference.

The transition to virtual care during the COVID-19 pandemic also necessitated the examination of virtual care from a digital health equity perspective [10]. Our team consulted with the CAMH Health Equity Office as well as the Provincial System Support Program (PSSP), which administers the Ontario Perception of Care Tool for Mental Health and Addictions (OPOC-MHA) [15], regarding the addition of sociodemographic questions to facilitate health equity analyses. Furthermore, 5 items from the OPOC-MHA were also added to facilitate alignment and comparison between the 2 tools.

The VCES sociodemographic questions ask about gender, age, geographic region, whether the client was born in Canada, racial or ethnic group, illness and disability, and other factors. Clients and families were also asked about their comfort with technology, where they accessed the virtual care appointment from (eg, home, health care organization), the type of device they used to access virtual care (eg, computer, smartphone), and which videoconference platform was used. The telephone was included as an option on the survey, as CAMH, like many health care settings, began providing more services by telephone during the pandemic, particularly for individuals experiencing barriers to video-based care. Finally, as recommended by Serhal et al [6], we added additional survey items to assess access to care, and physical and emotional safety during virtual care appointments. We also added an item about compassionate care, given our research group’s interest in compassion as a critical factor in the therapeutic relationship and inpatient and family experience of care [16,17].

The redesigned VCES consists of 22 items to assess the overall quality of care. These items are rated on a 4-point Likert scale (1=strongly disagree, 2=disagree, 3=agree, 4=strongly agree) with additional options including “not applicable” and “prefer not to answer.” The final question on the survey is an open-text question to elicit any additional feedback.

Following the adaption of the survey, the survey was revalidated using the 6 domains of health care quality of the Institute of Medicine (IOM) as a guiding framework. These 6 domains include being safe, effective, patient-centered, efficient, timely, and equitable [18]. The IOM domains were selected as a guiding framework to provide a comprehensive evaluation of client satisfaction and experience and to guide targeted quality improvement efforts across the different domains. The heath quality domain definitions [18] have been adapted in Table 1 below for the VCES.

Consent to participate was implied. The survey contained an introductory section explaining the purpose of the survey, informing potential respondents that participation was voluntary and anonymous, and informing them that the findings would be presented as grouped or aggregated data for any presentation or publication. There was no compensation offered to clients and families for completing the survey. This project received ethical approval from CAMH Quality Projects Ethics Review (QPER-2021-000).

In total, the survey was completed 181 times, with 99 surveys completed by clients, 16 by support persons on behalf of clients, and 22 by family members or caregivers. Respondent information is missing on 44 surveys. Sociodemographic characteristics are included in Table 2.

Most respondents used a computer (113/179, 62.4%) or tablet or smartphone (58/179, 32%). Only 5.5% (10/179) reported using a telephone (audio only). The majority of respondents were either comfortable with technology (64/179, 35.8%) or very comfortable (85/179, 47.5%).

The construct reliability, as measured by Cronbach α, was generally high. Timely was the only subscale with an α lower than 0.7; all others were above 0.8 (more details in Table 3). In all cases, the corrected item-total correlation was higher than 0.3. Considering the items in the model, item 9 (effective) and items 2 and 10 (timely) had lower corrected item-total correlation, with all the other items having correlations around 0.7 or above. Within the “timely” factor, the low item-total correlation was likely reflective of the small number of items associated with this domain.

Confirmatory Factor Analysis loadings are shown in Table 3 above. The model was adjusted after multiple imputations with 20 datasets. We show fit statistics using the mean and SD of these 20 datasets. The mean chi-square value was 437.5, with df=199 (P<.001). The mean RMSEA was 0.08 (SD 0.002), which is considered reasonable. The mean CFI was 0.987 (SD 0.001), the mean TLI was 0.985 (SD 0.001), and the mean SRMR was 0.04 (SD 0.001). These values are considered to be quite good. With the exception of item 9 in the “effective” factor, all the others have standardized loadings higher than 0.7.

The model shown in Table 3 had 2 respecifications: item 19 was moved from “effective” to “patient-centered,” and item 18 was moved from “patient-centered” to “safe.” These changes were based on the modification index and discussions with the project team; however, they did not meaningfully change the fit measures. While item 9 does not fit as well in the “effective factor,” it was maintained as is, as it represented the best conceptual fit.

The sample size for structural equation models is not well defined because of the high number of parameters of interest and broad range of model complexity. Considering our model with 5 factors with an average of 4 indicators per factor and loadings around 0.8, Wolf et al [31] found that a sample size of 120 tends to be sufficient for power above 80%. These results are negatively affected by the presence of missing values; as such, our sample of 181 is expected to have reasonable power for a CFA analysis.

This paper provides a valid and reliable measure to evaluate client and family experiences of virtual mental health and addiction care, focusing on 6 domains of health care quality. This survey has been used internally at CAMH to guide the next steps and recommendations for virtual mental health care and has been shared externally in both English and French to support quality measurement within other organizations. Information about the survey was shared through a webinar, and the survey was subsequently downloaded hundreds of times across Canada and internationally.

Given the widespread uptake of virtual care, this survey has broad applicability across settings that provide mental health and addiction care. The VCES can be used to guide targeted quality improvement initiatives across the 6 IOM domains of health care quality, including safe, effective, patient-centered, efficient, timely, and equitable care [18]. It also has the potential to be adapted to other health care specialties and contexts.

From our experience, we found the use of REDCap (Research Electronic Data Capture; Vanderbilt University) to be an effective means to disseminate the survey, collate results, and ensure anonymity. Some training and guidance regarding survey administration processes were required to ensure that the survey was delivered as part of the standard of care. We found it advantageous to delineate a time-limited window for survey dissemination and completion to limit time demands on both clients and clinicians. In terms of interpreting results, we set a cutoff of 3 (agree) and above as a positive result indicating high quality of care. Finally, there was a high degree of interest in receiving feedback on the survey results. We offered aggregated program-level reports once an adequate sample of surveys was completed in order to preserve respondent anonymity; this is particularly important as demographic questions attached to the survey could render individual results identifiable.

While the findings from the VCES provided invaluable and timely feedback, it is important not to limit patient and family feedback to a survey. The open text option on the survey provided a breadth of feedback not necessarily addressed within the existing survey questions. Analysis of this qualitative data may spur future adaptations to the survey and, on a practical level, may steer quality improvement initiatives resulting from survey findings. Furthermore, ongoing patient and family engagement in interpreting survey results and prioritizing and guiding quality improvement will be essential to making meaningful and impactful improvements.

Patient experience of virtual care should also be considered alongside provider experience of virtual care. To complement findings from the VCES and to capture another key perspective within health care, our team has developed and validated the Virtual Provider Experience Survey (VPES). These complementary tools provide key metrics to inform how health care organizations and systems can move toward the “quadruple aim,” a well-recognized framework for health care that seeks to improve population health outcomes, reduce health care costs, and improve patient, family, and provider experiences of care [32]. The consistent use of standardized tools across settings is a valuable way to inform policy and program planning with respect to virtual care [33] and to further our advancement toward the quadruple aim. Further practice-based research is also needed to examine the relationship between the experience of care and health outcomes for clients and families who access virtual mental health and addiction care and the cost-effectiveness of those services.

While the VCES was validated with 181 survey responses, it is possible that some clients or family members completed the survey more than once if they were involved in multiple programs or services during the evaluation period. In addition, when administering surveys, there is always the risk of self-selection and response bias. We sought to mitigate this risk by asking clinicians and administrative staff to send the survey link to all clients and families accessing virtual care at CAMH. A response rate can be beneficial to ascertain the extent of potential bias; however, we were unable to determine the response rate, as it is unknown how many clients and families received the link in their email correspondence from clinicians or administrative staff. It is quite likely that those who have a higher degree of familiarity or comfort with virtual care were more likely to complete an online survey sent by email.

We sought to address a notable gap in the literature by redesigning and validating a measure of client and family experiences of virtual mental health care. Client and family perspectives are needed to evaluate the current state of mental health service delivery and highlight areas that need improvement. By effectively addressing challenges as they emerge, it is anticipated that we will continue to move toward hybrid modalities of practice that leverage the strengths and benefits of telephone, video, and in-person care to effectively respond to unique client and family needs and circumstances [11].