The STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guideline for cohort studies was used to report the study. The methods are described in detail in Barnes et al [17]. We provide a brief overview below.

The analytical approach is detailed in Barnes et al [17] Briefly, SPSS (version 28; IBM Corp) was used to calculate the sensitivity, specificity, and reliability of the virtual assessment. For sensitivity and specificity, the findings of the in-person assessment completed by the initial clinician were compared to the findings of the virtual assessment completed by the initial clinician. For interrater reliability, the virtual assessment completed by the initial clinician was compared to the independently documented findings of the second clinician. For intrarater reliability, the virtual assessment findings documented by the second clinician at 2 times, approximately 1 month apart, were compared. Cohen Kappa values were documented for reliability. We calculated 95% CIs for each statistic. As the measures administered are concussion-specific, subgroup analyses for sensitivity, specificity, and reliability were conducted for concussion only and for nonconcussion participants.

Ethics approval was obtained from the Ottawa Health Sciences Network Research Ethics Board (20230311‐01H), the Bruyère Health Research Ethics Board (M16-22-006), and the University of Ottawa Board of Ethics (H-06-23-9348). Informed consent was obtained from all participants before participation in the study. Privacy and confidentiality were ensured by following institutional ethical protocols. All data were deidentified prior to analyses using unique study identifiers. Recordings of the virtual assessments were stored on secure servers and were only accessible to the research team and the second clinician rater. Participants received a CAD $30 (US $22) gift card and parking voucher for taking part in the study.

Figure 1 presents a CONSORT (Consolidated Standards of Reporting Trials) diagram outlining the number of participants approached and enrolled in this study.

We previously reported that 6 physical concussion measures, including the finger-to-nose test, VOMS, balance testing, cervical spine range of motion, and saccades, were deemed feasible and acceptable for virtual administration [16]. This study reports on the sensitivity, specificity, and reliability of the virtual administration of these measures. This is a critical advancement, as limited evidence exists on the psychometric properties of concussion-related measures in virtual environments. By directly examining reliability and diagnostic accuracy, this study provides foundational data and is essential to move beyond feasibility toward evidence-informed implementation. The findings indicate that properties vary from moderate to excellent for sensitivity and specificity, and poor to excellent for inter- and intrarater reliability.

Virtual assessments offer a potential benefit in concussion care in terms of improving accessibility, convenience, and cost-effectiveness, which is particularly important when in-person services are restricted, such as throughout the COVID-19 pandemic [12]. Highlighting these benefits may help address mixed perceptions of telehealth by demonstrating the tangible value that virtual assessments can add to both patients and health systems. While the benefits of virtual assessments are clear, documenting the reliability, sensitivity, and specificity metrics associated with the virtual administration of measures is crucial because clinicians rely on findings obtained on these assessments to make important clinical decisions, such as the need for directed treatment. Without this evidence, adoption of virtual care risks being undermined by uncertainty about accuracy. The findings of this study provide some insight into the important properties and identify the measures that offer the most promise when administering virtually. This, in turn, may inform clinical practice, ensuring that quality of care is maintained in a virtual environment.

The psychometric properties for in-person assessment of the 6 measures we tested have been reported previously and vary between studies. The sensitivity metrics associated with in-person administration range from poor (0.45 for the single-leg stance test) [22] to excellent (0.96 for the VOMS) [23]. Adequate sensitivity properties are important to ensure that the presence and magnitude of the deficits are appropriately identified by clinicians, which in turn aids in informing management [24]. The reliability of in-person administration of the measures ranges from moderate (κ=0.54 for finger to nose testing) [25] to excellent (intraclass correlation coefficient=0.90 for cervical spine range of motion evaluation) [26]. Acceptable reliability properties are required to ensure that measures yield consistent results required to make informed clinical decisions.

While acknowledging the variability in properties for in-person administration of measures, the findings of this study indicate that certain measures may be more suitable to administer virtually compared to others. The reliability, sensitivity, and specificity properties range from poor to excellent for all measures when administered virtually, with the VOMS change in symptoms measure showing the most promising metrics. Interrater reliability properties appear to be poor for most cervical spine range of motion evaluations, simpler balance tests (double leg stance, eyes open and closed), finger-to-nose testing, and oculomotor tests, including saccades and near point convergence measurement. These objective findings are in line with subjective concerns regarding the reliability and accuracy associated with the virtual assessment and the ability to identify subtle deficits over videoconferencing [27]. This highlights a potential area for development through the exploration of technological advancements to support completion of the virtual assessments, with a needed focus on ocular, simple balance, and coordination measures [28].

Reliability properties associated with the virtual administration of measures previously reported in the literature appear to be superior compared to those obtained in this study. Measures such as the 30-second arm curl test, 30-second chair stand test, 2-minute step test [29], knee and wrist joint range of motion [30], Berg Balance Scale, Timed Up and Go, Dynamic Gait Index [31], and the Tinetti Performance-Oriented Mobility Assessment gait scale [32] contain good to excellent reliability properties when administered in a virtual environment. Potential explanations for the variations in reliability metrics could include differences in technology and equipment used, differences in patient populations (with people with concussion potentially experiencing more subtle deficits that may be more challenging to identify on videoconferencing platforms), differences in measures used (with the concussion measures relying more on subjective interpretation by clinicians), and differences in methodological approaches (relying on recordings of videos compared to administering twice in the virtual environment).

When considering sensitivity and comparing the subjective perceptions of the participants, the perceived similarity appears to be comparable to the objective findings on the assessments, with 75% of participants perceiving as though similar results were obtained and sensitivity metrics above 75% for the majority of measures. This is superior to metrics reported in the literature for measures such as extraocular movements, gait, sensation, facial weakness [33], reach [34], and the Nine Hole Peg Insertion Test [35]. Superior sensitivity metrics reported in this study may be due to methodological differences (comparing the findings of the same clinician vs different clinicians), and clinician familiarity with the participants.

While there are clear concerns regarding the similarity of virtual and in-person administration of certain measures, previous studies have documented strong associations between in-person and virtual administration of stroke and multiple sclerosis measures [36-38].

In terms of participant perceptions of the assessments, a high level of confidence and perceived similarity was documented in this study. This is consistent with findings documented by Robb et al [39] who compared telemedicine and in-person visits, consisting of a clinical history interview and focused neurological examination (including gait, ocular movements, pronator drift, finger-to-nose, finger tapping, facial motor symmetry, and brief mental status evaluation) completed by a neurologist, for people with multiple sclerosis. Robb et al [39] reported perceived equivalence on the 2 approaches and highlighted the value in offering virtual visits as an alternative to in-person care. From the neurologist’s perspective, the virtual approach provided similar information when compared to the in-person visit.

While an understanding of psychometric properties associated with clinical measures is crucial, the selection of measures to use in virtual practice by clinicians also depends on the clinical utility characteristics. Previous work has reported that clinical instinct is prioritized over the use of standardized measures [40]. Physiotherapists may use standardized measures to quantify ability; however, clinical decisions regarding management typically rely upon observation [41]. Clinical relevance, as perceived by clinicians, acts as a facilitator to the use of certain measures in practice, and therefore, clinical utility may be of more importance to the use of measures rather than solely relying on empirical data [42]. Given these insights, clear communication of the properties associated with virtual administration of measures obtained in this study, along with emphasis on clinical relevance and ease of use for certain measures (such as the VOMS) compared to others (such as cervical spine range of motion), will be needed. It is recommended that clinicians exercise caution when using certain clinical measures virtually due to the variable psychometric properties. However, the integration of virtual assessments (using measures with promising properties) as a complement to in-person assessments may enhance the capacity to support patients through recovery. Further, there is potential for technological advancements, such as wearable sensors, to improve the accuracy of certain assessments. Development of a virtually appropriate battery of tests is needed. It should also be noted that several of the estimates in this study had wide confidence intervals reflecting variability in precision. These wide intervals highlight the need for cautious interpretation and reinforce the importance of future studies with larger sample sizes to narrow confidence intervals and strengthen clinical recommendations.

To address potential sources of bias, several standardization procedures were implemented. All study assessments were completed at the Ottawa Hospital Rehabilitation Center to ensure consistency in testing environment and equipment. The order of virtual and in-person assessments was randomized and counterbalanced to reduce rater and order effects. In addition, the virtual assessments were recorded and independently rated on 2 occasions by a second clinician to minimize single-rater bias and enable evaluation of both inter- and intrarater reliability.

However, certain metrics, such as sensitivity and specificity, may be overestimated as both the in-person and virtual assessments were completed in the same setting and with clinicians who were familiar with the patients. This limits the generalizability of the findings, as factors present in home environments with variable access to technology may impact psychometric properties associated with the measures. Technical challenges may contribute to more difficulties associated with completing virtual assessments in home environments [43]. Therefore, properties may differ if virtual assessments were conducted in a true virtual environment, such as in the home setting.

Sample bias may be present, as participants comfortable with technology may have been more inclined to participate when compared to those with limited experience or comfort with technology. Furthermore, rater bias may have contributed to properties observed between the in-person and virtual assessments; however, in attempts to address this issue, the order of the assessments was randomized and counterbalanced. In evaluating reliability, additional biases may be present due to duplicative testing in the in-person and virtual environments. We report on the top 6 physical concussion measures as identified by expert clinicians [27,44] and 4 of the most critical psychometric properties; however, future work should expand on the measures explored in the virtual environment and investigate additional properties such as responsiveness [45].

Some of the measures used in this study may not be relevant or validated in specific forms of acquired brain injury, such as the VOMS, which has not been explored for use in populations other than concussion. Therefore, properties for use in the nonconcussion population remain uncertain. Subgroup analyses (Multimedia Appendix 2), however, potentially support generalization beyond concussion for certain measures.

Clinical measures with acceptable psychometric properties are required for widespread adoption of such measures in practice. This method-comparison study reports on the reliability, sensitivity, and specificity associated with the virtual administration of certain physical concussion measures, including the finger-to-nose test, cervical spine range of motion, balance testing, VOMS, and saccades, along with the evaluation of effort. Metrics associated with these measures vary from poor to excellent. The virtual approach to concussion physical assessment may provide a promising approach to complement in-person care when barriers to attending face-to-face appointments exist. However, it is recommended that clinicians consider properties when interpreting certain measures, such as the VOMS and complex balance tests. Further research is needed to expand on these findings to include the exploration of other measures, additional psychometric properties, and the potential for technology to improve the ability to accurately and consistently identify deficits post concussion.