The items of the VCCQ were developed primarily based on the HCCQ [7]. The 15 items of the HCCQ were adapted to measure perceived autonomy-support in a virtual instead of face-to-face care setting. The adaptation of the items consisted of rewording items from a setting in which there is direct contact with a physician (eg, “My physician answers my questions fully and carefully”) to a setting in which the respondent interacts with a Web-based intervention (eg, “< name intervention> answers my questions fully and carefully”). Additionally, the target behavior was added when applicable to increase the specificity of the items (eg, “I feel that <name intervention> provides me with effective possibilities to <target behavior>”). The name of the new questionnaire, that is, virtual care climate questionnaire, aims to recognize the basis of most of its items, that is, the health care climate questionnaire. Yet, to ensure that a wide variety of autonomy-supportive strategies were represented in the VCCQ, four additional items were included based on the PASSES [8] (items 16 and 17) and a discussion with experts on motivation and Web-based health behavior change interventions (items 18 and 19).

Similar as in the HCCQ, a 7-point response scale was used with totally disagree (1) and totally agree (7) as endpoints. All items and possible responses were translated from English to Dutch.

To investigate the face validity of the VCCQ, a pretest was conducted among experts. Moreover, a pretest was conducted among the target population in order to identify improving alterations to the questionnaire.

To validate the VCCQ psychometrically, two studies were conducted. Study 1 was conducted in the context of a Web-based, computer-tailored intervention to reduce alcohol intake (Drinktest [15,17]) among Dutch adults who drink alcohol (occasionally). Study 2 was conducted in the context of a Web-based, computer-tailored intervention to reduce cannabis intake (Weed-check [18]) among a sample of Dutch students.

Sample characteristics of respondents participating in Study 1 are presented in Table 2. For Study 2, sample characteristics are presented in Table 3. For both studies, the flow of respondents is presented Multimedia Appendix 1.

In Study 1, participants’ average scores on the 23 VCCQ-items ranged between 3.68 (SD 1.66) and 5.59 (SD 1.29) (Table 4). Scores covered the entire range on the scale from 1 to 7. All items were skewed toward agreement that the virtual care climate is autonomy-supportive, though varied in the extent to which agreement was expressed (Multimedia Appendix 2). Only two items had ≥10 answers in each of the two response categories at the lower end of the scale. Inter-item correlations ranged from −.11 to .79 (P<.001) (Multimedia Appendix 3) . Two negatively worded items (13 and 23) showed nonsignificant associations with other items.

In Study 2, participants’ average scores on the 23 VCCQ-items ranged between 3.70 (SD 1.61) and 6.01 (SD 1.12) (Table 4). As in Study 1, scores covered the entire range of the scale. When compared with Study 1, items were less skewed toward agreement that the virtual care climate is autonomy-supportive; three items had <10 answers for the response category at the lower extreme and 14 items had <10 answers for the highest response category (Multimedia Appendix 2). Inter-item correlations (Spearman) ranged from −.22 (P<.001) to .67 (P<.001) (Multimedia Appendix 3) . The same two negatively worded items (13 and 23) as in Study 1 as well as item 18 and item 3 showed weak or no associations with other items.

Nonetheless, all items in both studies were included in the next steps for a more detailed diagnosis.

In Study 1, items 13 and 23 violated the assumption of unidimensionality (ie, item H<.30; Table 5); therefore, these items were excluded from further analyses. Table 5 therefore shows results for the resulting 21-item unidimensionality analysis. All items and the total scale had scores >.30 (H=.660, standard error, SE=0.029), hence the 21-item VCCQ scale could be considered unidimensional. In other words, its items measured a single underlying concept, as intended. No significant violations of monotonicity were found for the 21-item VCCQ (ie, all crit values <.40). Violations of IIO were only found for item 6; therefore, this item was excluded. All 20 remaining items fitted the IIO criterion.

In Study 2, items 13, 23, 18, and 3 were found to have item H<.30 (Table 5) and were therefore excluded. Table 5 shows the results of the unidimensionality analysis of the remaining 19 items. All item H s and the scale H were >.30 (H=. 445, SE=0.030). The results confirmed that the 19-item VCCQ scale could be considered unidimensional. No significant violations of monotonicity were found (ie, all crit values <.40) Violations of IIO were found for items 15, 17, and 2, which were therefore excluded, resulting in a final 16-item VCCQ.

With regard to LI, in Study 1 items 11 and 14 and in Study 2 items 16, 20, and 21 were flagged as possibly violating LI – whereas these items were retained in the questionnaire in this study as LI is still under construction, these items may be considered for exclusion when developing a VCCQ short-form.

Only 15 items remained in the VCCQ in both Study 1 and Study 2. For research as well as clinical purposes it is most desirable to keep the respondent burden of questionnaire completion as low as possible, and use the same items across interventions aimed at different behaviors. Therefore, we proceeded to the analyses of the questionnaire’s reliability and construct validity with a 15-item version of the VCCQ. The remaining 15 items maintained the breadth of the theoretical concept. That is, an item was only excluded when its content showed overlap with other items (six instances), it was excluded based on item response theory (IRT) results in both studies (one instance), or when the item was retrospectively deemed inappropriate in the context of Web-based, computer-tailored interventions (one instance). Remaining items are presented in bold font in Table 1. Item step response functions of the 15 remaining items are presented in Multimedia Appendix 4.

The CFA with data from Study 1 showed that the CFI of the one-factor model was 0.897, the TLI was 0.879, and the RMSEA was 0.128 (95% CI 0.115-0.140). In Study 2, results were similar: CFI=0.902; TLI=0.885; RMSEA=0.087 (95% CI 0.074-0.100). Chi-square tests were significant for both models (χ²₉₀=426.6 and 244.8, P<.001). The scale showed good internal consistency both in Study 1 (Cronbach alpha=.97 and omega=.97, H=.66, mean 4.9 [SD 1.0]) and Study 2 (Cronbach alpha=.92 and omega=.92, H=.66, mean 4.9 [SD 1.0]). CFA diagrams are presented in Multimedia Appendix 5.

The objective of this study was to develop and validate the VCCQ: the first instrument specifically intended to measure perceived support for autonomy in a virtual care setting. On the basis of on NIRT analyses, we found that several items needed to be excluded from the questionnaire: three items in Study 1 and seven items in Study 2. The final 15-item version included items that were adjusted from the original HCCQ [7] and PASSES [8] items, as well as new Internet-specific items based on expert consultation. This suggests that, as expected, neither the HCCQ nor the PASSES would have sufficiently covered the concept of perceived autonomy-support in a virtual care setting and provide support for the approach that was taken for item development, that is, to use both existing questionnaires and expert consultation as input.

The final 15-item VCCQ was characterized by unidimensionality, monotonicity, and IIO, as required by NIRT. Therefore, the items can be used to compute a total score that reflects the respondents’ relative positions on the latent construct of “perceived autonomy support,” and these scores can also be used for comparing groups. Most items also met the local independence criterion, though before any items can be excluded based on this new psychometric test, further research is needed. Yet the items flagged, based on the criterion of local independence in our studies, can be first candidates for exclusion from a short-form VCCQ. Furthermore, the CFA of the 15-item VCCQ showed an acceptable model fit consistent with MSA results. Fit values were slightly below the recommended thresholds [33,34], though there is a scientific debate on the use of and value that should be attached to these thresholds when determining a scale’s model fit [36]. Reliability of the 15-item VCCQ was confirmed by two alternative tests (alpha and omega). Therefore, this version can be used confidently in future research, while continuing investigations into its measurement properties in different contexts and populations.

All three negatively worded items (ie, 13, 18, and 23) were excluded from the questionnaire due to low inter-item correlations in both studies. While including such items in questionnaires is recommended to reduce acquiescence bias, it may lead to response inconsistencies and may prove difficult to specify in measurement models. Therefore, some experts advise against their use [37,38]. Excluding these items, together with other misfitting items, resulted in good measurement properties of the current VCCQ version without limiting the construct breadth—an item was only excluded when its content showed overlap with other items, it was excluded from the scale in both studies, or was retrospectively deemed inappropriate in the context of Web-based, computer-tailored interventions. Future psychometric work on the VCCQ could investigate the risk of acquiescence in reporting on perceived autonomy support and possible solutions.

The scale’s convergent validity was confirmed by positive associations with autonomous motivation for reducing alcohol intake and cannabis consumption, and with perceived competence for reducing alcohol intake. Divergent validity was confirmed by the nonsignificant association with perceived competence for learning, yet the expected negative or nonsignificant relationship with controlled motivation could not be confirmed. In fact, in both studies we found a significant positive association between perceived support for autonomy and controlled motivation, although weaker than the association with autonomous motivation in Study 1. Moreover, in both our studies, autonomous motivation showed a strong significant positive correlation with controlled motivation (r=.68 in Study 1; r=.61 in Study 2), supporting our idea that autonomous and controlled motivation may represent different motivational dimensions that can coexist within people. This has also been suggested in earlier SDT-based research into motivational profiles, suggesting not only a high quality (ie, high autonomous and low controlled motivation), but also a high quantity motivational profile (ie, high autonomous and high controlled motivation) that does not necessarily lead to inferior results in terms of learning and physical activity than the high-quality profile [39]. In addition, earlier research has shown that introjected motivation, a form of controlled motivation that is characterized by performance of a behavior to avoid feelings of guilt or anxiety or to attain ego enhancement such as pride [3], may have positive effects on health behavior change [40]. Thus, an increase in especially this type of controlled motivation from using an Web-based intervention may also be a desired outcome. In our studies, perceived support for autonomy as measured by the VCCQ showed a stronger association with the introjected motivation subscale (r=.35 in Study 1; r=.38 in Study 2) than with the extrinsic motivation subscale (r=.21 in Study 1; r=.34 in Study 2). Whether this can be the direct result of providing support for autonomy can, however, not be concluded from the present study due to its cross-sectional nature. Following this line of reasoning, an alternative explanation may be that both autonomously motivated and respondents with high levels of controlled motivation were included in the present studies, with both groups finding potentially different elements in the intervention as supportive of their autonomy. Longitudinal research that includes measurement of the different types of motivation both before and after participation in a virtual care intervention is needed to examine whether perceiving higher autonomy support as measured by the VCCQ indeed increases autonomous motivation and reduces or increases the different forms of controlled motivation, and to determine whether autonomy-supportive interventions differ in their effectiveness for people with different types of motivation—in terms of both motivational and behavioral outcome measures. In addition, longitudinal research may be used to obtain evidence related to the questionnaire’s predictive validity.

An important strength of this study is that before proceeding to classical test theory analyses to determine the VCCQ’s validity and reliability, we investigated the questionnaire’s structure using NIRT analyses [26]. A known limitation of factor analytic methods is that they do not take into account differences in item difficulty, which are considered in IRT analyses, including MSA [40]. Moreover, as other parametric methods, they attempt to estimate quantitative differences between respondents, and are therefore unnecessarily restrictive for constructs that only refer to differences in degree [26]; perceived autonomy support is arguably among these constructs and more adequately investigated via MSA. So far, not many studies have used NIRT analyses in addition to classical test theory analyses and we are unaware of any previous studies that did so in the context of SDT-based questionnaires.

The VCCQ was developed to measure perceived support for autonomy in a variety of virtual care settings, targeting different kinds of health behavior. Yet, in the present study it was tested for its reliability and validity solely in the context of Web-based interventions aimed at two addictive behaviors, namely alcohol consumption and cannabis use. To increase the generalizability of the results to interventions targeting other nonaddictive health behaviors such as physical activity and healthy dietary behaviors, future research could consider testing the validity and reliability of the VCCQ in the context of other Internet- based interventions. Moreover, as both studies described were cross-sectional, test-retest reliability and predictive validity of the VCCQ could not be ascertained. To investigate these properties, longitudinal research designs should be considered. An additional limitation pertains to the use of a scenario (ie, to ensure that participants made proper use of the Web-based interventions, they were instructed to imagine they wanted to change their behavior and that they used the Web-based intervention to this end). Whereas this may limit the ecological validity of the results presented, the aim of this specific study was to develop and validate a novel questionnaire, and not necessarily to actually help people to improve their health behavior. As a consequence, the exposure of people who truly wanted to change their behavior to interventions that were not necessarily developed to be autonomy-supportive was deemed undesirable. One of the uses of VCCQ is for guiding intervention development, and in such cases our choice of sample is actually recommended: people who are similar enough to the target group to be able to relate to the content, but not require the intervention as such (and thus be in a more vulnerable position). Due to the inclusion criteria set, the participants in the present studies can be considered very well-capable of imagining themselves wanting to change their behavior and of evaluating the interventions in terms of their autonomy-supportiveness. Therefore, the scenario used can be considered rather realistic. Another limitation that warrants attention concerns the use of an odd number of answering categories within the VCCQ, as also used in the HCCQ and PASSES, providing respondents with the possibility to equivocate when completing the questionnaire. In both studies, we prevented this from having a large influence on the psychometric analyses, by excluding respondents with a too high number of neutral answers (ie, respondents who had z-scores >3 for the number of neutral answers), though as is shown by the answering patterns presented in Multimedia Appendix 2 neutral answers were still rather common. The optimal number of response categories for questionnaires assessing support for autonomy in different contexts (virtual or face-to-face) is therefore a valuable research question for further studies. Finally, responding to the final 15 VCCQ-items within multimeasure surveys may still place a significant burden on respondents, potentially increasing nonresponse. Future research may therefore consider the development and validation of a VCCQ-short form, following examples from earlier research where the 15-item HCCQ was transformed into a valid and reliable six-item IOCQ (important other climate questionnaire) [41] and a six-item mHCCQ (modified HCCQ adapted for breast cancer patients) [42].

By developing the VCCQ, this study aimed to address the lack of validated tools that measure support for autonomy in the context of virtual care settings. Scientifically, the VCCQ enables further research into the role of autonomy-support in virtual care settings without virtual health care providers involved, a context in which evidence for its positive effects is currently lacking. Practically, using the VCCQ will help to identify how Web-based health behavior change interventions can most successfully support autonomy, subsequently increase autonomous forms of motivation, and ultimately promote a healthy lifestyle. Health behavior change interventions are increasingly provided via the Internet [9], and people tend to search for health-related information first and foremost through this medium [10]. While limited resources are available for the widespread implementation of effective lifestyle interventions and health care decision makers should select interventions based on their cost-effectiveness, Web-based health behavior change interventions are likely to be highly competitive when compared with other types of interventions [43,44]. Investigating the role of autonomy support in these settings is necessary, as it may lead to further improvements in the effectiveness of such interventions.

The VCCQ appeared to accurately assess participants’ perceived autonomy-support offered by two Web-based health behavior change interventions. Overall, the scale showed the expected properties and relationships with relevant concepts and the studies presented suggest this first version of the VCCQ to be reasonably valid and reliable. As a result, the current version should be used cautiously in future research and practice to measure perceived support for autonomy within a virtual care climate. However, future research efforts are required that focus on further investigating the VCCQ’s divergent validity, on determining the VCCQ’s validity and reliability when used in the context of Web-based interventions aimed at improving nonaddictive or other health behaviors, and on developing and validating a VCCQ-short form. A Dutch version is available from the authors upon request.