JMIR Publications

Journal of Medical Internet Research

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 08.05.17 in Vol 19, No 5 (2017): May

This paper is in the following e-collection/theme issue:

    Original Paper

    The Virtual Care Climate Questionnaire: Development and Validation of a Questionnaire Measuring Perceived Support for Autonomy in a Virtual Care Setting

    1Amsterdam School of Communication Research/ASCoR, Department of Communication Science, University of Amsterdam, Amsterdam, Netherlands

    2Health Services and Performance Research (HESPER EA 7425), University Claude Bernard Lyon 1, Lyon, France

    3Trimbos Institute, Netherlands Institute for Mental Health and Addiction, Utrecht, Netherlands

    4Department of Clinical and Social Sciences in Psychology, University of Rochester, Rochester, NY, United States

    5Healthy Living Center, Center for Community Health, Department of Medicine, University of Rochester, Rochester, NY, United States

    Corresponding Author:

    Eline Suzanne Smit, PhD

    Amsterdam School of Communication Research/ASCoR

    Department of Communication Science

    University of Amsterdam

    PO Box 15791

    Amsterdam, 1001 NG

    Netherlands

    Phone: 31 (0)6 25258558

    Email:


    ABSTRACT

    Background: Web-based health behavior change interventions may be more effective if they offer autonomy-supportive communication facilitating the internalization of motivation for health behavior change. Yet, at this moment no validated tools exist to assess user-perceived autonomy-support of such interventions.

    Objective: The aim of this study was to develop and validate the virtual climate care questionnaire (VCCQ), a measure of perceived autonomy-support in a virtual care setting.

    Methods: Items were developed based on existing questionnaires and expert consultation and were pretested among experts and target populations. The virtual climate care questionnaire was administered in relation to Web-based interventions aimed at reducing consumption of alcohol (Study 1; N=230) or cannabis (Study 2; N=228). Item properties, structural validity, and reliability were examined with item-response and classical test theory methods, and convergent and divergent validity via correlations with relevant concepts.

    Results: In Study 1, 20 of 23 items formed a one-dimensional scale (alpha=.97; omega=.97; H=.66; mean 4.9 [SD 1.0]; range 1-7) that met the assumptions of monotonicity and invariant item ordering. In Study 2, 16 items fitted these criteria (alpha=.92; H=.45; omega=.93; mean 4.2 [SD 1.1]; range 1-7). Only 15 items remained in the questionnaire in both studies, thus we proceeded to the analyses of the questionnaire’s reliability and construct validity with a 15-item version of the virtual climate care questionnaire. Convergent validity of the resulting 15-item virtual climate care questionnaire was confirmed by positive associations with autonomous motivation (Study 1: r=.66, P<.001; Study 2: r=.37, P<.001) and perceived competence for reducing alcohol intake (Study 1: r=.52, P<.001). Divergent validity could only be confirmed by the nonsignificant association with perceived competence for learning (Study 2: r=.05, P=.48).

    Conclusions: The virtual climate care questionnaire accurately assessed participants’ perceived autonomy-support offered by two Web-based health behavior change interventions. Overall, the scale showed the expected properties and relationships with relevant concepts, and the studies presented suggest this first version of the virtual climate care questionnaire to be reasonably valid and reliable. As a result, the current version may cautiously be used in future research and practice to measure perceived support for autonomy within a virtual care climate. Future research efforts are required that focus on further investigating the virtual climate care questionnaire's divergent validity, on determining the virtual climate care questionnaire’s validity and reliability when used in the context of Web-based interventions aimed at improving nonaddictive or other health behaviors, and on developing and validating a short form virtual climate care questionnaire.

    J Med Internet Res 2017;19(5):e155

    doi:10.2196/jmir.6714

    KEYWORDS



    Introduction

    Unhealthy lifestyle behaviors—for example, smoking tobacco or cannabis, consuming too much alcohol, overeating calories or not eating a sufficient amount of fruit and vegetables, or being insufficiently physically active—are a major cause of chronic illnesses like cancer, diabetes, and cardiovascular diseases [1]. They have a detrimental effect on quality of life, decrease work productivity, and put an enormous, preventable strain on health care [2]. Developing effective interventions to promote healthy lifestyle behaviors that prevent or delay these diseases, as well as making these interventions available on a large scale and to a great variety of people, is thus important.

    To improve one’s lifestyle, self-determination theory (SDT) [3] proposes that an autonomous form of motivation and perceiving competence for changing are imperative. When it concerns the improvement of health-related behaviors like those mentioned before, people are assumed to perceive themselves to be autonomous in their motivation to change when the behavior is accompanied by an experience of psychological freedom of choice. Supportive of this theoretical assumption, earlier research has found autonomous motivation to be an important predictor of health behavior change, its maintenance, and subsequent positive health outcomes [4]. According to SDT, someone’s autonomous motivation can be increased by providing support for autonomy. The concept of autonomy-support was first of all applied to a face-to-face context where one person, such as a health professional, interacts with another person, such as a patient, and tools have been developed to measure this concept in face-to-face settings. When provided by a health care professional, such as a general practitioner or lifestyle counselor, support for autonomy involves strategies like eliciting and acknowledging a person’s perspective, providing a clear rationale for change, offering choice, and using noncontrolling language [5,6]. Perceived support for autonomy from a health care professional is measured through instruments like the health care climate questionnaire (HCCQ) [7]. Similarly, perceived autonomy-support from a physical education teacher can be assessed by the perceived autonomy support scale for exercise settings (PASSES) [8].

    Nowadays, interventions aimed at a healthy lifestyle are increasingly delivered via the Internet [9,10], creating a virtual instead of face-to-face care climate. Web-based health behavior change interventions can successfully promote a healthy lifestyle [11] and have several advantages over and above face-to-face interventions; they are highly accessible, participants can use them at any convenient time, and many people can be reached at minimal cost [9]. It can be assumed to be equally important to provide support for autonomy within a virtual care climate in order to increase people’s autonomous motivation for initiating and maintaining health behavior change. However, instead of human interaction, Web-based interventions deliver this support in different formats and rather make use of computer-human interaction. For interventions that make use of virtual health care providers, like a virtual clinician [12] or computerized personal trainer [13], the interaction between the virtual health care provider and receiver of the intervention might resemble the interaction between two humans. Consequently, only slight adjustments to questionnaires like the HCCQ and PASSES might be sufficient. There is, however, an abundance of Web-based interventions that do not involve virtual care providers, but in which autonomy-support is integrated in the structure of the Web-based tool (for examples of such interventions, see [14-16]). In this context, the operationalization of the concept of perceived autonomy-support would need to be reconsidered to a larger extent and available measurement instruments would need to be adapted to this new context. To illustrate, one of the items from the HCCQ reads, “My physician listens to how I would like to do things;” whereas a physician or other person is indeed able to listen, a Web-based intervention where no virtual health care provider is involved does not possess this ability. As a consequence, the item would not be applicable.

    Although substantial evidence is available for the positive effects of perceived autonomy-support within the face-to-face setting [4,6], as well as some evidence for these effects when it concerns virtual health care providers [12,13], no such evidence exists for the role of autonomy-support in virtual care settings without virtual health care providers involved. To develop an evidence base for this growing field of Web-based interventions, an instrument is needed with adequate measurement properties specifically for assessing perceived support for autonomy in such settings. Therefore, the objective of this study was to develop and validate the virtual care climate questionnaire (VCCQ): the first measurement instrument of perceived support for autonomy in virtual care settings. By developing the VCCQ, this study aimed to fulfill the need for such tools and thus enable further research into how Web-based health behavior change interventions can successfully support autonomy, increase autonomous motivation, and ultimately promote a healthy lifestyle.


    Methods

    The VCCQ was constructed in three steps: item development, pretesting, and psychometric validation.

    Item Development

    The items of the VCCQ were developed primarily based on the HCCQ [7]. The 15 items of the HCCQ were adapted to measure perceived autonomy-support in a virtual instead of face-to-face care setting. The adaptation of the items consisted of rewording items from a setting in which there is direct contact with a physician (eg, “My physician answers my questions fully and carefully”) to a setting in which the respondent interacts with a Web-based intervention (eg, “< name intervention> answers my questions fully and carefully”). Additionally, the target behavior was added when applicable to increase the specificity of the items (eg, “I feel that <name intervention> provides me with effective possibilities to <target behavior>”). The name of the new questionnaire, that is, virtual care climate questionnaire, aims to recognize the basis of most of its items, that is, the health care climate questionnaire. Yet, to ensure that a wide variety of autonomy-supportive strategies were represented in the VCCQ, four additional items were included based on the PASSES [8] (items 16 and 17) and a discussion with experts on motivation and Web-based health behavior change interventions (items 18 and 19).

    Similar as in the HCCQ, a 7-point response scale was used with totally disagree (1) and totally agree (7) as endpoints. All items and possible responses were translated from English to Dutch.

    Pretesting

    To investigate the face validity of the VCCQ, a pretest was conducted among experts. Moreover, a pretest was conducted among the target population in order to identify improving alterations to the questionnaire.

    Sample and Procedure

    Pretests were conducted in the context of a Web-based, computer-tailored intervention aimed at reducing alcohol intake (Drinktest [15]) and took place among five experts as well as five Dutch adults who (occasionally) drink alcohol, with varying age and socioeconomic status.

    The experts were identified through the first author’s professional network and were considered experts when they had a track record in the field of motivation, health behavior, or Web-based health behavior change interventions. Experts were asked whether they thought the 19-item VCCQ measured the same concept as the HCCQ, whether items were properly reworded to fit a virtual care setting, and whether they thought the VCCQ was a comprehensive instrument to measure perceived autonomy-support in a virtual care setting. Furthermore, experts were asked to indicate any perceived ambiguities in wording and whether they felt any aspects of autonomy-support were not covered. Experts also gave feedback on the translation of the items from English to Dutch.

    The Dutch adults were invited to complete the VCCQ after they had visited and consulted the Web-based intervention Drinktest, and reflected on whether the questions and instructions in the VCCQ were clearly formulated and whether they thought the questionnaire had an acceptable length . During their participation, they were asked to take notes, which were discussed with them afterwards.

    Pretest Results

    Most experts indicated that, generally due to the rewording of the HCCQ-items to a virtual care setting, the VCCQ items did not always measure the underlying concept of perceived support for autonomy. To ensure the VCCQ truly measured perceived autonomy-support in a virtual care setting, experts suggested to primarily use the content of the HCCQ-items as a basis and reformulate items when necessary, instead of a literal translation and rewording to the virtual context. Furthermore, experts indicated the VCCQ to be reasonably comprehensive. Yet, to ensure the VCCQ included a wide range of autonomy-supportive strategies, four additional items were included based on their feedback (items 20-23). Moreover, based on both expert input and feedback from the participating Dutch adults, the wording of 14 VCCQ-items was simplified to guarantee comprehensibility by the target group. This resulted in the final 23-item questionnaire presented in Table 1.

    Table 1. The 23 items of the initial VCCQ (virtual care climate questionnaire) with their description and source.
    View this table

    Psychometric Validation

    To validate the VCCQ psychometrically, two studies were conducted. Study 1 was conducted in the context of a Web-based, computer-tailored intervention to reduce alcohol intake (Drinktest [15,17]) among Dutch adults who drink alcohol (occasionally). Study 2 was conducted in the context of a Web-based, computer-tailored intervention to reduce cannabis intake (Weed-check [18]) among a sample of Dutch students.

    Sample and Procedure
    Study 1

    The sample consisted of 230 Dutch adults who drink alcohol (occasionally). Participants were recruited via the ISO-certified (International Organization for Standardization, ISO) research panel PanelClix [19] and, after providing informed consent, completed the study through the Web. Before filling out the VCCQ, participants visited and consulted Drinktest [17], a Dutch, Web-based, computer-tailored intervention to reduce alcohol intake [15]. When consulting this intervention, participants answered a variety of questions concerning their current alcohol consumption as well as their cognitions related to this behavior, based on which they received feedback that is tailored to their personal situation and beliefs. This feedback was presented to participants on their computer screen, and participants were also to send this feedback to an email address of their choice as well as to print it. To ensure that participants made proper use of the Web-based intervention, they were instructed to imagine they wanted to reduce their alcohol intake and that they used the Web-based intervention to this end. For participants to be able to imagine wanting to reduce their alcohol intake, an inclusion criterion was set based on participants’ alcohol intake. Participants who indicated never to have drunk alcoholic beverages before or those who had not drunk any alcoholic beverages in the past 12 months were excluded from participation—as were respondents who did not complete the survey entirely, had missing data on key variables, did not engage with the intervention, or participated for too long in the survey (ie, had z-scores >3 for participation time). After visiting and consulting the Web-based intervention, they were informed that they would be asked for their opinions about—and personal experiences with—this intervention in a subsequent questionnaire (ie, the VCCQ) and to answer several related questions regarding their motivation and perceived competence for changing. Participants were rewarded for their approximately 15-minute participation by means of 100 Clix, the usual incentive used by PanelClix, worth about €1.25.

    Study 2

    The sample consisted of 228 Dutch students using cannabis. Participants were recruited via the student participation website of the faculty of Social and Behavioral Sciences of the University of Amsterdam and, after providing informed consent, completed the study through the Web. As part of the survey, participants visited and consulted Weed-check [18], a Dutch, Web-based, computer-tailored intervention to reduce cannabis consumption. Similar as in the intervention used for Study 1, as part of their consultation of this intervention participants answered questions concerning their cannabis consumption and their cognitions associated with cannabis consumption, based on which they received feedback that was tailored to their personal situation and beliefs. Again, feedback was visible on the participants’ computer screen, and options were provided to send the feedback to an email address and to print it. To ensure that participants made proper use of the Web-based intervention, they were instructed to imagine they wanted to have an overview of their cannabis usage and that they used the Web-based intervention to this end. In order for participants to be able to imagine wanting an overview of their cannabis consumption, an inclusion criterion was set based on participants’ cannabis usage. Participants who indicated never to have used cannabis before or those who had used cannabis but only for medical reasons were excluded—as were respondents who did not complete the survey entirely, participated twice, indicated to have no understanding of the intervention, participated for too long in the survey, or had a too high number of neutral answers (ie, had z-scores >3 for participation time and the number of neutral answers, respectively). After visiting and consulting the Web-based intervention, participants were informed that they would be asked for their opinions about—and personal experiences with—this intervention and to answer several related questions regarding their motivation and perceived competence for changing. As a reward for their approximately 20-minute participation, students could choose their own incentive (ie, 0.5 research credits or € 2.50).

    Measures

    Both studies assessed several common and study-specific background variables, autonomous motivation to change, and perceived competence concepts, as detailed below.

    Background Variables

    Participants’ age and gender were assessed via single items. Moreover, in Study 1 several characteristics of alcohol consumption were measured: the total number of weekdays consuming alcoholic beverages, average number of alcoholic beverages consumed during a typical weekday, total number of weekend days consuming alcoholic beverages, average number of alcoholic beverages consumed during a typical weekend day, and the total number of times consuming more than four alcoholic beverages in a day within the last 6 months. In Study 2, four characteristics of cannabis consumption were measured: number of times consuming cannabis in the last 12 months, cannabis use in the last 30 days, number of times consuming cannabis in the last 30 days, total number of joints on a typical day.

    Autonomous Motivation

    The degree to which one’s motivation to change is autonomous (vs controlled) was measured by the treatment self-regulation questionnaire (TSRQ) [20], consisting of 16 items relating to autonomous or controlled motivation, or amotivation. The autonomous motivation subscale (6 items) showed good internal consistency (Study 1: Cronbach alpha=.95 and omega=.93; Study 2: Cronbach alpha=.91 and omega=.91), as did the controlled motivation subscale (7 items; Study 1: Cronbach alpha=.93 and omega=.93; Study 2: Cronbach alpha =.92 and omega=.92).

    Perceived Competence for Changing

    Participants’ feelings of competence to change were measured by the perceived competence scale (PCS) [21] for reducing alcohol intake (Study 1) and cannabis intake (Study 2), respectively. The PCS asked participants the extent to which they agreed with the four statements, such as “I feel confident in my ability to reduce my alcohol intake” (Study 1), or “I am capable to reduce my cannabis consumption” (Study 2). All items were measured on a scale ranging from strongly disagree (1) to strongly agree (7), and the scale proved to be reliable (Study 1: Cronbach alpha=.95 and omega=.95; Study 2: Cronbach alpha=.93 and omega=.93). For divergent validity analysis purposes, in Study 2 perceived competence for learning was also assessed using the PCS [22] (Cronbach alpha=.94 and omega=.94).

    Analyses

    All statistical analyses were conducted using R statistical package [23]. Item selection followed several steps following the criteria described below.

    Item Descriptives and Inter-Item Correlations

    The first step was to investigate the distribution of responses to VCCQ items via descriptive statistics and examine the strength of association between items based on Spearman correlations. Items were flagged if they showed inadequate distributions (eg, <10 responses per response option [24]) or low associations with other items (eg, nonsignificant correlations with other items), but were kept in further analyses for detailed diagnosis.

    Nonparametric Item Response Theory Analyses

    To examine the VCCQ’s structural validity taking into account variations in item difficulty [25], nonparametric item response theory (NIRT) analyses (Mokken scale analysis, MSA) were conducted using the mokken package in R [26]. MSA examines whether an item set orders respondents accurately on a continuum representing a single latent trait (ie, in this study perceived autonomy-support). According to MSA, items can be considered a scale (fit the monotone homogeneity model) if three conditions are met: unidimensionality, monotonicity, and local independence (LI). If the items also meet a fourth criterion, invariant item ordering (IIO), the scale can be used for group comparison [25,27].

    Unidimensionality entails that items can be located on a single latent continuum in terms of probabilities of obtaining high scores [25]. It was examined by assessing coefficients of homogeneity (H; ranging from 0 to 1, from no association to perfect association considering item distributions) and via an automated item selection procedure (aisp) algorithm [26]. Loevinger’s scalability coefficients estimated item and scale homogeneity of the items (Hi coefficients and H coefficients) [28]. The aisp analysis performs an exploration of the scale unidimensionality at increasing levels of homogeneity [28]. Monotonicity implies that the probability of obtaining high scores on an item does not decrease as latent trait scores increase; crit values >40 are considered as violating monotonicity requirements [29] and then the recommended strategy is to remove the item with the most serious (highest) violation and rerun the analysis. LI means that associations between items are explained only by their relationship with the latent trait (perceived autonomy-support) [26]. At the moment, LI is still under research as a new MSA test [30]; until very recently, no tests were even available to test this assumption. Therefore, in this study no exclusion decisions were made based on LI results. If the items also show IIO, the scale represents a “person-free” item hierarchy in terms of their difficulty, that is, the order of items remains the same at different levels of the latent dimension [31]. LI and IIO are tested via dedicated functions that flag items that violate these criteria to be considered for exclusion. For monotonicity and IIO, minimum group size was set at 50; LI was examined with default parameter values.

    Confirmatory Factor Analysis and Reliability

    After selecting an item set fitting the MSA criterion, a confirmatory factor analysis (CFA) was conducted using the function cfa in the lavaan R package [32] to examine whether the unidimensional structure is supported by this alternative analysis, and thus allows comparisons with prior literature on autonomy-support scales; default parameter values were used (eg, maximum likelihood estimator, covariance matrix). Model diagnostics, parameter estimates, and goodness-of-fit indices were examined. Model fit was judged against the following criteria: Tucker-Lewis index (TLI) and Comparative Fit Index (CFI) >0.95; root mean square error of approximation (RMSEA) <0.06; and chi-square P value >.05 [33,34]. This analysis was followed by assessing the internal consistency of the final scale using Cronbach alpha and omega [35].

    Construct Validity

    The VCCQ’s construct validity was determined by examining its convergent and divergent validity, based on expectations based on theory [3] and evidence from earlier research [8]. In terms of convergent validity, the VCCQ was expected to positively correlate with the autonomous motivation subscale of the TSRQ as well as with perceived competence for changing (ie, for reducing one’s alcohol intake or cannabis consumption). In terms of divergent validity, the VCCQ was expected to show nonsignificant or negative correlations with the controlled motivation subscale of the TSRQ and with the nonhealth-related concept of perceived competence for learning (Study 2 only). Both studies had 90% power to detect bivariate correlations r=.21 and 80% power for r=.18, at alpha=.05.


    Results

    Samples

    Sample characteristics of respondents participating in Study 1 are presented in Table 2. For Study 2, sample characteristics are presented in Table 3. For both studies, the flow of respondents is presented Multimedia Appendix 1.

    Table 2. Demographic characteristics of respondents participating in Study 1 (N=230).
    View this table
    Table 3. Demographics characteristics of respondents participating in Study 2 (N=228).
    View this table

    Item Descriptives and Inter-Item Correlations

    In Study 1, participants’ average scores on the 23 VCCQ-items ranged between 3.68 (SD 1.66) and 5.59 (SD 1.29) (Table 4). Scores covered the entire range on the scale from 1 to 7. All items were skewed toward agreement that the virtual care climate is autonomy-supportive, though varied in the extent to which agreement was expressed (Multimedia Appendix 2). Only two items had ≥10 answers in each of the two response categories at the lower end of the scale. Inter-item correlations ranged from −.11 to .79 (P<.001) (Multimedia Appendix 3) . Two negatively worded items (13 and 23) showed nonsignificant associations with other items.

    In Study 2, participants’ average scores on the 23 VCCQ-items ranged between 3.70 (SD 1.61) and 6.01 (SD 1.12) (Table 4). As in Study 1, scores covered the entire range of the scale. When compared with Study 1, items were less skewed toward agreement that the virtual care climate is autonomy-supportive; three items had <10 answers for the response category at the lower extreme and 14 items had <10 answers for the highest response category (Multimedia Appendix 2). Inter-item correlations (Spearman) ranged from −.22 (P<.001) to .67 (P<.001) (Multimedia Appendix 3) . The same two negatively worded items (13 and 23) as in Study 1 as well as item 18 and item 3 showed weak or no associations with other items.

    Nonetheless, all items in both studies were included in the next steps for a more detailed diagnosis.

    Table 4. Descriptive statistics for the 23 VCCQ (virtual care climate questionnaire) items in Study 1 (N=230) and Study 2 (N=228).
    View this table

    Nonparametric Item Response Theory Analyses

    In Study 1, items 13 and 23 violated the assumption of unidimensionality (ie, item H<.30; Table 5); therefore, these items were excluded from further analyses. Table 5 therefore shows results for the resulting 21-item unidimensionality analysis. All items and the total scale had scores >.30 (H=.660, standard error, SE=0.029), hence the 21-item VCCQ scale could be considered unidimensional. In other words, its items measured a single underlying concept, as intended. No significant violations of monotonicity were found for the 21-item VCCQ (ie, all crit values <.40). Violations of IIO were only found for item 6; therefore, this item was excluded. All 20 remaining items fitted the IIO criterion.

    In Study 2, items 13, 23, 18, and 3 were found to have item H<.30 (Table 5) and were therefore excluded. Table 5 shows the results of the unidimensionality analysis of the remaining 19 items. All item H s and the scale H were >.30 (H=. 445, SE=0.030). The results confirmed that the 19-item VCCQ scale could be considered unidimensional. No significant violations of monotonicity were found (ie, all crit values <.40) Violations of IIO were found for items 15, 17, and 2, which were therefore excluded, resulting in a final 16-item VCCQ.

    With regard to LI, in Study 1 items 11 and 14 and in Study 2 items 16, 20, and 21 were flagged as possibly violating LI – whereas these items were retained in the questionnaire in this study as LI is still under construction, these items may be considered for exclusion when developing a VCCQ short-form.

    Table 5. Loevinger’s scalability coefficients (H) for the 23-item and 21-item VCCQ (virtual care climate questionnaire) in Study 1 (N=230) and the 23-item and 19-item VCCQ in Study 2 (N=228).
    View this table

    Only 15 items remained in the VCCQ in both Study 1 and Study 2. For research as well as clinical purposes it is most desirable to keep the respondent burden of questionnaire completion as low as possible, and use the same items across interventions aimed at different behaviors. Therefore, we proceeded to the analyses of the questionnaire’s reliability and construct validity with a 15-item version of the VCCQ. The remaining 15 items maintained the breadth of the theoretical concept. That is, an item was only excluded when its content showed overlap with other items (six instances), it was excluded based on item response theory (IRT) results in both studies (one instance), or when the item was retrospectively deemed inappropriate in the context of Web-based, computer-tailored interventions (one instance). Remaining items are presented in bold font in Table 1. Item step response functions of the 15 remaining items are presented in Multimedia Appendix 4.

    Confirmatory Factor Analysis and Reliability

    The CFA with data from Study 1 showed that the CFI of the one-factor model was 0.897, the TLI was 0.879, and the RMSEA was 0.128 (95% CI 0.115-0.140). In Study 2, results were similar: CFI=0.902; TLI=0.885; RMSEA=0.087 (95% CI 0.074-0.100). Chi-square tests were significant for both models (290=426.6 and 244.8, P<.001). The scale showed good internal consistency both in Study 1 (Cronbach alpha=.97 and omega=.97, H=.66, mean 4.9 [SD 1.0]) and Study 2 (Cronbach alpha=.92 and omega=.92, H=.66, mean 4.9 [SD 1.0]). CFA diagrams are presented in Multimedia Appendix 5.

    Construct Validity

    Convergent Validity

    In Study 1, the correlation between the final 15-item VCCQ and the autonomous motivation subscale of the TSRQ was strong and positive (r=.66, P<.001). That is, the extent to which participants perceived Drinktest to be autonomy-supportive—as measured by the VCCQ—was significantly and positively correlated with participants’ autonomous motivation to reduce their alcohol intake. Furthermore, the correlation between the VCCQ and PCS for reducing alcohol intake was also strong and positive (r=.52, P<.001). The extent to which participants perceived Drinktest to be autonomy-supportive was significantly and positively correlated to participants’ perceived competence for reducing their alcohol intake. These findings support the hypothesis of positive relationships between perceived autonomy-support in a virtual care setting and both autonomous motivations and perceived competence for reducing alcohol intake, indicating the convergent validity of the VCCQ.

    In Study 2, the correlation between the final 15-item VCCQ and the autonomous motivation subscale of the TSRQ was moderately strong and positive (r=.37, P<.001), whereas the correlation between the VCCQ and the PCS for cannabis reduction was weak and not significant (r=.01, P=.12).

    Divergent Validity

    In line with expectations, results from Study 1 showed that the correlation between the VCCQ and the controlled motivation subscale of the TSRQ was moderate, though contrary to expectations this correlation was positive (r=.29, P<.001). The extent to which participants perceived Drinktest to be autonomy-supportive was significantly and positively correlated with participants’ controlled motivations to reduce their alcohol intake—though the correlation was much weaker compared with the correlation between the VCCQ and autonomous motivation. Consequently, some support was found for the hypothesized divergent validity of the VCCQ.

    In Study 2, similar results were found: the correlation between the VCCQ and the controlled motivation subscale of the TSRQ was moderately strong and positive (r=.37, P<.001)—this correlation had a similar strength as the correlation between the VCCQ and autonomous motivation. In line with expectations, the association between the VCCQ and the PCS for learning was weak and not significant (r=.05, P=.48), supporting the divergent validity of the VCCQ. Together, however, the results concerning the scale’s divergent validity in this study were mixed.

    Table 6 summarizes results for convergent and divergent validity in both studies.

    Table 6. Pearson correlations between the virtual care climate questionnaire, the treatment self-regulation questionnaire autonomous and controlled motivation subscales, and the different perceived competence scales (N of Study 1=230; N of Study 2=228).
    View this table

    Discussion

    Principal Findings

    The objective of this study was to develop and validate the VCCQ: the first instrument specifically intended to measure perceived support for autonomy in a virtual care setting. On the basis of on NIRT analyses, we found that several items needed to be excluded from the questionnaire: three items in Study 1 and seven items in Study 2. The final 15-item version included items that were adjusted from the original HCCQ [7] and PASSES [8] items, as well as new Internet-specific items based on expert consultation. This suggests that, as expected, neither the HCCQ nor the PASSES would have sufficiently covered the concept of perceived autonomy-support in a virtual care setting and provide support for the approach that was taken for item development, that is, to use both existing questionnaires and expert consultation as input.

    The final 15-item VCCQ was characterized by unidimensionality, monotonicity, and IIO, as required by NIRT. Therefore, the items can be used to compute a total score that reflects the respondents’ relative positions on the latent construct of “perceived autonomy support,” and these scores can also be used for comparing groups. Most items also met the local independence criterion, though before any items can be excluded based on this new psychometric test, further research is needed. Yet the items flagged, based on the criterion of local independence in our studies, can be first candidates for exclusion from a short-form VCCQ. Furthermore, the CFA of the 15-item VCCQ showed an acceptable model fit consistent with MSA results. Fit values were slightly below the recommended thresholds [33,34], though there is a scientific debate on the use of and value that should be attached to these thresholds when determining a scale’s model fit [36]. Reliability of the 15-item VCCQ was confirmed by two alternative tests (alpha and omega). Therefore, this version can be used confidently in future research, while continuing investigations into its measurement properties in different contexts and populations.

    All three negatively worded items (ie, 13, 18, and 23) were excluded from the questionnaire due to low inter-item correlations in both studies. While including such items in questionnaires is recommended to reduce acquiescence bias, it may lead to response inconsistencies and may prove difficult to specify in measurement models. Therefore, some experts advise against their use [37,38]. Excluding these items, together with other misfitting items, resulted in good measurement properties of the current VCCQ version without limiting the construct breadth—an item was only excluded when its content showed overlap with other items, it was excluded from the scale in both studies, or was retrospectively deemed inappropriate in the context of Web-based, computer-tailored interventions. Future psychometric work on the VCCQ could investigate the risk of acquiescence in reporting on perceived autonomy support and possible solutions.

    The scale’s convergent validity was confirmed by positive associations with autonomous motivation for reducing alcohol intake and cannabis consumption, and with perceived competence for reducing alcohol intake. Divergent validity was confirmed by the nonsignificant association with perceived competence for learning, yet the expected negative or nonsignificant relationship with controlled motivation could not be confirmed. In fact, in both studies we found a significant positive association between perceived support for autonomy and controlled motivation, although weaker than the association with autonomous motivation in Study 1. Moreover, in both our studies, autonomous motivation showed a strong significant positive correlation with controlled motivation (r=.68 in Study 1; r=.61 in Study 2), supporting our idea that autonomous and controlled motivation may represent different motivational dimensions that can coexist within people. This has also been suggested in earlier SDT-based research into motivational profiles, suggesting not only a high quality (ie, high autonomous and low controlled motivation), but also a high quantity motivational profile (ie, high autonomous and high controlled motivation) that does not necessarily lead to inferior results in terms of learning and physical activity than the high-quality profile [39]. In addition, earlier research has shown that introjected motivation, a form of controlled motivation that is characterized by performance of a behavior to avoid feelings of guilt or anxiety or to attain ego enhancement such as pride [3], may have positive effects on health behavior change [40]. Thus, an increase in especially this type of controlled motivation from using an Web-based intervention may also be a desired outcome. In our studies, perceived support for autonomy as measured by the VCCQ showed a stronger association with the introjected motivation subscale (r=.35 in Study 1; r=.38 in Study 2) than with the extrinsic motivation subscale (r=.21 in Study 1; r=.34 in Study 2). Whether this can be the direct result of providing support for autonomy can, however, not be concluded from the present study due to its cross-sectional nature. Following this line of reasoning, an alternative explanation may be that both autonomously motivated and respondents with high levels of controlled motivation were included in the present studies, with both groups finding potentially different elements in the intervention as supportive of their autonomy. Longitudinal research that includes measurement of the different types of motivation both before and after participation in a virtual care intervention is needed to examine whether perceiving higher autonomy support as measured by the VCCQ indeed increases autonomous motivation and reduces or increases the different forms of controlled motivation, and to determine whether autonomy-supportive interventions differ in their effectiveness for people with different types of motivation—in terms of both motivational and behavioral outcome measures. In addition, longitudinal research may be used to obtain evidence related to the questionnaire’s predictive validity.

    Strengths and Limitations

    An important strength of this study is that before proceeding to classical test theory analyses to determine the VCCQ’s validity and reliability, we investigated the questionnaire’s structure using NIRT analyses [26]. A known limitation of factor analytic methods is that they do not take into account differences in item difficulty, which are considered in IRT analyses, including MSA [40]. Moreover, as other parametric methods, they attempt to estimate quantitative differences between respondents, and are therefore unnecessarily restrictive for constructs that only refer to differences in degree [26]; perceived autonomy support is arguably among these constructs and more adequately investigated via MSA. So far, not many studies have used NIRT analyses in addition to classical test theory analyses and we are unaware of any previous studies that did so in the context of SDT-based questionnaires.

    The VCCQ was developed to measure perceived support for autonomy in a variety of virtual care settings, targeting different kinds of health behavior. Yet, in the present study it was tested for its reliability and validity solely in the context of Web-based interventions aimed at two addictive behaviors, namely alcohol consumption and cannabis use. To increase the generalizability of the results to interventions targeting other nonaddictive health behaviors such as physical activity and healthy dietary behaviors, future research could consider testing the validity and reliability of the VCCQ in the context of other Internet- based interventions. Moreover, as both studies described were cross-sectional, test-retest reliability and predictive validity of the VCCQ could not be ascertained. To investigate these properties, longitudinal research designs should be considered. An additional limitation pertains to the use of a scenario (ie, to ensure that participants made proper use of the Web-based interventions, they were instructed to imagine they wanted to change their behavior and that they used the Web-based intervention to this end). Whereas this may limit the ecological validity of the results presented, the aim of this specific study was to develop and validate a novel questionnaire, and not necessarily to actually help people to improve their health behavior. As a consequence, the exposure of people who truly wanted to change their behavior to interventions that were not necessarily developed to be autonomy-supportive was deemed undesirable. One of the uses of VCCQ is for guiding intervention development, and in such cases our choice of sample is actually recommended: people who are similar enough to the target group to be able to relate to the content, but not require the intervention as such (and thus be in a more vulnerable position). Due to the inclusion criteria set, the participants in the present studies can be considered very well-capable of imagining themselves wanting to change their behavior and of evaluating the interventions in terms of their autonomy-supportiveness. Therefore, the scenario used can be considered rather realistic. Another limitation that warrants attention concerns the use of an odd number of answering categories within the VCCQ, as also used in the HCCQ and PASSES, providing respondents with the possibility to equivocate when completing the questionnaire. In both studies, we prevented this from having a large influence on the psychometric analyses, by excluding respondents with a too high number of neutral answers (ie, respondents who had z-scores >3 for the number of neutral answers), though as is shown by the answering patterns presented in Multimedia Appendix 2 neutral answers were still rather common. The optimal number of response categories for questionnaires assessing support for autonomy in different contexts (virtual or face-to-face) is therefore a valuable research question for further studies. Finally, responding to the final 15 VCCQ-items within multimeasure surveys may still place a significant burden on respondents, potentially increasing nonresponse. Future research may therefore consider the development and validation of a VCCQ-short form, following examples from earlier research where the 15-item HCCQ was transformed into a valid and reliable six-item IOCQ (important other climate questionnaire) [41] and a six-item mHCCQ (modified HCCQ adapted for breast cancer patients) [42].

    Conclusions

    By developing the VCCQ, this study aimed to address the lack of validated tools that measure support for autonomy in the context of virtual care settings. Scientifically, the VCCQ enables further research into the role of autonomy-support in virtual care settings without virtual health care providers involved, a context in which evidence for its positive effects is currently lacking. Practically, using the VCCQ will help to identify how Web-based health behavior change interventions can most successfully support autonomy, subsequently increase autonomous forms of motivation, and ultimately promote a healthy lifestyle. Health behavior change interventions are increasingly provided via the Internet [9], and people tend to search for health-related information first and foremost through this medium [10]. While limited resources are available for the widespread implementation of effective lifestyle interventions and health care decision makers should select interventions based on their cost-effectiveness, Web-based health behavior change interventions are likely to be highly competitive when compared with other types of interventions [43,44]. Investigating the role of autonomy support in these settings is necessary, as it may lead to further improvements in the effectiveness of such interventions.

    The VCCQ appeared to accurately assess participants’ perceived autonomy-support offered by two Web-based health behavior change interventions. Overall, the scale showed the expected properties and relationships with relevant concepts and the studies presented suggest this first version of the VCCQ to be reasonably valid and reliable. As a result, the current version should be used cautiously in future research and practice to measure perceived support for autonomy within a virtual care climate. However, future research efforts are required that focus on further investigating the VCCQ’s divergent validity, on determining the VCCQ’s validity and reliability when used in the context of Web-based interventions aimed at improving nonaddictive or other health behaviors, and on developing and validating a VCCQ-short form. A Dutch version is available from the authors upon request.

    Acknowledgments

    We would like to thank all experts and members of the target group for their participation in pretesting the VCCQ. Furthermore, we would like to thank Andrea Perez Ortiz for her help in conducting the pretest and in the data collection and preliminary analyses for Study 1. This study was supported by the Amsterdam School of Communication Research (ASCoR) and the Innovational Research Incentives Scheme Veni from NWO-MaGW (Netherlands Organization for Scientific Research—Division for the Social Sciences) accredited to the first author (project number 451-15-028).

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Flow of respondents who started participation.

    PDF File (Adobe PDF File), 97KB

    Multimedia Appendix 2

    Answering patterns for all 23 VCCQ (virtual care climate questionnaire) items.

    PDF File (Adobe PDF File), 358KB

    Multimedia Appendix 3

    Inter-item correlations.

    PDF File (Adobe PDF File), 181KB

    Multimedia Appendix 4

    Item Step Response Functions.

    PDF File (Adobe PDF File), 135KB

    Multimedia Appendix 5

    Confirmatory factor analysis diagrams.

    PDF File (Adobe PDF File), 243KB

    References

    1. World Health Organization. The World Health Report 2002: Reducing Risks, Promoting Healthy Life. Geneva: World Health Organization; 2002.
    2. Scarborough P, Bhatnagar P, Wickramasinghe KK, Allender S, Foster C, Rayner M. The economic burden of ill health due to diet, physical inactivity, smoking, alcohol and obesity in the UK: an update to 2006-07 NHS costs. J Public Health (Oxf) 2011 Dec;33(4):527-535 [FREE Full text] [CrossRef] [Medline]
    3. Ryan RM, Deci EL. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol 2000 Jan;55(1):68-78. [Medline]
    4. Ng JY, Ntoumanis N, Thøgersen-Ntoumani C, Deci EL, Ryan RM, Duda JL, et al. Self-determination theory applied to health contexts: a meta-analysis. Perspect Psychol Sci 2012 Jul;7(4):325-340. [CrossRef] [Medline]
    5. Deci EL, Eghrari H, Patrick BC, Leone DR. Facilitating internalization: the self-determination theory perspective. J Pers 1994 Mar;62(1):119-142. [Medline]
    6. Williams GC, Cox EM, Kouides R, Deci EL. Presenting the facts about smoking to adolescents: effects of an autonomy-supportive style. Arch Pediatr Adolesc Med 1999 Sep;153(9):959-964. [Medline]
    7. Williams GC, Gagné M, Ryan RM, Deci EL. Facilitating autonomous motivation for smoking cessation. Health Psychol 2002 Jan;21(1):40-50. [Medline]
    8. Hagger MS, Chatzisarantis NL, Hein V, Pihu M, Soós I, Karsai I. The perceived autonomy support scale for exercise settings (PASSES): Development, validity, and cross-cultural invariance in young people. Psychology of Sport and Exercise 2007 Sep;8(5):632-653. [CrossRef]
    9. Griffiths F, Lindenmeyer A, Powell J, Lowe P, Thorogood M. Why are health care interventions delivered over the internet? a systematic review of the published literature. J Med Internet Res 2006;8(2):e10 [FREE Full text] [CrossRef] [Medline]
    10. Van de Belt TH, Engelen LJ, Berben SA, Teerenstra S, Samsom M, Schoonhoven L. Internet and social media for health-related information and communication in health care: preferences of the Dutch general population. J Med Internet Res 2013;15(10):e220 [FREE Full text] [CrossRef] [Medline]
    11. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010;12(1):e4 [FREE Full text] [CrossRef] [Medline]
    12. Williams GC, Niemiec CP, Elliot AJ, LaGuardia JG, Gorin AA, Rigby CS. Virtual Look AHEAD program: initial support for a partly virtualized intensive lifestyle intervention in type 2 diabetes. Diabetes Care 2014 Aug;37(8):e169-e170 [FREE Full text] [CrossRef] [Medline]
    13. Patrick H, Canevello A. Methodological overview of a self-determination theory-based computerized intervention to promote leisure-time physical activity. Psychol Sport Exerc 2011 Jan;12(1):13-19 [FREE Full text] [CrossRef] [Medline]
    14. Smit ES, de Vries H, Hoving C. Effectiveness of a web-based multiple tailored smoking cessation program: a randomized controlled trial among Dutch adult smokers. J Med Internet Res 2012;14(3):e82 [FREE Full text] [CrossRef] [Medline]
    15. Boon B, Risselada A, Huiberts A, Riper H, Smit F. Curbing alcohol use in male adults through computer generated personalized advice: randomized controlled trial. J Med Internet Res 2011;13(2):e43 [FREE Full text] [CrossRef] [Medline]
    16. Schulz DN, Kremers SP, Vandelanotte C, van Adrichem MJ, Schneider F, Candel MJ, et al. Effects of a web-based tailored multiple-lifestyle intervention for adults: a two-year randomized controlled trial comparing sequential and simultaneous delivery modes. J Med Internet Res 2014;16(1):e26 [FREE Full text] [CrossRef] [Medline]
    17. Drinktest [Drinktest]. Utrecht, the Netherlands: Trimbos Institute   URL: http://www.drinktest.nl [accessed 2016-06-15] [WebCite Cache]
    18. Wietcheck [Weed-check]. Utrecht, the Netherlands: Trimbos Institute   URL: http://www.wietcheck.nl/index.cfm?act=home.tonen [accessed 2016-06-15] [WebCite Cache]
    19. Panelclix. Amsterdam, The Netherlands: PanelClix/iFlavours BV   URL: http://www.panelclix.nl [accessed 2016-06-15] [WebCite Cache]
    20. Levesque CS, Williams GC, Elliot D, Pickering MA, Bodenhamer B, Finley PJ. Validating the theoretical structure of the treatment self-regulation questionnaire (TSRQ) across three different health behaviors. Health Educ Res 2007 Oct;22(5):691-702 [FREE Full text] [CrossRef] [Medline]
    21. Williams GC, Freedman ZR, Deci EL. Supporting autonomy to motivate patients with diabetes for glucose control. Diabetes Care 1998 Oct;21(10):1644-1651. [Medline]
    22. Williams GC, Deci EL. Internalization of biopsychosocial values by medical students: a test of self-determination theory. J Pers Soc Psychol 1996 Apr;70(4):767-779. [Medline]
    23. R Core Team. R-project. Vienna, Austria: R Foundation for Statistical Computing R: A language and environment for statistical computing   URL: https://www.r-project.org/ [accessed 2016-07-12] [WebCite Cache]
    24. Linacre JM. Investigating rating scale category utility. J Outcome Meas 1999;3(2):103-122. [Medline]
    25. van Schuur WH. Mokken scale analysis: between the Guttman scale and parametric item response theory. Polit Anal 2003;11(2):139-163. [CrossRef]
    26. Van der Ark LA. Mokken scale analysis in R. J Stat Soft 2007;20(11):11-19. [CrossRef]
    27. Sijtsma K, Meijer RR, van der Ark AL. Mokken scale analysis as time goes by: An update for scaling practitioners. Pers Individ Dif 2011 Jan;50(1):31-37. [CrossRef]
    28. Hemker BT, Sijtsma K, Molenaar IW. Selection of unidimensional scales from a multidimensional item bank in the polytomous Mokken I RT model. Appl Psychol Meas 1995 Dec 01;19(4):337-352. [CrossRef]
    29. Sijtsma K, Molenaar I. Introduction to nonparametric item response theory. Thousand Oaks, CA: SAGE Publications; 2002.
    30. Sijtsma K, Straat J, van der Ark LA. Goodness-of-Fit Methods for Nonparametric IRT Models. In: Quantitative Psychology Research, Springer Proceedings in Mathematics & Statistics 140. Switzerland: Springer International Publishing; 2015:109-120.
    31. Stochl J, Jones PB, Croudace TJ. Mokken scale analysis of mental health and well-being questionnaire item responses: a non-parametric IRT method in empirical research for applied health researchers. BMC Med Res Methodol 2012;12:74 [FREE Full text] [CrossRef] [Medline]
    32. Rosseel Y. Lavaan: an R package for structural equation modeling. J Stat Soft 2012;48(2):1-36. [CrossRef]
    33. Jackson DL, Gillaspy JA, Purc-Stephenson R. Reporting practices in confirmatory factor analysis: an overview and some recommendations. Psychol Methods 2009 Mar;14(1):6-23. [CrossRef] [Medline]
    34. Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Equ Modeling 2009;6(1):1-55. [CrossRef]
    35. Dunn TJ, Baguley T, Brunsden V. From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation. Br J Psychol 2014 Aug;105(3):399-412. [CrossRef] [Medline]
    36. Marsh H, Hau K, Wen Z. In search of golden rules: comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler's (1999) findings. Struct Equ Modeling 2004 Jul;11(3):320-341. [CrossRef]
    37. Swain SD, Weathers D, Niedrich RW. Assessing three sources of misresponse to reversed likert items. J Mark Res 2008 Feb;45(1):116-131. [CrossRef]
    38. Wong N, Rindfleisch A, Burroughs JE. Do reverse-worded items confound measures in cross-cultural consumer research? the case of the material values scale. J Consum Res 2003 Jun 01;30(1):72-91. [CrossRef]
    39. Vansteenkiste M, Sierens E, Soenens B, Luyckx K, Lens W. Motivational profiles from a self-determination perspective: the quality of motivation matters. J Educ Psychol 2009;101(3):671-688. [CrossRef]
    40. Teixeira PJ, Carraça EV, Markland D, Silva MN, Ryan RM. Exercise, physical activity, and self-determination theory: a systematic review. Int J Behav Nutr Phys Act 2012;9:78 [FREE Full text] [CrossRef] [Medline]
    41. Williams GC, Lynch MF, McGregor HA, Ryan RM, Sharp D, Deci EL. Validation of the “Important Other” climate questionnaire: assessing autonomy support for health-related change. Fam Syst Health 2006;24(2):179-194. [CrossRef]
    42. Shumway D, Griffith KA, Jagsi R, Gabram SG, Williams GC, Resnicow K. Psychometric properties of a brief measure of autonomy support in breast cancer patients. BMC Med Inform Decis Mak 2015;15:51 [FREE Full text] [CrossRef] [Medline]
    43. Smit ES, Evers SM, de Vries H, Hoving C. Cost-effectiveness and cost-utility of internet-based computer tailoring for smoking cessation. J Med Internet Res 2013;15(3):e57 [FREE Full text] [CrossRef] [Medline]
    44. Schulz DN, Smit ES, Stanczyk NE, Kremers SP, de Vries H, Evers SM. Economic evaluation of a web-based tailored lifestyle intervention for adults: findings regarding cost-effectiveness and cost-utility from a randomized controlled trial. J Med Internet Res 2014;16(3):e91 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    CFA: confirmatory factor analysis
    CFI: comparative fit index
    HCCQ: health care climate questionnaire
    IIO: invariant item ordering
    IOCQ: important other climate questionnaire
    IRT: item response theory
    ISRF: item step response function
    LI: local independence
    MSA: Mokken scale analysis
    NIRT: nonparametric item response theory
    PASSES: perceived autonomy support scale for exercise settings
    PCS: perceived competence scale
    RMSEA: root mean square error of approximation
    SDT: self-determination theory
    TLI: Tucker-Lewis index
    TSRQ: treatment self-regulation questionnaire
    VCCQ: virtual care climate questionnaire


    Edited by G Eysenbach; submitted 29.09.16; peer-reviewed by K Zhao, L Vera, J Hensel, M Amato, G Papandonatos; comments to author 29.12.16; revised version received 19.02.17; accepted 20.03.17; published 08.05.17

    ©Eline Suzanne Smit, Alexandra Lelia Dima, Stephanie Annette Maria Immerzeel, Bas van den Putte, Geoffrey Colin Williams. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 08.05.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.