All adult female patients with an appointment were eligible to participate if they were at least 18 years of age, could speak and read English, and could provide informed consent. The study participants were recruited in 15 days over a period of three consecutive weeks in February of 2004. On recruitment days, all adult female patients with appointments were given a brief letter of invitation by the clinic receptionist at the time of arrival. These potential participants were then approached in the waiting area by a recruiter to confirm their eligibility and inquire about their interest in the study. Willing participants were taken to a separate room in the clinic, unaccompanied by family or friends, where they completed the survey after giving informed consent. At this time (T1), participants were also asked to consent to a subsequent contact after 2 weeks (T2) to administer the CLAS a second time. Participants sealed the survey in the provided envelope before returning it to the recruiter. Then, participants received health brochures (domestic violence, cancer, and heart health) with telephone numbers for domestic violence counsellors and the assaulted women’s helpline.

The survey included the CLAS, which is a 14-item scale that covers patients’ positive and negative perceptions about computer-based health-risk assessments [19]. Participants read a vignette about such a “computer survey” (Textbox 1) and rated each CLAS statement on a Likert-type scale of 1 to 5 (strongly agree, agree, not sure, disagree, strongly disagree). Other sections in the survey gathered information on sociodemographics (age, marital status, country of birth, years lived in Canada, highest education completed, employment status, and income), health (self-rated health, partner violence, and number of visits to family practice), exposure to computers (access and use), comfort in completing the survey, and English language abilities (see Table 1 for details).

We would like to know your opinion about a computer survey of patients. This survey is completed by patients on a computer before seeing their family doctor. The computer survey asks questions about lifestyle and health risks such as smoking, stress, conflict in relationships, and safety. The questions appear on the computer screen one by one. The patient answers by touching one of the options on the computer screen using a non-ink pen. Patients do not type or use any computer parts but only touch the screen to give answers. This computer survey uses simple day-to-day language of 5th grade reading level. The computer system prints (1) a summary of patient health risks for the doctor to review, and (2) an information sheet for the patients about their reported health risks. What is your opinion about such computer-based health-risk assessment of patients before seeing a family doctor?

The aim was to recruit a sample of 200 participants. As CLAS included 14 items, a sample of 200 was expected to generate an adequate subject-to-variable ratio of 14:1 to derive latent constructs. For factor analytical approaches, Gorsuch (1983) and others recommend a subject-to-variable ratio of five when the communalities are high and there are many variables for each factor [28,29]. If these conditions are not met, then a subject-to-variable ratio of 10 is recommended [28]. Others suggest that a sample of 150 should be considered sufficient when the factor analyzed solutions have several high loading markers (> 0.80) [30]. Our sample size is fair in meeting both of the established requirements (minimum sample size and sample size per item) for psychometric studies.

The CLAS items [19] were reverse coded prior to analyses so that 1 referred to “strongly disagree” and 5 to “strongly agree.” All analyses were conducted using Statistical Package for Social Sciences (SPSS) version 12(SPSS Inc, Chicago, IL, USA). Preliminary statistical procedures examined distributions of the individual items (eg, means, standard deviations, skewness, and kurtosis) and evidence of ceiling or floor effects. The quality of data was evaluated by percentage of missing responses, which were low; hence, we used the case deletion in subsequent analyses. The sampling adequacy was assessed by the Kaiser-Meyer-Olkin test.

Prior to reliability and validity analysis, we examined the latent structure of the scale. The latent constructs of the CLAS were examined by employing principal component analysis (PCA) [31,32]. The number of latent constructs or factors was determined using scree plots and the criterion of eigenvalues greater than 1.0. We considered three-factor, four-factor, and five-factor solutions with varimax rotation. Salient loadings were defined using a critical value of 0.38 [28].

The scale reliability was estimated by both internal consistency and test-retest reliability of the subscales. To examine homogeneity of items or internal consistency, item-total correlation [33] and change in Cronbach alpha coefficient upon item deletion were used [34]. For test-retest reliability, intraclass correlation (ICC) from a two-way random effects model was executed [35]. We also compared T2 participants to the remaining participants at T1 with respect to characteristics measured at T1 to assess the potential to generalize the reliability findings.

After factors were derived and reliability established, construct validitywas investigated. For this analysis, we tested hypotheses that were based on existing literature. Further details on the hypotheses are presented in the Results section under construct validity. The hypotheses were tested by using Pearson product moment (r _p), point biserial (r _pb), or Spearman rank (r _s) correlation analyses, as appropriate.

Among 361 women approached, 212 eligible women received the study details in privacy, 207 provided written consent (response rate 97.6%), and 202 returned the completed surveys. Participants had a mean age of 45.3 years (range 19 to 86) and 36% were immigrants, with the top two groups from Europe and Asia (Table 1). Almost 75% of the participants were in a current intimate relationship, and 77% had at least university education. Nearly 64% were currently employed and reported annual household income of at least Can $40,000; 87% of the participants had access to computers, and 66% used one every day. Self-perceived health was rated as “good” on a scale of one to five with a mean of 3.2 (SD 1). The mean number of visits to the family practice during the last year was 4.6 (median 3.5; mode 1; range 0 to 30).

The item means and standard deviations were acceptable, while three items were skewed (Table 2). These items were transformed and PCA was executed with and without transformations. As the two PCAs were similar in factor structure and factor loadings, we report PCA without transformed items in this paper. Sampling adequacy was indicated by a Kaiser-Meyer-Olkin test value of .82 and the absence of ceiling or floor effects. The factorability was indicated by correlation and partial correlation matrices.

On conducting the PCA, the first 10 eigenvalues were 4.7, 2.1, 1.1, 1.0, .85, .76, .69, .53, .50, and .44. Four factors emerged with eigenvalues greater than or equal to one, accounting for 63.7% of the total variance. Based on the scree plot, either a three-factor or four-factor solution was indicated. We considered three-, four-, and five-factor solutions, and the four-factor solution yielded the most interpretable results. A summary of the PCA with varimax rotation is presented in Table 3. The factors were named Benefits, Privacy-Barrier, Interaction-Barrier, and Interest. Three- and four-factor solutions were also compared for the internal consistency of the derived subscales. Although the Privacy-Barrier and Interaction-Barrier factors merged into one factor upon forcing a three-factor solution, the internal consistency of the subscales was higher in the four-factor solution than in the three-factor solution. This internal consistency comparison was based on the reliability coefficients adjusted for the length of the subscales [33].

Variances accounted for by the four identified factors (Benefits, Privacy-Barrier, Interaction-Barrier, and Interest) after the rotation were 33.6%, 15.0%, 8.0%, and 7.2%, respectively. The item “Computers can be trusted” in the first factor (Benefits) shared loading (.41) with the second factor (Privacy-Barrier) above the critical value of .38. Also, the item “Too many mistakes will be made with computer” in the second factor (Privacy-Barrier) shared loading (.39) with the third factor (Interaction-Barrier) above the critical value.

The Benefits factor consisted of six items with factor loadings ranging from .79 to .54. The items loading on this factor cover perceived benefits toward the quality of medical consultation and means of achieving the benefits. The Privacy-Barrier factor consisted of three items dealing with patient concerns about privacy, with loadings ranging from .82 to .60. The Interaction-Barrier factor consisted of three items covering patient concerns about interference in the interaction with the physician, with loadings ranging from .81 to .63. Although the Interest factor consisted of only two items, both items had strong factor weightings (ie, .80 and .79). The stability of this factor was also apparent during execution of the five-factor solution. Both items of this factor continued to load together while the fifth factor consisted of one item pulled from the Interaction-Barrier factor.

To estimate internal consistency reliability, we considered the following criteria for each subscale: (1) an item-total correlation of at least .3 for all items, (2) no increase in the Cronbach alpha coefficient if an item was deleted, and (3) general acceptability of the item means and standard deviations. All three criteria were met for the subscales (Table 4).

The Cronbach alpha coefficients for the four subscales Benefits, Privacy-Barrier, Interaction-Barrier, and Interest were .85, .70, .67, and .50, respectively. There was no increase in Cronbach alpha if items were deleted from the first three subscales. This analysis did not apply to the Interest subscale as it had two items only. The item-total correlation for the subscales Benefits, Privacy-Barrier, and Interaction-Barrier ranged from .77 to .57, .53 to .46, and .52 to .44, respectively. We also calculated the reliability coefficients adjusted for the length of subscale [33], given that the number of items loading on the subscales varied and that Cronbach alpha is sensitive to number of items. The adjusted reliability coefficients were .81, .80, and .75 for the Privacy-Barrier, Interaction-Barrier, and Interest subscales, respectively, where adjustment was made to assume six items as for the Benefits subscale. This analysis assumes that the new items would be similar to the old items with respect to content and reliability.

Scale reliability over time was assessed with the test-retest data (n = 52). At T2, 52 patients were successfully reached out of 145 T1 participants who consented to the second contact. The reduced participation at T2 was due to (1) the study requirement that the second administration of the CLAS occur within 2 weeks of the first administration, and (2) the fact that many patients were difficult to reach because they had provided telephone numbers at work. The T2 participants were similar to the other T1 participants (n = 150) on sociodemographic characteristics, including age, country of birth, number of years lived in Canada, education level, employment status, income, English language abilities, access to computers, computer use in the last month, relationship status, experiences of intimate partner violence, number of visits to family practice, and perceived health. However, the T2 participants were less likely to be employed than participants who consented but could not be reached for second contact (χ ² ₂= 7.0, P < .05). The time between T1 and T2 contacts averaged 16 days (SD 2.6, median 15, mode 15). The ICC analysis based on a two-way random effect model gave coefficients of .76 for the overall scale and .91, .82, .86, and .67 for the subscales of Benefits, Privacy-Barrier, Interaction-Barrier, and Interest, respectively. As the CLAS is a multidimensional scale, the test-retest reliability of the subscales was higher than the test-retest reliability of the overall scale.

To evaluate validity of the derived constructs, several hypotheses were formulated based on a literature review. We hypothesized that the Benefits factor would be positively associated with participants’ frequent use of computers as greater familiarity with computers is likely to increase peoples’ comfort and perceptions of the benefits [2]. Also, we hypothesized that patients with poorer health would perceive the benefits of computer-based screening as high due to the limited time available for lifestyle inquiries during their routine health care visits. As computer-based screening has been found specifically beneficial for socially sensitive issues [2-4,36-39], it was hypothesized that participants reporting victimization by intimate partners would perceive the benefits as high. Existing studies report that patients are likely to perceive barriers in using preventive health services if they have low socioeconomic status or are immigrants [40,41]. Accordingly, it was hypothesized that the Privacy-Barrier and Interaction-Barrier factors would be positively associated with participants’ non-Canadian-born status, low household income, unemployment, and lesser years of education. We also hypothesized that the Interest factor would be significantly associated with less use of computers and older age.

The hypotheses were tested by correlation analyses. The Benefits factor was positively associated with poorer self-perceived health and intimate partner victimization (r _p = .15, P = .03; r _pb= .19, P = .02) as hypothesized. However, it was not significantly associated with the use of computers, in contrast to our hypothesis. To explore further, we examined the mean scores of the Benefits subscale by participants’ frequency of computer use in the last month. Participants who used computers every day or two to three times a week somewhat agreed with the Benefits (mean 3.7, SD 0.67), while participants who used computers once a week or once a month (mean 3.5, SD 0.50) or not at all (mean 3.6, SD 0.67) seemed to neither agree nor disagree with the Benefits.

As hypothesized, the Privacy-Barrier and Interaction-Barrier factors had positive significant associations with participants’ non-Canadian-born status (r _pb= .19, P = .006; r _pb= .22, P = .001), low household income (r _p= .23, P = .002; r _p = .21, P = .004), and lower use of computers (r _s = .16, P = .03; r _s = .18, P = .01). Furthermore, older age at the time of immigration had a positive association with both the Privacy-Barrier and Interaction-Barrier factors (r _p = .27, P = .02; r _p = .28, P = .02). The Interaction-Barrier factor also had significant associations with participants’ unemployment status and lesser years of education (r _pb = .16, P = .03; r _s = .18, P < .01). The Interest factor had significant positive associations with older age (r _p = .16, P = .03) and less use of computers (r _s = .14, P = .04).

The use of a psychometrically validated scale is an essential element in facilitating clinical and policy decisions about the application of computer-based health-risk assessments. This is of particular importance for sensitive health risks and conditions where superiority of computer-based risk assessments over personal interviews is already well documented with respect to patient disclosure of socially sensitive information. These health risks and conditions include behaviors related to sex, alcohol, drugs, HIV, and violence [2-4,36-39]. A similar link is demonstrated in our study as a positive association between women’s victimization at the hands of their intimate partner and the Benefits subscale. Literature shows that women experiencing partner abuse seldom spontaneously disclose it to health care providers [43,44], who frequently fail to detect victims of abuse due to time pressure, priority of acute problems, and discomfort [45,46]. At the same time, clinicians’ questioning about abuse is the most significant predictor of women’s disclosure [47]. Computer-based screening matches abused women’s preferences for “direct questioning,” and it has limited dependency on physician time. Above all, it is a nonjudgmental and anonymous way of asking about socially sensitive health risks. Perhaps it explains why abused women in our study perceived higher benefits of the computer-based screening. Our future work will test the computer-based screening intervention in a family practice setting for the detection and disclosure of partner abuse.

The findings also highlight the complex nature of human behavior. Study participants perceived barriers in two distinct ways: barriers regarding privacy and barriers regarding interaction with physicians. At the implementation level, this underscores the need to measure both domains to understand and thereafter address effective use of computer-based health-risk assessments. At the theoretical level, this distinction is novel to the original conception of the scale. Possibly, patient attitudes have taken specific forms with the increasing use of computers. Recent studies reveal that use of the Internet for health information influences the way people relate to physicians, make medical decisions, and access health services [48,49]. In 2007, a telephone survey with 2479 Canadians examined their attitudes toward electronic health information and their privacy [50]. The survey found that 9 out of 10 people perceived the use of electronic health information as integral to the provision of high-quality care but had mixed confidence about the protection of health information. Future research should further examine the domains of privacy and interaction barriers in the use of computer-based health-risk assessments. Other studies report that patients’ perceptions toward computer-based lifestyle assessments are positively increased after they are provided the actual experience [2,19].

Our post hoc analyses indicate that study participants who were immigrants or had lower socioeconomic status perceived more barriers. This raises two critical questions: (1) Is this an extension of the “digital divide?”, and (2) What does it mean for implementation? The term “digital divide” stems from research and refers to “decreased access to information technologies, particularly the Internet, for racial and ethnic minorities, person with disabilities, rural populations, and those with low socioeconomic status” [51] (p 449). The digital divide requires vigilance when using certain health information technologies, such as the Internet [52,53]. In contrast, computer-based health-risk assessments in health care settings bridge the digital divide because these programs provide tailored health information to the patients at the point of care. They may play a positive role in addressing patients’ unequal access to health information and care—an anticipated impact similar to telemedicine [54].

Several limitations of this study should be noted. The CLAS predominantly measures the decision-making aspect of human behavior, though it has relevance for research on explaining and changing behavior regarding computerized assessments. Future studies should explore other aspects such as patient self-efficacy and cue-to-action. The construct of Interest would also benefit from further conceptual development. Further, our analysis of the construct validity is post hoc in nature. Many of the correlations were not strong even when significant. This is possibly due to our convenience-based use of a larger survey to select variables which in turn had a more distal than proximal relationship with the CLAS constructs. Although we found support for most of our hypothesized relationships, the Benefits subscale was not associated with the participants’ use of computers, contrary to our hypothesis. The study sample was relatively more educated than the average general population, and 87% of the participants had access to computers at home or work; almost a similar proportion reported using the computer every day or at least two to three times a week. Perhaps frequent use of computers makes people think critically about their advantages and disadvantages, leading to a cautious assessment of their benefits. On other side, it is also possible that computers have now become part of our everyday life and their benefits are taken for granted, reducing the level of perceived benefits seen a few years ago. Future research with larger samples should examine this further and establish the construct validity with a priori selection of variables. Also, it will be important to conduct a classic multitrait-multimethod study in which the four constructs on the CLAS are assessed via different methods (eg, peer ratings, behavior observations). This type of study will provide evidence for both convergent and discriminate aspects of the CLAS construct validity.

Caution is warranted regarding the generalizability of our study findings. We evaluated psychometric properties of the CLAS with female patients only. A future study involving both men and women is needed to ensure its applicability to all patients visiting primary health care settings. Further, patients were recruited from a single site. However, the collaborating clinic had several physicians and served a large number of diverse patients with estimated annual visits of 50,000. The study obtained a high response rate and, reassuringly, the participants were similar to females residing in Toronto in terms of immigration and marital status [55,56]. At the same time, study participants had relatively higher levels of income and education than the general population. The test-retest results of our study may have limited generalizability as participants in the second administration of the CLAS were more likely to be unemployed than the rest of participants. Nevertheless, the two groups were similar for all other sociodemographic and health-related variables that were measured. Further research is needed with a heterogeneous sample as an important next step to advance the generalizability of the scale.

This study of patients in a family practice setting advances our understanding of the properties, applicability, and generalizability of the CLAS. This is an important improvement over previous assessments of other scales that relied on samples of convenience or were not specific to patient populations. Furthermore, the phrasing of items in the CLAS is expected to allow people from different ethnocultural backgrounds to reply in a meaningful way, unlike some other existing scales. At the same time, future research with a heterogeneous sample is needed to enhance its generalizability by gender and socioeconomic status while examining the utility for low and high users of computers. In conclusion, this study is a step toward facilitating research and interventions for promoting patient acceptance of computer interactive technology.