One of the objectives of the study is to construct a measurement framework to assess the accessibility of consumer health information Web sites. As we discussed in the background section, two major specifications served as the normative guidelines for Web content accessibility design. The first-the W3C Web Content Accessibility Guideline 1.0 (WCAG)-is a stable international specification developed through a voluntary industry consensus. The US Access Board published the second specification-Electronic and Information Technology Accessibility Standards-in December 2000, pursuant to the US rulemaking process as required by Section 508 of the Rehabilitation Act Amendments of 1998 [31]. Both specifications offer checklists or rules that Web developers should follow with regard to content accessibility for people with disabilities. These two specifications largely overlap; only three of the checkpoints defined in Section 508 are not mentioned in the WCAG guideline 1.0. WCAG is more comprehensive than Section 508 on checkpoints of Web content accessibility, and it provides a priority level to each checkpoint to reflect severity of violations. Therefore, WCAG was used as the foundation for the accessibility metrics we developed.

The number of violations of each checkpoint is a component of our scoring method called the Web Accessibility Barrier (WAB) score. For example, a Web page with fewer accessibility checkpoint violations, e.g., providing an alternative description for an image object, would be considered to present fewer barriers for people with disabilities and will have a lower WAB score.

Because we are interested in automated evaluation of the degree of accessibility of a Web site, the subset of Web accessibility checkpoints demanding manual checking are not included in the calculation of the WAB score. For example, compliance to the rule "If you use color to convey information, make sure the information is also represented another way," cannot be verified until a manual check is done. For a list of Web accessibility rules that need to be manually checked, please see the WAI references [32].

WCAG attaches a three-point priority level to each checkpoint from its impact on Web accessibility. Priority 1 checkpoints mandate the largest level of compliance while Priority 3 checkpoints are optional for Web content developers. In weighting the calculation of the WAB score, we used the priority level in reverse order. The weighting factor for Priority 1 violations is 3, for Priority 2 violations is 2, and for Priority 3 violations is 1.

Using only the number of violations of Web accessibility checkpoints, however, may bias the results of the measurement. For example, a Web page with five "image without alternative text" violations may have 500 image objects embedded in the page and the Web page with one "image without alternative text" violation may have only one image object in the page. The developer of the first page may have already paid a great deal of attention to and put great effort into complying with the Web accessibility specifications while the developer of the second page may be completely unaware of accessibility. Therefore, the number of true violations of a checkpoint must be normalized against the number of potential violations of the checkpoint. In the last example, true violations are the image objects without alternative text, and the potential violations include all image objects on the page. Whenever a Web developer puts an image element into a Web page, he increases the potential that there could be a violation of the "alternative text" checkpoint. Table 1 explains the selection of potential violations from HTML code. The average WAB score of all Web pages within a Web site will be the WAB score of the Web site.

Figure 1 summarizes the calculation of the WAB score of a Web site as a formula. A higher score means there are more accessibility barriers on the site, while a lower score indicates fewer barriers. A score of zero denotes that the Web site does not violate any Web accessibility guidelines and should have no automatically detectable accessibility barriers to people with disabilities.

We employed several program tools to examine the true and potential violations of the Web pages. Bobby is a checking program that can examine a Web page and report violations of Web accessibility checkpoints [33]. It is the most widely used accessibility checking software package and has been around longest. Bobby was originally developed by the Center for Applied Special Technology [34], and is now maintained and distributed by Watchfire Corporation [35].

Bobby desktop version 4.0.1 was used in this study. The desktop version can check compliance with WCAG of an entire Web site or only certain layers from the main page of the Web site. The version 4.0.1 can check non-compliance issues with both WAI and Section 508 checkpoints. After checking a Web site, Bobby generates a report in eXtensible Markup Language (XML) format that can be further processed to extract data about true violations.

Bobby implements 91 distinct testing rules, each of which maps onto a specific WCAG checkpoint. The Bobby tests are classified into a number of different "checking" categories, as follows: (1) Full: Bobby automatically checks this rule and decides whether there is an error. (2) Partial: Bobby automatically performs some checking of the rule, but cannot decide the existence of violations. Instead, the line number is used as a warning to the testers. (3) Partial Once: Similar to the Partial category, but the warning is not specific to an individual line. (4) Ask Once: Bobby does not have a mechanism to check the rule, so the rule is presented as a reminder to the testers.

For all categories other than Full, a human tester must manually evaluate the Web site further to determine the WCAG compliance, which is not viable for a large scale Web site study like this one. We used only the 25 rules that Bobby implements with Full checking capacity for our evaluation. Even for the rules with "Full" checking capacity, we still could not determine the quality of the compliance with WCAG. For example, the Web page developer could simply put the file name of the image into the "alt" attribute of the <IMG> element to avoid a flag from Bobby. The quality of such compliance is much less acceptable than providing detailed description in the "ALT" attribute.

The data of corresponding potential violations for each checkpoint can be extracted using a Web crawler program, which is an automated program that follows hyperlinks to visit Web pages. We developed a lightweight Java-based Web crawler program to access Web pages at remote Web sites and determine the number of potential violations of Web accessibility checkpoints. We did not use the built-in Web crawler in Bobby because it cannot be customized to check potential violations of checkpoints in a Web page. We also made use of the "homemade" crawler as the basis for future development of tools for Web accessibility evaluation. For a list of rules for extracting data of potential violations, please see Table 1. Since the crawler embedded in Bobby and the "homemade" Web crawler may retrieve an unmatched number of pages for the different capacities of both crawlers, we only used the Web pages retrieved by both programs in the study.

All statistical analyses were performed with alpha value at 0.05 and power at 0.80. Descriptive statistics (means and standard deviation) were calculated for each variable considered in the study. Univariate statistics of the WAB scores were calculated at the level of each category. Then a one-way ANalysis Of VAriance (ANOVA) test was applied to the WAB scores at the level of the Web site's functional category. If the ANOVA test indicated a large difference in the WAB scores among different categories, the post hoc Bonferroni test of the WAB scores between different categories was conducted. The alpha level was adjusted for multiple comparisons in the Bonferroni test.

Google ranked Web sites with a sub-category from highest to lowest PageRank value. Therefore, we used the ranking sequence as the value of Web page importance for nonparametric Spearman correlation. Nonparametric Spearman correlation statistics were also conducted to measure the level of correlation between the WAB scores and the popularity of the Web sites. All statistical analyses were conducted using the SPSS 11.0 software package.

The Google subdirectory "Health/Consumer/Resources" lists 122 Web sites, 14 of which were excluded because their content are no longer healthcare related or they were not active during the study period. The assessing program retrieved 7,109 Web pages from the remaining 108 sites. Means and standard deviations of WAB scores for the remaining 108 Web sites were calculated. The average WAB score was 9.31 with standard deviation of 6.29. None of the 108 Web sites was absolutely accessible (WAB score = 0). The National Institutes of Health (NIH) Combined Health Information Database (CHID) Web site (http://chid.nih.gov/) achieved the lowest WAB score, i.e., it had the fewest accessibility barriers, of the sites tested (0.97), while a community Web site (http://www.discussyourhealth.com/) received the highest WAB score (24.99). The five most frequently violated WCAG checkpoints of all webpages were: "identify language of the text" (77.0%), "use a public text identifier in a DOCTYPE statement" (65.6%), "provide a summary for tables" (61.6%), "use relative sizing and positioning (% values) rather than absolute (pixels)" (60.0%), and "provide alternative text for all images" (52.2%).

Among the six functional categories of Web sites, government Web sites were most accessible and had the lowest WAB scores, and portal Web sites were least accessible to people with disabilities, indicated by higher WAB scores (Table 3).

The average scores of Web accessibility were calculated for each of the Web categories and the results indicate possible clustering among the six categories, as shown in Figure 2.

Statistically significant differences among the category groups were found using the ANOVA test on the WAB scores (F = 9.705, P < 0.001). In addition, the post hoc Bonferroni test found that the mean WAB scores of governmental and educational Web sites were significantly different from the rest of the categories (P < 0.001). There is no statistically significant difference between any two categories within each of the two clusters.

Furthermore, the Spearman correlation test indicates a statistically significant, though modest, correlation between the WAB score and the Alexa traffic ranking (r= 0.28, P < 0.01). No statistically significant correlation between the WAB score and the PageRank of Web sites was found (r= 0.15, P = 0.111) using the Spearman correlation test (Table 4). The correlation between the Alexa's traffic ranking and Google's PageRank was statistically significant (r= 0.32, P < 0.01) using the Spearman correlation test.

No consumer health information Web sites satisfied all of the Web accessibility requirements, which may be attributed to Web site developers knowing little about accessibility standards, the lack of effective and efficient evaluation and repair tools, and the pressure to update information on the Web site quickly. Web accessibility, if ever considered, is often an afterthought once Web content design is finished. This implies that program tools that produce efficient, effective post-hoc repairs of Web content accessibility violations, or an accessible proxy server that transforms and filters inaccessible online content for people with disabilities may be more accepted by both the developers and Web site visitors.

Of the sites providing health information, government sites followed by education sites are the most accessible. This compliance may be attributed to Section 508, since it is mandatory for all federal agencies [38]. High compliance among sites that fall under this mandate also indicates that legal activities would facilitate the removal of accessibility barriers for people with disabilities.

None of the tested Web sites, including the most accessible government sites, passed the WCAG guideline priority 1 checkpoints, even though the five most frequently violated checkpoints have technically uncomplicated solutions if designers pay attention to them. This may imply that the Web site editor simply overlooked the errors and, for such editors, an automatic Web site monitoring program could be very helpful in identifying and correcting these errors. Other possible reasons for such imperfection are the lack of integrated accessibility tools or functions within Web site editing software. Most Web site editing tools make it optional to strictly follow accessibility rules.

The education Web sites are the second most accessible category. Section 508 is not strictly mandatory for the information technology available on educational Web sites, but high awareness of WCAG rules and legal requirements on most campuses may contribute to better accessibility among the education Web sites. Furthermore, although Section 508 does not mandate all education Web sites, it does apply to educational programs and projects that receive federal funding, as many do, which may explain the high compliance to WCAG rules among education sites.

The accessibility of a Web site also correlates with its popularity, possibly implying that people with disabilities are more likely to visit sites that contain fewer or no barriers to them. A more accessible Web site may be more usable for the general population because it can also improve the efficiency, effectiveness, and ease of using the Web site [39]. Meanwhile, accessible Web pages will have better opportunities for indexing by Web search engines, which use programs called crawlers to access Web pages on the Internet and store Web page indexes in a database for fast Web information retrieval. Web crawlers work similarly to Web users who are blind and using screen reader programs. Therefore accessible Web pages will have more chances to be indexed by a Web crawler [40]. Subsequently the overall popularity of the Web sites increase since they attract a group of visitors who have difficulties accessing other sites containing more Web accessibility barriers. Other reasons for the correlation between accessibility and popularity include the possibility that people may take notice that a Web site is accessible and tend to visit it often, or Web developers of accessible Web sites spend more time ensuring their Web sites are appropriate in following other usability rules that make visiting easier for the public.

The correlation between Web accessibility and a Web site's importance was not statistically significant in our study, although the correlation between its importance and popularity was statistically significant. The measurement of the importance of a Web site was derived from comprehensive link analysis on the Web. It revealed the value of the Web site by measuring how many and what kind of other Web sites link into it. It does not necessarily reflect the value of other HTML elements, especially those Web accessibility related elements. A Web site can be very important in terms of PageRank because many other Web sites have links to it, even though it is not accessible to persons with disabilities when they directly visit it.

Please note that there are several limitations to this study. First, although this study attempts to comprehensively assess the accessibility of a Web site, it is not practical for some Web sites, especially those with large numbers of archived documents. The Bobby program often freezes when checking all layers of a Web site, and this resulted in the decision to check only a manageable two layers of Web pages in this study. A more robust tool needs to be adopted or developed for future studies.

Second, only the checkpoints of Web accessibility that can be examined automatically by a computer program were studied. Many other checkpoints require a manual check of pages to ensure the compliance of the content with the guidelines of Web accessibility. WAI proposed a comprehensive framework for evaluating Web content accessibility which requires multiple steps involving several evaluation tools to ensure the accuracy of the evaluation results. Although this type of evaluation is important for quality assurance of individual Web sites, the cost of such a large operation makes it impractical for an evaluation study involving many Web sites. This study assumes that the checkpoints that can be automatically evaluated will strongly correlate to the manual checkpoints and can be used as a surrogate assessment for accessibility of a Web site. Future studies might explore the agreement between these two groups of checkpoints.

Furthermore, the traffic ranking information provided from Alexa may skew towards users of Internet Explorer on a Windows operating system, underestimating the traffic to sites that are disproportionately accessed by people using other browsers or operating systems. The site most likely to suffer from this bias is AOL (America Online), since their members commonly use AOL browsers to access the site.

The WAB score in the study can be used to measure the degree of accessibility of a site. However, it should not be used as the only indicator for Web accessibility, which includes other checkpoints that can not be automatically assessed by computer programs. An experienced Web developer can fine-tune a Web site to produce a perfect WAB score. However, this does not necessarily mean that the Web site is entirely accessible to people with disabilities when they visit it.