Consumers use general-purpose search engines rather than medical sites or portals to find information, and most do not go beyond the first page of search results [13]. Therefore, we chose the ten most popular search engines (Table 1) to select Web sites that consumers are likely to encounter [14]. The three most popular herbs in the United States (in terms of dollars spent) [15], ginseng, ginkgo, and St. Johns wort, and their most common uses formed the search query. The following three queries were executed in each search engine on July 15, 2003: "ginseng and cancer," "ginkgo and memory loss," and "St. John's wort and depression." All Web sites listed on the first results page, including sponsored or paid links, were analyzed.

A Web site was included if it contained at least one sentence or phrase of health information on the search topic. Health information was defined as "information intended to be used to maintain or improve health, including to understand disease processes, health care issues, etc… to prevent, diagnose, or treat health problems, to be rehabilitated from the effect of diseases, or treatments, and to seek and select health care plans, providers, and other resources." [16] Duplicate URLs were removed. HTTrack [17], a Web site copier was used to permanently capture each Web site and every directly linked page.

In prior work, we assessed inter-rater agreement for popular technical quality criteria [1]. We assessed the degree to which two raters agreed upon the presence or absence of 22 quality criteria selected from Eysenbach's systematic review [17] of a sample of 21 CAM Web sites. Our preliminary analysis showed poor inter-rater agreement on 10 of the 22 criteria. Therefore, we created operational definitions for each of the criteria, decreased the allowed choices, and defined a location to look for the information. As a result, 15 out of the 22 quality criteria had acceptable inter-rater agreement (kappa > 0.6).

For this study, one evaluator (MW) analyzed all Web sites for compliance with 15 technical quality criteria (Table 2) that we previously determined to be reliably assessable. Therefore, in this study we did not re-calculate inter-observer reliability for these technical criteria.

First, a set of critical facts for each of the three herbs was determined by consensus of two clinically trained reviewers (SS, DS); please see appendices 1-3. The sets of critical facts were extracted from two independent sources of CAM information: the Physician Desk Reference (PDR) for Herbal Medicines [19] and the Sloan Kettering database of herbs [20]. After the sets of critical facts were determined, the CAM content displayed on each Web site was independently evaluated by both reviewers. Cases where reviewers disagreed were resolved by consensus. In order to minimize bias, materials identifying each Web site's origin, such as organization name, logo, footers, URLs and hyperlinks were removed. However, no changes were made to the design or layout.

In order to verify the concordance between reviewers, two additional clinically trained evaluators (validation reviewers), who were not aware of the study hypothesis or quality criteria tested, were given 30 randomly selected sites from the same sample looked at by primary reviewers (SS, DS). Inter-rater agreement between the validation reviewers was calculated. The validation reviewers were given the same critical facts documents as the primary reviewers and each validation reviewer assessed every site independently. After each reviewer independently evaluated the Web sites, inter-rater agreement was calculated between the two validation reviewers. Subsequently, cases of disagreement were resolved by consensus. A second inter-observer agreement measure was calculated between the pairs of reviewers (primary reviewers vs. validation reviewers) based on the consensus data.

Content on each page was scrutinized for the presence of misleading statements likely to cause physical harm (acts of commission) and for vital information that was missing (acts of omission). Commission may be thought of as a surrogate for accuracy, while omission has been referred to as completeness, coverage, or comprehensiveness [21]. We based our evaluation on the following framework, adapted from Markman [22]:

Statements that suggest use of higher doses of herbs than recommended in the critical facts documents (appendices 1-3) were categorized as causing "direct toxicity." Statements suggesting that the herb protects against disease and encouraging patients to self-medicate instead of seeing a physician were placed in the "delay in diagnosis or conventional treatment" category. Statements that project herbs as an "alternative to conventional treatment" (for example, "the herb is the first choice of treatment for the disease") were categorized as potentially causing "avoidance of conventional treatment." Statements that suggested using herbs with medications known to have drug interactions (for example, using St. John's wort with monoamine oxidase inhibitors) were classified as causing potential harm due to "interaction with conventional therapy." However, while evaluating potential physical harm due to omission of information about interactions, we did not expect Web sites to list all the drug interactions listed in the critical facts documents. Web sites that noted at least one drug interaction were considered not to omit drug interaction information. Web sites with vague statements such as "there are many interactions," were categorized as having "omitted drug interactions." Potential physical harm was present if any error of commission or omission was found.

We recognize that in addition to physical harm due to either commission or omission, CAM information on the Internet may cause other types of harm, such as emotional and financial. Emotional harm may occur because of inaccurate perception of disease or conventional therapy such as exaggeration of side effects of conventional treatment and presentation of alternative treatment as a "natural cure." Financial harm may be caused by the purchase of ineffective or harmful yet expensive CAM products. However, we did not evaluate emotional and financial harm in this study because of the inherent subjectivity involved, and difficulty in quantifying and assessing such measures.

The dichotomous (yes/no) dependent variables were: 1) physical harm from commission and 2) physical harm from omission. The independent variables were also dichotomous and consisted of the 15 technical quality criteria listed in Table 2. In addition, these 15 criteria were grouped into 5 categories [23]: authority, transparency and honesty, updating of information, editorial policy, and other. Web sites were classified into two groups based on whether they complied with the median number of quality criteria. The first group complied with six or fewer technical quality criteria, the second group complied with more than six technical quality criteria.

Inter-observer agreement measures were calculated to assess a) the degree to which validation reviewers agreed among themselves in their assessments of these dichotomous dependent variables (Table 3) and, b) the degree to which the validation reviewers agreed with the primary reviewers (Table 4). Cohen's kappa (K) is a commonly used measure of inter-observer agreement between two observers for dichotomous data. However, because K is affected in complex ways by the presence of bias between observers and by the distributions of data across the categories [24], we computed the prevalence-adjusted bias-adjusted kappa (PABAK), the bias index (BI) and the prevalence index (PI), as well as K, as recommended by Byrt et al [24].

The bias index (BI) is defined as the difference between the proportions of "Yes" for the two raters. The prevalence index (PI) is defined as the difference between the probability of "Yes" and the probability of "No." A BI close to 0 indicates less bias, while values closer to 1 (absolute value) indicate greater bias. Similarly, a PI close to 1 (absolute value) indicates high prevalence, while a PI closer to 0 indicates lower prevalence. The BI then measures the degree to which one reviewer tends to identify more or fewer occurrences than the other, while the PI measures the degree to which "Yes" agreements or "No" agreements predominate. The PABAK index of agreement between two observers is a measure that adjusts for both bias and prevalence. Although the derivation of the PABAK index is somewhat more complex, in practice it can be calculated as 2P₀ - 1, where P₀ is the proportion of observed agreement. Consequently, PABAK ranges from -1 to +1 and like K, a value of 0 represents no better than chance agreement, while magnitudes approaching 1 indicate maximal agreement.

Chi-square was calculated for each pairing of an independent variable with a dependent variable. Given the large number of statistical tests performed, significance was set at α<0.01. All analyses were performed using SPSS 11.0 statistical software.

Most Web sites did not comply with technical quality criteria. On average, a Web site complied with 6.3 (SD±2.6) of 15 criteria. One site failed to comply with any criteria, while three sites complied with 13 criteria. Only 27% of sites disclosed authorship, 36% provided references and 6% mentioned an editorial review process. Table 2 shows the number of Web sites that complied with each of the 15 quality criteria.

As shown in Table 3, agreement between the two evaluation reviewers was high (all PABAK > 0.67). Although there was little bias, there was a strong prevalence effect. Therefore, the two validation reviewers had a high degree of agreement for all measures of harm from commission and omission. Similarly, as shown in Table 4, consensus agreement between the primary and validation reviewers was also high (all PABAK > 0.73).

Potential physical harm from omission was more prevalent than from commission (97% vs. 25%, Table 5). However, a substantial number of Web sites (25%) displayed statements that could lead to physical harm. Statements that may cause toxicity if acted upon (direct toxicity) were present in 13% of CAM Web sites, while 7% of Web sites included statements encouraging the avoidance of conventional therapies. Eight percent of sites included information that may lead to harm from interactions if the advice were followed. Most CAM Web sites (97%) omitted vital information such as contraindications (89%) and drug interactions (83%).

We found that individual technical quality criteria did not identify sites with the potential to cause physical harm from commission or omission (Table 6). Similarly, when technical criteria were grouped into categories (such as authority, transparency and honesty, etc.), no significant association was found with potential physical harm (Table 7). Even when Web sites were classified into two groups, those complying with more criteria (≥ 6) versus fewer criteria (<6), there was no significant relationship. Overall, 44 hypotheses were tested but none were significant at the α<0.01 level, despite our study having 0.80 power to detect significance. Surprisingly, the presence of two quality criteria where a significant association was found at α<0.05 ("sources clear" and "editorial review process") indicated a greater chance of potential harm; the reverse of their original intent. However, it is possible that these two significant results may be due to chance since we conducted numerous statistical analyses.

We also explored the relationship between top level domain and potential harm. Seventy-seven percent of the 150 Web sites analyzed were commercial (.com), 10% organizational (.org), 7% network (.net), 3% educational (.edu), 2% governmental (.gov) and 1% unknown (numerical IP address only). Fisher's exact test statistic was calculated as expected values in some cases were <5, and significance was set at α = 0.05 level. Only the network top level domain had a significant relationship with physical harm from omission (Table 8). Of the 10 Web sites with the network top level domain, 20% did not contain harm from omission. In contrast, only 2% of sites that had a top level domain other than network did not have harm from omission (p<0.04). However, there was no statistically significant relationship between network and non-network sites with respect to physical harm from commission. Although there were few educational and government sites in our study, it is notable that there were no identified cases of potential harm by commission in these sites. As most Web sites were commercial, it is difficult to draw meaningful conclusions from this analysis.

In order to explore the relationships between Web sites that sold products and those that did not, two evaluators independently revisited each Web site and identified Web sites that allowed the ordering of products. Agreement between reviewers was high (K=0.95). Fifty-three percent of Web sites (n=79) sold products. There was no significant relationship between selling products and potential harm due to omission (P=0.56) or commission (P=0.02). Although not statistically significant at the α = 0.01 level, selling products was actually related to less harm from commission, the reverse of what we would expect. In fact 63% (n=24) of the harmful Web sites from commission were found on sites that did not sell products, while 37% (n=14) were found on Web sites that sold products. Therefore, in our sample there does not appear to be more harmful information on sites that sell products.