Published on in Vol 3, No 1 (2001):

The Use of Quality Benchmarking in Assessing Web Resources for the Dermatology Virtual Branch Library of the National electronic Library for Health (NeLH)

The Use of Quality Benchmarking in Assessing Web Resources for the Dermatology Virtual Branch Library of the National electronic Library for Health (NeLH)

The Use of Quality Benchmarking in Assessing Web Resources for the Dermatology Virtual Branch Library of the National electronic Library for Health (NeLH)

Authors of this article:

MN Kamel Boulos ;   AV Roudsari ;   C Gordon ;   JA Muir Gray

Original Paper

1Centre for Measurement and Information in Medicine, School of Informatics, City University, London, UK

2Royal Brompton & Harefield NHS Trust, London, UK

3Institute of Health Sciences, University of Oxford, Oxford, UK

Corresponding Author:

MN Kamel Boulos

Centre for Measurement and Information in Medicine

City University

Northampton Square





Background: In 1998, the U.K. National Health Service Information for Health Strategy proposed the implementation of a National electronic Library for Health to provide clinicians, healthcare managers and planners, patients and the public with easy, round the clock access to high quality, up-to-date electronic information on health and healthcare. The Virtual Branch Libraries are among the most important components of the National electronic Library for Health . They aim at creating online knowledge based communities, each concerned with some specific clinical and other health-related topics.

Objectives: This study is about the envisaged Dermatology Virtual Branch Libraries of the National electronic Library for Health . It aims at selecting suitable dermatology Web resources for inclusion in the forthcoming Virtual Branch Libraries after establishing preliminary quality benchmarking rules for this task. Psoriasis, being a common dermatological condition, has been chosen as a starting point.

Methods: Because quality is a principal concern of the National electronic Library for Health, the study includes a review of the major quality benchmarking systems available today for assessing health-related Web sites. The methodology of developing a quality benchmarking system has been also reviewed. Aided by metasearch Web tools, candidate resources were hand-selected in light of the reviewed benchmarking systems and specific criteria set by the authors.

Results: Over 90 professional and patient-oriented Web resources on psoriasis and dermatology in general are suggested for inclusion in the forthcoming Dermatology Virtual Branch Libraries. The idea of an all-in knowledge-hallmarking instrument for the National electronic Library for Health is also proposed based on the reviewed quality benchmarking systems.

Conclusions: Skilled, methodical, organized human reviewing, selection and filtering based on well-defined quality appraisal criteria seems likely to be the key ingredient in the envisaged National electronic Library for Health service. Furthermore, by promoting the application of agreed quality guidelines and codes of ethics by all health information providers and not just within the National electronic Library for Health, the overall quality of the Web will improve with time and the Web will ultimately become a reliable and integral part of the care space.

J Med Internet Res 2001;3(1):e5



The U.K. National electronic Library for Health (NeLH) [1] is a Web-based, "library without walls" project announced in 1998 as part of the National Health Service (NHS) "Information for Health" strategy of the U.K. Department of Health [2]. It will be one of the cornerstones of the new NHS health information system, with the ultimate goal of giving the people of the United Kingdom the best healthcare service in the world.

Backed by specialized and supporting services like the National Institute for Clinical Excellence (NICE) and other evidence and quality benchmarking and assurance systems, the NeLH will provide easy, 24-hour access to the best current knowledge and consequently help improve health and healthcare, clinical practice and patient choice [3].

The NeLH will also reduce the variations in healthcare delivery and quality from one part of the United Kingdom to another and empower the public and patients, reducing inequalities and ending "knowledge poverty" (in access and quality). Furthermore, the NeLH will help its users cope with the "knowledge overload", often of poor or uncertain quality, coming from raw sources like the World Wide Web and even from specialized services like MEDLINE.

The NeLH will include vortals known as Virtual Branch Libraries (VBLs). A VBL is a special collection of resources relevant to a specific community (or more than one community) of users and related to their particular interests and needs [4]. A Dermatology VBL is among the planned VBLs and will be based on the same high quality principles as the rest of the NeLH. Psoriasis, a common dermatological condition, has been chosen as a starting point.

Quality Benchmarking of Health-Related Web Resources

The Internet has become a major source of health information. No wonder the American Medical Informatics Association has a special Internet Working Group and a new peer-reviewed Journal of Medical Internet Research (JMIR) [5] was launched in 1999.

The importance of the Internet for healthcare professionals cannot be overlooked. Eysenbach et al [6] redefine cybermedicine as a new academic specialty at the crossroads of medical informatics and public health (medicine in cyberspace where cyberspace denotes the Internet). Compared with telemedicine (which is primarily curative medicine), cybermedicine is primarily preventive medicine and has changed the traditional model of preventive medicine and patient/community health promotion. Cybermedicine features mass patient education and patient-to-patient exchanges of information for patient education and self-support.

There is however a growing concern about the quality of health information on the Internet, which is frequently inaccurate or biased and sometimes even misleading and dangerous (Figure 1). Anyone can be a publisher on the Web, which is good for a democratic society but potentially problematic in professions such as medicine [7].

Figure 1. Royal Rife Research Society Web site, a blatant example of quackery on the Web [39]. The site offers a "miraculous" universal electronic cure for arthritis, diabetes, tumours and other afflictions (the end to all disease), and claims that the University of Southern California has sponsored research into this electronic therapy on the terminally ill, with astounding results
View this figure

The NeLH will not act as a censor, but rather as a quality filter powered by an explicit quality measurement system. This role is analogous to that of the renowned Goldsmith's Company (founded in London in 1327), which never traded gold, but was able through its independent hallmark stamping system to set the standards for the quality and purity of gold being traded. Knowledge hallmarks are needed to perform the function of gold hallmarks. The Cochrane logo for example has become a knowledge hallmark for systematic reviews and readers can always refer to the Cochrane Collaboration Handbook [8] to see the methods used to appraise and produce the Cochrane Reviews [9].

A Review of the Major Quality Benchmarking Systems Available Today

Some of the systems discussed below are mainly codes of ethics for resource providers, e.g. HONcode [10], while others are true quality-rating tools, e.g. the Oxford Centre for Evidence-Based Medicine Levels of Evidence [11]. Both approaches are important and there have been few attempts to combine a code of best practice with true quality rating under a single umbrella. Furthermore, two of the systems reviewed below, namely HONcode and med-PICS/MedCERTAIN [12,13], require target resource modification if they are to function optimally, e.g. adding the HONcode "active seal" logo and code or adding some PICS-labels or metatags to the section of Web pages in case of med-PICS. To use med-PICS consumers must also load the med-PICS rating description (a '.rat' file) into their Web browsers. The other systems described below provide quality evaluation schemes or checklists, sometimes very sophisticated, for selection of quality resources (they do not impose any modifications on target resources), e.g. OMNI [14], DISCERN [15] and QUICK [16]. Some systems also offer a directory service for quality resources, e.g. HON, OMNI and Medical Matrix [17].

Oxford Centre for Evidence-Based Medicine Levels of Evidence

Ball et al [11] from the Centre for Evidence-Based Medicine at Oxford define ten levels of evidence (1a, 1b, 1c, 2a, 2b, 2c, 3a, 3b, 4, 5) and map them to four grades of recommendation (A, B, C, D). A systematic review (with homogeneity) of randomized controlled trials is considered to be the most reliable source of evidence (top level), while an expert opinion (without explicit critical appraisal) is graded as the least trustworthy piece of evidence (level 5, Grade D). Systematic reviews act like a quality filter by purifying the findings of primary (raw) research, detecting bias and poor quality research and distilling the findings (the good reliable ones) into a single report.

The Health on the Net Foundation Code of Conduct (HONcode)

An international initiative launched in early 1996, Health On the Net Foundation (HON) is a not-for-profit organization headquartered in Geneva, Switzerland. The major HON sponsors are Sun Microsystems, the Swiss Institute of Bioinformatics and the State of Geneva.

Figure 2. Health on the Net Foundation logo and HONcode blue and red seals
View this figure

HON Code of Conduct (HONcode) [10] is a self-regulatory, voluntary certification system. HONcode does not rate the medical accuracy, validity or appropriateness of the information itself. It only defines a set of rules to hold Web site developers to basic ethical standards in the presentation of information and ensure readers always know the source and the purpose of the information they are reading. Compliant sites identify themselves by the blue-and-red HONcode hyperlink or "active" seal displayed usually at the bottom of their homepage (Figure 2).

HONcode addresses eight points: authority of the information provided, complementarity, confidentiality and privacy, proper attribution of sources, justifiability, transparency of authorship, transparency of financial sponsorship and honesty in advertising and editorial policy with emphasis on the importance of clearly separating advertising from editorial content. Dishonest operators may cut and paste the HONcode seal onto their Web sites. To check whether a given site featuring the HONcode seal is a bona fide HONcode subscriber, users can place their mouse cursor over the displayed HONcode seal and if the seal is authentic, they should see a special HONcode ID number appearing along the status bar of their browser. Clicking the seal should link them directly to a page on HON's site that summarizes the site's HONcode registration status. HON also has its own policing system and conducts random checks on subscribers.

Internet Healthcare Coalition e-Health Code of Ethics

The Internet Healthcare Coalition (IHC) (Figure 3) [18] is a not-for-profit organization with four major Internet health information providers, including Medscape, and Mediconsult, in addition to GlaxoSmithKline, the British pharmaceutical company, among its sponsors. (These sponsors should not pose any conflict of interest according to the Coalition's Statement of Independence).

Figure 3. Internet Healthcare Coalition logo
View this figure

In May 2000, the IHC e-Health Ethics Initiative introduced the Washington e-Health Code of Ethics which sets forth some important guiding principles for Internet healthcare sites and services grouped under eight main headings: candor; honesty; quality; informed consent; privacy; professionalism in online healthcare; responsible partnering; and accountability [19]. The full code can be downloaded from [accessed 27 August 2000].

Health Internet Ethics Consortium

In May 2000 the Hi-Ethics Consortium (Figure 4) [20], a coalition that brings together some of the most widely used health Internet sites, published their thoughts on what constitutes good Internet ethics. The Hi Ethics Consortium summarizes the key ethical principles of electronic health information publishing on the Web into a 14-point list. The complete list of Principles and a useful glossary is available [20].

Figure 4. Hi-Ethics Consortium Logo
View this figure

MedCERTAIN (med-PICS Certification and Rating of Trustful and Assessed Health Information on the Net)

One way to ensure that only quality medical Web sites are delivered to the consumers is to configure their Web browsers to filter out information that does not meet a defined standard [21].

MedCERTAIN (Figure 5) and the closely related metadata vocabulary med-PICS are initiatives for assessing, rating, labeling and filtering health information on the Web [12,13].

MedCERTAIN, an international not-for-profit project, was launched in 2000. Backed by the European Union under the "EU Action Plan for Safer Use of the Internet," it aims at improving health information quality on the Web through raising consumer awareness and industry self-governance. The underlying idea, proposed by G. Eysenbach, is to label medical Web pages with meta-information (information about information) using a standard computer-readable vocabulary based on the PICS (Platform for Internet Content Selection) labeling scheme [22].

Figure 5. med-PICS and MedCERTAIN logos
View this figure

These meta-information labels may be descriptive or evaluative. For example, a book review or critique is considered evaluative meta-information, while the table of contents is descriptive meta-information. Under the MedCERTAIN/med-PICS scheme, authors of medical Web pages are encouraged to include Unified Medical Language System (UMLS) codes in the header of their pages, as this can greatly enhance the quality of Web queries and increase the relevancy of search results. The proposed med-PICS vocabulary also includes definitions of target audience and country, type of information, contents rating, source assessment and advertising policy.

MedCERTAIN [13] will be a self- and third-party rating system. The project will develop and apply the necessary technologies to label health information on the Web, including exploring the next generation of PICS, which will apply Resource Description Framework (RDF), and eXtensible Markup Language (XML). A technical and organizational infrastructure will allow associations (e.g. medical societies) and individuals (e.g. medical domain experts) to rate (i.e. to assign meta-information to) health information on the Internet, in a collaborative, distributed and decentralized way.

Consumers will be able to use their browsers or additional software or search engines to retrieve this meta-information automatically in the background whenever they access a Web site. Based on this meta-information, access can be limited to quality assessed content on the Web and a disclaimer may be displayed if the consumer leaves the rated subset of the Web. The MedCERTAIN consortium will create different levels of certification for publishers of health information on the Web, ranging from simple quality seals indicating a "good standing" of the site (level 1) to quality seals indicating that the site has been peer-reviewed externally. Web sites that want to get a MedCERTAIN certification will have to commit themselves to the Washington Code of e-Health Ethics or other ethical codes (see above). A community of trusted assessors will rate information as they surf the Web, flagging fraudulent information and evaluating (peer-reviewing) information if authors apply for a "level-3" quality seal.

To make use of med-PICS (Figure 6) [12], a rating description must be loaded into the Web browser. Microsoft Internet Explorer simply needs a text file, with the file extension '.rat.' Once loaded, the user is presented with a simple interface through which quality requirements and personal preferences can be defined (Figure 6). Users then need to define which labeling bureau(x) they wish to check with before accessing any Web site.

Figure 6. Defining the target audience in Microsoft Internet Explorer Content Advisor using med-PICS after loading the proposed med-PICS rating file (current version as of May 2000 is named 'medv0-3.rat'). In this screenshot, the target audience has been defined as a doctor/GP
View this figure

Subsequently, when a user requests, through the browser, any Web page, the software filter not only fetches the document but also makes an inquiry to the label bureau requesting any labels that have been assigned to that site. Depending on what the labels say, the filter may display an alert or disclaimer [21].

OMNI (Organising Medical Networked Information)

Based at the University of Nottingham and backed by people from a wide array of backgrounds and institutions, including several other universities, the Department of Health, the Wellcome Trust and the British Library, OMNI (Figure 7), which started in 1994, is now a much-respected UK point of access to quality biomedical Internet resources [14]. OMNI has recently become one of BIOME's health and life sciences gateways [23].

OMNI's Advisory Group on Evaluation Criteria (AGEC) started by examining a number of different services available via the Internet that seek to provide access to selected and evaluated medical networked information resources. The group compared the criteria these services use for evaluating resources before producing their own OMNI Guidelines for Resource Evaluation which constitute one of the most comprehensive quality benchmarking checklists available today [24].

Resources are included that are either accessible directly as Web pages or can be downloaded across the network, e.g. downloadable tutoring software. As a rule, OMNI does not point to sites as a whole but to specific resources, so resources are identified and indexed at the level of individual Web pages and downloadable files (i.e., cataloguing on a per-page as opposed to per-site basis). This rule is clearly unsustainable in the case of databases, electronic journals and similar resources with a huge content base, where, unless an individual article or posting is particularly valuable in its own right and offers unique insights, OMNI will normally only point to the homepages of such resources.

Figure 7. OMNI logo
View this figure

OMNI's selection process starts by establishing the context of the candidate resource and examining its scope, target audiences, authority and provenance. The evaluation criteria can be grouped into two main sets: content evaluation criteria and access evaluation criteria.

Content evaluation criteria include coverage, accuracy of information content, currency/frequency and regularity of updating and uniqueness/comparison with other sources. Access evaluation criteria include accessibility, usability and charging policy (if there are access restrictions), any special requirements, software reliability (for software resources), copyright and redistribution issues, language, design and layout/user interface and finally user support/documentation. Each criterion consists of a series of questions (checklist) to which the assessors are supposed to find answers.

The resource evaluation process considers all the criteria listed above, some of which might not be fully satisfied (or might not be applicable to the type of resource in question). In the final analysis, it is the overall impression about the value of a resource to the OMNI user community that guides the assessors to recommend it for inclusion in the OMNI database.


The DISCERN Project (Figure 8) [15] was funded by The British Library and the NHS Research & Development Program at the University of Oxford.

Figure 8. DISCERN logo
View this figure

DISCERN is an instrument designed to help users of consumer health information judge the quality of written information about treatment choices. DISCERN is suitable for anyone who uses or produces information about treatment choices. To pass the DISCERN benchmark, a good quality publication about treatment choices must:

  • have explicit aims;
  • achieve its aims;
  • be relevant to consumers;
  • make sources of information explicit;
  • make date of information explicit;
  • be balanced and unbiased;
  • list additional sources of information;
  • refer to areas of uncertainty;
  • describe how treatment works;
  • describe the benefits of treatment;
  • describe the risks of treatment;
  • describe what would happen without treatment;
  • describe the effects of treatment choices on overall quality of life;
  • make it clear there may be more than one possible treatment choice;
  • provide support for shared decision-making.

C-H-i-Q (Centre for Health Information Quality - UK)

The Centre for Health Information Quality (C-H-i-Q) (Figure 9) was established in 1997 as part of the Patient Partnership Strategy, an NHS initiative acknowledging the need to "put patients first." C-H-i-Q has an international reputation, in particular in the area of appraisal of consumer health information and promoting good quality patient information [25]. C-H-i-Q has developed close ties with NICE to provide patients in the future with clear, concise "consumer-friendly" versions of NICE clinical guidelines.

Figure 9. C-H-i-Q logo
View this figure

Another major responsibility of the Center is to provide an appraisal service for NHS Direct Online, the NHS patient information service [26]. The following themes are on C-H-i-Q's checklist for resource evaluation:

  • accessibility (the information is in an appropriate format for the target audience);
  • accuracy (the information is based on the best available evidence);
  • appropriateness (the information communicates relevant messages);
  • availability (the information is available to the widest possible audience);
  • currency (the information is up-to-date);
  • legibility (written information is clearly presented);
  • originality (information has not already been produced for the same audience and in the same format);
  • patient involvement (the information is specifically designed to meet the needs of the patient);
  • readability (words and sentences are kept short where possible and jargon is minimized);
  • reliability (the information addresses all essential issues).

QUICK (The QUality Information ChecKlist)

Figure 10. Screenshot from the QUality Information ChecKlist Web site showing the eight points consumers have to consider when assessing the quality of information on Web sites
View this figure

The QUICK (QUality Information ChecKlist) Web site (Figure 10) [16] was developed by C-H-i-Q (Centre for Health Information Quality) in collaboration with HEA (the Health Education Authority (now replaced by the Health Development Agency, HDA). QUICK is a resource to help young people evaluate the information they find on the Internet. QUICK checklist includes the following eight questions:

  1. Is it clear who has written the information? Who is the author? Is it an organization or an individual person? Is there a way to contact them? An e-mail address on its own is not a proof that the author is a genuine expert on a subject or even who they claim to be as anyone can get an e-mail address especially with free services like Hotmail.
  2. Are the aims of the site clear? What are the aims of the site? What is it for? Who is it for?
  3. Does the site achieve its aims? Does the site do what it says it will?
  4. Is the site relevant to me?
  5. Can the information be checked? Is the author qualified to write the site? Has anyone else said the same things anywhere else? Is there any way of checking this out? If the information is new, is there any proof?
  6. When was the site produced? Is it up to date? Can you check to see if the information is up to date and not just the site?
  7. Is the information biased in any way? Has the site got a particular reason for wanting you to think in a particular way? Is it a balanced view or does it only give one opinion?
  8. Does the site tell you about choices open to you? Does the site give you advice? Does it tell you about other ideas?

Net Scoring: Criteria to Assess the Quality of Health Internet Information

Net Scoring (Figure 11) is a French quality benchmarking system in use by many prominent French institutions, including the Centre Hospitalier Universitaire de Rouen, which developed CISMeF (Catalogue et Index des Sites Médicaux Francophones), the French equivalent of OMNI [27]. Net Scoring [28] provides a set of criteria for assessing the quality of health information on the Internet. To ensure objectivity in the development of these criteria, a diverse group of individuals, including representatives of professional organizations, medical doctors, engineers and lawyers, was gathered. Net Scoring comprises 49 criteria grouped into eight categories: credibility, content, links, design, interactivity, quantitative aspects, ethics and accessibility. Each criterion is assigned a weight according to its importance. An essential criterion is rated from 0 to 10. An important criterion is rated from 0 to 5, while a minor criterion is rated from 0 to 2. It is noteworthy that Net Scoring considers the "use of metadata" an essential criterion. The total of these weighted criteria gives the overall score of a site (the maximum score a resource can achieve is 297 points). Net Scoring criteria are presented in Appendix 1.

Figure 11. Net Scoring logo
View this figure

Medical Matrix

Medical Matrix (Figure 12) is a medical Web directory service that aims to improve the global medical community's access to useful digital clinical medicine resources [17] (registration is required but is free).

The Medical Matrix Project assigns star ranks to Internet resources based on their clinical utility as follows (Table 1):

Table 1. Medical Matrix Star Ranking System
*Suitable, well-authored clinical content but lacking in substance, or currency. (1-10 points)
**Clinical content is generally reliable and up-to-date. Site design is logical and easy to use. Limited usefulness as a regular clinical resource. (11-20 points)
***Well-authored, accurate, current clinical content. Good site design, well-maintained and extensive functionality. Easily accessed and navigated by the routine user. An overall valuable clinical resource. (21-30 points)
****Outstanding site across all categories and a premier web page for the discipline. (31-40 points)
*****An award winning site for Medical Internet. (41-50 points)

Four main criteria are checked by the Medical Matrix Editorial Board when evaluating a resource:

  • Dimension/usefulness for clinical applications: The resource enhances the knowledge base of the target clinician. Resource documents have current clinical relevance and importance, intellectual and scientific strength, and clarity of presentation. (1-20 points)
  • Verifiability, clarity, and integrity: Resource document content is verifiable, endorsed, dated, current, and referenced. The material is original; the writing is clear; there is a minimization of bias; conclusions are reasonable and supported by evidence presented. The effort is ethical. The documents offer a comparison with relevant findings from other publications. Any conflict of interest is disclosed. (1-10 points)
  • Evidence-based criteria: Conclusions are based on studies that are methodologically sound, meet statistical validity criteria and are clinically relevant. Conclusions are rated against a "gold standard" in that they are founded upon randomized trials with appropriate follow-up and are based on studies that make an independent, blind comparison of tests. (1-10 points)
  • Media: Text, hypertext, or use of multimedia in the context of the resource. (1-10 points)
  • Feel/ease of access: Easy to follow in terms of composition, presentation on the Web and integration within a larger database. (1-5 points)
Figure 12. Medical Matrix logo
View this figure

Principles Governing American Medical Association (AMA) Web Sites

A standing committee composed of AMA staff members from the Scientific Publications and Multimedia, Publishing and Business Development, Ethical Standards, and Internet and Database Services areas (Figure 13) developed these guidelines, which were released in March 2000 [29]. The guidelines cover four areas, namely content, advertising and sponsorship, privacy and confidentiality, and e-commerce [29].

The principles for advertising and sponsorship cover many issues including guidance on the placement of digital advertisements. Just as a print advertisement would never be placed next to an editorial page on the same topic, a digital advertisement should never be adjacent to editorial content on the same topic. Readers should not be sent to a commercial site unless they choose to do so by clicking on an advertisement.

Figure 13. AMA logo
View this figure

The principles for privacy and confidentiality aim at maintaining Web site visitors' rights to privacy and the confidentiality of personal information. Tracking of personal medical and health information, i.e., medical conditions, health-seeking behaviors and questions, and requests about drug therapies or medical devices or information pertaining to them, could breach an individual's personal privacy and reveal an individual's health data. Identifying details should be omitted if they are not essential, but patient data should never be altered or falsified in an attempt to attain anonymity. Information about individual medical conditions must not be collected without the express permission of the site visitor.

The principles for privacy and confidentiality also state that e-mail alerts and newsletters should only be sent upon a visitor's explicit request and should always contain an "unsubscribe" option.

HSWG Criteria for Assessing the Quality of Health Information on the Internet

The Health Summit Working Group (HSWG), funded by the United States' Agency for Healthcare Research and Quality and the Health Information Technology Institute, selected, defined, ranked and evaluated seven major criteria for assessing the quality of Internet health information [30]. The credibility criterion covers the source, currency, relevance/utility, and editorial review process for the information. Resource content must be accurate (Table 2) and complete, and an appropriate disclaimer should be provided.

Disclosure includes informing the user of the purpose of the site, as well as any profiling or collection of information associated with using the site. Links are evaluated according to selection, architecture, content, and back linkages. Back linkages (also mentioned among Net Scoring criteria above) are links to one Web site from another. Many Web sites track and publish back linkages for the purpose of enhancing their credibility and marketability. The best way to evaluate back linkages is to examine the context in which they are used, that is, their purpose, relevance, credibility, and authority, as well as any associated bias. Design encompasses accessibility, logical organization (navigability), and internal search capability. Interactivity includes feedback mechanisms and means for exchange of information among users. The last criterion, caveats, involves clarification of whether the site function is to market products and services or is a primary information content provider.

Table 2. HSWG Hierarchy of Evidence
Validity of EvidenceWhat to Look For
++++ (Best Evidence)Randomized controlled trials
+++Non-randomized controlled trials
++Well designed cohort or case-control analysis
+Opinions of respected authorities, case reports, descriptive studies, reports of expert committees
No EvidenceMisrepresentation, fraud

Silberg's Core Criteria for Measuring Quality (Caveat lector and viewor)

Silberg [31] suggested the following four core criteria for measuring the quality of medical Web sites:

  • Authorship: The author(s) of a Web page, along with their affiliations and credentials, should be clearly stated. Ideally, there should be the facility to contact the author(s) by e-mail.
  • Attribution: If a Web site is quoting research or evidence then the source of this data must be explicitly stated.
  • Disclosure: The owner of the Web site must be prominently displayed, along with any sponsorship or advertising deals that could constitute a potential conflict of interest.
  • Currency: Web pages should indicate when the page was created, and when it was last updated.

The authors have noted that some webmasters use JavaScript within their pages to pick up the client machine's current date and display it after a "last updated" or similar string, fooling the reader into thinking that these pages were updated on the same day (i.e., a false impression of currency).

Price's PILOT Method for Evaluating Medical Web Sites

The PILOT method for evaluating medical Web sites consists of five criteria (Table 3) [32]:

Table 3. Price's Pilot Method
Purpose:If the site has a mission statement, read it. If not, read the home page and analyse the site\'s purpose. Does it inform and educate? Or is designed to persuade, sell, outrage or entertain?
Information:Truly useful medical Web sites offer valuable information and emphasise facts rather than opinion and testimonials. If the site is selling anything, ask yourself if that influences the content.
Links:The best sites want to inform you and are happy to recommend additional Web sites to further your knowledge in that topic or related topics. The best links are rated or reviewed.
Originator:Who is responsible for the information? Best bets for sound medical information are medical societies, consumer-advocacy groups, well-known hospitals, and government- and university-sponsored sites.
Timeliness:Medical information is only useful if it is current. Look for sites that update frequently.

Developing a Quality Benchmarking System for the NeLH

According to the Centre for Health Information Quality (C-H-i-Q) [25], a new discipline will probably emerge in the near future, with the development of an academic qualification in health information appraisal. Healthcare professionals trained in this domain will be the ideal NeLH "knowledge miners" or "quality assurance officers." We believe that a quality evaluation software wizard should be developed, possibly by pooling the best criteria and methods from all the quality benchmarking systems that are available today to produce an all-in knowledge-hallmarking instrument. The aim of this wizard will be to assist (not to replace) NeLH knowledge miners in their work and ensure that they consistently and explicitly follow all the required evaluation steps. This wizard should be also able to write evaluation results and details of the resource being evaluated (e.g. URL, score or rating, currency information, etc.) and any other relevant meta-information to the appropriate NeLH page or to the record pointing to this resource in the NeLH resource catalogue or database. To develop a good quality benchmarking system, a standard statement must first be defined. The standard statement is the hub on which the other elements of the standard revolve. It describes an agreed level of quality appropriate to the organizational and target readers needs. It specifies a desirable, acceptable and achievable quality level. The next step is to develop criteria that clearly and precisely specify the quality level that must be present in order to satisfy the standard. For any given quality standard, it is possible to write a great many criteria. The quality assurance team responsible for the development of the quality benchmarking system, therefore, has to choose criteria on the basis of a sound principle like AMOUR [33]. AMOUR defines a set of five clear questions/goals that each of the candidate criteria must answer:

  • is the criterion Achievable? The development group must choose between idealism (unattainable standards) and realism.
  • is it Measurable? A standard statement might not be worded in measurable terms but a criterion must be. When writing a criterion it is helpful to consider "how can a check be made that a given resource fulfils the criterion?" and find a clear and practical answer to this question.
  • is it Observable? For a phenomenon to be observable it must be detectable through the senses. If a criterion refers to unobservable (very abstract) objective, it will be almost impossible to determine whether or not this objective has been satisfied by any given resource under examination. A criterion which uses vague terms such as "appropriate content" or "information should not be biased" is very difficult to use (and very subjective).
  • is it Understandable? The development team should unambiguously and explicitly define the meanings of linguistic variables like "appropriate" or "suitable." The rule is always to be clear, objective and specific.
  • is it Reasonable? All involved or targeted "players" (information providers, patients and professionals) must be represented within the development group formulating the benchmarking system to ensure that all pertinent players find the final criteria reasonable.

Selection and Evaluation of Candidate Web Resources for the Dermatology VBL

The authors used Web Inspector, an offline metasearch tool [34], some specialized medical search engines and directories, e.g. HON MedHunt and HONselect [9], plus the personal knowledge of the first author (who is a dermatologist) about good dermatology Web resources to locate candidate sites.

Six major Internet search engines (Euroseek, Excite, Infoseek, Yahoo, All the Web and Google) were queried at the same time by Web Inspector (on 13 May 2000) using the keyword 'psoriasis'. The maximum number of results per search engine was set to 300. After the automatic removal by Web Inspector of all duplicate and dead links, 413 documents (containing the word 'psoriasis') remained for manual exploration by the authors.

We tried to include a balanced mix of resources that can be used to answer a variety of information queries and needs. The selected resources (Appendix 2) include sites intended mainly for patients, as well as sites targeting healthcare professionals. In addition to psoriasis-specific resources, the authors have also compiled a list of some of the very useful general dermatology resources on the Web for patients and healthcare professionals. These "general" resources can support continuing professional education and also provide different types of information on the rest of skin diseases.

Sites that are purely textual, searchable databases, atlases (dermatology image banks), sites that mix text, graphics and interactive elements (e.g. educational simulations) were all considered for inclusion, as long as they did not rely on sophisticated browser plug-ins and/or non-standard Web technologies to display their information. JavaScript (when properly implemented), Adobe Acrobat Reader and RealNetworks RealPlayer are examples of respectable, mainstream Web technologies and browser plug-ins or extensions. They are freely available, very stable, lightweight and supported on a wide variety of platforms, and therefore pages using them were not excluded. Web directories that are just link lists (pointers to other resources) were excluded unless they constituted a real resource guide offering additional evaluative or descriptive information. Information providers whose contributions to the knowledge base of dermatology have long been recognized as authoritative, e.g. the British Association of Dermatologists ( [accessed 27 August 2000], were preferred.

Sites displaying advertisements or sponsored by commercial bodies, e.g. sites endorsed by pharmaceutical companies, were typically excluded for obvious reasons, unless the author felt there were no real conflicts of interest, possibility of bias or any other hidden or malicious motives. For example, the author has included the New Zealand DermNet ( [accessed 27 August 2000], which remains a very good and unbiased resource, despite the fact that it displays pharmaceutical advertisements.

Sites that are not regularly updated were excluded, based on the concept that every piece of information should have an "expiry date" after which it should be revised and, if necessary, updated. But again exceptions to this rule had to be made to include some sites with inadequate currency information, but providing very good basic information that is less likely to change over time.

Deleting the right hand side of a resource URL in steps has proved very helpful in providing additional insight and information concerning the credibility of the authors and publishers of some resources, e.g. (the resource), (the author) and (the author's the academic institution).

Over 90 professional and patient-oriented Web resources on psoriasis and dermatology in general have been hand-selected for possible inclusion in the forthcoming Dermatology VBL (Appendix 2). Among the selected electronic Web-based atlases is Dermatology Online Atlas (DOIA) at the University of Erlangen, Germany (Figure 14) [35]. The history of this Atlas of Dermatology goes back to 1994, the early days of the World Wide Web. In these days, a German research project was launched to use this new technology for the benefit of medical education and teaching, especially in dermatology.

Figure 14
View this figure

This online atlas of dermatology surpasses any printed image atlas and integrates both interactive and dynamic components, with more than 3000 high quality images of more than 600 dermatological diagnoses and differential diagnoses. Morphological features of the images have been described by dermatologists. The site offers a database query interface for advanced image selection by different criteria, including localization, morphological features, age and gender. With each image or set of images covering the same diagnosis, direct links are provided to other images of suitable differential diagnoses within the atlas. Also provided are commented and rated links to external Web sites offering further information on the respective dermatological disease. Special links allow direct, fast and easy access to disease-related information in databases like MEDLINE and OMIM (Online Mendelian Inheritance in Man). The atlas can be also browsed in 'quizmode' for training [36]. While the authors did their best to observe the essence and spirit of all the quality benchmarking systems described above during the selection and evaluation process, the list of candidate resources (Appendix 2) should not be considered an exhaustive and final selection, but rather a first-pass filtering. The identified candidate sites should be subjected to more scrutiny to determine their final status, whether suitable or not for inclusion in the NeLH. Quality benchmarking of health-related resources will always depend on a human assessor, and as such, it will always have attached to it an inevitable element of subjectivity that cannot be corrected by simply providing numerical scales for rating a resource. These numerical scales can only offer apparent, but not true objectivity. According to Delamothe [37], rating the quality of medical Web sites may be impossible. One option he presents is to rate the process by which the content was produced rather than the content itself, i.e., rate the assessor or information provider rather than individual resources. Thus a medical journal's Web site containing peer-reviewed material would rate higher than a commercial site selling miracle cures for cancer.

Online (e.g. [accessed 27 August 2000] and offline (e.g. Web Inspector [34]) metasearch Web tools can greatly assist human NeLH knowledge miners by performing an automatic preliminary mining and filtering of Web pages on a given topic, thus facilitating the final filtering task of the human knowledge miner. By querying multiple search engines at the same time the potential for finding information is much greater. Moreover, users are spared the time spent manually visiting a number of search engines and re-keying their query or in following dead or duplicate links. However, the NeLH should always remain a human-maintained catalogue with value added contents. Skilled, methodical, organized human reviewing, selection and filtering based on well-defined quality appraisal criteria seems likely to be the key ingredient in the envisaged NeLH service. Proper training of human assessors to develop good discernment skills is as important as the quality benchmarking checklists themselves. The authors recommend that assessors take the "Internet Detective" interactive tutorial on evaluating the quality of Internet resources [38]. This tutorial is provided by the Social Science Information Gateway (SOSIG), which is part of the UK Resource Discovery Network. In the near future, MedCERTAIN [13] might become an international (or at least European) standard for rating health-related Web sites if it succeeds in setting the norm for quality benchmarking criteria and labeling infrastructure. By promoting the application of agreed quality guidelines and codes of ethics by all health information providers and not just within the NeLH, the overall quality of the Web should improve with time. This improvement in quality will always be the result of a collaborative world-wide effort based on the goodwill of "informed net citizens" and "professional obligation" of healthcare professionals and information providers, rather than on any enforced laws (i.e., self-regulation). Ultimately, the Web will become a reliable and integral part of the care space.

Conflicts of Interest

None declared.

Appendix 1

Net Scoring Criteria to Assess the Quality of Health Information
Table 4. Net Scoring Criteria to Assess the Quality of Health Information
1 Credibility
(100 points)
1.1 Source
1.1a Name, logo and references of the institution on each document of the site (essential criterion)
1.1b Name and title of author on each document of the site (essential criterion)
1.2 Disclosure 1.2a Context: source of financing, independence of the author(s) (essential criterion)
1.2b Conflict of interest (important criterion) 1.2c Influence, bias (important criterion)
1.3 Updating: currency information of the site including date of creation, date of last update and eventually date of last version (essential criterion)
1.4 Relevance/utility (essential criterion)
1.5 Editorial review process (essential criterion)
1.5a Webmastering process (important criterion)
1.5b Scientific review process (important criterion)
1.6. Target/purpose of the web site; access to the site (free or not, reserved or not) (important criterion)
1.7. Quality of the language and/or translation (important criterion)
1.8 Use of metadata (essential criterion)
2 Content
(79 points)
2.1 Accuracy (essential criterion)
2.2 Hierarchy of evidence (important criterion)
2.3 Original Source Stated (essential criterion)
2.4 Disclaimer (important criterion)
2.5 Logic organization (navigability) (essential criterion)
2.5a Quality of the internal search engine (important criterion)
2.5b General index (important criterion)
2.5c What\'s new page (important criterion)
2.5d Help page (minor criterion)
2.5e Map of the site (minor criterion)
2.7 Omissions noted (essential criterion)
2.8 Fast load of the site and its different pages (important criterion)
2.9 Clear display of available information categories (factual data, abstracts, full-text documents, catalogue, databases) (important criterion)
3 Hyperlinks
(52 points)
3.1 Selection (essential criterion)
3.2 Architecture (important criterion)
3.3 Content (essential criterion)
3.4 Back-links or number of hyperlinks to the resource in question (Web Impact Factor) (important criterion)
3.5 Regular verification that hyper-links are functioning, i.e., no broken links (important criterion)
3.6 In case of modification of the site structure, link between old and new HTML documents (important criterion)
3.7 Distinction between internal and external hyper-links (minor criterion)
4 Design
(20 points)
4.1 Design of the site (essential criterion)
4.2 Readability of the text, images, or video (important criterion)
4.3 Quality of the print (important criterion)
5 Interactivity
(17 points)
5.1 Mechanism for feedback: Email of the author on every document (essential criterion)
5.2 Forums, chat (minor criterion)
5.3 Users must be clearly informed if the site is using cookies or any similar technologies to trace their logging into the site and to retrieve information about their machines (even if the tracing is done in anonymous form). Users must be able to opt in or opt out of functions that track personal or machine information at any time (important criterion)
6 Quantitative aspects
(9 points)
6.1 Number of machines (IP addresses) visiting the site and number of visualised documents (important criterion)
6.2 Number of press releases (minor criterion)
6.3 Number of scientific production from the site, including bibliometric criteria (minor criterion)
7 Ethics
(20 points)
7.1 Liability of the reader (essential criterion)
7.2 Medical privacy (essential criterion)
8 Accessibility
(10 points)
8.1 Accessibility from the main search engines and catalogues (important criterion)
8.2 Intuitive address of a site (important criterion)
297 points maximum

Appendix 2

Patient Information
Table 5. List of candidate general dermatology and psoriasis Web resources for inclusion in the NeLH Dermatology VBL. Please note that URLs do not contain white spaces or line breaks
General Resources
SiteURL (all URLs have been accessed 27 August 2000)
NHS Direct (U.K.) [26]
U.K. Dermatology Patient Support Groups (U.K.)
New Zealand DermNet for Patients
American Academy of Dermatology Patient Information
American Academy of Family Physicians - Patient Information (Skin and Hair)
American Academy of Family Physicians - Patient Information (Skin Disorders)
Educational Information on Common Skin Diseases for Patients - National Skin Centre, Singapore
Psoriasis: Online Patient Information and Pamphlets
NHS Direct Audio clip on psoriasis from the College of Health Healthline Tapes service (requires RealPlayer -
PPP healthcare International-Health Info-psoriasis (U.K.)
NIH - Questions and Answers About Psoriasis (U.S.A.)
MEDLINEplus - Psoriasis (U.S.A.)
AAD Psoriasis Pamphlet (U.S.A.)
American Academy of Family Physicians: What Can Be Done for Psoriasis (U.S.A.)
Psoriasis at Medinfo (U.K.)
MCW HealthLink: Psoriasis (U.S.A.)
Psoriasis Research Institute: Facts about Psoriasis (U.S.A.)
Psoriatic Arthritis at the University of Washington (U.S.A.)
Psoriasis: Online Patient Information and Pamphlets
SiteURL (all URLs have been accessed 27 August 2000)
DermWeb Handout: Psoriasis: What is it and How is it Treated? (Canada)
National Skin Centre: Psoriasis (Singapore)
Beware of Web quackery and the so-called miraculous cures: Psoriasis Hall of PShame (U.S.A.)
Psoriasis: Online Organizations
The Psoriatic Arthropathy Alliance (U.K.)
National Psoriasis Foundation (U.S.A.)
Canadian Psoriasis Foundation
Psoriasis Society of Canada
Health Care Professionals
General Resources/Research Tools
Cochrane Skin Group
The British Association of Dermatologists
New Zealand DermNet for Dermatologists
New Zealand DermNet for General Practitioners
Skindex - Dynamic Dermatology (Dr. Thomas B. Fitzpatrick - U.S.A.)
Skin Diseases - Primary Care Clinical Practice Guidelines, University of California, San Francisco (U.S.A.)
Archives of Dermatology (Full text Journal - U.S.A.) Dermatology Medline Engine
Hardin Library for the Health Sciences, University of Iowa - PubMed Medline search Dermatology & Skin Diseases (U.S.A.)
Health On the Net Foundation: HONselect - Skin and Connective Tissue Diseases (Switzerland)
NIH Library - Skin Diseases Sites (U.S.A.)
General Resources/Research Tools
SiteURL (all URLs have been accessed 27 August 2000)
Department of Dermatology Links Page at the University of Wales College of Medicine, Cardiff (U.K.)
Matrix Dermatology Resources (U.S.A.)
DermWeb, the Home Page for the Division of Dermatology at the University of British Columbia (Canada)
Dermatology Medical Journals - WebMedLit
Medical Matrix: Dermatology (Requires registration; registration is free)
Skin and Connective Tissue Diseases - MeSH classified index, Karolinska Institute (Sweden)
Dermatonet for Professionals (France)
Psoriasis: Comprehensive Reviews
eMedicine Online Text: Psoriasis (U.S.A.)
The Merck Manual of Diagnosis and Therapy: Psoriasis (U.S.A.)
Psoriasis in Handbook of Dermatology Venereology, 2nd Edition, Social Hygiene Service, Department of Health, Hong Kong
Psoriasis: Genetics
Psoriasis Genetics Laboratory, University of Michigan (U.S.A.)
OMIM Psoriasis Inheritance and Genetics Research (U.S.A.)
Psoriasis: Photographs
Dermatology Online Atlas (DOIA) Erlangen: Psoriasis (Germany)
Psoriasis of the nails
Psoriasis: Research/Online Pre-formulated Queries
eBMJ - Psoriasis Articles (U.K.)
Honselect - Psoriasis (Switzerland)
Psoriasis: Research/Online Pre-formulated Queries
SiteURL (all URLs have been accessed 27 August 2000)
CliniWeb Search Results for Psoriasis (U.S.A.)
Medical World Search Results for Psoriasis (U.S.A.)
Archives of Dermatology Search Results for Psoriasis (free full text articles available) (U.S.A.)
NIH MEDLINE abstracts on psoriasis (All) (U.S.A.)[MAJR]+AND+Human[MESH]+AND+English[LANG]=d=1+Year=20
NIH MEDLINE abstracts on psoriasis (Review) (U.S.A.)[MAJR]+AND+Human[MESH]+AND+English[LANG]+AND+review[PTYP]=d=1+Year=20
NIH MEDLINE abstracts on psoriasis (Diagnosis) (U.S.A.)[MAJR]+AND+Human[MESH]+AND+English[LANG]+AND+(sensitivity+and+specificity[MESH]+OR+sensitivity[WORD]OR+diagnosis[SH]+OR+diagnostic+use[SH]+OR+specificity[WORD])=d=1+Year=20
NIH - Psoriasis info (U.S.A.)
Computer Image Analysing (CIA) System for Psoriasis Area Assessment (Research - Medical Informatics/Machine Vision - Finland)
Psoriasis: Treatment
Psoriasis treatments [Bandolier - May 1998; 51-5] (U.K.)
Topical capsaicin in Psoriasis [Bandolier - Jul 1996; 29-6] (U.K.)
Cochrane - Interventions for guttate psoriasis (U.K.)
Cochrane - Antistreptococcal interventions for guttate and chronic plaque psoriasis (U.K.)
Cochrane - Interventions for psoriatic arthritis (U.K.)
American Academy of dermatology Guidelines of Care for Psoriasis (U.S.A.)
American Academy of Family Physicians: Topical Psoriasis Therapy (U.S.A.)
Dermatology Times - Psoriasis Treatments (U.S.A.)
For Nurses: Psoriasis: what to tell your patients (NurseWeek, U.S.A.)
NIH MEDLINE abstracts on psoriasis (Treatment Research - U.S.A.)*+OR+therap*)
Professional Education
Online Textbooks, Lectures and Clinical Case Simulations
eMedicine Online Text - Dermatology (U.S.A.)
Handbook of Dermatology Venereology, 2nd Edition, Social Hygiene Service, Department of Health, Hong Kong
The Electronic Textbook of Dermatology (U.S.A.)
Interactive Dermatology Cases, UCLA Medical School (U.S.A.)
Simulated Dermatology Patient (Taiwan)
Multimedia Dermatology Course (Taiwan)
Dermatology and Dermatopathology Online Lectures, University of Rochester Medical Centre (U.S.A.)
Dermatology for Students (Skin Tumours and Other Skin Diseases), University of California Davis (U.S.A.)
Developmental Biology of the Skin: An Interactive Tutorial, University of Washington (U.S.A.)
Online Atlases/Image Banks
DermIS - DOIA Dermatology Online Atlas - English version. (Germany)
Dermatology Atlas - Loyola University Dermatology Medical Education Website (U.S.A.)
Dermatology Images, University of Kansas Medical Center (U.S.A.)
Skin Images, Harvard Medical School (U.S.A.)
Dermatology Images, University of Iowa (U.S.A.) (accessed 12 June 2000)
Dermatology Image Bank at the University of Utah School of Medicine (U.S.A.)
Dermatopathology Atlas, Indiana University (U.S.A.)
Dermatology and Rheumatology Online Atlas (Italy)
Anatomy of the skin (Electron Microscopy), Department of Dermatology, Mie University School Of Medicine (Japan)
Evidence-based Practice
Bigby M. Welcome to evidence-based medicine. Arch Dermatol 1998; 134:1516
Sharon E. Straus SE and Sackett DL. Bringing evidence to the clinic. Arch Dermatol 1998; 134:1519
Bigby M. Evidence-based medicine in a nutshell: a guide to finding and using the best evidence in caring for patients. Arch Dermatol 1998; 134:1609
Williams H. Adetugbo K, Li Wan Po A, Naldi L, Diepgen T and Murrell D. The Cochrane Skin Group: preparing, maintaining, and disseminating systematic reviews of clinical interventions in dermatology. Arch Dermatol 1998; 134:1620
Adetugbo K and Williams H. How well are randomized controlled trials reported in the dermatology literature? Arch Dermatol 2000; 136:381
The Outputs database of the RD Programme (DoH - U.K.)
Primary Care Clinical Practice Guideline, University of California, San Francisco (U.S.A.)

  1. ; U.K. National electronic Library for Health. U.K. National electronic Library for Health, NHS NeLH - architecture and strategy papers.   URL: [accessed 2000 August 27]
  2. The NHS' "Information for Health" strategy (1998-2005).   URL: [accessed 2000 August 27]
  3. National Institute for Clinical Excellence.   URL: [accessed 2000 August 27]
  4. Muir Gray JA, De Lusignan S. National electronic Library for Health (NeLH). BMJ 1999 Dec 4;319(7223):1476-1479 [FREE Full text] [Medline]
  5. Journal of Medical Internet Research.   URL: [accessed 2000 August 27]
  6. Eysenbach G, Sa ER, Diepgen TL. Cybermedicine. Interview by Clare Thompson. BMJ 1999 Nov 13;319(7220):1294 [FREE Full text] [Medline]
  7. Stead WW, Miller RA, Musen MA, Hersh WR. Integration and beyond: linking information from disparate sources and into workflow. J Am Med Inform Assoc 2000;7(2):135-145. [PMC] [Medline]
  8. Cochrane Reviewers' Handbook.   URL: [accessed 2001 Feb 27]
  9. Gray JAM. Hallmarks for quality of information. BMJ 1998;317:1500 [FREE Full text]
  10. HONcode.   URL: [accessed 2000 August 27]
  11. Ball B, Sackett D, Phillips B, Haynes B, Straus S. Levels of Evidence and Grades of Recommendations. 1999 Nov 18.   URL: [accessed 2000 August 27]
  12. MedPICS.   URL: [accessed 2000 August 27]
  13. MedCERTAIN.   URL: [accessed 2000 August 27]
  14. OMNI.   URL: [accessed 2000 August 27]
  15. DISCERN Project.   URL: [accessed 2000 August 27]
  16. QUICK (QUality Information ChecKlist).   URL: [accessed 2000 August 27]
  17. Medical Matrix.   URL: [accessed 2000 August 27]
  18. Internet Healthcare Coalition.   URL: [accessed 2000 August 27]
  19. Rippen H, Risk A. , for the e-Health Ethics Initiative. e-Health Code of Ethics. J Med Internet Res 2000;2(2):e9 [FREE Full text] [CrossRef]
  20. Hi-Ethics Consortium.   URL: [accessed 2000 August 27]
  21. Kiley R. Medical Information on the Internet: A Guide for Health Professionals (Book with CD-ROM). Edinburgh: W.B. Saunders Company; May 15, 1999.
  22. Eysenbach G, Diepgen TL. Towards quality management of medical information on the internet: evaluation, labelling, and filtering of information. BMJ 1998 Nov 28;317(7171):1496-1500 [FREE Full text] [Medline]
  23. BIOME.   URL: [accessed 2001 January 17]
  24. OMNI Guidelines for Resource Evaluation.   URL: [accessed 2000 August 27]
  25. Centre for Health Information Quality.   URL: [accessed 2000 August 27]
  26. NHS patient information service.   URL: [accessed 2000 August 27]
  27. Centre Hospitalier Universitaire de Rouen (CISMeF) Catalogue et Index des Sites Médicaux Francophones.   URL: [accessed 2000 August 27]
  28. Net Scoring.   URL: [accessed 2000 August 27]   URL: [accessed 2000 August 27]
  29. Winker MA, Flanagin A, Chi-lum B, White J, Andrews K, Kennett RL, et al. Guidelines for medical and health information sites on the internet: principles governing AMA web sites. American Medical Association. JAMA 2000;283(12):1600-1606 [FREE Full text] [Medline] [CrossRef]
  30. The Health Summit Working Group.   URL: [accessed 2000 August 27]
  31. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. JAMA 1997 Apr 16;277(15):1244-1245. [Medline] [CrossRef]
  32. Price J. Complete Idiot's Guide to Online Medical Resources (Complete Idiot's Guide). Indianapolis: Alpha; Mar 9, 2000.
  33. Wright CC, Whittington D, McKenna H, Ingleston LH, Moore AP, Ward K. Quality Assurance: An Introduction for Health Care Professionals. Edinburgh: Churchill Livingstone; 1992.
  34. Web Inspector.   URL: [accessed 2000 August 27]
  35. Dermatology Online Atlas (DOIA) at the University of Erlangen, Germany.   URL: [accessed 2000 August 27]
  36. Eysenbach G, Bauer J, Sager A, Bittorf A, Simon M, Diepgen T. An international dermatological image atlas on the WWW: practical use for undergraduate and continuing medical education, patient education and epidemiological research. Medinfo 1998;9 Pt 2(9 Pt 2):788-792. [Medline]
  37. Delamothe T. Quality of websites: kitemarking the west wind. BMJ 2000 Oct 7;321(7265):843-844 [FREE Full text] [Medline] [CrossRef]
  38. Social Science Information Gateway "Internet Detective".   URL: [accessed 2001 January 17]
  39. Royal Rife Research Society Web site.   URL: [accessed 2000 August 27]

AGEC: Advisory Group on Evaluation Criteria
C-H-i-Q: Centre for Health Information Quality
NeLH: National electronic Library for Health
NHS: National Health Service
NICE: National Institute for Clinical Excellence
RDF: Resource Description Framework
UMLS: Unified Medical Language System
VBL: Virtual Branch Libraries
vortals: Vertical portal targets a specific topic
XML: eXtensible Markup Language

Edited by G Eysenbach; submitted 11.09.00; peer-reviewed by M Hogarth, F McLellan; comments to author 16.01.01; accepted 23.02.01; published 17.03.01


© MN Kamel Boulos, AV Roudsari, C Gordon, JA Muir Gray. Originally published in the Journal of Medical Internet Research (, 17.3.2001. Except where otherwise noted, articles published in the Journal of Medical Internet Research are distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited, including full bibliographic details and the URL (see "please cite as" above), and this statement is included.