Published on in Vol 5, No 2 (2003):

The Evolution of Web-based Medical Information on Sore Throat: a Longitudinal Study

The Evolution of Web-based Medical Information on Sore Throat: a Longitudinal Study

The Evolution of Web-based Medical Information on Sore Throat: a Longitudinal Study

Original Paper

1Università Cattolica del Sacro Cuore, Istituto di Clinica Pediatrica, Rome, Italy

2LUISS "Guido Carli", Centro di Ricerca sui Sistemi Informativi, Rome, Italy

Corresponding Author:

Vincenzo Currò, MD

Istituto di Clinica Pediatrica

Università Cattolica del Sacro Cuore

Largo A Gemelli, 8

I-00168 Rome


Phone: +39 06 35510233

Fax:+39 06 3383211


Background: The content of a page can change and is likely to change over time; this is one of the useful qualities of the Web, but also a dangerous one.

Objective: To monitor the evolution of Web page contents on sore throat over a 3 year period.

Methods: Two medical doctors independently evaluated 34 Web pages on sore throat. Pages were found using a metasearch engine. The evaluation factors were: the adherence of medical contents to a gold standard (American Academy of Pediatrics recommendations) composed of 5 subfactors (epidemiological, clinical, complications, diagnosis, and therapy); the completeness of the contents in terms of considered/missed factors of the gold standard; references to medical literature; and a specified last update of the page. During the observation period these sites were revisited twice, after 28 and 39 months, to examine any changes therein since the first visit.

Results: The degree of adherence to the gold standard did not significantly change. Variations (both positive and negative) were recorded solely with regard to the update and references factors as well as with regard to the availability of the pages over time (18% disappeared during the observation period).

Conclusions: In 3 years medical contents have not changed significantly and despite the contemporary epochal Internet revolution (in terms of, eg, technology, graphics, and access) and the increase in the number of sites dealing with the issue of sore throat, there has been no corresponding qualitative increase in the contents of the pages monitored.

J Med Internet Res 2003;5(2):e10



The Internet is a relatively-recent phenomenon and its evolution and use by nonprofessionals have developed especially in the last 5 years. The usefulness of medical cyberspace continues to grow and links, search engines, and the presentation of information as well as users' Web-surfing skills all seem to be steadily improving. The use of the Internet is also rapidly increasing in Italian households. Our previous study carried out on a population of parents in 1999 showed that approximately 19% of an unselected sample of parents used the Internet to acquire medical information [1]. The ability to obtain medical information quickly, cheaply, and in the home represents an important development for better-informed participation in the care of children. From the point of view of pediatricians the dissemination of correct medical information to parents is an essential aspect of childcare. In our evaluation we did not consider such factors as usability and aesthetics. These areas, which are most certainly of interest in parents' consideration of a good Web page, are discussed in another study now in progress. Despite growing interest in evaluating health information on the Web, professional users say information quality is a problem [2- 6]. Eysenbach [7], in a recent systematic review [7] on what he called "infodemiology" [8] studies, i.e. studies where investigators evaluated the quality of health information on the web, reported that the greater part of studies (70%) concluded that quality is poor on the Web, while only 9% gave a positive statement.

The content of a page can change and is likely to change over time; this is one of the useful qualities of the Web, but also a dangerous one. Although many attempts have been made to assess, control, and assure the quality of Web-based medical information [9- 11], there is no single standard strategy for evaluating Web pages available to nonprofessional users such as parents. The Web is truly an international press and is essentially unregulated [12]. Although the cited authors provide relevant results, their analysis is on the status of the quality and not on its evolution over time. Since the Internet is continuously changing, our evaluation during a 3-year period could be an additional element for investigating this phenomenon. A contribution in this direction is offered by the study of Pandolfini et al [13] who replaced their study of 1997 [4] and reevaluated the quality of the same sites included in that study, as well as that of a more-recent sample of pages, using the same methodology. Li et al [14] conducted a prospective systematic review of Web sites related to back pain, using 5 search engines, during a 2-year period. In our study we carried out an evaluation of the content of medical Web sites directed to parents and we repeated the study of the same Web pages after a 24 and a 39 month period respectively.

On August 12 1998 we searched the World Wide Web for the first time using Metacrawler, a metasearch system that in 1998 combined 6 different search engines (Alta Vista, Excite, Infoseek, Lycos, Webcrawler, and Yahoo). Because of the differences among the search engines (eg, Infoseek is a random-search tool and Alta Vista ranks the requested references in order of relevance) a metasearcher combines such different approaches in a unique "independent" collection that could be linked to the way in which parents — who have variable experience, culture, and computer ability — perform the search.

We used the English words: sore throat and pharyngitis and the corresponding Italian translations: mal di gola and faringite(ie, the medical condition and its symptom, because the parents frequently use the same words for their research) without Boolean operators, as Metacrawler did not use them in the search; the results displayed 97 Web sites. As occurs in all Internet searches, some inappropriate documents appeared. We excluded all pages in which the words of our search appeared in a nonmedical context; ie, we excluded all pages created by commercial ventures and corporate sites sustained by advertising and sales of commercial products (eg, candies, oral spray, and herbal tea) as well as pages related to veterinary purposes. After the exclusion of the inappropriate sources, 34 Web sites were included in the study. These Web sites were all created by health care authorities (eg, American Medical Association and American Academy of Pediatrics), hospitals (eg, Children's Hospital of Iowa and St. James Hospital), departments of public health (eg, Hawaii, Illinois, and Bethesda), research foundations (eg, MayoClinic), or other lay health care organizations (eg, WellnessWeb, MedicineNet, HealthyNet). The list of evaluated sites is in Table 1.

To evaluate the quality of each Web site, we considered the adherence to some of the criteria suggested in literature available in 1998 on this topic: suitability of the medical contents, scientific citations, and date of creation and/or of the last update. We compared the medical contents of all these Web sites with the recommendations of the Committee on Infectious Diseases of the American Academy of Pediatrics [15], whose guidelines did not change between then and the last edition published in 2000 [16].

Score Criteria


The median score of the present factors was used as the global evaluation of adherence. Since our gold standard consists of 5 factors (epidemiological, clinical, complications, diagnosis, and therapy), each factor was evaluated according to the following scale: 1 (low, in case of errors or no adherence); 2 (medium-low adherence); 3 (medium adherence); 4 (medium-high adherence); and 5 (full adherence).


Completeness was determined according to the number of considered/missed factors of the gold standard: 1 (the worst, in case of 1 factor), 2 (bad, 2 factors), 3 (medium, 3 factors), 4 (good, 4 factors), 5 (the best, all factors).


References: 1 if the page listed some pertinent references about its contents.


Update: 1 if the page contained the date of creation (or its last update).

Textbox 1. Score Criteria

According to these criteria, the quality of the Web pages was independently evaluated by 2 medical doctors (VC, a senior pediatrician, and PSB, a resident) to validate the medical information for parents. Each doctor evaluated every Web site individually without knowing the site's address or author. When the 2 authors disagreed, to ensure the reliability of the rating they talked about the disagreement until they reached a common value.

On December 12, 2000 and on November 12, 2001 all 34 sites included in the study were revisited to see if the pages still existed or if they had been substituted with new pages. All pages were reevaluated with the same methodology, comparing the content with copies of the original pages. The missing pages were manually searched in the Web (by means of Altavista, Google, and other search engines) starting from the address and/or the contents of the previously-evaluated pages. The results were analyzed through Wilcoxon's paired signed rank test and the McNemar test.

Table 1. Sites evaluated and their respective homepage URL
OrganizationHomepage URL
Abersychan Surgery
American Academy of Pediatrics
American Medical Association
Association of State and Territorial Directors of Health Promotion and Public Health Education (ASTDHPPHE)
Baltimore County Public Schools
Beijing Scene Publishing
National Center for Emergency Medicine Informatics
Children\'s Medical Center of Dallas
Eastern Wyoming College
Group Health Cooperative
Hawaii Department of Health
HealthWorld Online
Housecall Medical Resources, Inc.
linois Department of Public Health
Kenyon College
MedicineNet, Inc.
Medscape Portals, Inc.
National Institute of Allergy and Infectious Diseases
Nightime Pediatrics Clinics, Inc.
Seton Hall University
St. James\'s Hospital
The Nemours Foundation
University of Iowa
University of Colorado at Boulder
University of Iowa Health Care
University of Missouri-Rolla
University of Rochester

During the observation period, 6 (18%) of the 34 pages monitored disappeared between August 1998 and November 2001; 3 pages were no longer available after 28 months with the remaining ones no longer active by 39 months. Table 2 summarizes the results of our survey for 1998, 2000, and 2001 and shows for each variable (adherence, completeness, update, and references) the number of Web sites for each score.

Table 2. Comparison of scores for 1998, 2000, and 2001
Score Criteriaand ScoreNumber of Web Sites
Number of sites evaluated343128

Because only 1 Web site showed a large degree of variation in terms of adherence and completeness to the gold standard, no statistical analyses were performed. Most variations relate exclusively to update and references; to evaluate their evolution, we made the Wilcoxon's paired signed rank test and the McNemar test for the pairs of years 1998-2000 and 1998-2001. The differences were not significant ( P> .05) for update and references for either pair of years. The pair 2000-2001 was not considered because there were not differences between respective values, the only difference was in the number of available sites.

If it is true that the Internet is continuously developing as regards the quantity of information available and the number of sites online, it seems that the same cannot be said for the quality of the information provided. This leads one to have doubts as to the effective greater dynamism of Web publications compared to publications printed on paper. Only 3 sites in our sample modified the pages relative to sore throat during the observation period and in 2 cases the changes did not have any significant impact on quality. With the exception of 1 site, the medical contents of the reevaluated sites remained similar to the initial version; apart from the foregoing, improvements were recorded only in aesthetics (2 sites) and a request for HON certification (1 site).

Of the 5 gold standard factors, the least common one is complication(about 60% of the sites) whereas clinical is in all sites. The factor that adheres least of all to the gold standard is therapy(72%). Overall average adherence is 81%. Only 12.5% of the sites give any indication of references. Only 33% of the sites specify the date of creation or of the last update. In our opinion it is very important for the reader to know the date the page was written; a recent date would imply possibly-better information, especially for the sites with a bad evaluation. In addition, some authors [7- 18,9] included the date of creation or update of the page among the items to consider when you want to evaluate health-related Web sites. In particular Abbott [17] included the update under the category of "content." Although this article was published after the beginning of our study, its conclusions reinforced our considerations. The low number of sites that dedicate space to complications is probably because these sites are addressed to more "impressionable" nonprofessionals (often parents) whose anxiety for their children's health dictates that it is better that matters like this are discussed personally and directly with a pediatrician. Of greater concern is the poor quality of therapeutic advice, since dealing with the issue inadequately means there is a risk of people treating themselves erroneously.

In the observation period the relevant changes were restricted to update(15 sites) and references(6 sites). An unexpected result was that the number of improved sites was the same as the number of those that became worse; hence the overall average result was the same as before. This leads one to wonder if such a trend is limited to the pages on sore throat or whether it is also applicable to other contexts.

It was not always trivial to find a missing page through the initial address or via a simple path starting from the home page. The Web administrator often made a major change in the location of the page in the site. In some cases there were no direct links to reach the new location; instead, the new page was only accessible by using a search engine In 1 case, the Web site on which the page was published ceased to exist.

Our research was limited in terms of the observation period, the number of monitored sites and the unique medical issue considered. The idea of conducting a broader and more-systematic analysis of the evolution over time of the pediatric information available to parents via a suitable "Internet observatory" is an important challenge. Unfortunately, the huge number of sites and issues to be monitored require a correspondingly huge amount of resources; nevertheless there is the need to continuously improve the efforts to provide the final user such services.

Conflicts of Interest

None declared.

  1. Currò V, Mauro V, Buonuomo PS, Mastroiacovo P, Bernabei A, D'Atri A. Accessing information on medical/health web sites by Italian families: a pilot study. Clin Pediatr (Phila) 2000 Jan;39(1):68-69.
  2. Pandolfini C, Impicciatore P, Bonati M. Parents on the web: risks for quality management of cough in children. Pediatrics 2000 Jan;105(1):e1. [Medline] [CrossRef]
  3. Mcclung HJ, Murray RD, Heitlinger LA. The Internet as a source for current patient information. Pediatrics 1998 Jun;101(6):e2 [FREE Full text] [Medline] [CrossRef]
  4. Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the World Wide Web: systematic survey of advice on managing fever in children at home. BMJ 1997 Jun 28;314(7098):1875-1879 [FREE Full text] [Medline]
  5. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA 2001 May 23;285(20):2612-2621. [Medline] [CrossRef]
  6. Griffiths KM, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ 2000 Dec 16;321(7275):1511-1515 [FREE Full text] [PMC] [Medline] [CrossRef]
  7. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002 May 22;287(20):2691-2700. [Medline] [CrossRef]
  8. Eysenbach G. Infodemiology: The epidemiology of (mis)information. Am J Med 2002 Dec 15;113(9):763-765. [Medline] [CrossRef]
  9. Eysenbach G, Diepgen TL. Labeling and filtering of medical information on the Internet. Methods Inf Med 1999 Jun;38(2):80-88. [Medline]
  10. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. JAMA 1997 Apr 16;277(15):1244-1245. [Medline] [CrossRef]
  11. Winker MA, Flanagin A, Chi-lum B, White J, Andrews K, Kennett RL, et al. Guidelines for medical and health information sites on the Internet: principles governing AMA web sites. JAMA 2000 Mar 22;283(12):1600-1606. [Medline] [CrossRef]
  12. Terry NP. Rating the raters: legal exposure of trustmark authorities in the context of consumer health informatics. J Med Internet Res 2000 Sep 19;2(3):e18 [FREE Full text] [Medline] [CrossRef]
  13. Pandolfini C, Bonati M. Follow up of quality of public oriented health information on the world wide web: systematic re-evaluation. BMJ 2002 Mar 9;324(7337):582-583 [FREE Full text] [PMC] [Medline] [CrossRef]
  14. Li L, Irvin E, Guzmán J, Bombardier C. Surfing for back pain patients: the nature and quality of back pain information on the Internet. Spine 2001 Mar 1;26(5):545-557. [Medline] [CrossRef]
  15. Peter G, editor. Group A streptococcal infections. In: 1997 Red Book Report of the Committee on Infectious Diseases. Elk Grove Village, IL: Rittenhouse Book Distributors; May 1, 1997:483-494.
  16. Pickering LK, editor. Group A streptococcal infections. In: 1997 Red Book Report of the Committee on Infectious Diseases. Elk Grove Village, IL: American Academy of Pediatrics; Jun 15, 2000:526-536.
  17. Abbott VP. Web page quality: can we measure it and what do we find? A report of exploratory findings. J Public Health Med 2000 Jun;22(2):191-197. [Medline] [CrossRef]
  18. ; Centrale Santé. NetScoring®: criteria to assess the quality of health Internet information. 2000 Jan.   URL: [accessed 2003 Mar 28]

Edited by G Eysenbach; submitted 06.09.02; peer-reviewed by LC Li; comments to author 08.11.02; revised version received 17.12.02; accepted 17.03.03; published 13.06.03


© Vincenzo Currò, Paola Sabrina Buonuomo, Paola De Rose, Roberta Onesimo, Andrea Vituzzi, Alessandro D'Atri. Originally published in the Journal of Medical Internet Research (, 13.6.2003. Except where otherwise noted, articles published in the Journal of Medical Internet Research are distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited, including full bibliographic details and the URL (see "please cite as" above), and this statement is included.