Published on 25.06.04 in Vol 6, No 2 (2004)
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/45, first published Jan 27, 2004.
Do the Leading Children's Hospitals have Quality Web Sites? A Description of Children's Hospital Web Sites
Background: Although leading children's hospitals are recognized as preeminent in the provision of health care to children, the quality of their Web sites has not been described.
Objective: To describe technical characteristics of the Web sites of leading children's hospitals.
Methods: This is a cross-sectional descriptive infodemiology study. Two reviewers independently reviewed and analyzed the Web sites of 26 nationally prominent children's hospitals in June 2003, using objective criteria based on accessibility (based on age and language), attribution, completeness, credibility, currency, disclosure, readability, and other technical elements.
Results: One-third of Web sites included content for children and adolescents. Twenty-four (92%) of the Web sites had health and disease-specific information. One-third contained only English, while two-thirds included other languages. All 26 Web sites included a disclaimer, although none had a requirement to read the disclaimer before accessing health and disease specific information. Twenty-four (92%) had search options. Although most (85%) listed a copyright date, only 10% listed the date last updated.
Conclusions: This is the first study to examine the Web sites of leading children's hospitals. Although the Web sites were designed for children's hospitals, only a few sites included content for children and adolescents. Primary care physicians who refer patients to these sites should be aware that many have limited content for children, and should assess them for other limitations, such as inconsistent documentation of disclaimers or failure to show the date of the last Web site update. These Web sites are a potentially useful source of patient information. However, as the public increasingly looks to the Internet for health information, children's hospitals need to keep up with increasingly high standards and demands of health-care consumers.
J Med Internet Res 2004;6(2):e20
- World Wide Web;
- children's hospitals;
- health information;
- quality indicators;
- health care
Although leading children's hospitals are recognized as preeminent in the provision of health care to children, the quality of their Web sites has not been described. Providers may be interested in referring parents and patients to the Internet for pediatric information and may look to the leading children's hospitals as a source. In this paper we seek to describe technical and content characteristics of the Web sites of leading children's hospitals.
The World Wide Web is becoming a popular source of health information for patients . A general rule for selecting an online source for health information is to "find a Web site that has a person, institution or organization in which you already have confidence" [ ].
The public and medical providers recognize leading, tertiary care, teaching hospitals as credible sources of information . Many of these institutions include children's hospitals. Because the leading children's hospitals in the United States are commonly held in high regard, a parent or patient might expect that they would also be reasonable sources of online health information. Our findings suggest that such academic Web sites may disappoint [ ].
It is not clear if the best children's hospitals that provide high quality care also have Web sites that provide high quality access and content. Although numerous systems for rating the quality of health information on the Internet have been developed [- ], to our knowledge, there has been no reported evaluation specifically assessing the Web sites of the leading children's hospitals. The criteria used in this study to assess quality involved the domains of accessibility, attribution, credibility, currency, and disclosure, and other Web site elements. The purpose of this paper is to describe the technical characteristics of these Web sites, in terms of quality and content, for the leading children's hospitals.
Our research questions are the following: Do children's hospitals that are considered to provide high quality care also have Web sites that provide high quality access and content? What are the technical characteristics of the Web sites of the leading children's hospitals?
We conducted a cross-sectional descriptive infodemiology analysis of the Web sites of the prominent children's hospitals in the United States. We selected 26 children's hospitals based on the 2002 United States News and World Report and the 2002 Child magazine rankings of the leading children's hospitals.
Although there are many methods for selecting leading medical institutions and children's hospitals, medical providers and the public are influenced by the United States News and World Report ranking of "America's Best Hospitals" [- ]. Another rating system, specific to pediatric hospitals, is published in Child magazine. For this study, we selected all 23 hospitals listed as leading children's hospitals from US News and World Report and all 10 leading pediatric hospitals from Child magazine. Together, these represent 26 distinct Web sites. The Internet addresses of these hospitals were published in the US News and World Report online. However, as these Web sites were not always specifically referring to the pediatric hospital, but rather to the parent medical center, reviewers searched for the correct address on the parent medical center's site, or by entering the hospital name into Google if a hospital was listed only in Child magazine ( ).
The US News and World Report list has been published and updated every year since 1990, and is the longest running annual ranking of hospital quality . The list also represents a common source for parents when finding medical information on the Internet [ ]. US News and World Report ranks hospitals in pediatrics based on reputation [ ]. The "America's Best Hospitals" methodology was devised in 1993 by the National Opinion Research Center at the University of Chicago [ ].
Child magazine has also published a list of leading children's hospitals that are full members of the National Association of Children's Hospitals and Related Institutions. Child first selects hospitals that received a score of at least 93 (91 in some circumstances) by the Joint Commission on Accreditation of Healthcare Organizations (JCAHO). These hospitals then complete a survey developed by Child advisory board members to identify the leading 10 children's hospitals .
Although these selected institutions are acknowledged as leaders, their Web sites are not necessarily the most popular (eg, as defined by the number of backlinks or a ranking in search engines such as Google). Our selection method assumes that people who are familiar with the non-Web reputations of these institutions may directly look up these institutions' Web sites, but they may not think critically about whether the sites are as reputable as the institutions themselves.
Two of the researchers (TK, MDC) independently reviewed each Web site using a set of objective criteria pre-determined by the authors. These included criteria in the domains of accessibility, attribution, credibility, currency, and disclosure, and other Web site elements. Specifically, we determined the presence or absence of the following: child-focused content and links for children, bilingual or multilingual content, health or disease specific information, references for medical information, posting of a "last update" and copyright date, an internal search engine, disclaimer and requirement to read it, option to make purchases or donations, and advertisements. For Web sites that included disease-specific information, we selected a basic text passage about asthma, and determined the readability using the Flesch-Kincaid Grade Level method, a commonly used computerized software program for scoring readability that is embedded in Microsoft Word .
Eysenbach et al have described five different types of criteria to evaluate the quality of a Web site . These include technical characteristics, readability, design, accuracy, and completeness. To evaluate the Web sites, we included technical characteristics, readability, and completeness criteria. We did not include criteria based on Web site design, since previous studies have reported kappa scores of only 0.08 and 0.23 [ ]. In addition, design criteria might not be valid for an analysis of these Web sites, since the pages might be designed for children. Since not all the Web sites offered disease-specific information, we did not include criteria for accuracy.
Data were abstracted from June 1, 2003 to June 30, 2003. Differences in classification were resolved by another reviewer (KLW or BR). We calculated kappa statistics for the dichotomous categories to describe the agreement in the initial classification of each of the characteristics. Simple counts and descriptive statistics are presented to describe the frequency of these characteristics on each hospital's Web site.
The hospital rankings from US News and World Report (n=23 hospitals) and from Child magazine (n=10 hospitals) are listed in, along with their Internet addresses. Combined, the two lists included a total of 26 hospitals. Seven hospitals appeared on both lists. All 26 leading hospitals in the initial sample had Web sites specific to pediatrics or to the children's hospital.
|Hospital||Web Site Address as Listed By US News||Pediatric or Children\'s Hospital Web Site Address|
|Children\'s Hospital Boston||childrenshospital.org||Same|
|Children\'s Hospital of Philadelphia||chop.edu||Same|
|Johns Hopkins Hospital||hopkinsmedicine.org||Hopkinschildrens.org|
|Children\'s Hospital, Denver||tchden.org||Same, also thechildrenshospital.org|
|Children\'s Hospital of NY Presbyterian||nyp.org||childrensnyp.org|
|Children\'s Hospital of Pittsburgh||chp.edu||Same|
|University Hospitals of Cleveland||uhhs.com||rainbowbabies.org|
|Texas Children\'s Hospital, Houston||txchildrens.org||Same|
|Children\'s Hospital Medical Center, Cincinnati||cincinnatichildrens.org||Same|
|Children\'s Memorial Hospital, Chicago||childrensmemorial.org||Same|
|Children\'s Hospital, Los Angeles||childrenshospitalla.org||Same|
|University of California, San Francisco Medical Center||ucsfhealth.org||ucsfhealth.org/childrens/index.html|
|UCLA (Mattel Children\'s Center)||healthcare.ucla.edu||peds.ucla.edu|
|Massachusetts General Hospital||mgh.harvard.edu||massgeneral.org/mghfc/|
|Lucile Packard Children\'s Hospital (Stanford)||stanfordhospital.org||lpch.org|
|Children\'s National Medical Center, DC||dcchildrens.com||Same|
|Children\'s Hospital and Medical Center, Seattle||seattlechildrens.org||Same|
|Duke University Medical Center||dukehealth.org||dukehealth.org/health_services/childrens_health.asp|
|Miami Children\'s Hospital||mch.org||Same|
|Yale-New Haven Hospital||ynhh.org||ynhh.org/ynhch/ynhch.html|
|University of Michigan Hospitals||med.umich.edu||med.umich.edu/mott|
|St. Christopher\'s Hospital, Philadelphia||stchristophershospital.com||Same|
|St Louis Children\'s Hospital||n/a||stlouischildrens.org|
|Children\'s Mercy Hospital, Kansas City||n/a||childrens-mercy.org|
|Primary Children\'s Medical Center, Salt Lake City||n/a||ihc.com/xp/ihc/primary|
* all addresses in this table have URLs (Uniform Resource Locators) prefixed with http//:www(HyperText Transfer Protocol; World Wide Web). n/a = not applicable, ie, the hospital was only listed in Child magazine, which did not list the URL
We measured accessibility of the Web sites for children, based on whether or not the Web site included information for children or recommended links. Although the Web sites were designed for children's hospitals, only one-third included content for children and adolescents. Accessibility was also examined with regard to multilingual content. One-third of the Web sites contained only English, while two-thirds included other languages.
In terms of completeness, 92% provided health or disease-specific information. Two-thirds (65%) provided additional or recommended Web sites. With respect to technical features, 92% of the sites allowed the user the option of searching the site.
All Web sites provided information about making a donation to the hospital; however, only one site (4%) had advertisements for organizations or companies other than the hospital itself .
Kappa statistic calculations revealed that the agreement between the two reviewers exceeded expected agreement for all variables assessed. Kappa ranged from 0.24 to 1.00.
|Domain:||Does the Web site have||n (%)||Κ|
|Accessibility (children)||content for children (educational or non-educational games)?||10 (38.5)||0.53|
|Accessibility (children)||recommended links for children?||9 (34.6)||0.49|
|Accessibility (teens)||recommended links or content for teens?||10 (38.5)||0.65|
|Accessibility (language)||English language only (no other languages)?||9 (34.6)||0.24|
|Attribution, Completeness||recommended links or resources for more information?||17 (65.4)||0.35|
|Completeness||Health or disease specific info?||24 (92.3)||0.34|
|Credibility, Conflict of Interest||purchase or donation option?||26 (100)||1.00|
|Credibility, Conflict of interest||advertisements other than for hospital itself?||1 (3.9)||0.47|
|Currency||copyright date on main (home) page?||22 (84.6)||0.90|
|Currency||has date last updated on main (home) page?||2(7.7)||0.34|
|Disclosure, Accessibility||requirement to read disclaimer prior to accessing health information?||0 (0)||n/a|
|Disclosure, Accessibility||requirement to log in prior to accessing health information?||0 (0)||n/a|
|Readability||8th grade or lower readability for disease specific info (asthma)?||8 of 21 (38.1)||n/a|
|Technical Features||search option?||24 (92.3)||1.00|
|Technical Features||option to email child or join an online community?||14 (53.8)||0.45|
* Kappa score, reflecting the agreement between the two raters. 1.0 represents perfect agreement.
This is the first study to examine the Web sites of the leading children's hospitals. Although all the commonly recognized leading children's hospitals have their own Web sites, style and content vary. Many of the Web sites lacked information for children. We also found that access to many sites was limited by the reading level and the language(s) in which the information was offered. In addition, although many had disease-specific information, the currency of such information was not described.
Given increasing use of the Internet as a source for health information by parents and patients , we expected that most of the Web sites for the leading children's hospitals would include pediatric health information, especially educational content intended specifically for children. However, this study shows that although the Web sites created by hospitals are dedicated to children, only one-third have information specifically for such an audience. Internet users with children (ie, parents), and pediatric providers who refer children to these sites for educational content would be disappointed by most of the sites.
In addition to being a useful and trusted source of patient information, these sites can easily guide and potentially link the parent or patient to information about a specialist at the hospital. As a result, children's hospitals are in a unique position to provide disease-specific information on the Internet, and theoretically may be more useful to health-care consumers than government sites (eg, Centers for Disease Control and Prevention or the National Institutes of Health) or private organization sites (eg, the American Lung Association), which traditionally do not contain links to providers or centers for care. By recommending high quality Web sites, pediatricians and other providers can assist parents and patients in becoming more involved in their own care and in learning about their health .
However, providing disease-specific information requires that such information be updated regularly. One reason leading children's hospitals might be considered "top" is because they remain current, on the cutting edge of medical research and technology. Because Web sites can easily be updated, users probably assume that information on the Internet is up-to-date .
Yet as the results of this study show, fewer than 10% of the Web sites of the leading children's hospitals assessed in this study posted the date the site was last updated on their home pages. It is not clear to the Web site audience how current the information is. Out-of-date information can contribute to inaccurate patient information. For example, McClung reviewed Internet sources regarding the treatment of childhood diarrhea and found that only 20% of sites, including those of traditional medical institutions, had information consistent with the most recent American Academy of Pediatrics guidelines for the management of acute diarrhea . Web sites should post the date of the most recent update.
Accessibility of the Web site was also measured in terms of language. The children's hospitals we surveyed were from different parts of the country, with different populations to serve. Although we only considered Web sites from the leading children's hospitals in the United States where English is the primary language, two-thirds of the Web sites did include languages other than English. This is a commendable effort on the part of the hospitals to reach out to their non-English-speaking patients. The differences in language availability may reflect the differences in the population of patients served by each hospital.
There are several limitations to this study. Although most of the Web sites in this study contain health and disease-specific information, we did not evaluate the accuracy of this information. Not all sites contained disease-specific information, and some of the sites had disease-specific content that did not differ from content on other sites, as it was purchased from a third party. Nonetheless, future investigation of disease-specific content would be necessary to evaluate on this criterion.
In addition, the low kappa scores for certain variables in this study may relate to Web site design. Because we were evaluating Web sites as opposed to specific Web pages, the range in kappas may reflect the differences in the ability to find the specific information among the different Web pages at one hospital Web site.
Another limitation of this study is that, although we evaluated the Web sites whose target audience is public, this was not a natural experiment using actual consumers of Internet-based pediatric health information. Further research can clarify how parents, for example, use the Internet for health information.
There are many criteria upon which a Web site can be evaluated. Our study did not ask whether sites had "contact us" information, which would attest to the accountability of the site. In addition, information on Web team composition would assist the user in learning who specifically authored the site. Although we did not include all the possible domains upon which a Web site can be evaluated, we chose several that are relevant to the pediatric community as well as those that are commonly employed in literature reports of Web site evaluation [- ].
This is the first study to examine the Web sites of the leading children's hospitals. Surprisingly, only one-third had links or content for either children or adolescents. All had disclaimers but none required users to read the disclaimer. Almost all of the Web sites contained health and disease-specific information, and many had multilingual information on their sites. The Web sites of the leading children's hospitals are a potentially useful source of patient information for primary care physicians to offer to their patients. However, this study indicates that the current Web sites of children's hospitals have several limitations.
This study suggests methods to improve Web sites for children's hospitals. Specifically, those responsible for such Web sites could provide educational content for children or provide quality links, as well as improve the readability levels of their content. In terms of technical features, Web sites should describe and maintain the currency of the information on their sites, and maintain appropriate disclaimers with adequate prompting of users to read such disclaimers. Finally, based on the population that the children's hospital serves, the Web site should provide reasonable multilingual options.
For providers interested in referring parents and patients to the Internet for pediatric information, this study demonstrates variability with respect to the leading children's hospital Web sites. These sites could be potential sources of additional information and patient education; however, providers should examine the extent that the Web sites they recommend meet the above quality criteria. As the public increasingly looks to the Internet for more health information, children's hospitals need to keep up with the increasing standards and demands of health-care consumers.
Conflicts of Interest
The authors are either employed by or students at two of the children's hospitals discussed in the current study.
- Fox S. The online health care revolution: How the web helps Americans take better care of themselves. Washington, DC: The Pew Internet & American Life Project; Nov 26, 2000. URL: http://www.pewinternet.org/pdfs/PIP_Health_Report.pdf
- ; Internet Healthcare Coalition. Tips for healthy surfing online: finding quality health information on the Internet. URL: http://www.ihealthcoalition.org/ [accessed 2003 May 15]
- Boscarino JA. The public's perception of quality hospitals II: Implications for patient surveys. Hosp Health Serv Adm 1992;37(1):13-35. [Medline]
- Murray PJ. Click here - and be disappointed? Evaluating Web sites. Comput Nurs 1996;14(5):260-261.
- Jadad AR, Gagliardi A. Rating health information on the Internet: navigating to knowledge or to Babel? JAMA 1998 Feb 25;279(8):611-614. [Medline] [CrossRef]
- Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related web sites: review. BMJ 1999 Mar 6;318(7184):647-649 [FREE Full text] [PMC] [Medline]
- Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. JAMA 1997 Apr 16;277(15):1244-1245. [Medline] [CrossRef]
- Wyatt JC. Commentary: measuring quality and impact of the World Wide Web. BMJ 1997 Jun 28;314(7098):1879-1881. [Medline]
- Chen J, Radford MJ, Wang Y, Marciniak TA, Krumholz HM. Do "America's Best Hospitals" perform better for acute myocardial infarction? N Engl J Med 1999 Jan 28;340(4):286-292 [FREE Full text] [Medline] [CrossRef]
- Green J, Wintfeld N, Krasner M, Wells C. In search of America's best hospitals. The promise and reality of quality assessment. JAMA 1997 Apr 9;277(14):1152-1155. [Medline] [CrossRef]
- Rosenthal GE, Chren MM, Lasek RJ, Landefeld CS. The annual guide to "America's best hospitals". Evidence of influence among health care leaders. J Gen Intern Med 1996 Jun;11(6):366-369. [Medline]
- Comarow A. Best hospitals: Methodology. URL: http://www.usnews.com/usnews/nycu/health/hosptl/methodology.htm [accessed 2003 May 15]
- O'Muircheartaigh C, Murphy J, Moore W. The 2002 index of hospital quality. National Organization for Research at the University of Chicago. 2002. URL: http://www.norc.org/new/IHQ2002.PDF
- Sangiorgio MP. The Best Children's Hospitals: Child's exclusive survey of the top pediatric care facilities in the US. URL: http://www.child.com/kids/health_nutrition/top_hospitals03.jsp?page=14 [accessed 2003 May 15]
- ; Institute for Simulation and Training. Software to measure readability comes from IST researcher's formula. URL: http://www.ist.ucf.edu/nlarchive/jan01.htm#soft [accessed 2003 May 15]
- Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287(20):2691-2700. [Medline] [CrossRef]
- Powell JA, Darvell M, Gray JAM. The doctor, the patient and the world-wide web: how the internet is changing healthcare. J R Soc Med 2003 Feb;96(2):74-76. [Medline] [CrossRef]
- Mcclung HJ, Murray RD, Heitlinger LA. The Internet as a source for current patient information. Pediatrics 1998 Jun;101(6):e2 [FREE Full text] [Medline] [CrossRef]
- ; Hi-Ethics, Inc. Health Internet ethics: Ethical principles for offering Internet health services to consumers. URL: http://www.hiethics.com/Principles/index.asp [accessed 2003 Jul 15]
- ; US Code Collection. Children's Online Privacy Protection. Legal Information Institute (15 U.S.C. 6501-6506, P.L. No 105-277, 122 Stat 2681-728). URL: http://www4.law.cornell.edu/uscode/15/ch91.html [accessed 2003 Jul 15]
- Dyer KA. Ethical challenges of medicine and health on the Internet: a review. J Med Internet Res 2001 Jun 28;3(2):e23 [FREE Full text] [Medline] [CrossRef]
- Rodrigues R. Ethical and legal issues in interactive health communications: a call for international cooperation. J Med Internet Res 2000 Mar 31;2(1):e8 [FREE Full text] [Medline] [CrossRef]
- Crocco AG, Villasis-keever M, Jadad AR. Analysis of cases of harm associated with use of health information on the internet. JAMA 2002 Jun 5;287(21):2869-2871. [Medline] [CrossRef]
Edited by G. Eysenbach; submitted 27.01.04; peer-reviewed by C Lehmann, A Spooner; comments to author 18.02.04; revised version received 19.03.04; accepted 15.04.04; published 25.06.04
© Terry Kind, Kathryn L Wheeler, Byanqa Robinson, Michael D Cabana. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 25.6.2004. Except where otherwise noted, articles published in the Journal of Medical Internet Research are distributed under the terms of the Creative Commons Attribution License (http://www.creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited, including full bibliographic details and the URL (see "please cite as" above), and this statement is included.