Adults with low literacy may encounter informational obstacles on the Internet when searching for health information, in part because most health Web sites require at least a high-school reading proficiency for optimal access.
The purpose of this study was to 1) determine how low-literacy adults independently access and evaluate health information on the Internet, 2) identify challenges and areas of proficiency in the Internet-searching skills of low-literacy adults.
Subjects (n=8) were enrolled in a reading assistance program at Bidwell Training Center in Pittsburgh, PA, and read at a 3rd to 8th grade level. Subjects conducted self-directed Internet searches for designated health topics while utilizing a think-aloud protocol. Subjects' keystrokes and comments were recorded using Camtasia Studio screen-capture software. The search terms used to find health information, the amount of time spent on each Web site, the number of Web sites accessed, the reading level of Web sites accessed, and the responses of subjects to questionnaires were assessed.
Subjects collectively answered 8 out of 24 questions correctly. Seven out of 8 subjects selected "sponsored sites"-paid Web advertisements-over search engine-generated links when answering health questions. On average, subjects accessed health Web sites written at or above a 10th grade reading level. Standard methodologies used for measuring health literacy and for promoting subjects to verbalize responses to Web-site form and content had limited utility in this population.
This study demonstrates that Web health information requires a reading level that prohibits optimal access by some low-literacy adults. These results highlight the low-literacy adult population as a potential audience for Web health information, and indicate some areas of difficulty that these individuals face when using the Internet and health Web sites to find information on specific health topics.
Although a tremendous volume of educational health materials is disseminated in the United States, not all Americans find this information accessible or usable. In particular, adults with poor health and low functional literacy face great risks of poor health outcomes and preventable disease progression [
The expense of poor health and low functional literacy on the health system is estimated at $73 billion each year [
However, the low-literacy population has largely been excluded from the veritable boom of Internet health resources. The expense of Internet services and personal computers may be too high for this population. In addition, most text-based health information on the Internet is too advanced to be optimally effective for low-literacy populations. On average, Internet health-education materials are written at a 10th grade or higher reading level, and 100% of English-language health Web sites examined in a 2001 study required at least high school-level reading proficiency [
While no studies to date have determined how many low-literacy adults regularly use the Internet to find health information, the dearth of educational materials suitable for these individuals may impair optimal usage and navigation. One study has reported interventions that enabled low-income HIV-positive individuals to use the Internet and to critically evaluate information that they encountered [
We conducted an observational study of low-literacy adults to assess how they searched for Internet health information in as close to a natural setting as possible. Our investigative questions include the following: if low- to mid-level literacy adults are given access to the Internet, can they find basic health information that they can understand? Will their search strategies be effective in identifying information that they can use and comprehend? How do they rate current health Web sites in relation to their needs and interests? Will they be able to conduct successful self-directed searches? In our investigation, we also categorized navigational strategies used by low-literacy adults and the reading level of materials they accessed.
We enrolled 13 adult literacy students (3rd to 8th grade reading levels) from Bidwell Training Center, a vocational school in Pittsburgh, Pennsylvania. The protocol used was approved by the University of Pittsburgh Institutional Review Board. Bidwell students are organized individually and/or in small groups for reading instruction; they meet together once a week for program announcements. The literacy program coordinator introduced the study to 20 students in this large-group setting. Thirteen interested students then self-selected into the study. All subjects participated in a computer skills workshop in May 2003, where they were presented with basic search and navigation strategies and learned how to use the Google search engine. We selected Google because it is a widely used search engine and has a "Did You Mean…" feature that corrects misspelled search terms. We anticipated that this might be a feature that low-literacy subjects would find particularly helpful. Among other topics, subjects were taught how to use the "Back" button and the "Forward" button, how to scroll down a page, how to identify links, and how to conduct basic searches. Each subject also filled out a brief questionnaire to give insight on their educational background, ethnicity, health insurance status, and previous experience with computers and the Internet. The questionnaire was written at a 3rd grade reading level (Flesch-Kincaid Reading Scale).
An investigator met individually with each of the participants within 3 weeks of the computer skills course for the observational portion of the study. Participants were 1) administered the REALM test (Rapid Estimate of Adult Literacy in Medicine) [
The investigator then asked the participant to use the Internet and Google search engine and think aloud while finding information on a subject of his or her choice. This preliminary question allowed participants to practice and review their Internet searching techniques. Participants were permitted to ask the investigator technical and navigation-related questions during this part of the study. These questions included, but were not limited to, whether to put spaces between words in search terms and how to initiate a search once a search term had been specified.
Participants were then asked to find answers on the Internet to 3 health-related questions developed by members of the research team. Participants were instructed to use the Google search engine so that their answers could be standardized. A committee consisting of a physician, a faculty member specializing in human-computer interactions, a community health educator, and an information sciences specialist compiled various answers to these questions that would qualify as accurate and complete. Subjects who were able to generate any of these answers during their online searches were considered to have answered the questions correctly; subjects who were not able to generate these answers were determined to have answered the question either incorrectly or incompletely. Examples of responses for each question that would have been considered correct are included in the
The investigator read the 3 questions aloud and also provided them to the participant in written form (Arial font, 20 pt):
1. Think of a health question you are interested in for yourself or for someone you know. Find out information about this question on the Internet.
2. Imagine that someone you care about has lung cancer. This person would like to know about treatments for lung cancer. Can you find out the three main types of treatments using the Internet?
3. Imagine that you are at a doctor's office and you are told you have a disease called diabetes (sometimes called sugar). You are given a pill called Metformin to take for it. What does Metformin do?
Subjects identified answers to the investigator, who then asked them to articulate the answers in their own words. Participants who seemed frustrated or unreceptive, or who asked to move to a new question were directed to the next task. Participants were allowed to use any Web sites they felt would help them answer the questions. Participants also were not provided with dictionaries-our objective was to examine how they navigated the Internet without assistance from external sources. Subjects were given up to 15 minutes to complete each task, as measured by the investigator. To minimize anxiety, they were not informed of the time limitation. After the 15-minute period, investigators used a series of prompts to gradually guide subjects, if necessary, to the next task.
Next, investigators accessed the colon and rectum cancer Web page on the American Cancer Society (ACS) Web site [
Investigators wrote notes on each participant's progress, and asked for participant feedback about the Internet both before and after searching the Internet. Investigators did not coach subjects on proper technical or navigational techniques after the initial practice question until subjects had completed their tasks. In 2 cases, investigators directed subjects to Google's "Did You Mean…" search term correction option in order to adjust for spelling mistakes; these subjects had repeatedly demonstrated very poor spelling proficiency before this intervention.
Camtasia Studio screen-capture software recorded individual keystrokes and think-aloud recordings. Questionnaires and think-aloud methods were used to ascertain the criteria used by participants in evaluating Internet health Web sites. Investigators also calculated the 1) literacy levels of Web sites accessed by the participants, 2) the amount of time spent on each Web site, 3) the number of questions answered thoroughly and correctly by each participant according to pre-determined standards, 4) the average number of sites used to answer each question, and 5) the number of participants who accessed sponsored sites, or paid advertisements appearing on the Google retrievals page, and how many used that information to answer questions.
Qualitative and quantitative results were analyzed in this study.
In this study, the subject population was reduced from 13 to 8. Two participants were excluded because they did not attend the one-on-one searching session with the investigator. Two other participants were excluded because they were non-native English speakers who did not understand the tasks presented to them. One participant was later excluded because technical problems prohibited the retrieval of her computer searches.
The average age of our 8 remaining participants was 41.5 years. Five subjects were male and 3 were female. Seven identified themselves as African Americans and 1 self-identified as of Asian descent. The Asian participant was an English-as-a-second language (ESL) speaker with a university education from his native country. Seven of the 8 participants reported having health insurance. Seven of the 8 also had at least some high school or trade school education; 1 participant did not report educational experience on the intake questionnaire.
Of these subjects, 2 reported on the intake questionnaire that they had never previously used a computer or the Internet. Two reported that they had previously used a computer, but had not used the Internet. Subjects generally used computers with greater frequency than they used the Internet. Three participants reported on the questionnaire that they used the Internet 2 or more times a week; they later said verbalized that their main online interests were news, sports, cars, and/or entertainment information. The other 5 participants reported on the questionnaire that they used the Internet either occasionally or not at all. Usage reports from the intake questionnaires are provided in
Self-reported, written questionnaire responses about prior Internet and computer usage by subjects (n=8)
Subject |
|
|
|
|
1 | No | "No where" [sic] | No | "I've never used the Internet" |
2 | Less than once a month | "When I was in jail" | No | (N/A) |
3 | 2 or more times a week | "At school, Bidwell Training Center in Ms. Cooper's class." | No | "At the Carnegie Library in Beechview where I live" |
4 | No | "No" | No | "No" |
5 | 2 or more times a week | "To type" | Yes; Less than once a month | "In school" |
6 | No; 2 or more times a week | "At home" | Once a week | "At home" |
7 | 2 or more times a week | "Home" | 2 or more times a week (at home) | "Home" |
8 | Once a week | "Different location" | 2 or more times a week | "Different location" |
As seen in
Participants reviewed their navigational skills during their preliminary question, where they were encouraged to look for information on any subject that interested them. They used Google to search for a variety of topics, ranging from entertainment to health-related information. Participants occasionally searched for information on more than one topic.
Participants used the search items listed in
Search terms used by subjects to answer preliminary questions and questions 1 to 3 (n=8)
Subject | Preliminary | Question 1 | Question 2 | Question 3 |
1 | lena horn |
Lung cancer | Lung cancer | Metformin |
2 | health care;health care mental | Sports and health | health care about lung cancer | A pill called metformin |
3 |
|
Herpes | Cancer | Metformin |
4 | Wwwsoulfood; wwwsoulfoodcom; soulfood | AIDS | lung caner |
Diabetes |
5 | Will Smith; sipers |
High blood | Lung cancer | Metformin |
6 |
Bi;;;dwell training center |
Health | health lung caner |
Health diabetes |
7 | sonny Rollins | Tuberculosis | Treatments for lung cancer | Metformin |
8 | Babyface recording artist | Pain | Cancer | Pdr |
* Physicians' Desk Reference
** English-as-a Second Language subject
† misspellings for: "lena horne," "bidwell training center," "lung cancer," and "spiders"; the Google correction option was used in two instances when the subject was prompted by investigators to amend search terms.
Questions 1 to 3 were given to our participants in writing, as well as orally; this may have affected their selection of search terms. For question 2, one participant wrote "treatments for lung cancer" in the search term box, a phrase that is written explicitly in that question. Another participant was similarly prompted by the wording of question 3 to write "a pill called metformin" as his search term.
Individuals who used the Internet at least once a week are labeled in
In general, this group found generating original search terms to be somewhat challenging. Many did not initially remember whether to put spaces between the words in search terms. Even a subject who reported using the Internet once a week hesitated when writing the search term for question 1, finally stating, "
Spelling of search terms was generally a problem for only 2 participants, one of whom (subject 7) spoke English as a second language. Subjects tended to self-correct for spelling in the search term box before pressing the "Google Search" button or Enter key. Several participants also had difficulty understanding what type of terms to put in. When conducting a preliminary search for information on the television show,
Nearly all participants retained skills such as scrolling and clicking on links from the computer workshop or previous Internet experiences. They also learned other navigational strategies through repetition and practice. For example, one participant who was conducting a preliminary search for information about Will Smith looked at the Google retrievals and stated, "
Six of the 8 participants did not venture past page 1 of the Google retrievals. One participant was surprised by the number of search results, saying, "
In question 1, participants were asked to use information on the Internet to find the answer to a health-related query of their choice. Most participants identified only a subject area, and did not clearly articulate a specific question despite verbal prompting by the investigators. Several participants initially stated a topic, but changed it as they retrieved unrelated material that they found more interesting. While recordings from the think-alouds would have been helpful in designating the search topics, we found that despite investigators' prompts and encouragement, subjects were very reluctant to verbally report their real-time experiences navigating through the Web. As one subject stated, "
Question 2 required participants to locate the 3 main types of lung cancer treatments (acceptable answers: chemotherapy, surgery, radiation). This question models the navigation of a typical Internet health-information seeker who searches for disease-related information. Of all 8 participants, only subject 5 was able to answer this question accurately and completely. Subject 3 verbalized one viable option-chemotherapy-based on information accessed online. The remaining participants either did not answer the question or identified an alternative medicine as one of the principal types of lung cancer treatments available.
Question 3 required participants to find out the role of metformin, or Glucophage, in diabetes treatment (one acceptable answer: metformin lowers sugar in the blood). This question models a doctor-patient interaction in which a patient who is prescribed an unfamiliar medication independently searches for information about its effects. Six of 8 participants were unable to find information on the Internet to answer the question. The 2 participants, subjects 3 and 7, who found the information, read directly from text on the site and did not articulate the information in their own words.
Surprisingly, subjects who reported sustained prior Internet experience in the questionnaire were no more successful at answering questions than subjects with little Internet experience. This could have been a result of the generalized search terms that they used to look for answers. Prior Internet experience does not seem to lead to satisfactory search/navigation skills for members of this group in searching for health information.
Sites used by subjects 3, 5, and 7 to successfully answer questions 2 and 3 were written at a 12th grade reading level (Flesch-Kincaid). It is noteworthy that these subjects were able to identify the answer in the text and read it aloud. In 2 out of 3 cases, they were unable to express these answers in their own words, which suggests a minimal comprehension of the material accessed.
Seven of the 8 participants accessed sponsored site information while attempting to answer questions. Businesses pay a service fee to Google to have their site names appear as sponsored sites when triggered by a particular search term or keyword. Sponsored sites are outlined in color and/or appear in boxes on the right side and heading of the Google retrievals page. In general, alternative treatments and commercial therapies and medications appear under this listing; many of these sites may contain information that is uncorroborated by legitimate scientific sources.
Five participants used information provided by the sponsored sites to answer questions. Two out of 3 of the subjects who used the Internet at least once a week also used this information to answer questions. Half of the participants searching for lung cancer cures arrived at the same site: an Asian dietary supplement site claiming to cure cancer by removing free radicals from the body [
Observational logs and records on the Camtasia software show little correlation between our subjects' ability to identify answers and the amount of text on a page; analysis using the Camtasia software also showed little conclusive difference in the amount of time that the subjects spent on each site despite variances in the amount of text on the pages accessed. Therefore, subjects did not seem to prefer or navigate towards Web pages/sites with less text.
Participants, on average, used between 1 and 2 Web sites to answer questions 1 to 3.
The Flesch-Kincaid reading scale used in this study scores text at a 1st to 12th grade reading level. Given this scale, sites ranked at the 12th grade level require
Average number of links used to answer questions
Avg. Number of Links Used (Average Total) | Avg. Number of Links Used (Subjects 1-5) | Avg. Number of Links Used (Subjects 6-8) | |
Preliminary | 1.875 | 2.4 | 1.0 |
Question 1 | 1.14 | 1.2 | 1.67 |
Question 2 | 1.82 | 1.8 | 2 |
Question 3 | 1.5 | 1.6 | 1.33 |
AVG. | 1.58 | 1.75 | 1.5 |
Average (rounded) reading level of sites accessed
Avg. Reading Level of Sites Accessed | Avg. Reading Level of Sites Accessed (Subjects 1-5) | Avg. Reading Level of Sites Accessed (Subjects 6-8) | |
Preliminary | 10.50 | 10.7 | 10.0 |
Question 1 | 10.50 | 9.4 | 11.2 |
Question 2 | 11.1 | 11.3 | 11.0 |
Question 3 | 11.8 | 11.8 | 11.9 |
AVG. | 11.0 | 10.8 | 11.0 |
Average time spent on sites
Avg. Total Time Spent Per Site (min) | Avg. Total Time Spent Per Site (Subjects 1-5) | Avg. Total Time Spent Per Site (Subjects 6-8) | |
Preliminary | 7.2 | 8.7 | 4.7 |
Question 1 | 10.3 | 10.6 | 9.8 |
Question 2 | 8.7 | 8.7 | 8.7 |
Question 3 | 6.6 | 8.3 | 5.8 |
AVG. | 8.2 | 9.1 | 7.25 |
Participants spent an overall average of 8.2 minutes on individual sites. All participants voluntarily finished answering questions 1 to 3 before the 15-minute time limit was reached.
After completion of these first 3 questions, subjects were directed to a specific site; question 4 was posed about information directly linked to that site. We chose to use the ACS colon and rectum cancer Web page site, which contains links to a variety of prevention resources written at 6.3-12.0 grade levels (Flesch-Kincaid Reading Scale). The page to which we directed subjects consists of a listing of links to defined topic areas, one of which was closely related in wording to question 4. On the ACS site, 5 out of 8 people were able to answer question 4 correctly. Three of the 5 reported prior Internet experience; 2 reported none. These subjects used 3.8 links on average to answer the question. The 3 subjects who did not access the material used 6.5 sites on average before they were either stopped by the investigator or quit voluntarily. Two of these subjects had never used the Internet prior to enrollment in the study.
While most participants were unable to answer all of the questions asked, 7 out of 8 reported feeling very comfortable or comfortable with their Internet searching experience. The eighth participant felt moderately comfortable. Also, 5 out of 8 found it at least moderately easy to find readable and understandable information on the Internet. Two of the remaining participants found it very difficult to find readable information, and one participant reported that finding understandable information is easy if the Web user has strong reading skills.
Despite their dependence on sponsored sites and alternative Web sites to answer questions, 7 out of 8 subjects reported that they found it very easy to locate trustworthy information on the Internet. The eighth subject noted that it is moderately easy to find information that is trustworthy on the Internet. However, one subject said, "
Subjects felt positive about continuing their online experiences, and all expressed some enthusiasm about improving their skills. One participant stated, "
This observational study is the first to examine Internet use by low-literacy adults seeking health information [
Searching strategies were sub-optimal in several respects. First, the search terms used by subjects were predominately non-specific (
Without guidance, subjects had difficulty generating original search terms that would yield specific results. A recent study reveals that adolescents used similarly general search terms when searching the Internet for health information [
Search strategies observed in this study were also sub-optimal because most subjects exhibited some unwillingness to click on links to Web sites on the Google retrievals page. On average, subjects clicked on one to two links to answer questions. Even when the subjects did not appropriately answer questions or only partially answered questions, most seemed reluctant to click on additional links on the Google retrievals page, and 7 of 8 did not go to subsequent retrievals pages. These results did not seem to correlate with prior Internet experience. Subjects also rarely re-typed search terms in order to access more relevant retrievals. These results differ from those of a previous observational Internet study, whose participants preferred to choose links from page-one retrievals and then re-type original search terms if they were unable to find appropriate information [
Another reason why subjects' generation of search terms and selection of links were so limited may have been because the subjects were not interested in the health materials or the questions. Subjects may have also found the Google retrievals page confusing and intimidating. While the think-alouds are inconclusive about which of these factors contributed most to the weak search strategies observed, the post-session questionnaire reveals that the majority of participants reported that it was easy to search the Internet. Future research may help to illuminate the factors that contribute to the inconsistencies between subjects' perceived unwillingness to explore the Internet's health resources and their positive feedback about navigating through these resources.
The
Despite the navigational difficulties observed on the ACS Web page, the ability of 5 subjects to correctly answer question 4 probably resulted from the fact that the information needed to answer question 4 was written at an 8th grade reading level-significantly lower than the11th grade reading level required on average to read information retrieved in the first 3 searches. This suggests that low-literacy individuals can identify and utilize easier-to-read materials on Web sites. The Internet may indeed be a useful health resource to this population if materials are written at an appropriate reading level. Considering the navigational struggles of our subjects, the actual process of locating low-literacy sites on the Web may prove a more daunting challenge to this population.
While most were able to competently navigate through lower literacy materials, subjects' comprehension of Internet health information was difficult to measure in our study. Some participants found correct answers and read them to the investigators directly from the Web text, but none were able to articulate the answer in their own words when prompted. In their analysis of the1992 National Adult Literacy Survey (NALS) results, Kirsch et al reported that low-literacy adults may successfully perform simple comprehension exercises such as locating a single piece of information from text, but often find it more difficult to integrate and synthesize that information [
In addition, several subjects seemed to compensate for their low literacy skills by using external information resources. One subject who examined a Web site on mental health law (12th grade level) expressed great enthusiasm about a particular topic that he said was presented on the site. A perusal of the site after the session showed that this topic was not addressed on any of the pages he had accessed. This participant may have compensated for his struggles in reading the site by citing facts with which he was personally familiar. Another subject used a similar approach when accessing a lung cancer site. When asked about the type of information he was reading, the subject responded that the page focused on smoking cessation. However, there were no smoking-related topics on the pages examined by the subject. The subject was able to correlate lung cancer with smoking, and may have relied on this information in order to answer the investigator's query. Overall, some subjects may have been able to rely less on actual comprehension skills and more on background knowledge in order to infer answers.
Positive Web-site and performance feedback reported by most of the participants could have also been fueled by a desire to compensate for reading and comprehension difficulties. Participants were aware that the majority of the investigators were affiliated with a local hospital system; some may have felt compelled to answer positively about Internet health information because they were reporting to health-care professionals. Additionally, the participants may have been unwilling or ashamed to admit that they had difficulty understanding the information on the Internet. Individuals with low literacy tend to be embarrassed by their reading inadequacies [
An alternative reason why subjects reported positive experiences on the Internet could be that subjects were unaware of the magnitude of their Internet searching difficulties. A study by Moon et al indicates that 70% of subjects told investigators that they read "really well," while in actuality, their mean REALM scores reflected a 7th to 8th grade reading level [
Subjects' reliance on sponsored-site information to answer questions, regardless of the high literacy level required to read those sites, suggests that other factors promote the selective advantage of sponsored sites over non-sponsored sites. In fact, the design of sponsored sites on the Google retrieval page follows many of the guidelines for creating optimal layouts for health information targeted to low-literacy adults [
Of concern is that subjects did not seem to differentiate between the information on the sponsored sites and information on non-sponsored sites. Subjects used these sites interchangeably to answer questions. One study suggests that critical interpretation of Web sites is based on the Internet acumen and interests of the information-seeker; if coupling the
Standard methodologies used in this study to determine health literacy and to generate continual feedback were sub-optimal. First, REALM test results were inconclusive. Subjects were placed into the literacy program at Bidwell Training Center after taking the national Tests for Adult Basic Education (TABE). However, in our study, these subjects tested significantly higher on the REALM than expected for individuals with the reading levels indicated by their TABE scores as reported by Bidwell Training Center (3rd to 8th grade reading skills). Subjects may have strong phonetic skills that help compensate for poor word recognition and comprehension. This observation is supported in a study by Wilson et al [
Whereas complete think-alouds could have helped us better understand subjects' navigational priorities and comprehension levels, the protocols we used in this study were ineffective at prompting verbalization. None of the participants consistently articulated their step-by-step navigational process at all points during their searching session. Investigators continually prompted the subjects through the exercise, but were unable to stimulate free-thinking, consistent, and self-motivated think-alouds. One potential explanation originates from the observation that our study population was not uniformly familiar with the Internet. Therefore, some subjects may have felt overly challenged by simultaneously learning how to use the Internet and verbalizing their navigational strategies. According to previous studies [
Overall, however, our subjects were very enthusiastic about learning how to use the Internet, and all indicated an interest in improving their skills for future use. In this study and other studies [
One caveat to the present study is that our sample size precluded the analysis of factors besides low literacy that could influence the results we observed. We believe, however, that our findings with this sample group in an observational study were representative of the way low-literacy adults interact with the Internet. It will be important to validate and analyze in a larger study the appeal of sponsored sites (as opposed to other retrieved links) to low-literacy adults. It will also be worthwhile to determine the relative importance of limited literacy in comparison to socioeconomic and cultural factors in effective use of the Internet by this population. Future work will identify the exact components of sites that engage and promote learning by low-literacy adults. Greater understanding of these factors will hasten the day when the Internet becomes an effective vehicle for optimizing the health knowledge and acumen for those at high risk of poor health outcomes.
We thank Derietra Neal Ferguson for helpful comments and advice.
The study was supported in part by the University of Pittsburgh Cancer Institute, and publication by NLM grant LM008302.
None declared.
American Cancer Society
National Adult Literacy Survey
Rapid Estimate of Adult Literacy in Medicine
Tests for Adult Basic Education