From March 20 to 22, 2018, we held a workshop that focused on health literacy in Austrian lower secondary students. The workshop was held by one medical student and one education researcher on 3 consecutive days and comprised 12 class hours in total. It took place in the computer laboratory of the school. As our understanding of health literacy includes internet-based information literacy and reading literacy, we addressed these topics explicitly. The participants of the workshop were between 12 and 14 years old, and all of them went to the same third-grade class of a secondary school in Austria. A total of 14 students (eight girls and six boys) participated, and all of them were born in Austria with German as their native language. The parents and the students themselves provided written consent for participation in the workshop and the study.

The control variables general cognitive abilities and reading competence were assessed prior to the workshop using the Culture Fair Intelligence Test, revised German version [37] and the Reading Speed and Reading Comprehension Test for Grades 6 to 12, German [38], respectively. In addition, we asked students about their habits of internet use.

We used the electronic health (eHealth) literacy scale (eHEALS), which has eight statements (Table 1) and an internal consistency of α=.88, to assess the students’ internet-related health literacy (pre- and postintervention; on the first day of the workshop and 2 weeks after the workshop) [39]. The statements were scored on a five-point Likert scale with response options ranging from strongly agree to strongly disagree. For further computations, we reversed the polarity of the scale so that a higher mean score indicated a higher eHealth literacy.

Further, we assessed the general self-efficacy of the participants with the General Self-Efficacy Scale, Reversed Version, German [40], which consists of 10 statements, with four response options for each statement ranging from strongly disagree to strongly agree. A higher mean score indicated a higher general self-efficacy with sufficient internal consistency between α=.78 and α=.79 [41].

At the beginning of the workshop, we asked the students to perform internet-based research on a health-related issue that was described in short narrative text in the fashion of a vignette with a research task. The topics addressed in the vignette were defined by the researchers and the class teacher. We divided the students into two groups (for didactical reasons, it was easier to work with smaller groups), and each group received one of two vignettes (vignette 1 and vignette 2).

Vignette 1 was as follows: “A friend of yours, his name is Klaus, meets with you to discuss something. So far, he has not talked to anyone about it, but now he wants to ask you for some advice. Klaus tells you that since his childhood, he has been bothered about the appearance of his nose, which he finds way too big and too crooked. He is considering having his nose corrected surgically. Now, he would like to know your opinion. You tell him that you would like to give him some advice, but first you want to know more about rhinoplasty. Try to inform yourself using the internet and make a profound decision about what advice you will give Klaus.”

Vignette 2 was as follows: “Imagine you have a friend named Claudia. Claudia has a large dark mole on her left cheek since birth. Making matters worse, this mole has dark bristly hair. Claudia tells you that this spot on her cheek has been bothering her since she can remember. In addition, she is afraid that the mole could become dangerous, as some moles do. Now, she asks you if you think she should get rid of the mole. You explain that you would like to give her some advice, but first want to know more about the condition. Try to inform yourself using the internet and make a profound decision about what advice you will give Claudia.”

Before and during their search the students did not receive any input on how to search efficiently on the internet or how to evaluate sources. We instructed them to take screenshots of all their Google searches (the students were most familiar with this search engine) and of all webpages they opened. These data were used to determine how they performed their internet-based search and to discuss their search behavior and their results with them later.

Following their search, the students wrote down the advice they would give to their friend (Klaus or Claudia) and why they would give that advice. Thereafter, we discussed the sources they chose and why they chose them. After this first day, we gathered the browser histories of all students and the screenshots they had taken during their searches, as well as the notes they had made on what advice they would have given their friend. On the second and third days of the workshop, the students received input and performed tasks regarding proper search, reading strategies, source evaluation, and reliable internet sources on health issues.

We clustered and evaluated the browser history and screenshots the students made on the first day of the workshop in order to analyze their internet search behavior. We assessed all visited websites following the classification system of Pérez et al [42] who suggested to distinguish four stages of information quality according to the following (independently rated) criteria: media quality (eg, A forum versus a website of the Ministry of Health), author’s motivation (eg, A surgeon with a private practice versus a researcher for evidence-based medicine), and author’s position (eg, A boulevard journalist versus a psychologist with specialization in body perception disorders). Two researchers rated all visited websites independently and subsequently compared and discussed their ratings. There was consensus regarding all websites, except one. For this website, a third researcher was consulted, and eventually, we were able to assign all of the websites to one of the following categories: (1) good (3 points; all three dimensions are reliable); (2) fair (2 points; two of the three dimensions are reliable); (3) poor (1 point; one of the three dimensions is reliable); and (4) bad (0 points; all of the three dimensions are unreliable).

Descriptive statistical analyses were performed using IBM SPSS Statistics 25 (IBM Corp, Armonk, New York, USA) [43].

All participating students had average IQ values (mean 99.77, SD 8.66). In relation to an age-equivalent norm sample, the average performance was within the norm for this cohort in reading comprehension (mean T value 50.07, SD 8.13), and the students were slightly faster than the norm sample in reading speed (mean T value 54.07, SD 5.62). There was a noticeable variation in reading competence among the students; however, reading competence had no relevant associations with seeking behavior-related variables, the eHEALS score, or self-efficacy. Furthermore, on comparing the reading comprehension score with the search behavior of the students, no systematic patterns were identified. A higher reading competence did not lead to a more sophisticated search or better source evaluation.

All students had access to web-enabled devices, such as tablets, smartphones, and computers, at home and reported searching for information on the internet at least several times a month.

The self-assessment of internet-based health literacy before the workshop (Table 1) resulted in a rather high mean overall eHEALS score (range 1-5) of 3.5 (SD 0.7). All students indicated that they knew how to find helpful health resources on the internet (Table 1, Q1). In contrast, their answers regarding the availability of health resources on the internet suggested that they did not know which health resources were available or they had little experience with internet-based health resources so far. A detailed inspection of the eHEALS scores (Table 2) showed a high diversity within the scores of the students. The average scores ranged from below the scale mean (student 1) to nearly the highest possible value (student 12). Furthermore, there were hardly any differences between female and male students.

All participants showed similar internet-based information–seeking behavior no matter how competent they rated themselves in the eHEALS questionnaire.

The participating 14 students entered a total of 44 search requests into Google and visited a total of 85 webpages for the task. Among these webpages, 2 (2%) were rated as good, 6 (7%) as fair, 37 (44%) as poor, and 40 (47%) as bad by independent raters. Hence, more than 90% of the webpages were classified as poor or bad. Almost half of the webpages visited were homepages of doctors with obvious commercial backgrounds. These were paid advertisements on the top of the search results list and were labelled accordingly. Only one student visited a webpage of a university providing evidence-based medical information, and another student visited a high-quality medical internet-based encyclopedia. Both webpages were rated as good (Table 3).

The mean number of Google searches with unique queries initiated by the participants was 3.14 (SD 1.99). The number of opened websites ranged from 2 to 16 (Table 4). We found no significant correlation (r_s=0.158, P=.59) between the self-assessed internet-based health literacy of the participants and the actual quality of their internet searches. The results of three students were particularly noticeable. Student 2 rated internet-based health literacy relatively low but performed six Google searches and visited eight websites with an average quality of 1.0. In contrast, student 12 had the highest eHEALS score and visited 16 websites, but the quality of the websites was poor or bad. Student 9 had the highest average website quality (1.5), and this student’s eHEALS score was relatively high.

Regarding the difference in eHEALS scores before and after the workshop, we found that the mean eHEALS score increased slightly from 3.5 (SD 0.7) before the workshop to 4.0 (SD 0.5) after the workshop. However, not all students had a higher eHEALS score after the workshop. Those students who estimated their internet-based health literacy as particularly high before the workshop had lower ratings after the workshop and vice versa. We also found a significant correlation (Spearman rank correlation coefficient) between the eHEALS score before the workshop and the students’ general self-efficacy (r_s=0.794, P=.002). Participants with a high self-efficacy rated themselves higher in the eHEALS as compared to those with lower self-efficacy. In contrast, participants with high self-efficacy rated their internet-based health literacy lower and those with low self-efficacy rated their internet-based health literacy higher after the workshop than before the workshop (Table 5).

After the workshop, on average, the participants rated themselves better for eHEALS questions Q2, Q3, Q4, Q6, and Q7, but worse for questions Q1, Q5, and Q8. In particular, the effects of the workshop were noticeable for Q1 (“I know how to find helpful health resources on the internet”) and Q3 (“I know what health resources are available on the internet”). The score for Q1 was high before the workshop and slightly lower after the workshop. This question was apparently particularly difficult for the students to answer before the workshop, as only 10 of the 14 students provided an answer. The score for Q3 changed in the opposite direction with an increase in the median (Table 6).

At the final interview after the workshop, the students acknowledged the usefulness of the workshop and the importance of critically reviewing information on the internet. For example, one student described the main insights as follows:

Another student pointed out a sense of achievement as follows:

Results from a brief evaluation questionnaire filled out immediately after the workshop by the students revealed that they were overall satisfied with the workshop and with what they had learned. They described the workshop as “beneficial,” “fun,” and “cool.”

The purpose of this study was to obtain initial insights into the internet-based health information–seeking behavior of children and adolescents aged between 12 and 14 years. We used an explorative approach to examine the internet-based health information–seeking behavior of the students during an interactive workshop. With this approach, we wanted to gain important insights into a previously sparsely investigated yet crucial subarea of health literacy.

At the beginning of the workshop, all students stated that they actually had limited knowledge about what kind of health information was available on the internet and where to find it. However, they thought that it would be easy to find health-related information on the internet and to evaluate this information.

We found no correlation between this self-assessment of the students and their actual seeking behavior. For example, there was one student who rated internet-based health literacy particularly low but performed a relatively good internet-based search. However, the overall internet-based seeking behavior was rather superficial for nearly all students, regardless of their mostly positive self-view. The first three results presented by the search engine were mainly visited, and the students did not consider whether the information was a paid advertisement. Paid advertisements were mainly by doctors, internet-based magazines, and forums, which were rated as poor or bad by experts. Among all the visited websites, only two were rated as good and the vast majority of websites were rated as either poor or bad. These results are in line with the findings of previous research [11] showing that youth tend to trust webpages merely because the owner is a doctor or has a similar profession, regardless of the intention (eg, selling products) of the authoring person.

The eHEALS score increased slightly after the workshop, indicating that the students had gained more confidence and competence in the areas of finding and evaluating internet-based health information. After the workshop, the participants were more skeptical about the quality of information on the internet and they were less trusting. This suggests an increased awareness of the students for the large amount of poor-quality health information on the internet.

The results of our study clearly support the findings of previous studies [11,21,44], indicating that students are not used to evaluating internet-based health information critically with objective criteria. This applies to internet users in general, as stated by Sun et al [12], and students in particular [11,21]. To understand the underlying mechanisms of webpages, students need information and training. This training should be given at a young age before they enter the critical phase of adolescence. As youth often develop risky behaviors at this age (eg, risky sexual behavior and smoking or substance abuse), they need to be prepared for critical health issues. Otherwise, the full effects and consequences of such behaviors will become clear in the later stages of life [45].

It is unlikely that the internet’s importance as a source of health information will decrease or that nonreliable health-related information will vanish from the internet. There is an urgent need to teach children and adolescents how to properly handle internet-based health information. This could be supported by implementing health literacy in the regular curriculum and developing training programs that foster appropriate and critical internet-based health information seeking. The benefits of such measures would be substantial, not least because increased health literacy can reduce costs for health care systems when individuals rely on high-quality information to make health decisions [46-48].

The most important limitation of the study is the small sample size. We cannot assume that the 14 students represent all other Austrian adolescents of similar age. Nevertheless, the results give rise to concerns because despite the low number of students in our evaluation, we expect the results to be reflected in a larger population of students at this age. To prove this assumption, further investigations in this area with larger samples are urgently needed. Additionally, to validate the results, a possible approach could be the application of a computer software that is being newly developed for this purpose.

The study was also limited by the use of the eHEALS, which is based on self-assessment and thus is not an objective instrument for measuring health literacy. The eHEALS scores may be confounded with other competencies, such as reading comprehension and general cognitive abilities. We tried to clarify this aspect, but it remains problematic, because this also applies to the concept of health literacy as a whole. Finally, we noticed a relevant correlation between the eHEALS score before the workshop and the students’ general self-efficacy, which might be a confounding factor in measuring health literacy.

The results of this study shed light on the very critical aspects of health literacy and information–seeking behavior. We showed that it is possible to draw the attention of students on the critical aspects of internet search, as well as to slightly improve their competence through a workshop. Although it is very difficult to promote health literacy over a limited time period of 3 days, we could raise awareness of this important topic among students and teachers. Finally, the results highlight that targeted promotion of health literacy, as well as further research is urgently needed, especially among children and adolescents.