This study analyzed the quality of dental caries-related information available on Brazilian websites. After the development of a specific search strategy, the websites were retrieved by Google Search, Yahoo!, Bing, and Baidu. Duplicates, nonspecific, inaccessible, and/or scientific links were excluded. The websites were evaluated by 2 independent examiners using the DISCERN questionnaire [40], the Journal of American Medical Association (JAMA) benchmark criteria [41], and the Flesch Reading Ease adapted to Brazilian Portuguese (FRE-BP) [42]. Furthermore, the websites were dichotomized by the nature of their authorship (health- or nonhealth-related authors). Finally, the websites’ identities were determined by cluster analysis in accordance with the combination of their respective contents (etiology, prevention, and/or treatment of dental caries).

The search strategy was designed with regard to the relevance of terms employed by the Internet users. Initially, a general query was performed on Google Search to confirm the link of Brazilian Portuguese words to dental caries issues. Additional terms automatically generated by the Keyword Planner were included in the analysis. The relevance of each one of 56 terms was subsequently examined in Google Trends by observing the monthly variation of their search volume index between the years 2004 and 2015, including all categories of Web queries performed in Brazil (Multimedia Appendix 1). After excluding 53 keywords with irrelevant volume searches, the final search strategy was constructed by the association of three terms (“cárie”+“carie”+“carie dentaria”), which correspond to synonyms and typos of dental caries written in Brazilian Portuguese.

The websites were selected through the 4 search engines with the largest market share: Google Search, Baidu, Yahoo!, and Bing [43]. On March 21, 2016, the searches were performed using a computer connected to the Internet, previously set up by clearing the cookies and history of each browser. Advanced queries were filtered by idiom (Portuguese) and country (Brazil). The retrieved links were ordered in rankings, considering the position of their appearance in each search engine tool.

Subsequently, the websites were accessed and registered using the WebCite [44], an online service that archive the information exactly as it was recovered, avoiding changes and updates for further analysis.

Finally, the websites were dichotomized according to the nature of their authorship in health- and nonhealth-related authors. Websites or blogs developed by dental or medical associations, universities, educational institutions, health companies, or health professionals were classified as health-related authors. All other pages were classified as nonhealth-related authors. Furthermore, the information presented on the websites was categorized as etiology, prevention, and/or treatment of dental caries. The presence or absence of these contents was graphically represented by the software Genesis (version 1.7.7, Graz, Austria), characterizing the identity of each website [45,46].

Two independent examiners (PEAA and MMC) evaluated the quality of websites using the DISCERN questionnaire [40] and the JAMA benchmark criteria [41]. The DISCERN questionnaire is commonly applied to assess the quality of written information on health treatment choices. The instrument is divided into the following 3 sections: (1) reliability of the publication, (2) specific details of the information about treatment choices, and (3) overall quality rating of the document. It consists of 16 questions with 5-level Likert scale, where the score “1” indicates that the criterion was not fulfilled and the score “5” indicates that the criterion was completely satisfied. The total DISCERN score varies between 15 and 80, as the second question must be disregarded when the first question is scored “1.” Typically, only the results of the first and second sections of this instrument are used to qualify the health content of documents, as follows: very poor (15-26), poor (27-38), fair (39-50), good (51-62), and excellent (63-75) [47].

The JAMA benchmark consists of a series of 4 qualitative criteria that refer to the description of the authorship (author’s name, affiliations, and credentials), attribution (effective references of content), currency (presence of dates of posts and updates of information), and disclosure (the statement of any potential conflicts of interest) of websites. For each fulfilled criterion, 1 point is given, with a total score of 0 to 4.

The websites that were divergently qualified by the examiners were reassessed to the achievement of a consensus score.

The FRE-BP [40] was used to assess the readability of the websites based on the following formula: FRE-BP=248.835−(84.6×syllables per word)−(1.015×words per sentence). These metrics were calculated using the online tool Readable.io (Readable.io, Bolney, England) [48] through the information of the respective Uniform Resource Locator (URL) of each website. All analyses were performed based on the overall written content downloaded from these links. The reading difficulty of a text is presented according to the following scores: very easy (75-100), easy (50-75), difficult (25-50), and very difficult (0-25).

Data were analyzed with the Statistical Package for Social Science (version 21.0; SPSS, Chicago, USA). Although the hypothesis of normal distribution of data was not confirmed by the Kolmogorov-Smirnov test, the statistical analysis was performed by the application of nonparametric tests. The internal consistency of DISCERN was determined by Cronbach alpha. The interrater reliability of DISCERN and JAMA scores provided by the independent examiners was assessed by intraclass correlation coefficient (ICC) for the absolute concordance. The correlations between distinct measures were demonstrated by the Spearman rank correlation coefficients. The significant differences between the dichotomized natures of websites were observed by Mann-Whitney U test. The clusters that emerged from the similarity of websites’ identities were determined by the hierarchical clustering analysis using the Ward minimum variance method. Distinct clusters were compared by Kruskal-Wallis test and post hoc Dunn test. P values of <.05 were considered significant for all analyses.

A total of 188 websites were obtained through the first links sequentially retrieved from Google Search (n=120), Baidu (n=25), Yahoo! (n=23), and Bing (n=20). Duplicates (n=63), nonspecific websites (n=21), inaccessible links (n=7), and scientific contents (n=22) were excluded. A total of 75 websites met the inclusion criteria for the analysis, as shown in Figure 1. As duplicates were also excluded sequentially, there was a great predominance of Google Search links among those that were effectively evaluated, as follows: Google Search (n=66), Baidu (n=1), Yahoo! (n=7), and Bing (n=1).

The instrument DISCERN displayed an excellent internal consistency (Cronbach alpha=.901), with values of Cronbach alpha varying between .884 and .904 if an item was deleted. A good absolute concordance level was observed between the examiners for the application of the instruments DISCERN questionnaire (ICC=0.73, 95% CI 0.38-0.86) and JAMA benchmark (ICC=0.72, 95% CI 0.56-0.82).

The DISCERN and JAMA scores for each website are depicted in the Multimedia Appendix 2. The contents of the websites were classified as of poor quality in accordance with both scores (Table 1), which were significantly correlated (ρ=.58, P<.001) (Table 2). In addition, the sum of partial DISCERN scores (sections 1 and 2) was strongly correlated with the scores of section 3 (ρ=.92, P<.001). A total of 20 websites scored ≥39, with a mean overall quality of 3.71. A digital encyclopedia opened to public contributors (“ Wikipedia ”) and a page specialized in dental health information (“ ident ”) showed the two highest DISCERN scores (>60). Only 9% (7/75) of the websites presented at least three required information displayed by JAMA benchmark criteria. The minor percentage of pages presented authorship (36%, 27/75), attribution (18.7%, 14/75), currency (17.3%, 13/75), and disclosure (40%, 30/75).

According to the FRE-BP scores, the websites were considered simple and accessible for most population (Table 1). In addition, the difficulty level in reading of websites was weakly and negatively correlated with DISCERN scores (Table 2). The ranking of the websites in the 4 engines was not correlated with DISCERN and FRE-BP scores. Distinctly, JAMA benchmark scores showed a weakly negative correlation with the ranking of the websites retrieved from Google Search (Table 2).

The scores of websites with health- and nonhealth-related authors were statistically similar, considering DISCERN (P=.29) and JAMA benchmark (P=.47) (Table 3). Nevertheless, the written documents produced by health-related authors were considered significantly more difficult than their counterparts.

The hierarchical clustering analysis yielded three distinct websites’ identities, as shown in Figure 2. Overall, websites containing contents of all 3 categories (cluster 1) showed higher quality scores than websites containing contents of only 1 (cluster 3) or 2 (cluster 2) categories. The DISCERN scores of cluster 1 were statistically higher than those of cluster 3 (P=.009) (Table 4). Additionally, the percentages of websites with health-related authors were 40.9% for cluster 1, 61.5% for cluster 2, and 48.1% for cluster 3.

To the best of our knowledge, this is the first study to assess the quality of dental caries-related information on Brazilian websites. In general, our results showed a predominance of low-quality contents, with a low rate of websites (26.7%) being classified as acceptable to high-quality levels (DISCERN≥39). These results are consistent with the results of similar studies [49,50]; Blizniuk et al [49] demonstrated lack of quality of dental caries-related information on English websites (DISCERN=44), whereas Leite and Correia [50] identified only 4 out of 75 Portuguese dental caries-related websites certified with the Health On the Net Foundation (HON) code, a trustworthy certification granted by a nongovernmental institution that evaluates the quality of health information on the Internet [51]. However, we did not consider the HON code in our methods, as only 1 Brazilian website was certified. Surprisingly, the content of this website was qualified as inadequate by both instruments utilized in this study.

We believe that the assessment of the quality of websites was improved by the simultaneous application of distinct criteria, particularly because they were only fairly correlated. Additionally, the outstanding internal consistency and interrater agreement of DISCERN when employed in these analyses are noteworthy. In contrast, the determination of the internal consistency of the JAMA benchmark is limited because it aims at the elucidation of specific data about technical and editorial production of websites. Although its four elements should be better interpreted individually, we calculated the central tendency measures of the JAMA benchmark to evaluate its relationship with other indicators.

The strict investigation executed by 2 health professionals may be linked to perception biases through the underestimation of the quality of websites; nevertheless, Griffiths and Christensen [52] revealed no significant differences between the scores of DISCERN given by professionals and laypersons. In addition, as cluster-based results were not influenced by the nature of websites’ authorship, the better performance of websites that published topics about etiology, prevention, and treatment of dental caries reflects the significant impact of the completeness of contents on the process of qualification of information. This finding is supported by the study of Diviani et al [29], who showed that the amount of information available on a website influences the perception of improved quality of digital contents by Internet users. No significant differences were found when comparing DISCERN scores with JAMA scores of websites of health- and nonhealth-related authors, suggesting that the quality criteria considered in these instruments were not a concern for the most dentists and/or dental companies during the production of electronic contents.

Although the negative correlation between FRE-BP and DISCERN scores was discrete, this trend should be regarded as an exacerbating factor for the impact of the low quality of information on the Internet users, as it demonstrates that more accessible contents are even worse in quality. This fact raises an important concern with regard to the high percentage of basic literate Brazilian youths (99%) and adults (93%) [53], as the understanding of medical information probably requires more advanced educational abilities [28]. In this context, the shared decision-making process could be deteriorated by the misunderstanding of health information and the development of harmful health beliefs.

Design considerations

According to the Vital Signs report [54], there are 4 types of individuals with regard to health care situation: (1) those who agree and accept the treatment decision, (2) those who access the Web to confirm the diagnosis given by a professional, (3) those who are involved in the decision-making process, and (4) those who are in complete control of their treatment relying on the information found by themselves. In this context, although health professionals are still considered the most important source of health advice, the easier and more affordable access to the Internet predisposes people to seek health counseling online [29]. Furthermore, 65% of health seekers frequently begin their searches using an engine bar instead of looking for information on specific portals [54]. Consequently, millions of health-related queries are entered in Google Search daily [55]; therefore, the methodological approach adopted for the construction of our search strategy probably improved the chances of retrieving the websites in a similar way to that usually performed by netizens.

To evaluate the correlation of the order of appearance of links in the search engines with the quality of their contents, we assessed a quite larger number of websites than the Internet users could be interested in [56]. For instance, Google’s PageRank uses more than 200 factors based on Larry Page’s algorithms to order the links by their relevance from the query [57]. In this study, the links found on the Google’s first page were represented by 5 blogs, 2 dental clinics, and 1 commercial website. Their contents were alarming, for example, with the description of dental treatments without dentists. In addition, the order of appearance of the websites was not correlated with DISCERN scores in the 4 different engines, that is, the algorithms created to retrieve the links associated with dental caries seemed to not have any relationship with the quality of available information, which may contribute even more with the deterioration of the health education process.

Limitations

This study presents some limitations. First, although laypersons could be interested in reading more specialized documents, the links associated to scientific publications, such as papers and books, were not considered into our analyses. This decision was based on two main reasons: (1) the DISCERN was developed to assess the quality of information presented to health seekers, that is, its application would be inadequate and unproductive to analyze scientific contents; and (2) probably, the great specificity and the technical language of scientific papers lead people to look for documents that explore more general knowledge about the disease. Second, it was not feasible to evaluate other types of website media, such as figures, films, and podcasts, as the DISCERN was specifically developed to assess the quality of written documents. Finally, the unknown audience of the websites prevented the determination of the impact of each source on the diffusion of dental knowledge, although our results were supposedly obtained from the most accessed websites, considering that they were the first dental caries-related links retrieved by search engines.

Challenges

The development of specific regulations in this field is arduous, requiring an intense debate to avoid the suppression of the rights of freedom of expression and opinion [58]. Likewise, it is almost impossible to control the publication of Web contents, particularly because of the diffusion of personal opinion contained in health blogs. The use of codes and seals for the certification of websites could be a good approach to indicate the useful health information, particularly if the accreditation is based on rigorous criteria and if the Internet users can easily view the certification on the website. Nevertheless, the continuous consumption of misleading knowledge could deteriorate the person-dentist relationship; hence, the professional should be prepared to face this challenge effectively, advising and encouraging their patients to explore information on recommended websites, warning people about the risks of health home practices, and contributing to the production of good quality electronic materials.

Conclusions

In conclusion, regarding the present sample of Brazilian websites, dental caries-related contents were considered simple, accessible, and of poor quality based on the results of FRE-BP, DISCERN, and JAMA benchmark scores, respectively. This pattern does not seem to rely on the natures of websites’ authorship but on the multiplicity of categories of information that they covered. These findings indicate the need for the development of special policies focused on the stimulus for the production and publication of Web health information, encouraging dentists to guide their patients to search for recommended oral health websites.