In Australia, the latest figures for cervical screening from 2017 to 2018 show a 2-year participation rate of 53% for women aged 25 to 69 years [1] and an overall incidence of cervical cancer at 7 cases per 100,000 women [2]. Prevention of cervical cancer through the Australian National Cervical Screening Program (NCSP) was first introduced in 1991 and screened women aged 18 to 69 years every 2 years using cytology-based screening (Pap smear). The program was renewed in December 2017, and women aged 25 to 74 years are screened every 5 years using primary human papillomavirus (HPV) screening (Cervical Screening Test) [3]. This renewal was based on a greater understanding of the natural history of HPV and cervical cancer, successful uptake of the HPV vaccination leading to a subsequent reduction in vaccine-related HPV types, evidence that the HPV test is more sensitive than the Pap smear, and economic modeling demonstrating HPV screening to be more cost-effective [3]. Recent modeling studies have predicted that the new program will reduce the incidence and mortality of cervical cancer in vaccinated women by 31% and 36%, respectively [3,4]. A limited understanding of the rationale behind these changes has been demonstrated by the general population [5,6]. This highlights a need for information explaining these changes that is easy to understand and access to not undermine the confidence women have in the screening program.

Almost 80% of Australians now use the internet as a source of health information, suggesting that the internet could be a powerful tool to educate and inform readers, with the potential to alleviate anxiety or concern [7]. As health models lean toward greater patient empowerment, patients may feel a greater sense of responsibility for their health care. However, because of the largely unrestricted nature of the internet and limited governance, there is a risk that users may be exposed to inaccurate, unreliable, biased, or potentially harmful information [8]. Exposure to information that is not presented in an accessible manner may cause unnecessary anxiety and distress. This could result in inappropriate care or people ignoring evidence-based recommendations because of being exposed to contradictory information [9].

Moreover, health information is only useful if it can be understood by the target population. The ability to read and understand written text is key for the comprehension of health information. Readability tools can measure the reading ability needed to understand the information presented. South Australia Health recommends that the readability level for health information is grade 8 (12-14 years old with 8 years of Australian education) [10,11]. In the United States, the National Institutes of Health and the American Medical Association recommend patient education materials to be written at or below the sixth-grade reading level (age 11 or 12 years), whereas the Joint Commission recommends a fifth-grade level (age 10 or 11 years) or lower [12]. There are limitations to using only readability measures to evaluate health care information [13], so it is important to ensure that web-based health information is of sufficient quality and is suitable to people from diverse backgrounds and varying levels of health literacy. Web-based health information may also require higher literacy levels than printed patient education materials, as previous studies have demonstrated that web-based patient education materials are often written above the recommended reading level of grades 6 to 8 [14-17], and these may be perceived as more difficult than print materials [13].

There is little to no value of health information that is trustworthy and credible if it cannot be easily understood and acted upon by the general population. A commonly cited disadvantage of web-based health information is the inability of consumers to evaluate the quality of websites [9,18]. In addition to being able to read health care information, consumers need to be able to understand the information presented, evaluate the credibility of websites, and find the information they are looking for.

To the best of our knowledge, no previous research has evaluated the readability of health information provided on websites about cervical screening in Australia, particularly regarding the changes to the NCSP. Therefore, this study aimed to evaluate the information available on the internet regarding the renewal of the NCSP in Australia, with a particular focus on website readability, understandability, design, credibility, and usability.

In April 2019, the term cervical screening was searched for in Australian versions of the 3 most popular search engines: Google, Yahoo, and Bing. The incognito window on Google Chrome was used to conduct the search, and browser history, cache, and cookies were cleared before running the search to ensure that previous searches would not impact the search results. The first 10 relevant results from the first 3 pages of results from each search engine were selected. Relevant websites (in the English language and related to cervical screening) were identified from each search engine, and any duplicate websites were removed and noted. Websites were excluded if they were advertisements, news reports, Wikipedia pages, social media pages, online discussion forums, blogs, videos, books, articles, or private websites.

Readability statistics were obtained by copying and pasting each website URL into a web-based analysis tool, the Readability Test Tool [27], which uses an algorithm to calculate readability measures. If there were several pages on a website, statistics for each page were obtained, and the mean was calculated. An exception to this was if a website had a publications and resources page; the statistics for this page were not calculated. The analysis was stratified by the intended audience: consumer focused versus health care provider focused. Websites were categorized according to their primary target audience (ie, websites with most content aimed toward consumers were treated as consumer focused).

The quality of websites was evaluated by 2 independent researchers using Microsoft Excel. A third researcher independently coded 40% (6/15) of the websites. Descriptive statistics were calculated using IBM SPSS version 25. Means and SDs were calculated for understandability and actionability (PEMAT), readability statistics (FRE, FKGL, and SMOG), and JAMA benchmark criteria. Interrater reliability between the 2 reviewers was assessed using the Cohen kappa coefficient. Correlations between PEMAT and readability measures were calculated using the Pearson correlation coefficient. The Kruskal-Wallis test was conducted to detect differences in mean readability, understandability, actionability, and JAMA benchmark scores between consumer-focused and health care provider–focused websites.

Ethical approval was not required, as the websites were in the public domain, and no human participants were involved.

A total of 49 websites were identified using Google (n=16), Yahoo (n=15), and Bing (n=18; Multimedia Appendix 3). After applying the exclusion criteria, 15 unique websites remained and were included in the evaluation (Figure 1). Duplicates (n=31) were excluded; therefore, a greater number of eligible websites were included from Google: Google (n=10), Yahoo (n=1), and Bing (n=4; full URLs can be found in Multimedia Appendix 3). The included websites were federal (Cancer Australia) or state government–owned (eg, Cancer Institute NSW), government-funded (Health Direct) or nongovernmental organizations (Cancer Council). Of the 15 included websites, 10 were targeted toward consumers, and 5 were targeted toward health care providers, including National Prescribing Service (NPS) MedicineWise that consisted of training modules for health care providers. In addition, 4 websites had information for both consumers and health care providers but were categorized according to the primary target audience. Abbreviations, target audience, and organizations of websites included in the analysis are described in Multimedia Appendix 4.

The mean FRE of consumer-focused websites was 51.8 (SD 13.3; Table 1), which is considered fairly difficult to read. Health care provider–focused websites were considered difficult to read, with a mean FRE of 43.7 (SD 7.3). Of the consumer-focused websites, the Cancer Council Australia Cervical Screening Consumer Site had the highest FRE (70.4; Table 2), and the Cancer Council Australia Main Site had the lowest FRE (37.8). Consumer-focused websites had a mean reading grade level of 7.8 (SD 1.5), compared with a mean reading grade level of 8.1 (SD 1.4; Table 1) for health care provider–focused sites. Health Direct was the only website to score below the US-recommended sixth-grade level using the FKGL (Table 2). The mean SMOG index of consumer-focused sites was 6.7 (SD 1.2), compared with a mean SMOG index of 7.5 (SD 0.7; Table 1) for health care provider–focused sites. The SMOG index ranged from 4.6 (Cancer Australia Cervical Screening) to 9.1 (Victoria Cervical Screening Program [CSP]). Overall, 13 out of 15 (87%) of all included websites (9 consumer focused and health care provider–focused) scored between the sixth and eighth grade reading level using the SMOG index. Victoria CSP had the highest grade level (FKGL and SMOG) and the lowest reading ease (FRE). The differences in the FRE, FKGL, and SMOG index scores of consumer-focused and health care provider–focused websites were not statistically significant.

The mean understandability score of consumer-focused websites was significantly higher than that of health care provider–focused websites (87.7%, SD 6.0% vs 64.9%, SD 13.8%; X²₁=6.9; P=.01). Overall, 90% (9/10) of consumer-focused websites and 80% (4/5) of health care-provider focused websites met or exceeded the 70% threshold for understandability. The mean PEMAT actionability score was significantly higher in consumer-focused websites than in health care provider–focused websites (58.1%, SD 19.1% vs 36.7%, SD 11.0%; X²₁=4.8; P=.04). Two consumer-focused websites met the threshold for actionability, whereas no health care provider-focused website was found to have actionable information. The NCSP Main Site and Jean Hailes were the only 2 websites to score over 70% in both the understandability and actionability domains. Interrater reliability was substantial for PEMAT ratings with 0.73 for understandability and 0.75 for actionability.

All 15 websites made their purpose evident and defined terms; however, only 13% (2/15; Table 3) of websites were judged as using visual aids whenever possible, and 53% (8/15) of the websites were judged to use the active voice. All consumer-focused websites, and 60% (3/5; Table 3) of health care provider-focused websites identified at least one action for the reader to take. Overall, 53% (8/15) of websites broke down the action into explicit steps, and 7% (1/15) of websites provided tangible tools whenever possible to help the reader take action. A correlation analysis was conducted between the FRE and PEMAT understandability to determine if websites that were easier to read were also easier to understand. There was no significant correlation found between reading ease and understandability.

Overall, 3 websites met all 4 of the JAMA benchmark criteria (Cancer Council Australia Main Site, NCSP Clinical Guidelines, and the Royal Australian College of General Practitioners [RACGP]). The mean number of JAMA benchmark criteria satisfied was 2.4, ranging from 1 to 4. Overall, currency was the most well-adhered principle, with 8 of 10 consumer-focused and 4 of 5 health care provider–focused websites providing update information (Table 4). Authorship was the most commonly missed criteria, with only 3 consumer-focused and 3 health care provider–focused websites properly attributing the authors of information, including their qualifications and affiliations. There was no significant difference in the mean number of criteria satisfied between consumer-focused and health care provider–focused websites.

The HONcode was only present on 2 websites (NPS MedicineWise and Health Direct; Table 2). No websites displayed an explicit conflict of interest statement (represented as part of the disclosure criteria); therefore, the existence of any actual or potential conflicts cannot be determined. RACGP provided a link to a conflict of interest management policy, which states that no one with a conflict of interest can be involved in the decision-making process. NCSP Clinical Guidelines provided a register of all interests, leaving it up to the end user to determine the presence and significance of any conflict.

All websites had consistent use of white space, separating paragraphs of information into smaller chunks of text. All websites used distinct and easy-to-read font styles and had clear subheadings in a text size larger than the main text. An adaptable font size was available in 2 of 15 websites (Cancer Council Australia Main Site and Cancer Australia Cervical Cancer). All 15 websites used a responsive web design. Of 15 websites, 3 (NSW CSP, Cancer Council Australia Cervical Screening Consumer Site, and Jean Hailes) used illustrations to reinforce key messages.

Of 15 websites, 3 (NCSP Main Site, Cancer Council Australia Cervical Screening Consumer Site, and NSW CSP) provided answers to all 5 simulation questions. The question most frequently answered by websites was question 5: “Who needs the new test?” NPS MedicineWise, RACGP, National Cancer Screening Register, and Queensland CSP (B) did not provide answers to any of the questions. The time taken to find the answers ranged from approximately 10 seconds to 2 min. The answers were found the fastest on NSW CSP (average time of 34 seconds per question) and the slowest on NCSP Clinical Guidelines (average time of 146 seconds per question).

This study is the first to evaluate web-based information available to consumers about the renewal of the Australian NCSP. Overall, the readability of consumer-focused websites providing information about the renewed NCSP was fairly difficult to read by the general population (ie, below the threshold of 60); however, there was high variability in scores. Overall, the evaluated websites demonstrated a high level of understandability. The Cancer Council Australia Cervical Screening Consumer Site scored highest for understandability assessed using the PEMAT. There was no significant correlation between reading ease and understandability, demonstrating that they measure different constructs. This is in line with previous studies and highlights the importance of using both measures to evaluate quality [17]. Most websites met some criteria to assess their credibility, and all websites demonstrated some thought had been given to the design of the websites. Of the 5 key questions that could be frequently searched by consumers, the following 3 websites provided answers to all questions: NCSP Main Site, Cancer Council Australia Cervical Screening Consumer Site, and NSW CSP. These findings demonstrate great variability in the readability, understandability, and credibility of websites available on the internet, which provide both consumers and health care providers with information about the changes to the NCSP.

Across all measures, the Cancer Council Australia Cervical Screening Consumer Site was judged to be the most accessible website overall. This site was considered the easiest to read based on the FRE score and had a reading level between grade 6 and 7 using both the FKGL and SMOG index, which is in line with recommendations from South Australia Health [11]. Furthermore, it scored highly on the understandability criterion of the PEMAT and was one of only three websites to answer all 5 simulation questions. This website was developed by Cancer Council Australia in response to the renewal of the NCSP and has a clear purpose to educate women about the changes to cervical screening and seeks to answer possible frequently answered questions. It provides comprehensive content about the changes in an accessible manner. Australian women with questions or concerns about the renewed NCSP may benefit from being directed to this website by their health care provider.

This study used validated tools and a combination of several measures to independently evaluate websites providing information about cervical screening. Objective readability measures provide limited data about the health literacy level required to understand information, so by combining this with the PEMAT, this enabled us to determine how well people of various backgrounds and levels of health literacy may be able to understand and act upon the provided health information. These tools were selected because the items were most relevant to the health information we were evaluating. Other available tools, such as DISCERN [28], have a number of items aimed at treatment-related information, which are not relevant for the content of cervical screening information. A limitation of our study is the small number of websites included in the analysis, which may have prevented seeing differences in readability between health care provider–focused and consumer-focused websites. However, the search term cervical screening returned over 50,000,000 results on Google alone, and as internet users rarely look past the first 3 results pages, it is likely that the websites included in our analysis would be those most commonly accessed by consumers [18]. Some of the measures we used are subjective (eg, PEMAT), but by having 2 independent evaluators score all 15 websites and a third evaluator score 40% (6/15) of websites, this will have helped ease any discrepancies, and the interrater reliability between the evaluators was rated as substantial. The websites in this study were only examined using their content and were not examined for accuracy of the information presented, as this was not an aim of the study. This study was limited to the evaluation of printable material on the websites; therefore, information delivered in video format was excluded.

This study is particularly relevant in light of the renewed NCSP and the public misconception about the rationale behind the changes [5,6]. Our findings show that most consumer-focused web-based information on cervical screening in Australia is written at a reading level inaccessible to the general population. This is similar to previous studies that have found web-based health information to often be written above the recommended reading level [29]. This may be because the authors have insufficient awareness of health literacy considerations and readability [30].

In this study, websites with the lowest readability tended to be those targeted toward health care providers. Making the target audience of the website more explicit may help avoid potential confusion for consumers and the frustration of not having questions answered. Although this may be obvious for some websites (eg, NCSP Clinical Guidelines), health consumers should be able to determine at a glance the target audience of the information they are reading for all websites. This may minimize their exposure to inaccessible material. Sites aimed toward health care providers could provide links to guide consumers to reputable consumer-focused websites (eg, NCSP Main Site or Cancer Council Australia Cervical Screening Consumer Site).

The high understandability scores indicate that web-based cervical screening information can be understood relatively easily by consumers of diverse backgrounds and levels of health literacy [23] and demonstrate that the PEMAT and readability tools measure different constructs. Greater awareness of how the mode of delivery may affect consumers’ understanding of the health information could help to create and distribute better patient materials. The low actionability scores suggest that it may be difficult for consumers to act upon the information they read. Most websites failed to provide tangible tools or visual aids and lacked explicit instructions for the user to follow (eg, a tool to help consumers decide whether they are eligible for screening under the new guidelines). The general purpose of these websites is to inform and educate the reader about the changes rather than encouraging an action. Consequently, the actionability domain of the PEMAT may have limited applicability to this context. The large range in PEMAT scores may be partially explained by the fact that some websites were solely targeted toward health care providers. As such, the content on these websites (NCSP Clinical Guidelines, RACGP, and NPS MedicineWise) would not be considered patient education materials, and consequently, the PEMAT score may not be an accurate measure of quality. Most existing literature on web-based health information is related to disease and treatment information, where there may be more explicit actions to take than for cervical screening (eg, different treatment options for a chronic disease). This may, in part, contribute to the low actionability scores for our websites, but because of actionability not being an aim of these websites, less weight was given to these scores.

Noting the presence of the HONcode logo is a simple way for consumers to recognize that the information they are reading is trustworthy, but because HONcode certification is a voluntary registration process, it is possible that websites may comply with the principles despite not being certified [10]. As most of the included websites are federal or state government owned, they would likely be instantly recognized as trustworthy and credible sources of information; therefore, there may be no added benefit of HONcode certification. It is unclear how recognizable the HONcode is among both consumers and website developers. The only 2 websites displaying the HONcode are government-funded not-for-profit organizations (Health Direct and NPS Medicine Wise). As these sites may not be as easily recognized as reputable sources, HONcode certification may be more valuable; however, there is little value if a consumer does not know what it is or how to interpret it.

Most websites failed to provide adequate detail about the authors of the information or to provide sources and references for information. This information was seldom displayed prominently, often located on the about us or disclaimer pages, which are rarely accessed by consumers [18]. Sources that were provided were rarely primary research and often linked to related websites, raising the potential of consumers to be exposed to inaccurate, inaccessible, or unreliable content if the linked websites are of poor quality or lack credibility. It could be argued, however, that the omission of primary sources on consumer-focused websites may be a deliberate decision by the website developer. Primary research papers are often behind paywalls or written at a reading level inaccessible to the general public. This has the potential to cause greater confusion and may offer little benefit to the reader. Furthermore, consumers generally judge the credibility of websites based on the hosting organization (ie, whether it is from a reputable and recognizable organization such as the Australian Government or Cancer Council Australia), language, and professional-looking design [18]. Web-based information can be biased, with little to no scientific evidence [18]. In addition to providing credit to the original source, appropriate attribution can help the end user evaluate the quality and ensure the trustworthiness of information on the web. This leads to a discussion of whether there should be mandatory criteria to follow when setting up websites to ensure their credibility. Furthermore, it raises the question of whether details about sources, authors, and conflicts of interest can be displayed in a way that is more accessible and meaningful to health consumers.

The 3 websites that satisfied all 4 JAMA benchmarks were targeted specifically toward health care providers. If websites targeted toward health consumers lack the aforementioned principles, it raises potential issues. The general population, particularly those of lower health literacy, may be less equipped than health care providers to evaluate and discredit websites. It is critical that consumer-focused websites are credible, but it may be necessary for a tradeoff between accessibility, readability, understandability, and credibility. This area requires further research to minimize this tradeoff, as each component contributes toward consumers’ experience of accessing health information on the web. Appropriate attribution and references are a particularly important credibility indicator of health care provider–focused websites; however, they may be considered less relevant for consumer-focused sites. Existing credibility assessment tools (such as the JAMA benchmark criteria and the HONcode criteria) are limited in their ability to evaluate consumer-focused websites. The development of a new credibility framework for consumer-focused websites could mitigate this limitation. Taking into consideration other factors that influence credibility, such as placing more weight on the hosting organization, could enable consumers to discredit unreliable information without reducing the overall quality.

Owing to an increasing interest in the literature about conflicts of interest, it was interesting to find that none of the websites in this study explicitly disclosed a conflict of interest. It is important to note that the absence of a conflict of interest statement is not equivalent to having no conflict. There is increasing research into the management of conflicts of interest and whether or not disclosure is the most appropriate strategy [31]. Disclosing conflicts merely indicates the possibility of a bias; however, it does not provide guidance for resolving it or how the conflict was managed. For the consumer, a lack of disclosure may hinder their ability to mitigate any potential risk of bias, as conflicts of interest may impact decision making [31].

Simulation questions were helpful in establishing whether these websites explained the changes to the CSP. The majority of these websites did not answer the questions in detail, raising concerns about whether consumers reading the information would be fully informed about the changes. A total of 3 websites provided answers to all the questions (NCSP Main Site, Cancer Council Australia Cervical Screening Consumer Site, and NSW CSP), with the time taken to find answers to these questions being the quickest for the NCSP Main Site. The websites that either did not provide answers to the simulation questions or it took a long time to find the answers were those aimed toward health care providers, further highlighting the need for the target audience of the website to be made more explicit.

Finally, all websites were designed to improve the accessibility of information through the use of white space and appropriate font style. The layout and design of materials can be a contributing factor to how consumers comprehend the information or how accessible the information is [22]. Only 2 websites showed adaptable font size, which may cause potential difficulties for those with impaired sight. An important consideration when designing websites now is also about how the interface looks on mobile phones, given the increasing number of consumers who will view health information on their mobile phones [7]. It was therefore reassuring to observe that all websites used responsive web design. The use of illustrations can reinforce messages and aid interpretation of written text or they can distract away from the purpose [26]. The NSW CSP and Cancer Council Australia Cervical Screening Consumer Site use photographs of culturally and demographically diverse women, which may help consumers understand that the information is relevant to them.

The findings from this study can help health care providers direct their patients toward websites that have information about the renewal of the CSP, which is easy to read and has high understandability (eg, Cancer Council Australia Cervical Screening Consumer Site). Encouragingly, web-based information about the renewed CSP is generally of good quality. There is a need to promote increased awareness of the importance of web-based health information that is credible, user friendly, and easily understood by people with wide levels of health literacy. Web-based information should be produced with particular consideration to people with low health literacy, with those responsible for creating these websites to be accountable for ensuring that this information is accessible, accurate, and credible. There may be potential for web-based health information to be improved through the introduction of mandatory criteria to ensure the credibility and quality of consumer-focused websites.