We utilized several steps to create our website database in order to minimize bias and most broadly capture what the general public would encounter while searching on the web (Figure 1). First, we used Google’s Adwords tool, a free Google advertising tool that allows advertisers to associate their ads with search terms based on the popularity of the term. For a given topic, the keyword tool identifies the most commonly used search terms with a topic based on historical search data [26]. We selected 5 search terms for each of our 3 keyword combinations (abortion safety, abortion depression, and abortion infertility) based on their frequency as identified by Google Adwords and relevance (Figure 2). Next, we conducted Google searches using these phrases on August 31, 2018, and compiled website results for each of our search terms. In order to remove search term personalization, each search was performed using a cookie-cleared Google Chrome (version 10.8) browser in incognito mode combined with a free virtual private network called Windscribe (version 1.82) in order to reduce locational bias as well as anonymize our internet protocol addresses. For each search phrase, we extracted the top 25 page-ranked results, noting multiple occurrences. Each link was then imported into a spreadsheet using MozBar (Moz Inc), a Chrome extension that also produced titles, brief excerpts, and a search position number for each link based on its proximity to the top of the Google page. We produced a database for each of our topics (safety, depression, infertility) based on this search methodology.

We next classified and rated each website in the database. We rated websites on 5 metrics: (1) slant, (2) slant clarity, (3) trustworthiness, (4) type, and (5) domain. For slant, a website was determined to have a pro-choice or anti-choice bias if either (a) information was given in a biased or dramatic fashion (eg, an anti-choice site describing the procedure as “tearing the baby from the womb”), or (b) the website displayed an opinion regarding the provision of abortion and its legality (eg, a pro-choice site stating that abortion should be free and legal for all women); otherwise, the website was considered neutral. We assigned a slant clarity rating based on how easy it was to discern the website’s slant (obvious, in-between, or difficult/can’t tell). We scored trustworthiness on a rating scale from 1 to 5, with 5 being most trustworthy based on factors including the content and the source of the content (eg, creators, references, or affiliation with health care organizations or health care professionals) [27-29]. These factors were informed by existing systematic criteria developed by prior studies of media credibility and disinformation [30]. The domain was the generic top-level domain of the site (.com/.net, .edu, .gov, .org, or a domain that identified the website as international, such as .uk or .nz). Finally, investigators loosely categorized websites based on 7 descriptive “types,” including forum, feature, scholarly article, resource page, news article, blog, and video. Two investigators independently coded all of the websites. In the event of disagreement, a third researcher arbitrated the categorization. In the case of trustworthiness, the scores from investigators were averaged. All three researchers have published abortion-related research previously.

First, we created a visualization of the websites and their parent search terms and topics using RAWGraphs. Next, we compared website characteristics overall and by search topic (safety, depression, or infertility) using Pearson chi-square test or Fisher exact test for all categorical classifications to compare characteristics across topics. We summarized trustworthiness, measured on a 5-point scale, using the median and IQR. For each search topic, we also assessed the distribution of websites based on the content’s slant for the 5 query terms, and then compared the trustworthiness of query term results using Kruskal-Wallis and Wilcoxon rank sum tests. We used box-and-whisker plots to visually compare trustworthiness by slant and by domain type. Finally, we tabulated websites according to their slant and slant clarity, comparing distributions using Fisher exact test, and then used a bar graph to examine website counts graphically. The study was reviewed and approved by the Oregon Health and Science University’s institutional review board and was deemed not human subjects research.

Our study of websites that provide information about safety of abortion or subsequent risks of depression or infertility found that overall, 35.4% (112/316) of sites had an anti-choice slant. Anti-choice sites were less trustworthy, and the clarity of their stance was also more difficult to determine. Our data also show that people seeking information about the safety and potential risks of abortion are likely to find substantial amounts of untrustworthy abortion information; the overall median trustworthiness score of these sites was 2.7 (on a scale of 1-5), with a wide range (IQR 1.7-3.7).

Our results are consistent with other examples of intentionally disseminated abortion disinformation. CPCs have an antiabortion agenda but use neutral language and advertising to appear as if they are a normal medical clinic [20]. In our study, almost two-thirds (71/112, 63.4%) of anti-choice sites were categorized as resource pages and many derived from web addresses that appeared generic and neutral. Moreover, one state health department site rated as anti-choice, which demonstrates that even official governmental sites can be biased sources of information. These findings suggest that anti-choice sites may be trying to obscure their biases or appear as neutral information brokers. Based on the high number of “difficult to tell” sites in our data sets, we need more research to understand the characteristics of anti-choice sites that create this effect.

We know that in the case of abortion, false or misleading information continues to play a role in public debates and government legislation. For example, Targeted Regulation of Abortion Providers (TRAP) laws, or costly and burdensome regulations aiming to restrict access to abortions by dictating how, by whom, and when abortions can be provided, are often based on false health claims but framed as protecting patient safety [31]. The internet is one of the most important propagators of false information. Previous research has shown that media is more influential among young people (aged 13 to 29 years) than friends, family, and health care providers when it comes to learning about abortion [32]. However, there is also evidence that exposing the public to more evidence-based information on abortion can change opinions about the provision and regulation of abortion services [33]. Thus, efforts to address the spread of false information and strengthen the presence of evidence-based information on abortion can have real impact. Our results can help inform clinicians and others who work in the abortion field about likely information gaps [34]. Moreover, health organizations that seek to disseminate evidence-based information may want to conduct regular search audits such as this in order to optimize their search positions to reach a broader audience.

Our study results should be interpreted with the following limitations in mind. We used expert ratings to gain insight into the abortion online media ecosystem. All of our experts are engaged in abortion-related research and two-thirds of our experts are abortion providers. We recognize that non-experts may have rated and classified these same websites differently. However, our expert ratings provide a benchmark for future work that explores how non-experts perceive the information contained in these websites. We also acknowledge that trustworthiness scores could be confounded with slant and that sites with obvious anti-choice slants may have been rated with lower trustworthiness scores based on the impression of bias. However, it is difficult to blind these components from each other and we wanted to provide some kind of measure of quality for the information we encountered. We recognize that our study provides only a cross-sectional snapshot of the internet. While specific site rankings may be driven by news cycles, we believe the overall picture is unlikely to change [35]. We also acknowledge that while we took measures to anonymize the search results, individuals will receive different results based on their previous search history, and the majority will not scroll past the first page. However, our database sizes captured approximately the first 8 to 12 pages of search results for a given topic. Finally, while our focus was on websites, websites alone do not fully address the digital means through which abortion information is shared. In particular, social networking platforms are critical sources [36]. The strengths of our study include the use of a systematic and rigorous approach to create our website database, incorporation of multiple topic areas, and focus on content trustworthiness and bias.

Our results provide insight into the online abortion ecosystem. We find that anti-choice sites are prevalent, often hard to identify as anti-choice (difficult slant clarity), and less trustworthy than neutral and pro-choice sites. Additionally, the search terms a user chooses may play a substantial role in the quality and bias of websites they see. Our results help us understand how the internet may impact public perceptions and knowledge about the safety of abortion and potential risks of depression and infertility. Our findings suggest that web searches may lead people to perceive abortion procedures as more dangerous and riskier than they actually are.