Website Usability Analysis of United States Academic Medical Centers

Background: Many healthcare organizations are now tasked with providing web-based health resources and information. Usability refers to the ease of user experience on a website. Our objective was to adapt pre-existing usability scoring systems for United States academic medical center websites, then apply this system to a sample for the purposes of testing this system and deriving insights from the results on potential areas of improvement. Objective: The primary aims of the study were to: 1) Adapt a pre-existing usability scoring methodology to academic medical centers; 2) apply and test this adapted usability scoring methodology on a sample set of academic medical center website and; 3) derive recommendations from these results on potential areas of improvements for our sample of academic medical centers websites. Methods: All website usability testing took place from 1 June 2020 to 15 December 2020. We replicated a methodology developed in previous literature and applied it to academic medical centers. Our sample included 73 U.S. academic medical centers. Usability was split into four broad categories: Accessibility (ability of those with low levels of computer literacy to access and navigate hospital’s Web presence), Marketing (ability to be found through search engines, examining the relevance of descriptions to the links provided), Content Quality (grammar, frequency of info updates, material relevancy, and readability), and Technology (download speed, quality of the programming code, and website infrastructure). Using these tools, we scored each website in each category. The composite of?key factors in each category contributed to an overall “General Usability” score for each website. An overall score was then calculated by applying a weighted percentage across all factors and used for the final Overall Usability ranking. Results: The category with the highest average score was Technology, with a 0.82


Please make my preprint PDF available to anyone at any time (recommended).
Please make my preprint PDF available only to logged-in users; I understand that my title and abstract will remain visible to all users.Only make the preprint title and abstract visible.No, I do not wish to publish my submitted manuscript as a preprint.2) If accepted for publication in a JMIR journal, would you like the PDF to be visible to the public?
Yes, please make my accepted manuscript PDF available to anyone at any time (Recommended).
Yes, but please make my accepted manuscript PDF available only to logged-in users; I understand that the title and abstract will remain v Yes, but only make the title and abstract visible (see Important note, above).I understand that if I later pay to participate in <a href="http

Introduction Background
A medical center's website is often the first point of contact with the public; its initial impact is responsible for returning users and attracting new visitors [1,2].It has potential to be the first step in improving patient satisfaction as well as attracting new patients [3].In a time when information is expected to be readily available, medical centers use their websites as a key tool for both patient communication and education [4][5][6].The users expect to find current and reliable information on websites that is easily accessible in order to make health-related decisions [7].With many sources available (e.g., Web MD), medical centers are looking to improve their internet presence to better engage with potential consumers [3].

Website Usability
Improving website usability is a noteworthy way medical centers can improve their internet presence to attract and retain more users, and thus reach a larger audience with accurate and reliable information.Usability goes beyond surface-level design, but refers broadly [8] to a product's "user experience", such as ease of navigation or encountered problems within a website [9].It addresses the question of how easy or pleasing a website is to use (which can influence how many users engage with it), the level of engagement, and a website's ability to achieve other objectives.When users are not able to easily access and use a website, they are unlikely to continue using that given source.Alternatively, improved usability can enhance the reach of a website.For this reason websites are facing the increasing need to conform to user expectations, desires, and requirements [10,11].

Usability for Academic Medical Centers
Studies have sought to apply usability analyses to e-commerce, e-government, mobile news apps, and library websites [15][16][17][18].In healthcare, other studies have looked at usability for hospitals, children's hospitals, digital health centers, residencies, and cancer center websites [3,[19][20][21].However, to our knowledge, no studies of usability have been conducted exclusively on academic medical centers in the United States, which included all websites falling under the academic medical centers' domain.Academic medical centers are the intersection of health professional schools, patient care, and academic research.Being that an academic medical center is comprised of numerous institutions that function in their own part to be a part of a greater whole, academic medical centers play a key role in the advancement of medical care [22].Web presence is the way in which an academic medical center can demonstrate the advancements among their health professional schools, patient care, and academic research.It is important for these organizations to utilize usability metrics to not only improve user experience, but to represent themselves well as leaders in innovation.

Objectives
The primary aims of the study were to: 1) Adapt a usability scoring methodology to academic medical centers; 2) apply and test this adapted usability scoring methodology on a sample set of digital health center website and; 3) derive recommendations from these results on potential areas of improvements for our sample of academic medical center websites.

Sample selection
Our focus was on academic health centers in the United States.Indexing every academic medical center in the United States was not in the scope of our review, rather, this study focused on 74 academic medical centers listed on the Association of Academic Health Centers' (AAHC) members page [23].A link was provided for each of the academic medical centers listed, and the links were used to navigate to the appropriate academic medical centers page.Some of the links navigated to the affiliated university's website, therefore, terms such as, "patient care," "health," "healthcare," "clinics," "university health," and "hospitals," were used to find the appropriate academic medical centers main webpage.One link was removed from analysis, University of California System, as this link provided a list of academic medical centers in the University of California System that were already included in the current analysis.This provided a total of 73 academic medical centers that were used for usability testing (Figure 1).

Overview
All website usability testing took place from 1 June 2020 to 15 December 2020.We began by adapting a methodology developed in previous healthcare website ranking literature [3,19,29].We kept the weighted percentages from the previous studies and applied specific formulas to those calculated percentages to create a relative scale for comparing usability scoring (see Multimedia Appendix 1).Of note, previous methodology used a consistent scoring system that used the relative maximum as the reference point of 1 so that every usability test fell between 0-1 [29].It was found one of these tests did not follow this format, so our methodology improved previous testing and applied the same reference range to the values.
Usability was sorted into four broad categories to ensure quantifiable, objective, actionable recommendations for websites: 1) Accessibility -ability of those with low levels of computer literacy to access and navigate hospital's Web presence 2) Marketing -ability to be found through search engines, examining the relevance of descriptions to the links provided 3) Content Quality -grammar, frequency of info updates, material relevancy, and readability 4) Technology -download speed, quality of the programming code, and website infrastructure [3,19].
Each of these categories represent areas of usability in which academic medical centers can communicate more effectively with their audiences.The factors that contribute to each category were originally discussed in Huerta et al [3].To add to the comprehensive nature of this study, we further defined these categories by utilizing supported tools that bolster the credibility and reproducibility of the results of this analysis.These tools were chosen based on their ability to address each of the individual factors in the category criteria and their availability for public ease of use and implementation.

Analysis
All websites were analyzed using a set of established usability tools.These tools were chosen based on their ability to specifically assess the individual factors selected.The tools were also chosen based on their ability to meet industry standards for evaluating the selected factor and for their relative ease-of-use.The process for utilizing each tool was based on the tool's specific instruction manual.The authors collaboratively ensured the proper use of each tool and the proper value to be recorded.There were a total of 2 researchers that collected data and each underwent specific training on utilizing the suite of analytic tools and data entry.Questions and discrepancies were addressed by the project supervisor as they arose.Factors, such as speed, that can vary second-to-second were averaged across two separate tools to provide the most accurate values possible.Overall, all of the tools were run on a total of two computers, to minimize as many outlying technology errors as possible.The selected tools can be viewed in Multimedia Appendix 1.
We began building a database of the top-level uniform resource locators (URLs) associated with each website in our data set.The web crawler processes the URL and creates a topographical map of the website, including all its subpages.For instance, a top-level domain, corresponding to a website's homepage, may be associated with the URL: www.healthcare.org.A subpage of this center might be a page on the team members and be associated with the URL: www.healthcare.org/team.There may be other subpages for specific topics such as the emergency medicine department, the pediatric department, and so forth.Once the web crawler has created a topographical map of a website, that website can then be analyzed for page errors, amount of page content, metadata (i.e.titles, keywords, and descriptions), or other pre-programmed factors [24].
Using these usability tools, we scored each website on four categories: 1) Accessibility; 2) Content Quality; 3) Marketing; 4) Technology.The composite of key factors in each of the four categories contributed to an overall "General Usability" score for each website.Lastly, an overall score was calculated by applying a weighted percentage across all factors and used for the final ranking system.
Below we describe each of the categories we evaluated, and their contributed significance.

Accessibility
Accessibility is a category that refers to how well a website engages a broad audience with varied levels of technical ability, literacy, and disability.This category includes the following components: meta description, functionality, readability, and the overall layout of the website.A meta description refers to the "snippet" page summary presented in a search engine result.Functionality refers to a website's ability to provide content that appeals to a broad range of literacy levels.Functionality also includes features that allow users to access different parts of a website.It is estimated that 43% of American adults have basic or below basic literacy levels [25].The usability of assistive technologies such as screen readers and magnifiers for websites is also assessed by the Accessibility category [26].
In this study, we utilized tools that apply algorithmic scales to rank website reading difficulty and to determine the grade level required to comprehend a website's content.

Content Quality
The Content Quality category refers to our assessment of the attributes of the content on a website.This can include the relevancy of the written information to that particular point in time on a specific topic, generated metadata, and the use of the website's multimedia for imagery.For instance, a website dedicated to supplying information on current closed-loop insulin pumps for diabetics may be evaluated on its ability to provide relevant, fact-driven, answers to questions that people are seeking (i.e.relative costs, ease of use, etc.).The multimedia on a website may also be evaluated, such as for quality (e.g.resolution) and available metadata function to add support to the composed content.Content quality also includes written text, and may evaluate grammar and spelling.

Marketing
The Marketing category refers to our assessment of the discoverability of a website, with a particular emphasis on its Search Engine Results Pages (SERP), which refers to the websites served to users when they search for something online using a search engine, such as Google.Higher placement in search results can lead to greater visibility, and SERPs are considered by some one of the most important elements of digital marketing.The field of Search Engine Optimization (SEO) deals with optimizing a website to place better in SERPs, and effectively implementing SEO may allow healthcare websites to uphold a corporate image as industry leaders [27].Technical SEO auditing, specifically, was beyond the scope of this study.

Technology
The Technology category refers to our assessment of the technical functionality of a website, as opposed to its content.It evaluates the quality of a website's technology and technological design and performance, including its front-end design and user experience to back-end coding infrastructure and server management.The front-end is what the users of the site view when browsing a website.It also involves analysis of the hypertext markup language (HTML) elements to ensure that the user has an easily navigated layout and that the site can be scalable across devices (i.e.computers, mobile phones, and tablets).The back-end involves the programming code upon which the website runs.This code and other website components, such as its databases, are stored on servers, which functionally allow people to view websites from their own devices.The servers also affect the speed of the site, e.g.how quickly it loads for users, which can play a crucial role in gaining and maintaining users and followers.For instance, a previous study conducted by Google showed that a website that takes longer than 3 seconds to load on a mobile device will lose approximately 53% of its users; problematically, that same study revealed that the average mobile website speed is upwards of 18 seconds [27].

General Usability
This was a composite of all the metrics from the prior four categories.This category aims to answer the question: "How good is my website?"This metric may serve as a starting point for healthcare Gale et al organizations to perform an initial audit of their website to look for areas of improvement.

Overall Usability
An overall usability rank order calculation was included to create a comprehensive evaluation of all major and minor factors across all of the five aforementioned categories.From there we assigned a percentage weight to create an all-inclusive usability ranking system.

Results
Scores were assigned to all (N=73) academic medical centers found on the AAHC member list [23]. The

Comparison with Prior Work
This study assessed academic medical center websites utilizing methodology outlined by Calvano et.al. in a publication that ranked usability of digital healthcare center websites [29].Previous studies enabled Calvano et.al. to assess website usability trends including that of hospitals, digital health centers and children's hospitals [3,19,29].In previous studies, Content Quality was the highest scoring category.This was postulated to reflect the healthcare industry's emphasis on providing adequate amounts of information for website users.However, this is accompanied by a lower investment in other usability categories.
Another major trend in previous studies was that the Technology category was the lowest scoring category [3,20,21,29].Interestingly, when these study methods were applied to academic medical center websites, we found the opposite.Technology was the highest scoring category and Content Quality was the lowest.One possible explanation for this is that the significant amount of time elapsed between studies enabled website technology to be updated.In addition, when compared to community hospitals and digital health centers, academic medical centers are larger institutions with more financial capital to invest in website functionality.Academic medical centers also have more expansive websites which may cause difficulty in monitoring the quality of the large amount of content produced.
With reference to academic medical centers, we assert that an increased level of importance should be placed on monitoring content quality, thereby ensuring a higher standard of information presented to the general public.As medical advancements continue to become more complex, and patients become more comfortable with technology use, a larger number of individuals will turn to the internet for assistance in understanding key medical concepts.Therefore, it is of utmost importance that academic medical centers realize their vital role in providing targeted, and relevant, content for their website users.This includes providing germane, concise answers to common medical questions people may be searching for to obtain new users and maintain the existing ones.
Another difference between previous studies and our findings involved the Accessibility category.In the assessment of children's hospital websites, previous studies found Accessibility to the lowest scoring category [19].We found Accessibility to be the second highest scoring category.One possible explanation for this might be that academic medical centers better understand the importance of creating content that is built at the proper literacy levels and technical complexity to ensure ease of access for a broad audience.Academic medical centers tend to serve as leaders in the medical community and are often looked to for guidance from the general public in areas of medical concern, such as the current COVID-19 pandemic.With this in mind, academic medical centers may understand the importance of creating content that is easily comprehended by a range of website users.
Marketing scores were noted to be lower than originally anticipated [19].Pertaining to healthcare, Marketing is important to ensure users are able to easily locate an organization's website within search engine results.It is imperative that academic medical centers employ search engine optimization techniques to enable improved public visibility of their information compared to less authoritative sources.
A specific goal of this research is to promote standardization of website analysis across the healthcare industry, as it has been neglected previously despite being an important facet of other sectors [12][13][14].Recent technological advancements have driven down costs in medicine and increased quality of care [22].Usability analysis is an important element in this process and enables healthcare organizations to improve their website presence.With the heightened awareness of technology's importance in healthcare due to the COVID-19 pandemic, the Web presence of academic medical centers is even more essential to ensuring proper dissemination of health information.

Limitations
This study has several limitations.Not all of the social media accounts were directly accessible from the website, making them more difficult to find through Twitter and Facebook's respective search engines.In most cases, an affiliated Facebook or Twitter page was found.A total of 5 academic medical centers were not associated with a Twitter page and a total of 2 academic medical centers were not associated with either a Facebook or Twitter page.For the academic medical centers that did not have affiliated Facebook or Twitter pages, this either was because one had not been created for the institution or because a link was not provided on their webpage and was not able to be found using the social media's search function.
Assessments of a website's speed can vary depending on the time of the day or the day of data collection.This could be due to changes to the website's servers, internet connectivity, or computer hardware.To minimize sampling bias, the same computer on the same network was used when measuring parameters such as website speed.
All of this information was collected over the course of several months and therefore, some measures may have changed since initial evaluation.
The AAHC provides their own definition of an academic medical center which includes three main components: "...an allopathic or osteopathic medical school; one or more other health profession schools or programs…, and one or more owned or affiliated teaching hospitals or health systems" [30].In accordance with the definition of an academic medical center provided by the AAHC, a clinic webpage would not be considered an academic medical center.However, for consistency of using the full sample of academic medical centers provided on the AACH website, we decided to use clinic webpages for usability analysis if no other institution could be found that better met the definition of an academic medical center.In total, four clinic pages were used in analysis.For example, in the case of Des Moines University -Osteopathic Medical Center, only the university's clinic page was able to be found via their webpage, therefore this was the link that was used in usability analysis.
Our data set was much larger than previous studies [21,29] because we analyzed entire institution websites; many of which had between 500,000 and over one million URLs.Websites were cut off at 750,000 URLs during web crawling as this was felt to be a sufficient sample size without running into random access memory (RAM) limitations.

Conclusion
As an increasing number of individuals look to the internet for medical information, responsibility will be placed on academic medical centers to maintain high quality websites, given their status as respected sources of current medical information and research.This study offers an analysis of the overall need for improvement in website usability by academic medical centers.The average General Usability score was 0.62, showing the necessity for improving usability measures.Academic medical centers may benefit from taking steps to improve various components of their websites in order to reach their audiences.A suggested step is for these organizations to perform periodic usability audits of their websites to identify areas for improvement.Several of these institutions have significant room for improvement of their overall usability, specifically with Content Quality and Marketing, the lowest scoring categories in this analysis.Using content audit tools, institutions can gather data regarding their webpage content and improve upon a number of factors including: ensuring every webpage has a title of appropriate length, that every webpage has an H1 heading, and that the meta-descriptions on the page are concise.Content audits of webpages should be focused on improving the quality of information presented by enhancing aspects such as navigability.Navigability of information can be improved upon by fixing broken web links that are cited in the website, adding alternative text to images, and correctly utilizing keywords on pages.In terms of marketing, we also recommend confirming that Facebook pages and other social media links are highly visible on home screens.In addition, we recommend ensuring that there is a sufficient quantity of content on those social media pages.These and other web page improvements will lead to enhanced usability of web pages for patients and academicians alike, thereby furthering the quality of healthcare information available online.

Table 1 .
category with the highest average score was Technology, with a 0.82 (standard deviation of 0.068, standard error of 0.008).Accessibility was the second highest scoring subcategory, with an average score of 0.77 (standard deviation of 0.059, standard error of 0.007).The third highest scoring subcategory was Marketing with an average score of 0.43 (standard deviation of 0.066, standard error of 0.008).The lowest performing category was Content Quality, with an average of 0.22 (standard deviation of 0.070, standard error of 0.008).The summary statistics across all five categories are presented in Table1and a description of the usability tools used in each of the categories can be found in Multimedia Appendix 1. Academic medical center websites: Summary statistics from usability analysis.Technology -Rosalind Franklin University of Medicine and Science; 5) General Usability -University of Southern California.The overall usability top performing website was the University of Southern California.