Background: As patients are increasingly searching for information about their medical condition on the internet, there is a need for health professionals to be able to guide patients toward reliable and suitable information sources on the internet.
Objective: The aim of the study was to develop a clinical tool for health care professionals to assess the usability and quality of the content of websites containing medical information that could be recommended to patients.
Methods: A 3-round modified electronic Delphi (eDelphi) study was conducted with 20 health care professionals.
Results: In round one of the eDelphi study, of the 68 items initially created, 41 items (29 on usability and 12 on content) were rated as important or very important by more than half of the panel and thus selected for further evaluation in round two. In round two, of the 41 items chosen from round 1, 19 were selected (9 on usability and 10 on content) as important or very important by more than half of the panel for further evaluation. As a result of round three, 2 items were combined as a single item, leaving the instrument with 18 items in total (8 on usability and 10 on content). The tool is freely accessible online.
Conclusions: The CUE-tool can be used to (1) evaluate the usability and reliability of the content of websites before recommending them to patients as a good information source; (2) identify websites that do not have reliable content or may be difficult for patients to use; (3) develop quality websites by using the criteria in the CUE-tool; and (4) identify different qualities between different websites.
Patients are increasingly searching for information about their medical condition on the internet. In high-income countries 75% of the population reported that they search the internet for health information [, ]. Using the internet as a source for different types of health-related information (eg, reading about medications, reading about other persons’ health experiences, watching health-related videos, and signing up for different health email updates) is more common among patients with a chronic illness compared with people without a chronic illness [ ]. Health-related information on the internet is gradually replacing health professionals as important sources of reliable and independent information regarding health and treatment. There are advantages related to online patient education. For example, patients with chronic diseases reported, after seeking information from disease-specific websites, that they were taking their medications more regularly and adhering to treatment to a greater extent [ ]. However, when patients lack guidance, there is a risk that their condition becomes worsened by, for example, waiting for too long seeking health care with symptoms of deterioration [ ].
Although websites and smartphone apps are available with reliable information, there are also numerous examples of low-quality information regarding health and medical problems accessible through the internet . Furthermore, the information could be very reliable (eg, peer-reviewed open access articles) but less suitable or understandable for the average patient.
Self-care (a rational process involving purposeful choices and behaviors, reflecting knowledge and thought ) is essential in the management of most illnesses, and knowledge about own health condition has been identified as an important prerequisite for successful self-care. To gain knowledge, the patient needs access to reliable and understandable information [ ]. In order to gain knowledge, whether from the internet or other sources, the patient needs to be able to both read and understand the information [ ]. As the information available on the internet is excessive, patients increasingly turn to health care for help with choosing reliable websites [ ]. Besides, the information available on websites is not always of acceptable quality or usable for the patient. Therefore, there is a need for health professionals to be able to guide patients toward reliable and suitable information sources on the internet.
An important result of a comprehensive website evaluation  was the need for a practical, easy-to-use tool to evaluate websites. There are a number of different tools to evaluate websites; however, these tools seldom consider both the reliability and appropriateness of the information as well as the readability/comprehensibility of the information at the same time.
In this study, we aimed to develop a clinical tool for health care professionals to assess the usability and quality of the content of websites containing medical information that could be recommended to patients.
Electronic Delphi Study
For the development of a tool to assess the quality of websites containing health-related medical information for patients, a 3-round modified electronic Delphi (eDelphi) study was conducted (). An eDelphi study is a structured process distributing a series of questionnaires during several rounds to gather information and set priorities or gain consensus regarding a specific issue [ , ]. The eDelphi technique allows the inclusion of a large number of individuals across diverse geographical locations without physically meeting them. The eDelphi technique is often conducted via online web surveys, offering a number of advantages, as they are quick to set up, relatively low cost, and provide high level of data security [ ]. Systematic feedback, structured information flow, and iteration and anonymity are the main characteristics of an eDelphi technique. Systematic feedback of experts’ responses takes place in-between rounds by informing individual experts about the group opinions. Iteration takes place by presenting feedback via a certain number of rounds [ ].
Procedures and Participants
The panel consisted of health care professionals, selected based on their publications within patient education or that they were members of the CESAR (Collaboration and Exchange in Swedish cardiovascular caring Academic Research) network, a professional research network in Sweden. We aimed for a multidisciplinary panel of health care professionals, with diversity in gender and from variety of countries. The health care professionals that participated in the first round were also approached for the second and third rounds. Study data were collected and managed using REDCap electronic data capture tools hosted at the University of Iceland .
Before round one of the study’s eDelphi, items on usability and content were created based on available literature on website evaluation and the general information needs of patients. The theoretical perspective of empowering patient education  guided the development of the items in the content part of the tool. In empowering patient education, the emphasis is on patients’ knowledge expectations and the knowledge patients receive. The more patient expectations of knowledge are met with the received knowledge, the more possibilities there are for empowerment and self-management [ ].
A total of 68 items were created for the first round, of which 58 were based on the available literature on tools for evaluation of websites and 10 on content from the multidimensional empowering patient education .
The panel was asked to rate the importance of each item on a 4-point scale (response options from 1 [Very important] to 4 [Not important at all]) and invited to comment on each item.
Items were selected for round two if more than half of the participants rated them as important or very important and the same method was applied for round three. In round three the panel was asked to answer 5 additional questions on the developed tool:
1. Are there any items missing?
2. Are there any items unclear?
3. Please look at the examples given in the tool. Do you agree on the examples or do you have suggestions for better examples?
4. Please give your thoughts about the end product of the tool.
5. Do you have a suggestion on a name of the instrument that is easy to use and reflects the area of use?
After the development of the tool, it was translated from English to Swedish, Icelandic, and Dutch and its face feasibility tested by nurses and allied professionals at the EuroHeartCare Conference 2019, Icelandic Nurses’ Association Conference Hjúkrun 2019, Icelandic nursing students, and in 2 primary care centers and 1 in-hospital heart failure clinic in Sweden. These health care professionals could choose any website including patient information, in any language, and they were asked if the aim of the development of the CUE-tool was met and their opinion about the usability of the scale. All of the health care professionals agreed that the aim was met, and they were positive about using the tool to evaluate websites. No adjustments were needed.
In total, 20 out of 34 health care professionals, from 5 countries, responded to the invitation to participate in the study in round one and 18 of them responded in rounds two and three. The panel came from Sweden, the Netherlands, Iceland, Norway, and Finland. The panel members’ age ranged from 30 to 69 (mean 46 [SD 11]) years and 15/20 were female (75%). Among the panel members, 13 had a doctorate, 4 had a master’s degree, and 1 had a bachelor’s degree ().
|Demographics||Round 1 (N=20)|
|Female, n (%)||15 (75)|
|Age (years), mean (SD)||46 (11)|
|Education, n (%)|
|Master’s degree||4 (20)|
|Bachelor’s degree||1 (5)|
|Experience in patient education (years), mean (SD)||15 (10)|
|Main area of work role, n (%)|
|Clinical and research||6 (30)|
In round one of the eDelphi study, of the 68 items initially created, 41 (29 on usability and 12 on content) were rated as important or very important by more than half of the panel and selected for further evaluation in round two ().
In round two, of the 41 items chosen from round 1, 19 were selected (9 on usability and 10 on content) as important or very important by more than half of the panel, and thus these items were considered for further evaluation.
As a result of round three, 2 items were put together as a single item, leaving the instrument with 18 items in total (). The panel did not miss any items in the tool, and all the items were clear. Examples given in the instrument to clarify items were also agreed on.
As an end product, the panel preferred a scoring system with 2 separate summative scores, 1 for usability and 1 for content, ranging from 0 to 100. From the suggestions made by the panel on a name for the tool, the authors chose the CUE-tool as an acronym for “The Credible and Usable Evaluation of patient education tool for web-sites.”
The nurses and allied professionals (N = 100) who tested the CUE-tool (on paper) when evaluating websites for patient educational purposes had no problems using it. All items were clear and the tool was seen as an addition to practice. To have the tool online with a summative scoring system was seen as an asset. The tool is freely accessible online  and a copy of the tool is also presented as . A summative scoring and reliability assessment will be performed in the future.
Because patients are increasingly searching for information about their medical condition online, it was important to develop a tool for health care professionals so they can advise patients on suitable and reliable websites from which they can seek disease-related information.
The CUE-tool is an easy-to-use website evaluation tool that helps the user to evaluate both the website’s usability and quality of its content. A recent review  evaluated patients’ preferences for the design features of an effective online education website and found that the information should be patient tailored, interactive, and the content credible and clearly presented. Patients also found multimedia and high interpretability to be essential design features of online patient education websites for chronic disease management. All these features are assessed when using the CUE-tool.
The composition of the eDelphi panel may have affected the results of this study and the development of the tools because the majority were researchers, although of different backgrounds (age and profession). However, most of them had previous work experience as clinical nurses at hospitals, in both wards and outpatient clinics. Although we used patient knowledge expectations while developing the content items in the CUE-tool by using the theoretical perspective of empowering patient education , we did not include patients in the eDelphi panel. This decision was made because the tool was designed to be used as a clinical tool to assist health care professionals in finding websites that could be recommended to patients. Although the CUE-tool is developed to be used as a clinical tool, it is also useful outside of health care, for example, in patient organizations or in developing websites for patient education.
The use of empowering patient education  as the theoretical foundation of the content items and performing an eDelphi study to develop the CUE-tool may provide specific future possibilities of its applications in (nursing) research and education besides its clinical usefulness. For example, the CUE-tool can be used to (1) evaluate the usability and reliability of the content of websites before recommending them to patients as a good information source; (2) identify websites that do not have reliable content or may be difficult for patients to use; (3) develop quality websites using the criteria in the CUE-tool; (4) identify different qualities between different websites. Accurate and detailed assessments of available websites providing health information can be a valuable resource in teaching strategies to increase knowledge and self-care of patients. Accordingly, health care professionals can create new teaching interventions and revise curricula based on reliable websites identified in these assessments. For further validation, we will include patients’ perspectives and the reliability of the scoring system of the CUE-tool will be assessed in future research.
Conflicts of Interest
The CUE-tool: The Credible and Usable Evaluation of patient education tool for websites.PDF File (Adobe PDF File), 136 KB
- Bujnowska-Fedak M, Waligóra J, Mastalerz-Migas A. The Internet as a Source of Health Information and Services. Adv Exp Med Biol 2019;1211:1-16. [CrossRef] [Medline]
- Madrigal L, Escoffery C. Electronic Health Behaviors Among US Adults With Chronic Disease: Cross-Sectional Survey. J Med Internet Res 2019 Mar 05;21(3):e11240 [FREE Full text] [CrossRef] [Medline]
- Dolor RJ, Yancy WS, Owen WF, Matchar DB, Samsa GP, Pollak KI, et al. Hypertension Improvement Project (HIP): study protocol and implementation challenges. Trials 2009 Mar 26;10(1):13 [FREE Full text] [CrossRef] [Medline]
- Vasaroangrong T, Thosingha O, Riegel B, Ruangsetakit C, Viwatwongkasem C. Factors influencing prehospital delay time among patients with peripheral arterial occlusive disease. Eur J Cardiovasc Nurs 2016 Jun 20;15(4):285-293. [CrossRef] [Medline]
- Keselman A, Arnott Smith C, Murcko AC, Kaufman DR. Evaluating the Quality of Health Information in a Changing Digital Ecosystem. J Med Internet Res 2019 Feb 08;21(2):e11129 [FREE Full text] [CrossRef] [Medline]
- Riegel B, Carlson B, Moser DK, Sebern M, Hicks FD, Roland V. Psychometric testing of the self-care of heart failure index. J Card Fail 2004 Aug;10(4):350-360. [CrossRef] [Medline]
- Riegel B, Jaarsma T, Strömberg A. A middle-range theory of self-care of chronic illness. ANS Adv Nurs Sci 2012;35(3):194-204. [CrossRef] [Medline]
- Speros C. Health literacy: concept analysis. J Adv Nurs 2005 Jul;50(6):633-640. [CrossRef] [Medline]
- Ingadottir B, Blondal K, Jaarsma T, Thylen I. Perceptions about traditional and novel methods to learn about postoperative pain management: a qualitative study. J Adv Nurs 2016 Dec 13;72(11):2672-2683. [CrossRef] [Medline]
- Klompstra L, Johansson Östbring M, Jaarsma T, Ågren S, Fridlund B, Hjelm C, et al. The Appropriateness and Presentation of Commonly Available Cardiovascular Web Pages Providing Information About Cardiovascular Diseases. CIN: Computers, Informatics, Nursing 2019;37(10):493-497. [CrossRef]
- Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs 2006 Jan;53(2):205-212. [CrossRef] [Medline]
- Keeney S, Hasson F, McKenna H. The Delphi Technique in Nursing and Health Research. Chichester, West Sussex, UK: John Wiley & Sons; 2011.
- Gill FJ, Leslie GD, Grech C, Latour JM. Using a web-based survey tool to undertake a Delphi study: application for nurse education research. Nurse Educ Today 2013 Dec;33(11):1322-1328. [CrossRef] [Medline]
- Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009 May;42(2):377-381. [CrossRef] [Medline]
- Valkeapää K, Klemetti S, Cabrera E, Cano S, Charalambous A, Copanitsanou P, et al. Knowledge expectations of surgical orthopaedic patients: a European survey. Int J Nurs Pract 2014 Dec;20(6):597-607. [CrossRef] [Medline]
- CUE-Tool. URL: https://liu.se/en/research/cesar-network/cue-verktyget [accessed 2021-02-07]
- Win KT, Hassan NM, Oinas-Kukkonen H, Probst Y. Online Patient Education for Chronic Disease Management: Consumer Perspectives. J Med Syst 2016 May 4;40(4):88. [CrossRef] [Medline]
|CESAR: Collaboration and Exchange in Swedish cardiovascular caring Academic Research|
|CUE-tool: The Credible and Usable Evaluation of patient education tool for websites|
Edited by G Eysenbach; submitted 20.07.20; peer-reviewed by N Sharifzadeh, J Taylor; comments to author 17.10.20; revised version received 30.11.20; accepted 24.12.20; published 17.02.21Copyright
©Leonie Klompstra, Maria Liljeroos, Johan Lundgren, Brynja Ingadottir. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 17.02.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.