The Info Without Side Effects (iWISE) checklist was developed using a sequential, multistage mixed methods design, in which the findings from each step iteratively informed focus and data collection of subsequent stages [17]. The development process comprised five steps: (1) a comprehensive literature review, (2) an expert Delphi study, (3) cognitive interviews with lay users, (4) application tests with laypersons, and (5) predictive validation with a test set (see Figure 1 for a short overview of the development process). For a detailed description of the process, please see the Detailed Methods Description section in Multimedia Appendix 1.

We developed a protocol a priori and published and registered it in the Open Science Framework retrospectively [18]. During the study, we made a few amendments to the protocol.

The study followed the University for Continuing Education Krems’ privacy policy (Article 13 General Data Protection Regulation) and was approved by the university’s ethics commission (EK GZ 23/2021‐2024). Interview participants received written study information and provided verbal informed consent; they received approximately US $35 voucher as compensation. Transcripts were deidentified by replacing all names and other identifiable details with alphanumeric codes. To ensure comprehensive reporting, we adhered to the Standards for Reporting Qualitative Research (SRQR) [19] and the American Psychological Association Style Journal Article Reporting Standards for mixed methods research [20].

We conducted a tailored literature search [21] led by an experienced information specialist (IK) to identify existing checklists and tools as well as conceptual literature relevant to evaluate (online) health information. The iterative search was conducted between March and September 2021 and combined a variety of search methods:

One person screened all the results, and a second person verified the inclusion or exclusion decision. From each included (systematic) review on tools and checklists, we extracted the tools’ and checklists’ names and references.

First, 4 team researchers (CK-K, EK, IK, and UG) merged duplicate and similarly worded items and translated the English items into German. We standardized the item wording by phrasing all items as questions and aligning the responses so that “yes” indicated trustworthiness. The process was done by one person and checked by a second person, and discrepancies or conflicts were resolved by discussion.

Second, we rated each item’s appropriateness (relevance for assessing trustworthiness) and applicability (ease of use without prior knowledge) on 5-point scales in an Excel matrix. In total, 8 researchers (EK, FKA, BL, UG, BK, CK-K, IK, and Isolde Sommer) independently rated all items. Items rated 1‐2 by all were excluded, and the highest-rated per subcategory were retained, aiming for about 50 items for the Delphi study.

We conducted an expert Delphi study to reach consensus on the most relevant checklist items [21,22]. Experts with backgrounds in patient communication, health information, and online communication were identified through internet searches, existing contacts, team suggestions, and snowballing. Eligible experts were contacted via email and invited to participate. Of the 12 invited German-speaking experts from Germany and Austria, 6 participated (4 from Austria and 2 from Germany; 4 women and 2 men).

In an initial online meeting (May 2022), we presented the study aims, procedures, and categories of the preliminary item list. The experts rated each item’s appropriateness and applicability (1‐5 scale) using an Excel matrix, suggested rewording, and proposed new items. The mean, SD, and range were calculated for each item, and revisions were reviewed in a second Delphi round. The final selection was based on the expert ratings and content relevance. Only minor wording changes were suggested. The result of the 2-step Delphi study was an interim checklist version 1.

We used cognitive interviewing to test the interim checklist version 1 with laypersons, focusing on item comprehension and usability. Both think-aloud and verbal probing techniques were applied [23,24].

The database and web searches identified 200 documents that were further evaluated. We also identified 11 relevant reviews or pertinent papers, of which we checked the included tools and checklists (Table S2 in Multimedia Appendix 1). Altogether, 73 documents met our eligibility criteria and were included for data extraction. Figure 2 shows a flowchart of all the documents and items through the development process of the iWISE checklist.

We performed a data extraction of the general information for all 73 included documents. Of these, 36 were journal papers, 27 were web pages, 3 were books or book sections, 4 were reports, and 2 were conference papers, and they spanned the literature from 1997 to 2021 (see Tables S3 and S4 in Multimedia Appendix 1for a complete list of all included references). Of the 73 documents, 46 (63%) were tools or checklists, more than half of which were aimed at laypersons, consumers, or patients (27/46, 59%), and approximately one-third at experts (13/46, 28%), that is, health information producers or health care professionals, and 3 (7%) did not have a specific target audience.

We extracted a total of 1740 items and categorized them into 5 content categories or marked them as excluded. Table 1 shows the subcategories for each category along with a description. The largest number of items (450/1740, 26%) belonged to the transparency category, followed by the presentation of information category (402/1740, 23%). The least number of items belonged to the functional or technical aspects category (163/1740, 9%), which is unsurprising, as this was not our search’s main focus (see Table S1 in Multimedia Appendix 1 for the number of items in each subcategory and respective examples). About 14% (237/1740) of the items had to be excluded, either because of the wrong scope, the need for prior knowledge, the lack of generalizability, or the need to use an external source to apply the item, all of which we defined a priori as reasons for exclusion.

The first step in consolidating the item list resulted in a reduction from 1740 to 449 items. A further reduction step using the research team members’ ratings of the items’ appropriateness and applicability resulted in a list of 46 items (see Table S1 in Multimedia Appendix 1 for the content distribution).

The descriptive results of the 46 items rated in Delphi round 1 are shown in Table S5 in Multimedia Appendix 1. Based on the expert feedback, the items were revised from questions to statements, and 6 new items were suggested. In round 2, the experts rated 52 items (the revised 46 plus 6 additions); the results are provided in Table S6 in Multimedia Appendix 1. The applicability ratings showed greater variation than the appropriateness ratings across the rounds (Table S7 in Multimedia Appendix 1).

After completing both rounds, the research team met to discuss the results and to select items. No fixed a priori criteria were set, as the decisions required a comprehensive review of the data. The experts generally rated appropriateness higher than applicability (mean ranges: 3.2‐5.0 vs 2.2‐4.8). We used a stepwise selection approach: first excluding items with a mean appropriateness ≤4.0 or applicability ≤3.0, then removing items with an applicability from 3.0 to 4.0. We ensured that all content categories remained represented and reinstated the highest-rated item where necessary (eg, an item on quality assurance with an applicability rating of 3.17). Redundant items were removed.

This process resulted in 23 items forming the interim checklist version 1. The selected items had higher average ratings for both appropriateness and applicability than the excluded items (Table 2).

We conducted 2 rounds of cognitive interviews, involving 9 persons in the first round and 10 in the second. We managed to recruit a diverse set of persons for inclusion, with 58% (11/19) women, 58% (11/19) between 40 and 64 years of age, 11% (2/19) and 26% (5/19) with a lower educational background of either compulsory school only or apprenticeship or vocational secondary school, respectively, and 21% (4/19) with a migration background (Table 3).

The first-round results showed that many words used in the interim checklist version 1 were difficult to understand for the users, and some items were rated as less important to the lay users than to the experts. Therefore, we made large modifications to the checklist: we reworded 17 of the 23 items, deleted 1 item, and added another; we revised the instructions on how to interpret the checklist results; and we reordered the items in a way in which the information on a web page would usually be presented to avoid too much scrolling by the users.

In the second round, further modifications to the interim checklist version 1 were implemented, for example, adding subtitles to structure the checklist and provide more guidance to the users. Two additional items were tested with the last 7 cognitive interviews.

After analysis and discussions among the whole research team, a second interim checklist version was created for use in the application test (Table 4). The wording of 13 items was changed, or text was added to the item to give tips on where to find the respective information on the web page. Furthermore, the order of the items was changed once more.

Already in the first round, lay users made remarks on single items that were less important to them than to the experts, for example, “It is clear to which target group the health information applies.” As this was an item deemed important by the experts in the Delphi study, we kept it for the application test but added an example because the term target group was ambiguous for some users. In general, many of the suggested modifications led to shorter and more straightforward sentences and the explicit use of examples (ie, “for example”). We also removed redundancies and used words that were unlikely to be misunderstood, and we avoided negative constructions, as these are difficult to understand in German. During the analysis, we realized that some items may be more important for health information creators than for health information consumers, for example, one item that we dropped after the cognitive interviews: “The health information mentions how well the differences between men and women have been researched.”

We conducted application tests with 20 further lay users (see participant characteristics in Table 3). All 20 lay users returned quantitative and qualitative feedback on using the interim checklist version 2. The mean answer to the general question “On a scale from 1 to 10: how easy or difficult did you find it to use the checklist?” (1=very easy and 10=very difficult) was 4.0 (SD 1.8; range 2-9). We asked the participants to check the items they found difficult to answer, and 4 items were marked by either 8, 9, or 11 participants (40%‐55%; Table 4). Explanations as to why these items were difficult were also given. All this information was taken into consideration in our discussion on the final selection of items for the checklist. As evidenced in Table 4, none of the 4 items that were considered difficult by nearly half or more than half of the lay users made it to the final checklist. The reasons for the difficulties in applying certain items were manifold. For example, regarding item 23 “The health information states whether and how readers were involved in creating the information,” over half of the lay users stated that it was either difficult to find that information on the web page, or the concept of reader involvement in creating health information was unclear to them. Furthermore, no one rated this item as especially important.

Conversely, 4 items were also rated as especially important by 9, 11, or 14 participants (45%‐70%; Table 4), and 3 of these made it to the final checklist. Multiple users had trouble understanding item 14 “The health information indicates how well a fact is scientifically proven,” and 7 also marked it as a difficult item. However, more than half also rated it as especially important, and this item made it to the final checklist.

For item 19 “It is clear that the health information is up to date,” one-third of the participants demanded a clear time frame for their judgment of up-to-dateness. Since the time frame depends on the health information topic and is highly variable, this item was not included in the final checklist, but a more neutral item on currency was included (“It is clear when the health information was created or updated.”).

Of the 100 health information web pages in the test set, 37 were factually correct; that is, had little or no difference between the claimed and factual strength of evidence; 63 web pages were incorrect. The regression coefficients of the model for predicting the correctness of the health information web pages are shown in Table 4, along with the estimated errors and 96% CIs. The higher the values, the better the respective item’s predictive validity in predicting the trustworthiness of a health information web page.

Table 4 also shows the agreement among the lay users and among the experts, respectively. Based on the initial results, we tested a few item combinations until we arrived at the final checklist (Textbox 1). As stated earlier, all the quantitative and qualitative results were integrated into our final decision.

Some items with the highest regression coefficients for predicting a correct health information web page were not chosen because of other reasons. For example, item 23 “The health information states whether and how readers were involved in creating the information” with the highest predictability values was only fulfilled in 8 of 100 health information web pages. Therefore, the practical value of this item’s predictive validity is low. Furthermore, more than half of the lay users perceived this item as difficult to answer, and the agreement among the lay users in answering this item was very low (Table 4). The situation was similar with item 22: “It is described how the information was created. For example, which studies were considered and why, and which were not.” While this had a good predictive value, 8 of 20 lay users perceived it to be difficult to answer; therefore, we decided not to include it in the final checklist, as only items that are easily answered and clear to laypeople are useful in the final checklist.

After several discussion rounds among the whole research team, we arrived at the final iWISE checklist with 7 items (Textbox 1).

According to our statistical model, the probability that the health information is correct if all of these items are marked with “yes” is 99.9%. The probability that the health information is correct if all of the items are marked with “no” is 20.4%. However, this requires that the items are marked correctly by the user, which, in practice, may not be the case.

We published the iWISE checklist in German on our project website [32] and also provided brief explanatory notes for each item to help users apply the checklist. We also created a video explaining all 7 items. The English version of the iWISE checklist, including explanations for lay users, is available in Multimedia Appendix 1 and can also be downloaded as a PDF [33].

In this mixed methods study, we developed and validated the iWISE checklist, a concise and easy-to-use tool enabling laypersons to evaluate the trustworthiness of health information on websites. To our best knowledge, iWISE is the first checklist that can be applied by laypersons in everyday information-seeking situations without the need for prior training. Consistent with our aims, we developed iWISE in a multistage development process, combining evidence synthesis, expert review, iterative cognitive testing, application tests, and predictive validation, which resulted in 7 items that lay users can both understand and reliably apply. To further support lay users, we also provide brief, plain-language explanations for each item, including examples and tips on where to find the relevant information on a web page.

The final iWISE checklist contains 7 items that assess key aspects of health information trustworthiness, including the absence of advertising, a balanced presentation of information, the limited use of professional jargon, an origination from an independent organization, the citation of sources, the mention of scientific validation, and the presence of a publication date. The checklist demonstrated the ability to distinguish between evidence-based and nonevidence-based health information web pages in the German language: the validation testing showed that when all items were marked with “yes,” there was a nearly 100% probability that the health information was also factually correct.

There are 3 main features that characterize the iWISE checklist: it is more user-friendly for laypersons than prior checklists, it was developed with the extensive involvement of laypersons, and it reliably identifies trustworthy health information on web pages, as demonstrated by the validation.

Our checklist has some limitations. With 100 web pages, our validation test set of health information articles was relatively small. We tested our checklist with a limited set of only 10 popular health questions—and with 10 articles for each of these questions, respectively. Our test set might therefore be biased and not be representative of the broad mass of health information that a typical user encounters when searching the web. A different, more varied, and larger test set might have resulted in a slightly different combination of checklist items. However, iWISE is comparable with other checklists that have either used a smaller validation test set or that deal with one health topic or both, for example, the MAPPinfo checklist was tested on 57 web pages [11], the Quality Evaluation Scoring Tool on 45 online articles on Alzheimer disease treatment and prevention [37], and Dobbins et al [38] used 120 web resources and another 107 in a second validation round on healthy aging.

The predictive validity results, one of the factors we considered in selecting the final checklist items, are based on the ratings from the application test with the research team. The laypersons’ ratings were not integrated in this validation step. However, the perspectives of the laypersons were considered in other ways, and we specifically ensured that all items were well understood by our lay user group.

Our checklist has only been validated for information on treatments or health decisions. It has not been tested on websites lacking interventional information. The checklist might therefore not be valid for judging the accuracy of information on symptoms for the diagnosis of a disease or on the health risks of noninterventional hazard exposure.

The iWISE checklist was tested with German-speaking laypersons in Austria and validated on a test set of websites in the German language. We cannot rule out that the checklist works differently when tested in other languages. In addition, our checklist is not tested with social media postings or health information videos and cannot be used for evaluating medication leaflets.

The iWISE checklist offers an accessible and empirically validated way for laypersons to judge the trustworthiness of online health information. By reducing a complex evaluation process to 7 intuitive items, iWISE has the potential to strengthen critical health literacy in everyday decision-making. In a digital environment where misinformation is widespread, such an accessible tool can help individuals navigate health claims with greater confidence while also encouraging health information providers to adopt more transparent and balanced communication practices.

Because iWISE was developed and validated in a German-language context, future research should examine its applicability across other languages and cultural settings. Research teams in other countries may wish to validate or adapt the checklist for their own health information environments. In addition, the checklist was developed for web pages and should be tested for use with social media posts and health information videos, where misinformation is especially prevalent. Building on this need, our team has already initiated a follow-up project to develop and evaluate a complementary tool specifically tailored to assessing the reliability of health information on social media platforms.