We conducted an observational descriptive assessment and scoring using nominal group technique (NGT) [37,38] on a global sample of COVID-19 dashboards. Each dashboard was reviewed using a study-specific assessment tool that was piloted and validated among a panel of scorers (n=17) prior to its use [37,38]. NGT was chosen over other consensus methods (eg, Delphi) for scorers to independently appraise a subset of dashboards using the assessment tool and collectively discuss what makes them actionable through a series of workshops [38,39]. All workshops were conducted virtually rather than face-to-face in accordance with pandemic-related public health measures.

Our search strategy for dashboards aimed to be thorough but not exhaustive. This was in line with our aim of exploring the state of the art of public web-based COVID-19 dashboards. An initial list of dashboards was collected through sampling conducted from May 19 to June 30, 2020. Three methods were applied: (1) surveying the authors; (2) surveying other international networks of public health, health services, and system researchers and practitioners (Young Forum Gastein, European Public Health Association, and European Network of Medical Residents in Public Health); and (3) snowballing of sources identified through (1) and (2). The sampling survey was developed using a Google Forms data collection tool and disseminated by email (Multimedia Appendix 1).

The consolidated list of dashboards was screened by one team member with the aims to confirm the inclusion criteria were met; exclude duplicates; and assess the available languages for each dashboard against the panel’s competencies. Dashboards were labeled as red (exclude), green (include), or yellow (obtain second opinion). A second team member assessed dashboards labeled yellow, from which a final joint decision on inclusion or exclusion was made.

An assessment tool was developed by drawing primarily on two existing theoretical models. From communication sciences, we applied the Lasswell model (1948) [41], which states that for mass communication processes to be understood, each element of “who (says) what (to) whom (in) which channel (with) what effect” has to be presented and understood. These five elements—the communicator, message, medium, audience, and effect—informed the basis of the assessment tool’s considerations. We tailored these considerations to the communication of COVID-19 data by drawing on the emerging discipline of performance intelligence in health [36,42]. Specifically, we incorporated key considerations from a definition of actionability and its notions of fitness for purpose and use (Barbazza et al, unpublished data, 2021). The resulting considerations are in line with existing health information instruments (eg, [43,44]), although they were tailored to the aims of the study.

These considerations were clustered to depict COVID-19 dashboards by their general characteristics and a description of why, what, and how data is communicated, followed by an appraisal of their overall actionability (Table 1). Actionability scores were defined on a Likert scale from “not actionable” (score=1) to “extremely actionable” (score=5) and assigned based on the scorer’s judgement of the considerations assessed and their expert opinion of the dashboard’s fitness for purpose and use. Scores were accompanied by a written statement explaining the rationale behind the response. In line with the study’s aim to consolidate key features of highly actionable dashboards, the scoring was merely a means to this end: the panel’s individual appraisal of actionability facilitated the clustering of the actionability of the dashboards as low (score=1 or 2) or high (score=4 or 5) for further collective deliberation on their common features.

An Excel-based tool (Microsoft Corporation) was developed to record our findings. Each consideration of the assessment tool was formulated as a question with defined answer options. The tool included the underlying theory for the considerations by referring back to the concepts applied and available evidence [1,2,5,16,30,31,33,45-55] (Multimedia Appendix 2) to remind the panel of the significance of each consideration and aid the assessment and scoring process.

A prototype of the assessment tool was piloted by two authors on five dashboards. The extracted data were reviewed jointly with two other team members. This resulted in refinements to the phrasing of the questions and answer options. A second iteration of the assessment tool was then piloted with the panel of scorers on a sample of 18 dashboards representing a range of contexts, levels, and organization types. Each dashboard was independently reviewed by two scorers. Prior to piloting, a virtual training session with the panel of scorers was organized, recorded, and disseminated to serve as a resource. Each scorer was given six days (June 17-22, 2020) to review their two assigned pilot dashboards.

The pilot data were reviewed to assess the consistency of responses (ie, scorers of the same dashboard recorded equivalent answers) and meaningfulness of the answers (ie, the answer categories were meaningfully differentiated between dashboards). Where possible, the open-ended answer options of the tool were further specified into categorical values based on recurrent themes in the pilot data set. Definitions were added for key terms based on comments by the scorers. The reviewed pilots and tool amendments were returned to the panel of scorers, and a follow-up meeting was organized to discuss the reviews.

Each scorer was assigned between 5 and 12 dashboards to assess. The dashboards were distributed with first order priority given to the language competencies of each scorer. To synchronize the assessment, the scorers were given a 2-week period to complete data extraction. The assessment was limited to each dashboard’s main page, and a “one-click-away policy” was applied by which content accessible within one click of the main page was also assessed. To store a record of the dashboard on the date it was reviewed, the main page of each dashboard was archived, generating a permanent and publicly available record of its contents [56].

The data records from each scorer were consolidated by the lead authors into a master data set for analysis and subsequently underwent a series of data quality checks to detect data entry errors, inconsistences, or missed fields. In all instances where errors were detected, corrections were suggested and discussed jointly; once agreed upon, the changes were entered into the master data set.

The findings were totaled and averaged by research question. Free text fields and comments were analyzed in a deductive and inductive approach: topics explored in the tool (Multimedia Appendix 2) were used to guide the deductive thematic analysis [57], and new themes that emerged were identified using an inductive approach [58]. This included an analysis of indicator titles using an existing classification of types of pandemic-related information [3]. Due to the observed variability in phrasing of indicator titles and calculations, key performance indicators were grouped by themes.

A workshop with the panel of scorers was organized to discuss the preliminary results and distribution of actionability scores. During the workshop, panelists individually shared the rationale for their scoring of dashboards with low (score=1 or 2) and high (score=4 or 5) actionability. The common features of dashboards scored as highly actionable were discussed to further calibrate the panel’s scoring of the actionability. From this discussion, a working list of actionability features was consolidated.

All panelists were returned their original data records and given 1 week to revisit their initial actionability scoring, drawing on the discussion during the workshop. Panelists were given the opportunity to increase each score, lower it, or leave it the same. Following rescoring, the distributions of the scores were recalculated. The data records for the top dashboards (score=5) following this second round were consolidated and provided to the panel, together with the working set of actionability features. A second consensus workshop was convened and, in a similar way to the previous workshop, a round table was conducted for each scorer to share their views. This was followed by a joint discussion to reach agreement on the common features of highly actionable dashboards.

Using our multimethod search strategy, we initially identified 265 COVID-19 dashboards. More than 40 respondents contributed to the sampling survey, including all members of the study team and international public health experts. Following screening of each dashboard’s main page, 103 dashboards were excluded. The remaining 162 dashboards were distributed among the panel of scorers for full review. During the assessment process, 5 additional dashboards were excluded and 1 new dashboard was included. A final total of 158 dashboards was included for further analysis (Figure 1).

Data extraction and the first round of scoring were conducted in a 2.5-week period between July 6 and 23, 2020. The data extract and archived version of each dashboard were referred to throughout the study. Therefore, any updates following this date were not accounted for. The 158 dashboards were assessed in 22 different languages, predominately in English (n=85, 53.8%), followed by Russian (n=11, 7.0%), Spanish (n=9, 5.7%), French (n=9, 5.7%), and Chinese (n=6, 3.8%). A full listing of the dashboards assessed is available in Multimedia Appendix 3.

Table 2 summarizes key characteristics of the 158 dashboards assessed. Our sample included dashboards reporting on 53 countries in all 6 WHO regions [59]. On the date of the review, the severity of the pandemic with regard to total cases and deaths varied widely between location as reported in Multimedia Appendix 3.

More than half of the dashboards (93/158, 58.9%) were developed for use at the national level. Nearly two-thirds of the dashboards (100/158, 63.3%) were developed by government authorities at the national, regional, or municipal level. New initiatives or organizations formed in response to COVID-19 accounted for 10.1% (16/158) of the dashboards assessed [29,60-74].

With regard to language, only one-fifth of the dashboards were available in more than one language with full functionality (32/158, 20.3%). In terms of their scope of information, gauged according to the content of the dashboard as well as information to which users were redirected through affiliate links, almost all the dashboards were epidemiological in focus (156/158, 98.7%), followed by providing information on infection control measures and health system management (65/158, 41.1%, and 49/158, 31.0%, respectively).

A quarter of the dashboards (45/158, 28.5%) explicitly stated the intended purpose of their reporting. Of these 45 dashboards, the statements spanned three main themes: (1) high-level reporting to create trust and aid overall compliance (25/45, 56%); (2) sub-national reporting targeting policy interventions, including benchmarking (12/45, 27%); and (3) individual-risk assessment (8/45, 18%).

Only 14.6% (23/158) of the dashboards explicitly stated the intended audience (end users). Target users predominately included the general public (20/23, 87%) and, in a few instances, more specific audiences such as travelers or subject matter experts (6/23, 26%). When examined by the level of reporting, national-level dashboards were less likely to explicitly state the intended audience (9/93, 10%), while international- and municipal-level dashboards were more likely to do so (7/25, 28%, and 2/7, 29%, respectively).

Of the 158 dashboards assessed, 20 (12.7%) reported both the purpose and intended user explicitly. The profiles of these dashboards, in terms of their levels of reporting and the types of organizations that developed them, did not differ from the characteristics of the general sample. For the remainder of the analysis, the sample of dashboards was aggregated rather than subdivided by the intended purpose of the use and audience, due to the limited explicit statements of both.

Table 4 summarizes the types of analysis and presentation of data. The dashboards predominately reported indicators over time (138/158, 87.4%), and most of these breakdowns were by day (128/138, 92.8%). Of the dashboards, 40% reported data on two geographic levels (eg, national and regional or regional and municipal). In the case of national-level dashboards (n=93), geographic breakdowns predominately included regional comparisons (73/93, 79%), with some municipal-level (28/93, 30%) and international-level (25/93, 27%) comparisons. Breakdowns by neighborhood (post–code-level) were reported in only a few instances (4/93, 4%).

In addition to geographic breakdowns, more than half of the dashboards (96/158, 60.8%) analyzed data by other breakdowns: on average, three types of breakdowns were included. Of these 96 dashboards, the most common breakdowns included by age (79/96, 82%), sex (71/96, 74%), and mode of transmission (26/96, 27%). Other breakdowns, although less frequently reported, included race, ethnicity, long-term care facilities, health care workers, comorbidities, and socioeconomic status.

As per our inclusion criteria, all dashboards used some form of visualization. On average, two types of visualizations were included per dashboard. These included graphs or charts (134/158, 84.8%), maps (111/158, 70.3%), and tables (95/158, 60.1%). Almost half of the dashboards (76/158, 48.1%) did not include written descriptions to clarify either the quality or meaning of the data, while 31/158 dashboards (19.6%) provided both.

More than half of the dashboards (104/158, 65.8%) used some technique to simplify the data. In these 104 dashboards, color-coding was most often used (n=93, 89.4%), followed by size variation (n=40, 38.5%). The majority of dashboards (126/158, 79.7%) included some element of user interaction. These elements mostly included the possibility to present more information (eg, pop-up windows), change the information (eg, different breakdowns), or change the display (eg, switch from table to map).

In the first round of scoring, 21 of the 158 dashboards assessed (13.3%) were scored with the highest actionability score (score=5), and 18 dashboards (11.4%) received the lowest score (score=1), for a mean score of 3.01 (SD 1.20). The second round of scoring resulted in a final total of 20 dashboards that were scored as the most actionable. A quarter of the dashboards (40/158, 25.3%) were scored differently: 24 scored lower, and 16 scored higher. All 17 panelists completed both rounds of scoring. Details on the distribution of scoring by panelist and between rounds are summarized in Multimedia Appendix 5.

The panel workshop following the first round of scoring resulted in a total of 18 features that characterized highly actionable dashboards. After rescoring, these features were further discussed among the panel to consolidate the list in terms of their description and importance as well as its consistency and completeness as a set. A final total of seven key features common to highly actionable dashboards were agreed upon (Table 5). There was consensus among the panelists that some dashboards excelled in certain features over others. These dashboards are noted as illustrative examples.

With this study, we set out to assess the state of the art of public web-based COVID-19 dashboards globally during the initial stage of the pandemic (July 2020) and identify features common to the dashboards that were found to be highly actionable. We assessed 158 dashboards, each operating in a different context. Their differences aside, the dashboards analyzed in this study ultimately share a common aim: to serve as both a communication tool and call for individual and collective action to respond to the COVID-19 pandemic. Despite their contextual differences (or because of them), our results indicate that some dashboards fulfill their function of communicating, informing decision-making, and supporting behavior change better than others. Moreover, while it is also clear there is no single approach to developing a dashboard, our results suggest that introducing certain features may enhance the actionability of a dashboard.

Knowing the audience and their information needs was identified as a key actionability feature, which corresponds with the Lasswell model for effective communication ([1,41]; Barbazza et al, unpublished data, 2021). However, clear reporting of a dashboard’s purpose (its “why”) and audience (for “whom”) was infrequent. This may be explained in part by the fact that the majority of the dashboards were developed by public authorities and hosted on existing web pages. Hence, the target audience (citizens) and the aim (constitutional mandate to protect health) may be considered implicit. However, without clarity on the intended use and user of a dashboard, its development is steered by the potential to be useful rather than addressing a specific information need [32,93-95].

“What” a dashboard communicates through its content is not a neutral window into the available data. It is the result of judgment, discernment, and choice [14]. The average of 5 indicator themes reported per dashboard can be considered to be a manageable volume and is in line with the evidence that “less is more” [33,47]. It is the breadth of types of information presented that is concerningly narrow, with only a handful of dashboards addressing the WHO-recommended four types of information needed for a complete picture of the pandemic [3]. For example, indicators reporting on population behavioral insights gauge the compliance of citizens with infection control measures; thus, they are an important tool for maintaining public trust. However, in our sample, this type of information was rarely reported. This may be due to data infrastructure limitations and the limited availability of these data, especially in the early phases of the pandemic. Similarly, less than half of the dashboards reported on health system management indicators, despite the importance of these indicators in informing the management of both COVID-19 and non–COVID-19 services. Dashboards that did report on these non-epidemiological types of information may serve as inspiration for drawing on innovative data sources and indicators [28,60].

Clarity around data sources and indicator calculations (metadata) are critical for overall quality, credibility, and trustworthiness of reporting [46,48,49]. For transparency on how data were collected and insights into “what lies behind” the reported indicators, providing explicit data sources and calculations should be considered a minimum requirement. Nonetheless, our findings signal that these provisions are not a given. Further efforts are needed internationally and nationally to standardize indicator calculations and set requirements of what constitutes good practice in public reporting of pandemic-related data.

In terms of “how” content is presented, dashboards should be viewed as tools for making clear links between current trends and past policy decisions and individual behavior. Doing so connects change-points and actions, which has been found to contribute to an indicator’s use [96,97]. It also serves to leverage the two-way communication potential of dashboards. Dashboards that fail to make the connection between the past and present miss the opportunity to communicate the effects of users’ decision-making back to them. Beyond describing the past and present, only a handful of dashboards went further and employed predictive analytics by illustrating different future scenarios of “what could happen.” The lack of precision of predictive models and simulations early in the pandemic likely stunted their use. Use of both descriptive and predictive approaches to dashboard design and tighter links between infection control policies and their effects should be further explored into the next phases of the pandemic.

We found frequent use of different display options and interactive techniques among the dashboards assessed. However, the analysis of data by location and by population subgroups was limited overall, which may restrict their utility for individual-level decision-making purposes. The challenge to report data locally and disaggregate the data by relevant breakdowns such as age, sex, socioeconomic status, and ethnic or racial groups may be in large part due to data infrastructure limitations and perceived legal obstacles [98]. Without collecting, registering, and using data about meaningful population subgroups, there is a risk of not being informed about these important (and modifiable) differences [98].

Finally, an actionable dashboard is based on complete, timely, and transparent data that is prepared, contextualized, and presented so that it can be used as information [99]. Our assessment found an overall underuse of known and proven delivery techniques, in particular, the use of explanatory narratives. Plain language text to clarify complicated information has proven to make end users more motivated and confident in using information in their decision-making [1,47,54]. Although commonly used software for the development of dashboards (eg, ArcGIS) has served to optimize their single-screen design, the embedding of narratives into templates may be useful for improving interpretation.

Future research could explore the following points. First, recognizing the highly dynamic nature of COVID-19 dashboards, a follow-up study could provide insights into how dashboards have evolved over time, given improvements in disease prevention, testing, and treatment as well as data infrastructure. Second, exploring across official municipal, regional, and national dashboards in a given context was beyond the scope of this study; however, such an exploration may offer insights into the possibility of tailoring dashboards at different levels to specific purposes and audiences. Third, this study has pursued a theoretically informed expert-based appraisal of actionability. A study from the perspective of the target audience is therefore merited and needed to obtain insights from firsthand use. Finally, the developed assessment tool could be used within a specific country context to analyze actions needed to implement the identified features.

To our knowledge, this is the most comprehensive summary of COVID-19 dashboards and assessment of their actionability published to date. The search for COVID-19 dashboards was wide-reaching and used multiple methods to amass a global sample. The approach tapped into a unique and highly specialized international network dedicated to health care performance intelligence, allowing for an expert, context-aware, and multicultural team. The multilanguage competencies of the panel made it possible for the dashboards to be reviewed in their original languages for high-quality data extraction. Through detailed data extraction and a structured process of scoring with joint deliberation, we have identified a set of timely and pragmatic features for optimizing dashboards. This is also the first study to our knowledge on the use of dashboards for public reporting from a communication and health care performance intelligence perspective. Importantly, the study was conducted at pace with the ongoing COVID-19 pandemic to ensure the potential for findings to inform the continued development of dashboards in combination with other communication tools.

We acknowledge the following potential limitations. First, the sample of dashboards is ultimately a subset of publicly available web-based COVID-19 reporting. The sample is also skewed to locations in the European and Pan-American regions, which account for two-thirds of the dashboards reviewed. This can be attributed in part to factors including the thorough but not exhaustive sampling strategy applied; the exclusion of dashboards beyond the 22 language competencies of the panel (ie, Arabic and Hindi); and the focus on web-based dashboards to the exclusion of those exclusively on mobile apps (common to Asian countries). As an exploratory study, reasonable diversity of locations, in combination with different levels (scales) of focus and types of organizations, took precedent and was achieved. Nonetheless, the findings may not be generalizable to all contexts. Second, despite our best efforts to obtain a snapshot of COVID-19 dashboards in a common 2-week period, the severity and specific phase of the pandemic inevitably varied greatly on the date of the review as described. Our approach to assess rather than evaluate the impact of COVID-19 dashboards mitigates the significance of these differences on our findings. Third, the appraised actionability of the dashboards ultimately does not confirm their use in practice, and evaluating this was beyond the scope of this study.

This study has taken stock of the vast landscape of public web-based COVID-19 dashboards; this is a testament to the advancements in health information systems and digitalization of our societies, coupled with the responsibility and imperative to publicly report health information. As could be expected, the 158 dashboards in our sample, spanning a total of 53 countries, are diverse. They have different contexts and levels of focus, purposes, and audiences. They draw from various data sources, offer different content and use a range of ways—albeit at times limited—to break down data and to visualize, simplify, and interact with information. Their actionability also differs, signaling that their fitness for use by decision makers is not a guarantee. The number of dashboards appraised as highly actionable in the period of July 2020 when the dashboards in this study were assessed signals that work is still needed to optimize the use of dashboards. There is no one-size-fits-all template or model to accomplish this. Dashboards must be purpose-driven and context-specific. We urge those working on COVID-19 dashboards to consider the seven features identified in our study and adopt them as called for. By doing so, they stand to fully leverage the potential advantages of public reporting and its use for decision-making and behavior change needed to address the current pandemic.