Connected or overlapping terms are often used to describe digital interventions in health and care services, including eHealth, mobile health (mHealth), digital health, telehealth, and telemedicine [1]. The term digital health has become an established umbrella term for these [2] and implies “the use of information and communication technologies to improve human health, health care services, and wellness for both individuals and populations” [3]. The term health and welfare technology (HWT) [4], now broadly used in the Nordic countries among others, adds more detail to the digital health concept. HWT is defined as “a technology-based intervention that aims at maintaining or promoting health, well-being, quality of life, and/or increasing efficiency in the service delivery system of welfare, social, and health care services, while improving working conditions of the staff” [5]. The combination of digital health and HWT thereby encompasses broad and burgeoning interdisciplinary fields of research that may not always prioritize established scientific literature databases when publicizing results.

The Cochrane Handbook for Systematic Reviews of Interventions, considered the gold standard in conducting such reviews, recommends that “searches for studies should be as extensive as possible in order to reduce the risk of publication bias and to identify as much relevant evidence as possible” [6]. Bibliographic databases mainly index studies published in peer-reviewed journals. Depending on the question and the scope of the review, however, the proportion of relevant studies not published in scholarly journals may vary greatly. Those not indexed in bibliographic databases are frequently referred to as gray literature.

An often-cited definition of gray literature is “that which is produced on all levels of government, academics, business, and industry in print and electronic formats, but which is not controlled by commercial publishers (ie, where publishing is not the primary activity of the producing body)” [7]. Gray literature, according to this definition, is very diverse and could encompass self-published studies from research institutes, short conference abstracts, theses and dissertations, and ongoing research (trial registers), as well as government and committee reports, among others [8]. With today’s digital development and new media landscape, the related term gray data is sometimes used, including user-generated content such as blogs and social media [9].

Methodological handbooks for literature reviews promote inclusion of gray literature to increase quality and depth of reviews [6,10,11]. This can be achieved by identifying ongoing studies with useful data as well as finding additional sources for evidence above and beyond what is normally found in commercially published material. Gray literature, therefore, has the potential to reduce publication bias, which occurs when the published research is not representative of all the research that has been conducted [12].

Industry sponsors and technology developers may have different motives and timelines for work conducted in the field of digital health and HWT that may not coincide with traditional academic processes, scientific methods, or the conduct of rigorously controlled trials. In areas where the technology is developing fast, other study types than randomized controlled trials (RCT) may be preferred when evaluating interventions to allow for more rapid dissemination of results [13].

Even completed RCTs are not always published or retrievable. Al-Durra et al [14] found that 27% of all RCTs in digital health remained unpublished 5 years after completion. Industry-sponsored trials are less often published in scientific journals compared to nonsponsored or publicly funded trials, as found in a study of ClinicalTrials.gov [15]. Whether this is due to time constraints or unwillingness to widely disseminate findings, gray literature searches may reduce potential publication bias by identifying such unpublished, or ongoing, studies [16].

Moreover, rigorous and holistic assessments of evidence for interventions may not be found in bibliographic databases either. For example, publications from organizations that compile and assess evidence-based practice, such as the UK-based National Institute for Health and Care Excellence, and other health technology assessment (HTA) organizations will likely be missed if searches are restricted to bibliographic databases.

Another reason for including gray literature is that it is widely used. A recent study on nursing journals shows that gray literature accounts for about 10% of all citations [17], while another study showed that more than 10% of UK governmental publications in the field of health care are cited in the publication database Scopus [18]. Farrah and Mierzwinski-Urban [19] found that about 47% of the references on reports on novel nondrug health technologies were found in gray literature, and a review by Song et al [20] found that approximately half of all medical and health-related studies are not published in peer-reviewed journals. This can be particularly true in literature regarding new and emerging health technologies, but the reasons for not publishing studies can vary. In the Song et al [20] review, the main reason given for nonpublication was nonsubmission, where 85% of unpublished studies had not been submitted to journals due to lack of time or low priority. The next most common reason was that studies were incomplete or still ongoing. There may also be alternative incentives for publishing research in other sources than academic journals. In a study concerning the production of gray literature in the Australian public sector [21], the main reason organizations gave for publication output was to provide an evidence base to inform policy or practice, and the most effective channel for achieving this was felt to be publishing via the organization’s own website.

Gray literature can make a valuable contribution to the results in reviews of literature, although its potential relevance and contribution to the review’s results are often dependent upon the review type and its aim. Its strongest contribution may be in the conduct of scoping reviews, where it is a recommended source to search along with scientific bibliographic databases. The aim of a scoping review is to provide an overview of a field of research, quite often with a focus on providing a general picture on the available literature for a topic [22,23] or understanding the context for an intervention [9]. Gray literature suits both these purposes well. As assessment of study quality or risk of bias is generally not performed [24], gray literature may also be more easily and reliably integrated alongside scientific publications in these types of reviews.

Including unpublished studies in meta-analyses, where studies in a systematic literature review are statistically analyzed, is generally considered to improve precision in the overall result. However, it is less clear if the inclusion of gray literature influences the effect size or statistical significance [25-27]. Another advantage of including gray literature in meta-analyses reported by Halfpenny et al [26] is that they found more data in trial registers than in the published study concerning the same trial; including gray literature therefore yielded a fuller and more complete result. Integration of some types of nontrial-based gray literature alongside published studies may be more challenging, however, as requirements on the rigor and format of statistical presentations can be highly variable in comparison.

Gray literature can also strengthen systematic literature reviews without meta-analyses, especially when the focus of the review is on evaluating how new interventions function in practice—an aspect particularly relevant for digital health and HWT. This may be particularly apparent in study areas where RCTs are not standard or common such as in interdisciplinary fields [28]—a description that also fits digital health and HWT well. Such areas may have a greater proportion of lower-quality research-based evidence and a higher context specificity that is important in implementation of interventions. This is where gray literature can provide a valuable supplement to other findings [29]. Examples of this include a study by Adams et al [9] regarding public health interventions, were most or all studies were found in gray literature, demonstrating its importance in investigations of effectiveness. In a systematic review of process and implementation of participatory ergonomics, Mahood et al [22] found that the several gray literature studies included alongside peer-reviewed literature resulted in a broader view of the evidence. Similarly, a systematic review by Cooper et al [30] of public health and environmental enhancement discovered that when comparing the results retrieved from searching in several bibliographic databases with a gray literature search (foremost via web searching) the latter contributed to a greater extent to the overall synthesis of the review. A result that might be related to the interdisciplinary nature of the subject for the review, sometimes not easily found when searching more subject-specific bibliographic databases.

There are disadvantages to using gray literature, however. One is the time and resources it takes to search for it. Even though the internet has made many types of gray literature more accessible via, for example, web pages and online repositories, gray literature can still be hard to find due to its comparative deficits in metadata and indexing. The sustainability of gray literature digital domains is also a concern, with links and documents more prone to disappear than with bibliographic databases. Gray literature in many cases lacks time-saving functionality—for example, when exporting references to references management tools. It is also often less structured (eg, lacking abstracts), which makes any screening process less efficient [8,23,31].

Before undertaking a gray literature search, one should therefore consider whether the benefits outweigh the disadvantages. One example is the recommendation created by Jefferson et al [32] on whether to include the gray literature of regulatory data in Cochrane reviews concerning drugs or biologics. Key considerations include the burden of disease and number of people using or likely to use an intervention.

Because of the extra time required, Cochrane recommends limiting the use of gray literature when conducting rapid reviews and instead focusing mainly on study registers [33]. For the same reason, Cochrane recommends less frequent searches of gray literature than among publication databases when conducting and updating living reviews [34].

Concerns about the quality of gray literature are often raised. Literature that in many cases has not been subjected to peer review and editorial examination is often viewed as lower quality and less credible. Some types of gray literature (eg, theses and governmental reports) do, however, receive a thorough quality check before being published [23]. In the previously mentioned study of Australian gray literature [21], two-thirds of the organizations stated that they had their work externally or internally reviewed prior to publication.

Like the evidence hierarchies within scientific literature, gray literature can be sorted according to the source’s perceived usefulness. For example, Adams et al [16] described the connection between credibility, retrievability, and outlet control in gray literature by arranging it in 3 tiers (Figure 1). The first and most credible tier consists of government reports, books, and other highly traceable publication types, followed by a second tier that includes company publications and nongovernmental organization studies. The third and final tier includes publications with little or no outlet control, such as blogs and tweets, which would rarely be included in studies of effectiveness.

When assessing bias and study quality of scientific literature, tools like the Cochrane risk-of-bias 2.0 tool for randomized trials [35] and equivalents for other types of studies are well established. The same tools can certainly be used when assessing gray literature [22,36,37]. Other forms of assessment that focus more on the value of the information may be better suited to some kinds of gray literature [9]. One approach used in several reviews [38-40] is the AACODS checklist [41], which, in line with the list’s acronym, evaluates the concepts of authority, accuracy, coverage, objectivity, date/time period, and significance of the content in a context specific to gray literature.

There are no generally established guidelines regarding methods and search strategies for gray literature [8,12,23]. Since the extent of gray literature is vast, making a search plan prior to the search helps estimate the time and resources needed to conduct the search.

The search plan should correspond to the aim of the search and identify the sources likely to yield the most value. Three key considerations when creating a search plan are as follows:

Furthermore, time period and geographic coverage should be considered [12]. A systematic gray literature search, as with any other literature search, should use a clear population, intervention, comparison, and outcome format. Searches should also be as reproducible as possible, although gray literature searches often present challenges to this. Webpages and even repositories are less stable than scientific bibliographic databases and documents and links vanish or cease to exist more regularly.

Searches for gray literature should be documented and reported in a similar manner to searches in bibliographic databases. The aim of this documentation is to allow the search results to be reproduced by others. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [77] and extension PRISMA-S [71] provide guidance on how searches should be documented and reported when conducting a systematic review. The checklist Grey Matters [45] from the Canadian Agency for Drugs and Technologies in Health also includes elements of how to document and report a gray literature search.

According to PRISMA-S [71], gray literature searches should be briefly described in the methods section of the systematic review and complemented with more detailed information in supplementary material.

The inclusion of gray literature in the two reviews brought both advantages and disadvantages, but the former clearly outweighed the latter. As most included studies came from gray literature, the volume of literature to base conclusions on was considerably larger—a systematic review of only two studies would have otherwise been the reality in one case. The included gray literature studies were also highly context-specific and had excellent adherence to the reviews’ aims, perhaps even more so than the studies from the scientific database. It was, however, essentially impossible to consolidate results statistically in any meaningful way due to the lack of rigor in reporting statistics in the gray literature, which at least one journal reviewer deemed a disadvantage. It was also difficult to use the entire search strategy due to challenges with the search engines at the gray literature sources. Both the conduct of the gray literature searches and the subsequent import of found studies to the review software were far more time-consuming than in scientific publication databases; this was a far more manual process that could not use specialized applications that read the abundant metadata found in scientific databases to automate workflows. The time to review and extract data from gray literature studies was also far more time consuming due to less easily identifiable inclusion and exclusion criteria and less standardized publication formats. It is worth noting that, while the inclusion of Nordic sources and specific language searches made the findings more context-specific, it may reduce the applicability of the results elsewhere. As well, reproducing the gray literature searches may be difficult if not impossible for other researchers due to the inability to interpret specific search results and the prospective fluctuation of available documents in the searched sources. Generally, reproduction of results on effectiveness is a challenge due to complexities in implementation of HWT in specific contexts.

Gray literature is worth considering when conducting many types of reviews related to HWT interventions as they can help reduce publication bias and include evidence for interventions that are not typically indexed in bibliographic databases. Including gray literature is particularly relevant for digital health and HWT, where studies tend to evaluate adoption and use using non-RCT study designs and for purposes other than primarily academic publishing. Caution needs to be taken when drawing conclusions from results in the gray literature due to a potential bias that may rise from less rigorous research design, methods, and data analyses. A risk of bias tool or checklist should be used, as with other literature. Finally, using a systematic approach when searching the gray literature is highly recommended. Planning the search and documenting it in a similar manner as when searching in bibliographic databases will save time and effort and make the search transparent and reproducible.