Companies were identified using the Digital Health Venture Funding Database maintained by Rock Health Inc, a digital health venture fund and advisory firm, which has been used in prior studies [25-27]. The database includes all digital health companies with headquarters in the United States that have raised at least one venture funding round of US $2 million or more since 2011. Our analysis included companies that sell products targeting the prevention, diagnosis, or treatment phases of the care continuum, which raised at least one round of funding between 2011 and 2020. Digital health companies are defined as those that build and sell digital technologies in health care [28,29].

Total venture funding, clinical area(s) of focus, care continuum phase(s), and customer data were collected for each company. Clinical areas represented 1 of 20 specific clinical domains (eg, cardiovascular, nephrology). Care continuum phase was defined according to the following: prevention, diagnosis, or treatment. Customer type referred to the category of buyer for a company’s products, such as payer (ie, health insurance companies), biopharma (ie, pharmaceutical or biopharmaceutical companies), and medical devices (ie, companies that manufacture medical devices). Companies can be categorized into multiple categories (ie, companies may address multiple phases of the care continuum or multiple clinical areas). Company variables were gathered from the Digital Health Venture Funding Database, which Rock Health maintains using publicly available information such as company websites, press releases, and US Securities and Exchange Commission filings. Data for companies were collected through August 3, 2021.

The number and type of claims, defined as unique quantitative statements about product outcomes, made on a company’s website were collected in the following categories: clinical, economic, and engagement. The definitions of these claim types are detailed in Table 1. Data were obtained by reviewing all pages of a company’s website, excluding links to external pages such as press releases, between May 3, 2021, and August 3, 2021.

We collected regulatory data from the Food and Drug Administration (FDA), including the number of 510(k), De Novo, and premarket approval filings (where the company was listed as the “Requester” on fda.gov). In addition, we collected the number and type of clinical trials by searching ClinicalTrials.gov where the company was listed as “Sponsor / Collaborator.” Data on both FDA filings and clinical trials were collected between July 1, 2021, and September 2, 2021, through a combination of web scraping and manual searching. Data collection on claims was completed by at least 3 authors, with blinded cross-review to ensure consistency in data collection.

A “clinical robustness” score was calculated for each company, defined as the sum of the number of regulatory filings and clinical trials. Each regulatory filing and clinical trial was weighted equally in the calculation.

All data were stored in Microsoft Excel (Microsoft Corp), where all descriptive and statistical analyses including coefficient of correlation between variables were calculated.

There were 224 companies in the cohort, with an average age of 7.7 years. Collectively, these companies have raised a total of US $8.2 billion in venture capital funding since 2011. The companies spanned 3 phases of the care continuum, with 25 offering solutions for prevention, 106 offering solutions for diagnosis, and 110 companies offering solutions for treatment (Multimedia Appendix 1). Some companies offered products across multiple phases of the care continuum and were therefore counted in multiple categories. Average company funding was similar across phases of care—prevention companies raised US $35.3 million; diagnosis companies raised US $37.8 million; and treatment companies raised US $37.9 million.

The average clinical robustness score for all companies was 2.5 (clinical trials: 1.8; regulatory filings: 0.8). The median clinical robustness score was 1, with 98 companies (44%) having a score of 0 and 34 companies (15%) having a score of 1. Diagnosis companies had the highest average clinical robustness scores (2.8), followed by treatment companies (2.2), and then prevention companies (1.9).

The average clinical robustness score of companies that sold to employers was 3.1, compared to 2.0, 2.2, and 2.7 for those companies that sold to payers, consumers, and providers, respectively (Multimedia Appendix 2). Fifteen of the 18 clinical areas had a higher average number of clinical trials than regulatory filings. The distribution of companies by clinical robustness score can be found in Table 2.

The average number of total claims for all companies was 1.3 (0.5 clinical, 0.4 economic, and 0.4 engagement) with the median number of total claims equal to 1 (0 clinical, 0 economic, 0 engagement) (Multimedia Appendix 3). The median number of claims of any type was zero for companies that sold to consumers and providers. The median number of economic and engagement claims for companies that sold to payers was also zero. Companies that sold to employers made more clinical, economic, and engagement claims than companies that sold to all other customer types (Figure 1). Many companies (43%) made zero claims.

There was no correlation between clinical robustness and number of clinical claims (r²=0.02), clinical robustness and total funding (r²=0.08), or clinical robustness and company age (r²=0.18). In addition, there was no strong correlation between clinical areas that had higher clinical robustness scores and clinical areas where companies have received higher average funding (r²=0.07).

Our findings indicate that many venture-backed startups in digital health have limited clinical robustness as measured by regulatory filings and clinical trials. There was, however, a sizable minority (20%) that had a score of 5 or more, suggesting a small population of rigorously tested solutions (Table 2). Although this subpopulation may portend progress, the lack of meaningful clinical validation for nearly half of digital health companies (44% had a clinical robustness score of 0) highlighted a major gap in health care technology today. The lack of overall correlation between a company’s total venture funding and its clinical robustness score similarly highlighted a significant asymmetry in how companies are potentially valued in today’s marketplace (ie, no correlation between clinical impact and funding) (Figure 2). However, it is possible that funding amounts reflect future anticipated value rather than current value [30].

Although average clinical robustness was quite low across the population, there was significant variation across clinical areas (eg, high in cardiovascular and nephrology and low in oncology and primary care) (Figure 3). This may reflect the varying levels of technological maturity of digital health solutions across clinical disciplines. Prior literature points to significant differences in technological maturity between well-funded clinical areas such as diabetes [27] and less well-funded areas such as reproductive and maternal health [31].

The average and median number of all claims was also low. These findings indicated that digital health companies largely did not share outcomes publicly. This may have reflected a desire to keep this data private, but more likely represented a lack of meaningful analyses of any impact (clinical, economic, or engagement) since this data could be used as a competitive differentiator if shared publicly. Separately, we identified 32 companies that had one or more clinical claims and a clinical robustness score of 0. These findings may suggest a disconnect between marketing and evidence. Additional future research could examine the links between public claims and clinical trials or regulatory filings (ie, are individual claims directly supported by clinical trials or regulatory filings?).

Companies that sold to employers made more claims and had higher clinical robustness scores compared to other customer types (Figure 1). This may suggest that this customer base is currently the most competitive and/or has the highest entry standards, despite accounting for a relatively small percentage (<12%) of companies in the sample. A competitive employer market for digital health tools is unsurprising given the rising cost burden of health care on employers [32] and evidence that employers are increasingly offering digital health benefits to improve health outcomes [33,34] and contain health care costs [35,36].

Our definition of clinical robustness was limited to clinical trials and regulatory findings, which remain proxies for effectiveness and require companies to register their activities. However, we believe these elements offer a better estimation of clinical rigor than publications, which can lag in timing and may not always relate to clinical outcomes. We chose to focus on clinical trials and regulatory filings because they were publicly available data and are generally undertaken to demonstrate an impact on clinical outcomes. Our approach to measuring clinical robustness equally accounts for clinical trials that demonstrate a technology does and does not work. Additionally, our data collection methodology could have missed clinical trials that were not registered on ClinicalTrials.gov or regulatory filings that were submitted to the FDA under an individual’s name rather than the company name (both scenarios are atypical). Future research could incorporate condition- or disease-specific metrics of effectiveness that are then standardized across clinical areas to provide a more accurate measure of clinical impact.

In addition, our cohort only included venture-backed companies above US $2 million in funding. This may have resulted in selection bias by excluding both ends of the spectrum (ie, not including earlier stage companies or large conglomerate technology companies). Although our population excludes these companies, we believe that our sample represents the most comprehensive assessment of outcomes or claims across the digital health landscape and encompasses the majority of funded companies and activity in this space. As a point of reference, the average seed stage deal size in 2021 was US $3.5 million, 75% higher than our minimum funding threshold [37].

Despite the hundreds of digital health companies targeting the myriad of needs across the care continuum, clinical robustness and public communication of claims remains low across much of the sector. These results highlight a significant opportunity for companies to differentiate themselves and for customers to demand greater validation for the products and services they purchase.