Assessing the Clinical Robustness of Digital Health Startups: Cross-sectional Observational Analysis

Background The digital health sector has experienced rapid growth over the past decade. However, health care technology stakeholders lack a comprehensive understanding of clinical robustness and claims across the industry. Objective This analysis aimed to examine the clinical robustness and public claims made by digital health companies. Methods A cross-sectional observational analysis was conducted using company data from the Rock Health Digital Health Venture Funding Database, the US Food and Drug Administration, and the US National Library of Medicine. Companies were included if they sell products targeting the prevention, diagnosis, or treatment phases of the care continuum. Clinical robustness was defined using regulatory filings and clinical trials completed by each company. Public claims data included clinical, economic, and engagement claims regarding product outcomes made by each company on its website. Results A total of 224 digital health companies with an average age of 7.7 years were included in our cohort. Average clinical robustness was 2.5 (1.8 clinical trials and 0.8 regulatory filings) with a median score of 1. Ninety-eight (44%) companies had a clinical robustness score of 0, while 45 (20%) companies had a clinical robustness score of 5 or more. The average number of public claims was 1.3 (0.5 clinical, 0.4 economic, and 0.4 engagement); the median number of claims was 1. No correlation was observed between clinical robustness and number of clinical claims (r2=0.02), clinical robustness and total funding (r2=0.08), or clinical robustness and company age (r2=0.18). Conclusions Many digital health companies have a low level of clinical robustness and do not make many claims as measured by regulatory filings, clinical trials, and public data shared online. Companies and customers may benefit from investing in greater clinical validation efforts.


Introduction
The digital health sector has grown rapidly over the past decade [1,2]. There are more than 1,900 digital health startup companies in the US that have raised more than $2M in venture funding, which in total have raised more than $77B in venture capital funding since 2011 [3].
While growth is apparent, the ability to measure impact is not. Several studies have highlighted the need for greater clinical validation [4,5] and found that many solutions were not supported by robust clinical evidence [6,7] and demonstrated mixed results on cost savings and cost-effectiveness [8,9]. In addition, there is evidence that some claims made by digital health companies have been misleading [10][11][12], with a few highly publicized cases resulting in legal action by the Federal Trade Commission and state attorneys general [13][14][15]. Most studies focusing on clinical impact are narrowly defined to specific clinical therapeutic areas (e.g., diabetes, cardiac arrhythmia, etc.), making it difficult to extrapolate findings to the broader field of digital health [16][17][18][19][20][21][22]. Additional limitations include a small sample of companies and the use of publications as a proxy for clinical impact. Other studies in digital health have examined larger clinical trends, such as growth in clinical trials, but these often lack data on clinical and customer focus [23,24].
As a result, the ability to understand clinical impact is often either too narrow or too broad. In addition, no studies have examined both clinical rigor and public claims made by companies. To address these limitations, we sought to comprehensively examine the topic of clinical robustness in digital health companies by measuring a more comprehensive definition of clinical rigor and examining companies' public claims across the most in-depth database of US-based digital health companies. These findings provide additional context for all stakeholders in health technology that rely on a more accurate characterization of digital health solutions.

Population
Companies were identified using the Digital Health Venture Funding Database maintained by Rock Health, Inc., a digital health venture fund and advisory firm, which has been used in prior studies [25][26][27]. The database includes all digital health companies with headquarters in the US that have raised at least one venture funding round of $2M or more since 2011. Our analysis included companies that sell products targeting the prevention, diagnosis, or treatment phases of the care continuum that raised at least one round of funding between 2011 and 2020. Digital health companies are defined as those that build and sell digital technologies in healthcare [28,29].

Company Variables
Total venture funding, clinical area(s) of focus, care continuum phase(s), and customer(s) data were collected for each company. Clinical areas represented one of 20 specific clinical domains (e.g., cardiovascular, nephrology, etc.). Care continuum phase was defined according to the following: prevention, diagnosis, or treatment. Customer type referred to the category of buyer for a company's products, such a payer (i.e., health insurance companies), biopharma (i.e., pharmaceutical or biopharmaceutical companies), and medical devices (i.e., companies that manufacture medical devices). Companies can be categorized into multiple categories (companies may address multiple phases of the care continuum or multiple clinical areas). Company variables were gathered from the Digital Health Venture Funding Database, which Rock Health maintains using publicly available information such as company websites, press releases, and SEC filings. Data for companies were collected through August 3, 2021.

Claims Variables
The number and type of claims, defined as unique quantitative statements about product outcomes, made on a company's website were collected in the following categories: clinical, economic, and engagement. The definitions of these claim types are detailed in Table 1. Data was obtained by reviewing all pages of a company's website, excluding links to external pages such as press releases, between May 3, 2021 and August 3, 2021.

Clinical Robustness Variables
We collected regulatory data from the Food and Drug Administration (FDA), including the number of 510(k), De Novo, and PMA filings (where the company was listed as the "Requester" on fda.gov). In addition, we collected the number and type of clinical trials by searching www.clinicaltrials.gov where the company was listed as "Sponsor / Collaborator." Data on both FDA filings and clinical trials were collected between July 1, 2021 and September 2, 2021 through a combination of web scraping and manual searching. Data collection on claims was completed by at least 3 authors, with blinded cross-review to ensure consistency in data collection.
A "clinical robustness" score was calculated for each company, defined as the sum of the number of regulatory filings and clinical trials. Each regulatory filing and clinical trial was weighted equally in the calculation.

Data and Statistical Analysis
All data was stored in Microsoft Excel, where all descriptive and statistical analyses, including coefficient of correlation between variables were calculated.

Population Characteristics
There were 224 companies in the cohort, with an average age of 7.7 years. Collectively, these companies have raised a total of $8.2B in venture capital funding since 2011. The companies spanned three phases of the care continuum -25 offering solutions for prevention, 106 offering solutions for diagnosis, and 110 companies offering solutions for treatment ( Table 2). Some companies offered products across multiple phases of the care continuum and were therefore counted in multiple categories. Average company funding was similar across phases of care -prevention companies raised $35.3M; diagnosis companies raised $37.8M; and treatment companies raised $37.9M.

Clinical Robustness
The average clinical robustness score for all companies was 2.5 (clinical trials,1.8; regulatory filings, 0.8). The median clinical robustness score was 1, with ninety-eight companies (44%) having a score of 0 and thirty-four companies (15%) having a score of 1. Diagnosis companies had the highest average clinical robustness scores (2.8), followed by treatment companies (2.2), and then prevention companies (1.9).
The average clinical robustness score of companies that sold to employers was 3.1, compared to 2.0, 2.2, and 2.7 for those companies that sold to payers, consumers, and providers respectively ( Table 3). Fifteen of the eighteen clinical areas had a higher average number of clinical trials than regulatory filings.   Table  4). The median number of claims of any type was zero for companies that sold to consumers and providers. The median number of economic and engagement claims for companies that sold to payers was also zero.
Companies that sold to employers made more clinical, economic, and engagement claims, than companies that sold to all other customer types ( Figure 4). Many companies (43%) made zero claims.

Clinical Robustness and Claims
There was no correlation between clinical robustness and number of clinical claims (r-squared = 0.02), clinical robustness and total funding (r-squared = 0.08), or clinical robustness and company age (r-squared = 0.18). In addition, there was no strong correlation between clinical areas that had higher clinical robustness scores and clinical areas where companies have received higher average funding (r-squared = 0.07).

Principle Findings
Our findings indicate that many venture-backed startups in digital health have limited clinical robustness as measured by regulatory filings and clinical trials. There was, however, a sizable minority (20%) that had a score of 5 or more suggesting a small population of rigorously tested solutions (Figure 1). While this subpopulation may portend progress, the lack of meaningful clinical validation for nearly half of digital health companies (44% had a clinical robustness score of 0) highlighted a major gap in healthcare technology today. The lack of overall correlation between a company's total venture funding and its clinical robustness score similarly highlighted a significant asymmetry in how companies are potentially valued in today's marketplace (i.e., no correlation between clinical impact and funding) ( Figure 2). However, it is possible that funding amounts reflect future anticipated value, instead of current value [30].
While average clinical robustness was quite low across the population, there was significant variation across clinical areas (e.g., high in cardiovascular and nephrology and low in oncology and primary care) (Figure 3). This may reflect the varying levels of technological maturity of digital health solutions across clinical disciplines. Prior literature points to significant differences in technological maturity between well-funded clinical areas such as diabetes [31] and less-well funded areas such as reproductive and maternal health [32].
The average and median number of all claims was also low. These findings indicated that digital health companies largely did not share outcomes publicly. This may have reflected a desire to keep this data private, but more likely represented a lack of meaningful analyses of any impact (clinical, economic, or engagement) since this data could be used as a competitive differentiator if shared publicly. Separately, we identified 32 companies that had one or more clinical claim and a clinical robustness score of 0. These findings may suggest a disconnect between marketing and evidence. Additional future research could examine the links between public claims and clinical trials or regulatory filings (i.e., are individual claims directly supported by clinical trials or regulatory filings?).
Companies that sold to employers made more claims and had higher clinical robustness scores compared to other customer types ( Figure 4). This may suggest that this customer base is currently the most competitive and/or has the highest entry standards, despite accounting for a relatively small percentage (< 12%) of companies in the sample. A competitive employer market for digital health tools is unsurprising given the rising cost burden of healthcare on employers [33] and evidence that employers are increasingly offering digital health benefits to improve health outcomes [34,35] and contain healthcare costs [36,37].

Limitations
Our definition of clinical robustness was limited to clinical trials and regulatory findings, which remain proxies for effectiveness and require companies to register their activities. However, we believe these elements offer a better estimation of clinical rigor than publications, which can both lag in timing and may not always relate to clinical outcomes. We chose to focus on clinical trials and regulatory filings because they were publicly available data and are generally undertaken to demonstrate an impact on clinical outcomes. Our approach to measuring clinical robustness equally accounts for clinical trials that demonstrate a technology does and does not work. Additionally, our data collection methodology could have missed clinical trials that were not registered on clinicaltrials.gov or regulatory filings that were submitted to the FDA under an individual's name, rather than the company name (both scenarios are atypical). Future research could incorporate condition or disease specific metrics of effectiveness that are then standardized across clinical areas to provide a more accurate measure of clinical impact.
In addition, our cohort only included venture-backed companies above $2M in funding. This may have resulted in selection bias by excluding both ends of the spectrum (i.e., not including earlier stage companies or large conglomerate technology companies). While our population excludes these companies, we believe that our sample represents the most comprehensive assessment of outcomes or claims across the digital health landscape and encompasses the majority of funded companies and activity in this space. As a point of reference, the average seed stage deal size in 2021 was $3.5M, 75% higher than our minimum funding threshold [38].

Conclusions
Despite the hundreds of digital health companies targeting the myriad of needs across the care continuum, clinical robustness and public communication of claims remains low across much of the sector. These results highlight a significant opportunity for companies to differentiate themselves and for customers to demand greater validation for the products and services they purchase.

Figures
Histogram of number of companies by clinical robustness score.