Published on in Vol 25 (2023)

Preprints (earlier versions) of this paper are available at, first published .
A Narrow Definition of Clinical Robustness

A Narrow Definition of Clinical Robustness

A Narrow Definition of Clinical Robustness

Letter to the Editor

Included Health, San Francisco, CA, United States

*all authors contributed equally

Corresponding Author:

Jaclyn Marshall, MS

Included Health

1 California Street, Ste 2300

San Francisco, CA, 94111

United States

Phone: 1 855 431 5533


We read with great interest the article “Assessing the Clinical Robustness of Digital Health Startups: Cross-Sectional Observational Analysis” by Day et al [1]. We appreciate the authors highlighting the importance of assessing the quality and impact of the growing digital health industry. However, we believe the definition of “clinical robustness” was far too narrow and not equally applicable to the heterogeneous group of digital health startups in the study.

The authors applied a unique approach using publicly available data from and Food and Drug Administration (FDA) filings to assess the clinical robustness of 224 digital health startups. While we acknowledge that defining clinical robustness in this industry is inherently complicated, we do not believe it is adequately assessed by summing the number of clinical trials and FDA filings. This is particularly true for startups whose services are not subject to FDA approval. Employed at a digital health company focused on patient navigation and virtual care, we argue much can be gained by including 3 additional components, at a minimum, when assessing clinical robustness: accreditations, externally validated methodologies, and peer-reviewed studies demonstrating impact.

First, accreditations can serve as an important proxy measure for high-quality clinical care. There are multiple well-cited frameworks for high-quality clinical care today, including the Quadruple Aim and the Institute of Medicine’s 6 aims for improvement [2,3]. Large, formally recognized accrediting bodies have used these frameworks to hold providers accountable to performing against diverse, complex care priorities. Achievements of relevant accreditations, such as the National Committee for Quality Assurance, demonstrate a commitment to a clinically robust approach to care delivery.

Second, it is not uncommon for digital health startups to rely on objective, external validation of their assumptions and results by reputable third parties. For example, a third party validated Included Health’s provider match algorithm. These studies demonstrate a startup’s commitment to a robust approach for product design and evaluation.

Finally, some digital health startups have made an investment in rigorous research that is not captured in but disseminated through peer-reviewed literature. The authors acknowledged the lack of peer-reviewed literature as a study limitation. Without this component, however, well-designed studies with meaningful outcomes are overlooked. For example, we conducted a randomized controlled trial on antibiotic stewardship that found a reduction in antibiotic prescribing rates [4]. In another study, we demonstrated reductions in blood pressure among patients with elevated blood pressure who had a video visit [5]. We believe that a well-designed literature review, including terms to identify studies of interest, is a feasible and necessary addition to capture these important findings.

A clinical robustness definition that applies to all of digital health is not an easy task, and we applaud the authors for attempting this across a diverse group of startups. However, we believe that the conclusions are insufficient given the narrow definition of clinical robustness. A broader definition that varies by type of digital health company (eg, care delivery, medical technology) is critical to appropriately assess the clinical robustness of any digital health startup. 

Conflicts of Interest

All authors are employed by Included Health. AP holds a volunteer faculty position at the University of California San Francisco.

Editorial Notice

The corresponding author of “Assessing the Clinical Robustness of Digital Health Startups: Cross-sectional Observational Analysis” declined to respond to this letter.

  1. Day S, Shah V, Kaganoff S, Powelson S, Mathews SC. Assessing the clinical robustness of digital health startups: cross-sectional observational analysis. J Med Internet Res. Jul 20, 2022;24(6):e37677. [FREE Full text] [CrossRef] [Medline]
  2. Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med. Nov 2014;12(6):573-576. [FREE Full text] [CrossRef] [Medline]
  3. Institute of Medicine, Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington DC. National Academies Press; 2001.
  4. Du Yan L, Dean K, Park D, Thompson J, Tong I, Liu C, et al. Education vs clinician feedback on antibiotic prescriptions for acute respiratory infections in telemedicine: a randomized controlled trial. J Gen Intern Med. Mar 26, 2021;36(2):305-312. [FREE Full text] [CrossRef] [Medline]
  5. Taylor P, Berg C, Thompson J, Dean K, Yuan T, Nallamshetty S, et al. Effective access to care in a crisis period: hypertension control during the COVID-19 pandemic by telemedicine. Mayo Clin Proc Innov Qual Outcomes. Feb 2022;6(1):19-26. [FREE Full text] [CrossRef] [Medline]

FDA: Food and Drug Administration

Edited by T Leung; submitted 05.08.22; peer-reviewed by S Mathews; accepted 01.08.23; published 21.09.23.


©Jaclyn Marshall, Alexandra Yurkovic, Todd Thames, Ami Parekh. Originally published in the Journal of Medical Internet Research (, 21.09.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.