This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
Digital clinical measures collected via various digital sensing technologies such as smartphones, smartwatches, wearables, ingestibles, and implantables are increasingly used by individuals and clinicians to capture health outcomes or behavioral and physiological characteristics of individuals. Although academia is taking an active role in evaluating digital sensing products, academic contributions to advancing the safe, effective, ethical, and equitable use of digital clinical measures are poorly characterized.
We performed a systematic review to characterize the nature of academic research on digital clinical measures and to compare and contrast the types of sensors used and the sources of funding support for specific subareas of this research.
We conducted a PubMed search using a range of search terms to retrieve peer-reviewed articles reporting US-led academic research on digital clinical measures between January 2019 and February 2021. We screened each publication against specific inclusion and exclusion criteria. We then identified and categorized research studies based on the types of academic research, sensors used, and funding sources. Finally, we compared and contrasted the funding support for these specific subareas of research and sensor types.
The search retrieved 4240 articles of interest. Following the screening, 295 articles remained for data extraction and categorization. The top five research subareas included operations research (research analysis; n=225, 76%), analytical validation (n=173, 59%), usability and utility (data visualization; n=123, 42%), verification (n=93, 32%), and clinical validation (n=83, 28%). The three most underrepresented areas of research into digital clinical measures were ethics (n=0, 0%), security (n=1, 0.5%), and data rights and governance (n=1, 0.5%). Movement and activity trackers were the most commonly studied sensor type, and physiological (mechanical) sensors were the least frequently studied. We found that government agencies are providing the most funding for research on digital clinical measures (n=192, 65%), followed by independent foundations (n=109, 37%) and industries (n=56, 19%), with the remaining 12% (n=36) of these studies completely unfunded.
Specific subareas of academic research related to digital clinical measures are not keeping pace with the rapid expansion and adoption of digital sensing products. An integrated and coordinated effort is required across academia, academic partners, and academic funders to establish the field of digital clinical measures as an evidence-based field worthy of our trust.
Digital clinical measures are health outcomes or physiological characteristics of an individual’s health, wellness, or condition that are collected digitally with a sensor [
In recent years, digital clinical measures have drawn substantial interest from industry, government agencies, academia, and nonprofit institutions, as digital sensing tools, including consumer products and medical devices, are becoming increasingly popular. Consumer products such as smartwatches and smartphones have become part of daily life for many Americans. These have emerged as popular and multipurpose real-time physiological monitoring products capable of measuring sleep and stress in addition to the more traditional actigraphy and heart rate monitoring. In 2020, 26% of Americans owned a smartwatch [
To support the development and assessment of digital consumer products and medical devices, the volume of academic research has increased across the total product life cycle of digital clinical measures [
For these reasons, a multi-stakeholder group of experts collaborating on The Playbook [
We conducted a systematic search of peer-reviewed literature indexed in PubMed and published between January 1, 2019, and February 24, 2021. For the purposes of this review, we did not restrict the scope of our search to any single digital clinical measure or area of academic research. A multi-stakeholder team of clinical, academic, technical, and operational experts developed the search terms (
US-led research (ie, ≥50% US-based authors)
Academic research with at least one US-based academic researcher (industry-only research articles were excluded)
Peer-reviewed journals and full-length conference articles (systematic reviews, meta-analyses, editorials, opinion pieces, case reports, and case studies were excluded)
All portable biometric monitoring technologies (BioMeTs) that rely upon a biometric sensor, such as microphones and accelerometers [
Papers published between January 1, 2019, and February 24, 2021
Data fields extracted from identified academic research.
Field | Definition | Allowed values |
Title | N/Aa | Free text |
Authors | Last name, first name | Free text |
Author affiliation | N/A | Free text |
Journal | Name | Free text |
Year | N/A | 2019, 2020, 2021 |
DOI | Digital object identifier: a unique alphanumeric string used to identify content and provide a persistent link to the manuscript’s online location. | Free text |
Nature of academic research | Academic research measured here by the publication of peer-reviewed journals and full-length conference articles by study teams that include researchers from either a university or academic institute and society or nonprofit foundation. | Verification, analytical validation, measure identification, clinical validation, security, ethics, data rights and governance, usability and utility (human factors/behavioral economics), standards, usability and utility (data visualization), economic feasibility, operations (care), operations (research design), operations (research analysis), and operations (data) |
Digital clinical measure | Health outcomes or physiological characteristics of an individual’s health, wellness, or condition that are collected digitally with a sensor [ |
Biochemical, movement and activity, physiological (electrical, mechanical, optics and imaging) |
Funding sources | Funding information | Government, industry, independent foundation, and unfunded |
aN/A: not applicable.
Following the PubMed search, we conducted a multistep review process to screen articles for inclusion following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [
Following the screening phase, seven analysts (authors MMHS, KR, AB, AVK, AF, YJ, and WKW) extracted data from the articles included in the data extraction phase and categorized each publication as described in
The categories to subgroup the “nature of academic research” included verification, analytical validation, measure identification, clinical validation, security, ethics, data rights and governance, usability and utility (human factors and behavioral economics), standards, usability and utility (data visualization), economic feasibility, operations (care), operations (research design), operations (research analysis), and operations (data).
The categories to subgroup “digital clinical measures” included biochemical, movement and activity, and physiological (electrical, mechanical, and optics and imaging).
“Funding sources” were subgrouped by government, industry, independent foundation, and unfunded. Articles with missing funding information were categorized as unfunded. The details of these categories are given in
Categories for data extraction.
Category | Definitions | Reference | |||
|
|||||
|
Verification | Evaluates and demonstrates the performance of a sensor technology within a BioMeTa, and the sample-level data it generates, against a prespecified set of criteria | [ |
||
|
Analytical validation | Evaluates the performance of the algorithm, and the ability of this component of the BioMeT to measure, detect, or predict physiological or behavioral metrics | [ |
||
|
Clinical validation | Evaluates whether a BioMeT acceptably identifies, measures, or predicts a meaningful clinical, biological, physical, functional state, or experience in the stated context of use (which includes a specified population) | [ |
||
|
Measure identification | Research studies to identify key variables from the information extracted from digital sensors, to support decision-making | [ |
||
|
Security | Research studies to assess the risks associated with digital clinical measures and taking necessary measures for information security | [ |
||
|
Data rights and governance | Research studies to assess the data access, privacy, and sharing (following the FAIRb guiding principle) | [ |
||
|
Ethics | Research studies to ensure equity and justice during every step of the development and deployment of digital clinical measures (eg, reduce health disparities or racial injustice) | [ |
||
|
Usability and utility (human factors/behavioral economics) | Research studies to investigate human factors associated with digital clinical measures (eg, how usable, useful, or unobtrusive a digital clinical measure can be for an end user). It involves surveys from the participants on user experience. | [ |
||
|
Standards | Involves standardization of the data extracted from digital clinical measures for interoperability | [ |
||
|
Usability and utility (data visualization) | Involves data visualization/result presentation for all end uses | [ |
||
|
Economic feasibility | Research studies to investigate economic feasibility of a digital clinical measure | [ |
||
|
Operations (care) | Involves clinicians and economists to design clinical workflow and corresponding evaluation that is typically done for a clinical trial | [ |
||
|
Operations (research design) | Involves clinicians and biostatisticians to design a research study and execution plan, which is typically done for a clinical trial via power analysis and statistical analysis plan | [ |
||
|
Operations (research analysis) | Involves analyzing data from digital clinical measures (eg, data analyst or data scientists) | [ |
||
|
Operations (data) | Involves monitoring data and metadata from digital clinical measures (eg, bioinformatics) | [ |
||
|
|||||
|
Biochemical | Senses biochemicals (eg, sweat sensor or continuous glucose monitors) | [ |
||
|
Movement and activity | Tracks movement and activity (eg, step count or actigraph) | [ |
||
|
Physiological (electrical) | Senses electrical signals related to physiological phenomena (eg, electrocardiography, electroencephalography, electromyography, bioimpedance, electrodermal activity, or electroooculography) | [ |
||
|
Physiological (mechanical) | Senses mechanical signals related to physiological phenomena (eg, phonocardiography, speech, lung sounds, joint acoustic emission, seismocardiography, or ballistocardiography) | [ |
||
|
Physiological (optics and imaging) | Senses optical signals related to physiological phenomena (eg, photoplethysmography, camera for blood volume pulse, or bioradar) | [ |
||
|
|||||
|
Government | US Government funding agencies | [ |
||
|
Industry | Pharma, tech, and medical device industry | [ |
||
|
Independent foundation | Universities, private nonprofits, societies, and independent associations | [ |
||
|
Unfunded | Investigator initiated with no funding sources explicitly stated |
|
aBioMeT: biometric monitoring technology.
bFAIR: Findable, Accessible, Interoperable, and Reusable.
For the data extraction process, each publication was reviewed by at least three of the seven analysts (MMHS, KR, AB, AVK, AF, WKW, and YJ). Each publication was assigned to one or more categories of a particular criterion as a result of two or more votes for a particular category for each publication. This method of subgrouping was used to reduce the impact of individual analyst subjectivity at this stage. Following the initial categorization, articles falling into the government funding subgroup were further categorized by US government agency (ie, National Institutes of Health [NIH], National Science Foundation [NSF], Department of Defense [DOD], Veteran Affairs [VA], National Aeronautics and Space Administration [NASA], Department of Energy [DOE], and “Other”). The “Other” category constitutes government funding sources that were listed for just one article in our pool. Articles with NIH funding were further subgrouped by NIH institutes and centers [
Following the data extraction process, we performed Pearson chi-square tests with one categorical variable to determine whether the representation of academic research studies varies significantly within the following categories: academic research, digital sensors, and funding sources. We assumed equal representation for all categories as the null hypothesis. In this work, we considered
Our initial search on PubMed retrieved 4240 articles (
Article screening process and diagram following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) review methodology.
The 295 articles used for analysis were categorized by research study type, sensor type, and funding source, including broad US government funding sources and specific NIH funding sources (
Categorization by sensor types (
Distribution of articles across (a) research study types, (b) different sensing modalities, (c) different funding sources, and (d) different government funding agencies. The bars are showing the percentage of studies, with 100% equivalent to 295 papers included in the data extraction process for (a-c) and 100% equivalent to 192 papers with government funding for (d). The text on top of the bars in all the plots showing the actual number of articles per category. Others in (d): state governments, National Institute of Justice, US Department of Agriculture, National Institute of Food and Agriculture. One article can be grouped into multiple categories for (a-d). DOD: Department of Defense; DOE: Department of Energy; Gov: governance; NASA: National Aeronautics and Space Administration; NIH: National Institutes of Health; NSF: National Science Foundation; Ops: operations; Phys: physiological; U&U: usability and utility; Visual: visualization; VA: Veteran Affairs.
Studies categorized by funding source (
For studies receiving government funding, the NIH was the most frequent contributor in terms of the number of articles funded—66% (n=126) of the studies with government funding were funded by the NIH (
Of the articles that reported receiving funding from independent foundations (n=109), 75 (69%) studies received funding from institutional funds at universities, 46 (42%) studies received funding from private nonprofits (eg, Bill and Melinda Gates Foundation or Chan Zuckerberg Initiative), and 6 (6%) received funding from societies and associations (eg, American Heart Association).
To understand whether specific funding types may be driving specific sectors of digital clinical measures research, and where a lack of funding may be contributing to low research output, we explored the distribution of funding across different research study types (
Distribution of articles across different NIH institutes and centers. The bars are showing the percentage of studies, with 100% equivalent to 126 papers with NIH funding. The text on top of the bars showing the actual number of articles per category. One article can be grouped into multiple categories. FIC: Fogarty International Center; NBIB: National Institute of Biomedical Imaging and Bioengineering; NCATS: National Center for Advancing Translational Sciences; NCCIH: National Center for Complementary and Integrative Health; NEI: National Eye Institute; NHLBI: National Heart, Lung, and Blood Institute; NIA: National Institute on Aging; NIAAA: National Institute on Alcohol Abuse and Alcoholism; NICHD: National Institute of Child Health and Human Development; NIDA: National Institute on Drug Abuse; NIDCD: National Institute on Deafness and Other Communication Disorders; NIDCR: National Institute of Dental and Craniofacial Research; NIDDK: National Institute of Diabetes and Digestive and Kidney Diseases; NIEHS: National Institute of Environmental Health Sciences; NIGMS: National Institute of General Medical Sciences; NIH: National Institutes of Health; NIMH: National Institute of Mental Health; NIMHD: National Institute on Minority Health and Health Disparities; NINDS: National Institute of Neurological Disorders and Stroke; NLM: National Library of Medicine; OAR: Office of AIDS Research; OD: Office of Dietary Supplements.
Distribution of funding sources across different research study types, with the heat map color showing the percentage of studies (100% equivalent to 295 papers included in the data extraction process) and the text in each cell showing the actual number of articles per category. One article can be grouped into multiple categories. Gov: governance; Ops: operations; U&U: usability and utility; Visual: visualization.
Similar to the previous analysis subdividing research topics by funding type, we sought to understand whether the volume of literature surrounding particular types of digital sensors is related to funding. Therefore, we subdivided the articles in the different digital sensor categories from
Distribution of funding sources across different digital sensor types, with the heat map color showing the percentage of studies (100% equivalent to 295 papers included in the data extraction process) and the text in each cell showing the actual number of articles per category. One article can be grouped into multiple categories. Phys: physiological.
In this systematic review, we describe the nature of academic research related to digital clinical measures and the distribution of funding across different types of academic research and sensing modalities.
Verification, analytical validation, and clinical validation studies [
Research studies examining the usability and utility of digital sensing products are relatively common compared to publications reporting research into security, data rights and governance, and economic feasibility, which ought to trend together [
Research into the operational aspects of deploying digital clinical measures is the largest single study type identified by our review. Although digital clinical measures cannot add value unless they are successfully operationalized during routine clinical care and in clinical trials, focusing academic research on deployment without first ensuring that the digital clinical measures are fit-for-purpose and trustworthy leaves the entire field of digital health at risk of collecting vast swaths of data that, at best, are of no value and, at worst, could cause harm. During the rapid acceleration of digital clinical measurements, research into the selection and development of high-quality measures and tools must be a primary focus of academic research in this new field.
Research related to movement and activity sensors are most common when we parse the article pool by sensor type. This finding is consistent with other literature where digital measures of activity have been found to be most commonly used to answer clinical questions [
Our review indicates that government agencies and independent foundations are funding most of the academic research studies related to digital clinical measures. Industry funding was relatively low, and this is likely due to our definition of academic studies that excludes studies that only have industry-affiliated authors without academic research partnerships. Of the government agencies, the NIH is funding most of the academic research studies, which is consistent with previous research examining funding of US biomedical research [
After operational research, analytical validation is the most common government-funded study type in digital clinical measurement. This is critically important as analytical validation includes examination of algorithmic bias [
Although movement and activity sensors are the most used sensors in academic research, these sensors are still the second most likely to be unfunded (n=19, 15%), suggesting that academic researchers are pursuing research into movement and activity sensors even when funding may not exist. This is again praiseworthy but not sustainable, considering the rapid adoption of these sensors in our daily life [
Our review has several limitations. First, we have focused only on academic research led by US-based academic researchers. Future research should expand beyond the United States to examine trends in academic research into digital clinical measures globally. Second, we searched only one database (PubMed) to retrieve articles for this review. PubMed only indexes research related to life sciences and biomedicine [
This review reports the current state of academic research on the rapidly expanding and highly promising field of digital clinical measures. Substantial work is being done in areas such as validation and operations, with a paucity of research in other areas like security and ethics. Future studies should investigate why critical research into the safe, effective, ethical, and equitable advancement of digital clinical measures is largely absent from the published literature. Both academic researchers and funding agencies should focus on the subareas of academic research on digital clinical measures that are underrepresented and relatively underfunded to ensure that funding priorities adequately reflect the evidentiary needs of the field.
Academic research related to digital clinical measures is not keeping pace with the rapid expansion and adoption of digital sensing products. Although substantial foundational research validating the performance of digital clinical measures is being conducted, academic studies of security, data rights and governance, economic feasibility, ethics, and standards necessary to advance the field are lagging. These areas must be bolstered to minimize the growing chasm between the promised benefits of digital clinical measures and their potential risks. As expected, research funding appears to be associated with increased research publications. An integrated and coordinated effort is required across academia, academic partners, and academic funders to establish the field of digital clinical measures as an evidence-based field worthy of our trust.
PubMed search terms.
Python code.
List of articles excluded after full-text review.
List of included papers with categories.
Digital Medicine Society
Department of Defence
Department of Energy
National Aeronautics and Space Administration
National Institutes of Health
National Science Foundation
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
Veteran Affairs
The authors would like to thank Sarah Park, librarian for engineering and computer science at Duke University, for her support in the development of the search terms. This study was funded, in part, by the Digital Medicine Society. This project has been made possible, in part, by grant 2020-218599 from the Chan Zuckerberg Initiative Doner-Advised Fund, an advised fund of Silicon Valley Community Foundation.
JCG and JD conceived the study. MMHS, JCG, TH, JP, SC, JC, and JD participated in the study’s design and coordination, and drafted the search terms and inclusion criteria. MMHS ran the search on PubMed. MMHS, KR, and AB participated in the screening phase. MMHS, KR, AB, AVK, AF, YJ, and WKW participated in the data extraction phase. MMHS, KR, AB, AVK, and AF standardized the extracted data. MMHS carried out the data analysis and created visual representations. MMHS, JCG, and JD drafted the manuscript. All authors read and approved the final manuscript.
TH is employed by Activinsights Ltd. The other authors declare no conflicts of interest.