This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Clinical trials are key to advancing evidence-based medical research. The medical research literature has identified the impact of publication bias in clinical trials. Selective publication for positive outcomes or nonpublication of negative results could misdirect subsequent research and result in literature reviews leaning toward positive outcomes. Digital health trials face specific challenges, including a high attrition rate, usability issues, and insufficient formative research. These challenges may contribute to nonpublication of the trial results. To our knowledge, no study has thus far reported the nonpublication rates of digital health trials.
The primary research objective was to evaluate the nonpublication rate of digital health randomized clinical trials registered in ClinicalTrials.gov. Our secondary research objective was to determine whether industry funding contributes to nonpublication of digital health trials.
To identify digital health trials, a list of 47 search terms was developed through an iterative process and applied to the “Title,” “Interventions,” and “Outcome Measures” fields of registered trials with completion dates between April 1, 2010, and April 1, 2013. The search was based on the full dataset exported from the ClinlicalTrials.gov database, with 265,657 trials entries downloaded on February 10, 2018, to allow publication of studies within 5 years of trial completion. We identified publications related to the results of the trials through a comprehensive approach that included an automated and manual publication-identification process.
In total, 6717 articles matched the
In the domain of digital health, 27% of registered clinical trials results are unpublished, which is lower than nonpublication rates in other fields. There are substantial differences in nonpublication rates between trials funded by industry and nonindustry sponsors. Further research is required to define the determinants and reasons for nonpublication and, more importantly, to articulate the impact and risk of publication bias in the field of digital health trials.
Empirical observations demonstrate that not all clinical studies successfully publish their results in peer-reviewed journals. Perhaps, the earliest indication of publication bias in the area of scientific research was in 1979 by Robert Rosenthal with the term “file drawer problem,” acknowledging the existence of selective publication bias for studies with positive and significant results [
In 2008, a study of publication rates of clinical trials supporting successful new Food and Drug Administration drug applications found that over half of all the included trials were unpublished 5 years after obtaining approval from the Food and Drug Administration [
The registration of clinical trials, first proposed by Simes in 1986 [
Since its establishment in the year 2000, the ClinicalTrials.gov website, which is maintained by the United States National Library of Medicine at the National Institutes of Health, has become the world’s largest clinical trial registry, with 286,717 registered trials, 60% of which are non-US–based as of October 11, 2018 [
A number of studies have analyzed and reported the characteristics of publication rates of clinical trials registered in ClinicalTrials.gov [
The primary research objective was to examine the prevalence and characteristics of the nonpublication rate among digital health randomized clinical trials registered in the ClinicalTrials.gov database. The secondary research objective was to determine whether industry funding contributes to nonpublication of trials. Considering that the ClinicalTrials.gov registry is a US–based registry including 60% of non-US–based trials, we intended to explore differences in the nonpublication rate and trial size between US- and non-US–based trials [
The ClinicalTrials.gov website provides free, global open access to the online registry database through a comprehensive website search page as well as download capabilities; for example, all registration information for a given trial can be downloaded in XML format via a Web service interface. For our study, we downloaded the entire ClinicalTrials.gov online database, with 265,657 registered clinical trials entries, on February 10, 2018.
The research included all eHealth-, mHealth-, telehealth-, and digital health-related randomized clinical trials that are registered in the ClinicalTrials.gov website and include any information and communication technology component, such as cellular phones, mobile phones, smart phones; devices and computer-assisted interventions; internet, online websites, and mobile applications; blogs and social media components; and emails, messages, and texts.
We also included interventional and behavioral trials with or without the results. We limited our inclusion criteria to trials with latest completion dates between April 1, 2010, and April 1, 2013. The latest date between trials’ primary completion date and completion date fields was considered the latest completion date. Details regarding the evaluation of the latest completion date of trials are described in
Our search allowed for almost 5 years of a “publication lag period” between the stated trial completion date (up to April 1, 2013) and the search date for published reports (February 10, 2018). This strategy allowed us to account for longer publication cycles that may take up to several years, as indicated in prior studies [
Our search excluded registered clinical trials that were not randomized or only focused on electronic record-management systems such as electronic medical records, electronic health records, and hospital information systems as well as back-end integration systems, middleware applications, and Web services. Registered clinical trials that only reported on internet, Web-based, online, and computer-based surveys as well as television or online advertisement were also excluded. In addition, the search excluded registered clinical trials that focused only on biotechnology, bioinformatics analysis, and sequencing techniques. Finally, trials on medical devices and those only related to diagnostic imaging device, computerized neuropsychological, cognition, and oxygen assessment tools were excluded.
The search terms and phrases were conceptually derived from the inclusion criteria. A complete list of included search terms and phrases was developed through an iterative process (
The “condition” field in ClinicalTrials.gov was defined as “the disease, disorder, syndrome, illness, or injury that is being studied” [
The data exported from the ClinicalTrials.gov database includes a field “Why_Stopped” that indicates the reasons for trial discontinuation. This field is populated for trials with a withdrawn, suspended, and terminated recruitment status. We extracted and evaluated the textual content of this field as part of our recruitment analysis. Details of classification of the reasons for trial discontinuations are indicated in
We analyzed the descriptions of the 556 included randomized clinical trials to identify the major type of technology that was utilized within the respective interventions. Details of major technology classifications of the trials are indicated in
Analysis of randomized clinical trials by their lead sponsor information.
Lead sponsor category (N=556) | Trials, n (%) | |
Foundations, Institutes, and Research Centers | 72 (12.9%) | |
Hospitals and Medical Centers | 102 (18.3%) | |
United States Federal Government | 25 (4.5%) | |
University | 301 (54.1%) | |
Other | 18 (3.2%) | |
38 (6.8%) | ||
Insurance | 6 (15.8%) | |
Pharmaceuticals | 2 (5.3%) | |
Technology and Services | 29 (76.3%) | |
Telecommunication | 1 (3.1%) |
The XML data exported from the ClinicalTrials.gov database did not include an explicit field to indicate whether the trial was registered prospectively. We compared each trial’s “study_first_submitted” field to the “start_date” field in order to determine if the trial was registered prospectively or retrospectively. The “study_first_submitted” field indicates the dates when the trial’s primary investigator first submitted the trial record to ClinicalTrials.gov, whereas the “start_date” field indicates the date when the first participant was enrolled in the study [
The data exported from the ClinicalTrials.gov database includes a field “Has Results” to indicate whether results have been submitted for the underlying study. The XML export of the trial metadata also includes the field “FirstReceived_Results_Date,” which is the date on which the study’s first results were received. These fields are maintained by the primary investigators of the respective trials and, in many cases, as explained in the “Limitations” section, this field is updated voluntarily by the primary investigator and seems to be inconsistent. Our analysis showed that only 61 (11%) of all included 556 randomized clinical trials reported results in the ClinicalTrials.Gov database.
We defined a comprehensive and specific categorization of the funding sources of trials. We analyzed the content of the “Lead_Sponsor” field, available in trials’ XML files exported from ClinicalTrials.gov, which comprises information regarding the entity or individual that sponsors the clinical study [
We exported all the contents of the 556 included registered randomized clinical trials from the ClinicalTrials.gov website in XML format and then identified existing publications by two processes: automated and manual identification processes. The automated identification process considered all publications referenced in the trial's registry record as well as a PubMed search according to each trial’s National Clinical Trial registration number. The manual identification process was a multistep process aimed to search trial publications by key trial attributes and author details in two major bibliographic databases (PubMed and Medline) as well as the Google search engine. We only considered the results of a clinical trial to be “published” if at least one of the primary outcome measures was reported. Complete details of the publication-identification processes are described in
We exported the entire ClinlicalTrials.gov database, with 265,657 registered clinical trials entries as of February 10, 2018, into a local Structured Query Language server database. The 47 indicated search terms and phrases were then applied in the Structured Query Language server database as follows:
For every search term and phrase, identify matching records by the [Title] OR [Interventions] OR [Outcome Measures] fields. We identified 6717 matching trials.
Apply the latest completion date criteria between April 1, 2010, and April 1, 2013. We obtained 803 matching trials.
After screening against all inclusion and exclusion criteria, 247 registered clinical trials were excluded as per the following breakdown:
149 trials were not randomized.
52 trials had false-positive matching terms. For example, the registered clinical trial NCT01287377 examined the association between nicotine patch messaging and smoking cessation. The trial term “messaging” was a false-positive match to one of our search terms.
17 trials were only related to computerized neuropsychological, cognition, and oxygen assessment tools.
11 trials focused only on internet, Web-based, online, and computer-based surveys.
9 trials were limited to the phone call intervention component.
5 trials were related to scanners and diagnostic imaging devices.
3 trials were related to television or online advertisement.
1 trial was related to electronic medical record systems.
Finally, 556 studies were included after screening.
A summary of the search results is presented in
In summary, 406 of 556 (73%) trials were associated with identified outcome publications and 150 of 556 (27%) trials did not have any identified publications or their identified publications did not report any of their primary outcomes. Only 6 of the 556 (1.1%) published trials did not report any of the primary outcome measures indicated in the trial’s registration protocols (
We conducted a statistical descriptive analysis, describing and summarizing the characteristics of all the 556 included registered randomized clinical trials by the following standard data elements exported from and defined by the ClinicalTrials.gov database: age group, condition, country, gender, intervention model, lead sponsor, masking, recruitment status, start date, study arms, study results, trial phase, and trial size [
We examined the relationship between trial characteristics and the nonpublication rate using bivariate and multivariate analyses. For bivariate analysis, we used the Pearson Chi-square statistical test, and for multivariate analyses, we used binary logistics regression in SPSS (IBM Corporation, Armonk, NY). The results of this analysis are depicted in
The Pearson Chi-square test and binary logistic regression test results reported significant relationships (
Trials included from the search results.
Results of the publication-identification process. *NCT: National Clinical Trial.
The Pearson Chi-square test results showed a significant association (
The Pearson Chi-square test results showed significant differences (
Only 38 (6.8%) of the 556 included registered randomized clinical trials were funded by industry sponsors. We observed a trend of 1.5 times higher nonpublication rate for industry-funded trials than non-industry-funded trials. However, this trend was not statistically significant (
Our Pearson Chi-square test results showed significant differences (
We examined the relationship between prospective trial registrations and trial nonpublication rates. Results of the Pearson Chi-square test showed a statistically significant relationship (
Relationship between the characteristics of randomized clinical trials and nonpublication rate.
Trial characteristics | Unpublished RCTsa/Total RCTsa, n (%) | P valueb | Binary logistic regression | ||
P value | Odds ratio (95% CI) | ||||
Overall | 150/556 (27%) | — | — | — | |
0.52 | 0.36 | ||||
Adult | 27/97 (27.8%) | — | 0.47 | 0.689 (0.254 to 1.871) | |
Adult/Senior | 90/312 (28.8%) | — | 0.73 | 0.864 (0.337 to 1.987) | |
Child | 0/2 (0%) | — | 0.99 | >999.999 (0 to >999.999)c | |
Child/Adult | 20/79 (25.3%) | — | 0.29 | 1.738 (0.627 to 4.821) | |
Child/Adult/Senior | 13/66 (19.7%) | — | — | Reference | |
0.005 | 0.01 | ||||
Cancer | 14/31 (45.2%) | — | 0.1 | 0.414 (0.740 to 16.173) | |
Chronic pain and chronic conditions (including diabetes, asthma, and COPDd) | 24/81 (29.6%) | — | 0.52 | 0.752 (0.317 to 1.784) | |
Heart disease, hypertension, and stroke | 15/53 (28.3%) | — | 0.8 | 1.130 (0.436 to 2.931) | |
Mental health, neurodevelopmental disorders, Alzheimer, dementia, and epilepsy | 14/78 (17.9%) | — | 0.31 | 1.585 (0.648 to 3.877) | |
Multiconditions | 23/53 (43.4%) | — | 0.11 | 0.480 (0.197 to 1.165) | |
Obesity, weight management, nutrition, and physical activity | 17/60 (28.3%) | — | 0.11 | 2.455 (0.810 to 7.438) | |
Smoking, alcohol consumption, substance abuse, and addiction | 9/57 (15.8%) | — | 0.12 | 3.458 (0.740 to 16.173) | |
Others | 34/143 (23.8%) | — | — | Reference | |
<.001 | <.001 | ||||
Outside the United States | 39/218 (17.9%) | — | <.001 | 3.317 (1.845 to 5.964) | |
United States | 111/338 (32.8%) | — | — | Reference | |
<.001 | 0.02 | ||||
≤5th percentile (up to 26 participants) | 15/29 (51.7%) | — | 0.99 | >999.999 (0 to >999.999)c | |
Between the 5th and 50th percentile (between 27 and 148 participants) | 58/244 (23.8%) | — | 0.99 | >999.999 (0 to >999.999)c | |
Between the 50th and 95th percentile (between 149-1962 participants) | 59/246 (24%) | — | 0.99 | >999.999 (0 to >999.999)c | |
>95th percentile (more than 1962 participants) | 8/27 (29.6%) | — | 0.99 | >999.999 (0 to >999.999)c | |
Undefined | 10/10 (100%) | — | — | Reference | |
0.14 | 0.21 | ||||
<1 month | 13/56 (23.2%) | — | — | Reference | |
1-3 months | 34/138 (24.6%) | — | 0.44 | 1.436 (0.574 to 3.595) | |
4-6 months | 32/171 (18.7%) | — | 0.62 | 0.792 (0.314 to 1.997) | |
6-12 months | 45/128 (35.2%) | — | 0.39 | 0.670 (0.272 to 1.653) | |
12-24 months | 12/40 (30%) | — | 0.89 | 1.085 (0.330 to 3.570) | |
>24 months | 5/17 (29.4%) | — | 0.9 | 0.908 (0.200 to 4.124) | |
Undefined | 9/60 (15%) | — | 0.19 | 2.199 (0.673 to 7.185) | |
0.98 | 0.64 | ||||
Both | 132/491 (26.9%) | — | 0.88 | 0.877 (0.168 to 4.567) | |
Female | 15/55 (27.3%) | — | 0.76 | 1.318 (0.225 to 7.738) | |
Male | 3/10 (30%) | — | — | Reference | |
0.09 | 0.29 | ||||
Single assignment | 14/33 (42.4%) | — | 0.99 | 1.475 (0.929 to 2.343) | |
Crossover assignment | 4/21 (19%) | — | 0.99 | <.001 (<.001 to >999.999)c | |
Parallel assignment | 121/464 (26.1%) | — | 0.99 | <.001 (<.001 to >999.999)c | |
Factorial assignment | 11/32 (34.4%) | — | 0.99 | <.001 (<.001 to >999.999)c | |
Undefined | 0/6 (0%) | — | — | Reference | |
0.07 | 0.06 | ||||
Before 2012 | 63/269 (23.4%) | — | 0.06 | 1.636 (0.987 to 2.714) | |
On or after 2012 | 87/287 (30.3%) | — | — | Reference | |
0.07 | 0.3 | ||||
No | 135/518 (26.1%) | — | 0.3 | 1.609 (0.650 to 3.986) | |
Yes | 15/38 (39.5%) | — | — | Reference | |
0.67 | 0.58 | ||||
Computer-based intervention (offline) | 27/97 (27.8%) | — | 0.99 | 0.995 (0.119 to 8.299) | |
Email notifications | 7/24 (29.2%) | — | 0.88 | 0.834 (0.082 to 8.444) | |
Mobile phone application | 5/14 (35.7%) | — | 0.84 | 0.771 (0.058 to 10.204) | |
Telemonitoring devices | 16/64 (25%) | — | 0.54 | 1.950 (0.226 to 16.842) | |
Text messaging | 9/53 (17%) | — | 0.61 | 1.799 (0.188 to 17.215) | |
Web-based intervention | 84/294 (28.6%) | — | 0.93 | 0.914 (0.114 to 7.336) | |
Wii | 2/10 (20%) | — | — | Reference | |
0.41 | 0.41 | ||||
Open label | 86/319 (26.7%) | — | 0.07 | 12.986 (0.786 to 213.344) | |
Single label | 53/177 (29.9%) | — | 0.12 | 9.041 (0.546 to 149.7930) | |
Double label | 7/30 (23.3%) | — | 0.07 | 15.213 (0.781 to 296.201) | |
Triple label | 1/16 (6.3%) | — | 0.99 | >999.999 (0 to >999.999)c | |
Quadruple label | 1/7 (14.3%) | — | 0.17 | 13.859 (0.332 to 578.089) | |
Undefined | 2/7 (28.6%) | — | — | Reference | |
0.01 | 0.004 | ||||
0/I | 5/31 (16.1%) | — | 0.08 | 3.112 (0.876 to 11.054) | |
I/II or II | 8/56 (14.3%) | — | 0.01 | 3.882 (1.460 to 10.318) | |
II/III, III, or IV | 17/42 (40.5%) | — | 0.13 | 0.512 (0.217 to 1.208) | |
Undefined | 120/427 (28.1%) | — | — | Reference | |
0.16 | 0.25 | ||||
Adherence to treatment | 11/26 (42.3%) | — | 0.69 | 0.761 (0.202 to 2.868) | |
Clinical evaluation | 76/316 (24%) | — | 0.42 | 1.386 (0.631 to 3.044) | |
Drug, tobacco, and alcohol use | 10/41 (24.1%) | — | 0.81 | 0.813 (0.148 to 4.475) | |
Physical activity and diet intake | 9/30 (30%) | — | 0.97 | 1.022 (0.330 to 3.161) | |
Process evaluation | 13/58 (22.4%) | — | 0.04 | 2.924 (1.036 to 8.250) | |
Undefined | 1/3 (33.3%) | — | 0.3 | 1.341 (0.782 to 2.297) | |
Vital measurement | 30/82 (36.6%) | — | — | Reference | |
0.006 | 0.29 | ||||
Retrospective | 93/393 (23.7%) | — | 0.29 | 1.341 (0.782 to 2.297) | |
Prospective | 57/163 (35%) | — | — | Reference | |
<.001 | <.001 | ||||
Active, not recruiting | 0/1 (0%) | — | 0.99 | >999.999 (0 to >999.999)c | |
Completed | 105/468 (22.4%) | — | 0.002 | 3.303 (1.564 to 6.976) | |
Suspended | 3/4 (75%) | — | 0.21 | 0.188 (0.014 to 2.497) | |
Terminated | 11/17 (64.7%) | — | 0.21 | 0.403 (0.098 to 1.656) | |
Unknown status | 21/56 (37.5%) | — | 0.99 | >999.999 (0 to >999.999)c | |
Withdrawn | 10/10 (100%) | — | — | Reference | |
0.71 | 0.99 | ||||
After 2008 | 109/413 (26.4%) | — | 0.99 | <.001 (<.001 to >999.999)c | |
On or Before 2008 | 41/142 (28.9%) | — | 0.99 | <.001 (<.001 to >999.999)c | |
Undefined | 0/1 (0%) | — | — | Reference | |
0.11 | 0.4 | ||||
One | 8/18 (44.4%) | — | 0.17 | 0.240 (0.032 to 1.820) | |
Two | 101/410 (24.6%) | — | 0.63 | 1.486 (0.296 to 7.459) | |
Three | 27/75 (36%) | — | 0.74 | 0.756 (0.143 to 3.999) | |
Four or more | 11/38 (28.9%) | — | 0.78 | 1.295 (0.219 to 7.646) | |
Undefined | 3/15 (20%) | — | — | Reference | |
0.86 | 0.79 | ||||
No | 133/495 (26.9%) | — | 0.79 | 1.113 (0.512 to 2.420) | |
Yes | 17/61 (27.9%) | — | — | Reference |
aRCT: randomized controlled trial.
b
cNonconvergence was reported after 20 iterations possibly due to quasicomplete separation. Logistic regression model was not appropriate for this variable level value.
dThe median of the latest completion date year was 2012.
eThe cut-off point for the year of start date was set at 2008, the year when the 7th Declaration of Helsinki was adopted.
Results of the Pearson Chi-square test between start date of trials and prospective trial registration.
Trial start date | Prospective trial registrations/total, n (%) | |
Before or on 2008 | 73/142 (51.4%) | <.001 |
After 2008 | 90/414 (21.7%) | <.001 |
Results of the Pearson Chi-square test showed a statistically significant relationship (
Summary of reasons for discontinuation.
Reason for discontinuation | Trials (N=31), n (%) |
Recruitment challenges | 9 (29%) |
Funding challenges | 6 (19%) |
New study priorities | 3 (10%) |
Primary investigator/staff attrition | 2 (6%) |
Drop out | 2 (6%) |
Technical challenges | 2 (6%) |
Primary investigator/staff attrition and funding challenges | 2 (6%) |
Not provided | 5 (16%) |
Our analysis showed that recruitment and funding challenges are major factors contributing to discontinuation of trials and their nonpublication rates. Details of the classification of discontinuation reasons are provided in
Results of the Pearson Chi-square test showed no statistically significant relationship (
We aimed to analyze the duration required to publish trial results for the 556 included trials. We measured the time to publication as the duration in years between the start date of trials and their respective publication date, which we then reported along with the number of published trials and cumulative nonpublication rates on a biyearly scale (
The majority of our 556 included trials were published within 6 and 8 years of the trial’s start date (356 [64%] and 393 [70.7%], respectively). A total of 148 (26.6%) trials were published in the fourth year of the trial. We also observed that half of our included trials were published between the fourth and fifth year after the trial start date.
No enrollment values were identified for ten trials in the ClinicalTrials.gov database, and we could not identify any publications for these trials. We stratified all trials into four strata by size at the 5th, 50th, and 95th percentiles and found a statistically significant difference between the nonpublication rate of trials and trial size. The highest nonpublication rate was 51.7% for small trials that enrolled no more than 26 participants (at the 5th percentile), whereas the lowest nonpublication rate was 23.8% for trials that enrolled between 27 and 148 participants (between the 5th and 50th percentile).
The Pearson Chi-square test showed a statistically significant relationship between the nonpublication rate and trial size (
Analysis of trial publication cycles (duration).
Time to publication (start date to publication date), years | Published trials (N=556), n (%) | Cumulative nonpublication rate (N=556), % |
2 | 108 (19.4%) | 80.6 |
4 | 148 (26.6%) | 54 |
6 | 100 (18%) | 36 |
8 | 37 (6.7%) | 29.3 |
10 | 9 (1.6%) | 27.7 |
<15 | 3 (1%) | 27.2 |
Time to publication of registered clinical trials in digital health.
The research literature has identified the impact and risks of publication bias for researchers, clinicians, healthcare professionals, and health policy decision makers
as well as a number of factors contributing to nonpublication and discontinuation of clinical trials [
In the domain of digital health, we analyzed the nonpublication rate among 556 randomized clinical trials that were registered in ClinicalTrials.gov, with the latest completion date between April 2010 and April 2013. We found that 27% of all included trials remain unpublished 5 years after the latest completion date. Our finding is in line with a similar study of large randomized clinical trials, with at least 500 enrolled participants, that reported a 29% nonpublication rate [
As part of our publication-identification process, we compared the published outcomes and primary outcomes of trials indicated in the trial registration entries in ClinicalTrials.gov. Only 6 of the 556 (1.1%) published trials did not report any of the primary outcome measures indicated in the trial registration protocols. Our finding is substantially different and should not be compared to findings from other studies that reported that 40%–62% of clinical trials had at least one change in primary outcome when comparing trial publications and protocols [
We reported a statistically significant relationship between the nonpublication rate and eight different condition groups in the Pearson Chi-square test (
We also found that half of our included trials enrolled ≥148 participants, which is similar to other findings from two different studies: 46% of trials included ≥160 participants, and 45% of trials included ≥100 participants [
Randomized clinical trials are usually conducted in a series of phases, 0 to IV, to examine the intervention efficacy, safety, and adverse events over various periods and sizes of population samples [
In our study, we reported a statistically significant relationship between the trial recruitment status and trial nonpublication rate, and completed trials were 3.3 times more likely to be published (
We analyzed the nonpublication rate with regard to the start date year of trials, stratified according to their start before or after 2008, when the 7th revision of the Declaration of Helsinki was adopted [
We postulate that the nonpublication rate may be higher for trials registered prospectively, as the primary investigator would register a trial before the enrollment of any participant, without knowing if the trial would be completed successfully or the results would ultimately be published. The Pearson Chi-square test showed a statistically significant relationship (
Most of our included trials were published within 6 to 8 years after the trial start date (356 [64%] and 393 [70.7%], respectively). We also observed that half of our included trials were published between the fourth and fifth year of the trial start date. The timelines of our findings are comparable to those of a 2007 study that analyzed time to publication of clinical trials (also measured from the start to publication date) and reported that clinical trials with statistically significant positive results were published 4-5 years after their start date, whereas trials with negative results were published in 6-8 years [
When we analyzed the funding sources of trials, we found that only a small number of trials (38 [6.8%] of our included trials) were funded by the industry. This finding is in contrast with the results of other studies, in which most included trials were funded by the industry. A study of delayed and nonpublication of randomized clinical trials on vaccines reported that 85% of their included trials were funded by the industry [
We observed a trend of 1.5 times higher nonpublication rates among industry-funded trials than among non-industry-funded trials. However, the trend was not statistically significant, which may be explained by the small sample size. We also found that the ratio of industry-funded trials in the United States is five times higher than that of international trials. Although these findings may be interpreted by the predominantly privately funded healthcare system in the United States, they could also be attributed to the scale of the digital health industry in the United States compared to the rest of the world, with US–based digital health startups holding 75% of the global market shares between 2013 and 2017 [
Despite ICMJE–mandated trial registration since 2005, not all randomized trials are registered [
In this study, the ClinicalTrials.gov database was the sole data source of trial registrations. The choice was driven by feasibility challenges with limited research resources available for this study initiative and broader and global adoption of the ClinicalTrials.gov registry within the biomedical research enterprise. There are many other trials registries such as the European Clinical Trials Registry [
Our publication-identification process was conducted between June 29, 2016, and February 10, 2018, for all included 556 randomized clinical trials. Therefore, our findings did not include studies published after February 10, 2018. This study includes trials based on their completion date and primary completion date declared in the registry record in ClinicalTrials.gov. When not provided, we considered the latest completion date as described in
From our study of 556 randomized clinical trials in the field of digital health that are registered in the ClinicalTrials.gov database, we found that nonpublication of trials is prevalent, with almost a third (150, 27%) of all included trials remaining unpublished 5 years after their completion date. There are distinct differences in nonpublication rates between US- and non-US–based trials and according to the funding sources (industry sponsors vs non-industry sponsors). Further research is required to define the rationale behind the nonpublication rates from the perspectives of primary investigators and, more importantly, to articulate the impact and risk of publication bias in the field of digital health clinical trials. Future studies could also include nonrandomized trials such as projects published in protocols (such as JMIR Research Protocols).
It is not clear whether the research or technology failed, or if the results were disappointing and scholars did not write up a report, or if reports were rejected by journals; however, given the multitude of potential publication venues, and increased transparency in publishing, the former seems more likely. Scholarly communication is evolving, and short reports of failed trials may not always be published in peer-reviewed journals, but may be found in preprint servers. With the growing popularity of preprints, future analyses may also include searches for draft reports on preprint servers (such as preprints.jmir.org) to include unpublished reports, which may further shed light on why trials failed or remained unpublished. In the meantime, a general recommendation would be to conduct thorough formative research and pilot studies before conducting a full randomized controlled trial to reduce the risk of failure such as having insufficient power due to lack of participant engagement and nonuse attrition [
Evaluation of the latest completion date of trials.
Determination of search terms and phrases.
Classification of trial condition groups.
Classification of reasons for discontinuation of trials.
Classification of major technologies used in trials.
Identification of publications.
Global distribution of all included trials.
International Committee of Medical Journal Editors
odds ratio
randomized controlled trial
GE is the editor-in-chief of the Journal of Medical Internet Research (and publisher at JMIR Publications) but was not involved in the peer-review or decision-making process for this paper. The associate editor handling this manuscript and the reviewers were blinded and not aware of the co-authorship of GE. As owner of JMIR Publications, GE may benefit from increased publication rates of digital health trials. The other authors declare no conflicts of interests.