The Karma system is currently undergoing maintenance (Monday, January 29, 2018).
The maintenance period has been extended to 8PM EST.

Karma Credits will not be available for redeeming during maintenance.
Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 05.04.18 in Vol 20, No 4 (2018): April

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/9368, first published Nov 08, 2017.

This paper is in the following e-collection/theme issue:

    Original Paper

    Impact of Low Back Pain Clinical Trials Measured by the Altmetric Score: Cross-Sectional Study

    1Masters and Doctoral Programs in Physical Therapy, Universidade Cidade de São Paulo, São Paulo, Brazil

    2Musculoskeletal Health Sydney, School of Public Health, The University of Sydney, Sydney, Australia

    Corresponding Author:

    Amanda Costa Araujo, MSc

    Masters and Doctoral Programs in Physical Therapy

    Universidade Cidade de São Paulo

    Tatuapé

    448 Cesario Galeno St

    São Paulo, 03070-000

    Brazil

    Phone: 55 11 2476 5749

    Email:


    ABSTRACT

    Background: There is interest from authors and publishers in sharing the results of their studies over the Internet in order to increase their readership. In this way, articles tend to be discussed and the impact of these articles tends to be increased. In order to measure this type of impact, a new score (named Altmetric) was created. Altmetric aims to understand the individual impact of each article through the attention attracted online.

    Objective: The primary objective of this study was to analyze potential factors related with the publishing journal and the publishing trial that could be associated with Altmetric scores on a random sample of low back pain randomized controlled trials (RCTs). The secondary objective of this study was to describe the characteristics of these trials and their Altmetric scores.

    Methods: We searched for all low back pain RCTs indexed on the Physiotherapy Evidence Database (PEDro; www.pedro.org.au) published between 2010 and 2015. A total of 200 articles were randomly selected, and we extracted data related to the publishing trial, the publishing journal, methodological quality of the trials (measured by the 0-10 item PEDro scale), and total and individual scores of Altmetric mentioned and Altmetric reader. The study was a cross-sectional study, and multivariate regression models and descriptive statistics were used.

    Results: A total of four variables were associated with Altmetric mentioned score: impact factor (β-coefficient=3.4 points), number of years since publication (β-coefficient=–4.9 points), number of citations divided by years since publication (β-coefficient=5.2 points), and descriptive title (β-coefficient=–29.4 points). Only one independent variable was associated with Altmetric reader score: number of citations divided by years since publication (β-coefficient=10.1 points, 95% CI 7.74-12.46). We also found that the majority of articles were published in English, with a descriptive title, and published in open access journals endorsing the Consolidated Standards of Reporting Trials (CONSORT) statement.

    Conclusions: Researchers should preferably select high impact factor journals for submission and use declarative or interrogative titles, as these factors are likely to increase the visibility of their studies in social media.

    J Med Internet Res 2018;20(4):e86

    doi:10.2196/jmir.9368

    KEYWORDS



    Introduction

    There is growing interest from both authors and publishers in sharing the results of their studies over the Internet in order to increase their readership [1,2]. Similarly, consumers of research (including clinicians and patients) also share articles that they found interesting, useful, or controversial with their peers over the Internet. One of the ways that articles can be disseminated on a large scale is by sharing them on social media, such as Facebook, Twitter, Instagram, and others. Another way involves the use of reference manager websites such as Mendeley or Connotea [2]. Both methods offer new approaches to access, read, and discuss research articles; consequently, the dissemination of these articles increases [2-4].

    The most traditional way of quantifying the scientific impact of an article or a journal is through the number of citations in peer-reviewed journals [5]. However, indices related to the number of journal citations do not necessarily reflect a greater dissemination of the content of articles to clinicians and patients. Until very recently, the impact of scientific articles on social media and reference managers was not quantified. In order to measure this type of impact a new score (named Altmetric) was created [2,6].

    Altmetric is a tool developed by a group of British researchers [7]. Altmetric aims to understand the individual impact of each article through the attention attracted online (eg, on social media and reference managers) [6]. The Altmetric attention score is composed by two independent scoring systems: the Altmetric mentioned score and the Altmetric reader score [7]. The Altmetric mentioned score for an article reflects how widely an article is mentioned in a range of media, including social media (eg, Facebook, Twitter), newspapers, encyclopedias (eg, Wikipedia), online platforms (eg, Faculty1000 and Publication Peer-Reviews), videos on YouTube, sites on questions and answers (eg, Q&A stack overflow), and policy documents or PDF documents available over the Internet . Each of these mentions receives different weights to reflect the relative reach of each source, and contributes to the total score. For example, each mention on Facebook counts as 0.25 points while a mention on Twitter counts as 1.0 point. The Altmetric reader score can be visualized by clicking the Altmetric “donut” symbol (ie, a visual representation of the Altmetric score) and summing the number of readers. This score is hidden in the donut. The second score is the Altmetric reader score which measures the impact on online reference managers such as Mendeley, CiteULike and Connotea [7] . This score has identical weights for all reference managers (ie, 1.0 point for each mention). Readers can easily identify the Altmetric mentioned in the websites of most journals by clicking on the Altmetric “donut.”

    Most journal articles about Altmetric published to date are only introductory tutorials or editorials [2,4-6,8]. Smith et al [9] published a discussion paper about the importance of Altmetric in the field of health sciences that aimed to quantify the social impact of these articles. Patthi et al [10] published a systematic review that retrieved seven articles published between 2010 to 2016 in the dental area that aimed to analyze the correlations between journal citations measured by the Web of Science website and Altmetric mentioned scores. This review concluded that journal citations and Altmetric scores are positively correlated (between r=0.30 and 0.61) [10]. Finally, Rosenkrantz et al [11] also observed positive correlations between citations and Altmetric scores in radiology journals.

    There are three articles [12-14] that have measured the correlation among Altmetric, Tweets, blogs, and Mendeley (a reference management software). These studies observed a large increase in the number of Tweets and blogs related to scientific journals, and these variables were correlated with the Altmetric score . Previously published articles [13,15,16] showed that the number of Tweets can predict citations within the first three days of article publication. These findings indicate that Altmetric scores are likely to be correlated with the journal’s impact factor [8]. Rinald [17] published a tutorial about the benefits of open access journals with regards to visibility on social media. However, more research is needed to identify potential variables that might be associated with Altmetric scores, such as the journal impact factor, number of years since publication, study quality, and open access articles.

    To our knowledge, there is no study describing the characteristics of randomized controlled trials (RCTs) and their Altmetric scores or predictive factors of Altmetric score. In this study, trials of nonpharmacological interventions for low back pain were chosen by the authors because back pain has the largest amount of evidence in the field of musculoskeletal health [18]. Additionally, back pain is extremely prevalent [19-21] and involves high costs [19,20,22]. According to a study that ranks the most disabling diseases in the world [21], low back pain has been one of the highest ranking musculoskeletal diseases since 1990 [21].

    Therefore, the primary objective was to analyze potential factors related to the publishing journal (eg, online access) and the publishing trial (eg, trial quality) that could be associated with Altmetric scores in a random sample of low back pain RCTs. The secondary objective of this study was to describe the characteristics of these trials and their Altmetric scores.


    Methods

    Study Design

    This is a cross-sectional study.

    Search Strategies

    We selected a random sample of 200 low back pain RCTs from the Physiotherapy Evidence Database (PEDro) [23]. We have chosen PEDro because this is the most comprehensive database of physiotherapy trials [24,25], and also because the PEDro scale has acceptably high reliability and validity [18]. In addition, all trials indexed on PEDro are rated for methodological quality using the 0-to-10-point PEDro scale [24-27]. The items are described below:

    1. Eligibility criteria
    2. Random allocation
    3. Concealed allocation
    4. Baseline comparability
    5. Blinding of subjects
    6. Blinding of therapists
    7. Blinding of outcome assessors
    8. Completeness of follow up
    9. Intention to treat analysis
    10. Between-group statistical comparisons
    11. Presentation of point measures and measures of variability

    Textbox 1. Data sources and details that were extracted.
    View this box

    The total PEDro score is computed by summing yes responses to items 2-11. The first item does not count in the final score because this is related to external validity. All trials on PEDro are rated by at least two trained raters and, in cases of disagreement, a final arbitration is performed by a senior rater.

    On February 1, 2016 we identified all low back pain trials indexed on PEDro that were published in the time period of 2010-2015 and selected a random sample of 40%. We excluded trial protocols, preliminary analyses of trials, and secondary analyses. The search strategy is described as follows:

    Strategy search: “2010 until 2015” [year of publication] and “low back pain” [part of body] and “pain” [problem] and “clinical trial” [method].

    Data Extraction

    Several pieces of data were extracted, as detailed in Textbox 1.

    Data related to Altmetric scores and number of citations divided by years since publication were collected on May 10, 2016 for all articles because these scores are extremely dynamic.

    Statistical Analyses

    The number of years since publication of the article and the number of citations were determined as of May 10, 2016. The number of citations was normalized by the number of years since publication (number of citations divided by years since publication), as it is expected that older manuscripts are more likely to have a larger number of citations compared to newer ones. Descriptive statistics were used to present most of the data.

    Separate multivariate regression models were built to predict (1) Altmetric mentioned score and (2) Altmetric reader score. The independent variables in both models were: impact factor, paper was published as open access (yes/no), total PEDro score, number of years since publication, normalized citation count, and type of title. These variables were chosen because it seems plausible that they would be associated with Altmetric scores. For example, we choose the variable number of years since publication and total PEDro score because they are related to the number of accesses on PEDro [18].

    Initially, univariate regression analyses were performed and all variables that reached a P value of <0.20 were retained for inclusion in the multivariate model. Multivariate regression models were then built and the final model contained only variables that reached a statistical significance of P<0.05. The results were expressed as R² indexes (explained variability of the model) and the individual contribution of each variable was expressed through the presentation of β-coefficients and their respective 95% CIs. We used the Statistical Package for Social Sciences (SPSS) version 19 for the analyses.


    Results

    Selection of Eligible Articles

    A total of 537 clinical trials were retrieved using the search strategy. Sixty-seven articles were excluded because they were related to conditions other than low back pain or were related to studies in progress. From the remaining 470 articles, 200 were randomly selected for analyses (Figure 1).

    Descriptive Characteristics of Articles

    Table 1 presents the characteristics of the trials. From the 200 articles, 186 had an Altmetric score with a mean mentioned score of 18.2 (SD 41.3) and a mean reader score of 34.9 (SD 41.6). Most of the articles were published in English, had a descriptive title (title describing the aim, but does not reveal the main conclusions) and were published as open access in journals that endorse the CONSORT statement. In addition, the mean impact factor of the journals publishing these trials was 2.1 (SD 2.6) with a mean total PEDro score of 5.8 points (SD 1.6; Table 1).

    Predictive Factors

    The univariate analysis for Altmetric mentioned score showed that being published in an open access journal was not independently associated with Altmetric mentioned score (Table 2). The final multivariate model is presented in Table 3. Four variables were associated with Altmetric mentioned score: impact factor (β-coefficient=3.4 points), number of years since publication (β coefficient=–4.9), number of citations divided by years since publication (β-coefficient=5.2 points), and descriptive title (β-coefficient=–29.4 points). This model accounts for 28% of the explained variance. The interpretation of this model is that older articles and those with descriptive titles were associated with a lower Altmetric mentioned score, whereas articles from journals with a higher impact factor and with greater citations were associated with a higher Altmetric mentioned score.

    The univariate analysis for Altmetric reader score showed that being published in an open access journal was not associated with Altmetric reader score (Table 4). The multivariate analysis showed that one independent variable was associated with Altmetric reader score: number of citations divided by years since publication (β-coefficient=10.1 points, 95% CI 7.74-12.46). This single variable accounted for 31% of the explained variance.

    Figure 1. Flow diagram.
    View this figure
    Table 1. Characteristics of the included trails (n=200), Altmetric mentioned and Altmetric reader scores. Categorical data were expressed as numbers (percentage). Continuous normal data were expressed as means (SD). PEDro: Physiotherapy Evidence Database.
    View this table
    Table 2. Univariate model to predict characteristics that were associated with Altmetric mentioned score. PEDro: Physiotherapy Evidence Database.
    View this table
    Table 3. Final multivariate model to predict characteristics that were associated with Altmetric mentioned score.
    View this table
    Table 4. Univariate model to predict characteristics that were associated with Altmetric reader score. PEDro: Physiotherapy Evidence Database.
    View this table

    Discussion

    Principal Findings

    The primary objective of this study was to analyze potential factors that could be associated with Altmetric score. The secondary objective was to describe the characteristics of low back pain RCTs and their Altmetric scores. We found that trials with interrogative/declarative titles, those published in higher impact factor journals, those published more recently, and those with a larger number of citations were associated with a higher Altmetric mentioned score. We observed that the number of citations was also associated with a higher Altmetric reader score. Finally, we found that the Altmetric reader score was higher than Altmetric mentioned score. Most of the articles were published in English, had descriptive titles, and were published as open access in journals that endorse the CONSORT statement.

    There are three previous articles that have measured correlations between the number of citations and Altmetric scores in medical journals [10,11,29], and numerous others studying the relationship between tweets (the main Altmetric score component) and citations ([15] being the first). The conclusions of these articles are very similar to our study: there is an association between citations and Altmetric scores. This information confirms that conventional measures of scientific impact (based on citations) are associated with social impact (based on social media). The difference between our study and these three previous studies is that we used multivariate regression analyses rather than simple correlations. We believe that this approach allowed us to focus on the key independent variables, and the beta coefficients we provide are more interpretable than correlation coefficients. For example, we can predict that for every citation received, 5.2 and 10.1 points will be added to the Altmetric mentioned and reader scores, respectively. Scientific impact appears to follow the social impact in back pain trials.

    We observed an association between Altmetric mentioned score and the journal’s impact factor. The journal’s impact factor is a measure that reflects the number of citations of scientific articles published in the journal divided for the two previous years [5]. We might infer that journals with higher impact factors have more credibility to a wider range of readers, health care providers, and media, which may reflect a large number of posts in social media. Many of these journals may have well-developed media strategies, such as preparation and distribution of releases to the media. This action encourages the promotion of their papers in newspapers, blogs, and social media more rapidly and efficiently than journals that do not do this. These journals also format the online versions of articles so that a reader can easily click an icon to post details about the study on social media, usually by embedding a key figure from the article in the post.

    We also observed that trials with declarative/interrogative titles were associated with higher Altmetric mentioned scores than those with descriptive titles. Our study is, to our knowledge, the first that has investigated the effect of the format of the articles title on Altmetric scores. There is evidence that articles with shorter titles are more likely to be highly cited [30-32]. Jamali et al [33] concluded that articles with interrogative titles are also associated with a larger number of citations and downloads [33]. Therefore, authors should be aware that shorter and interrogative titles should be considered in order to attract a wider audience for their manuscripts. Finally, we observed that papers published more recently also have a higher Altmetric score; it seems that recent studies are more likely to be shared. This finding should be investigated further in future studies.

    The strength of this study is the use of a representative sample of trials (N=200, or 40% of all trial reports indexed on PEDro and categorized as “low back pain”). A possible limitation of this study is related to external validity, as our dataset contains only articles about low back pain. It would be important to replicate our study in other health disciplines.

    Conclusion

    Our study brings new insights for authors on how to increase the visibility of their articles. First, researchers should preferably select high impact factor journals for submission and use declarative or interrogative titles, as these factors are likely to increase the visibility of their studies in social media. Furthermore, we suggest new studies that use different research designs (eg, systematic reviews and guidelines) in order to externally validate our findings.

    Acknowledgments

    We would like to thank Anne M Moseley for her insightful contribution on this project, Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) for funding ACA’s PhD scholarship, and Australia’s National Health and Medical Research Council for funding CGM’s fellowship.

    Conflicts of Interest

    None declared.

    References

    1. Thelwall M, Haustein S, Larivière V, Sugimoto CR. Do altmetrics work? Twitter and ten other social web services. PLoS One 2013;8(5):e64841 [FREE Full text] [CrossRef] [Medline]
    2. Melero R. Altmetrics - a complement to conventional metrics. Biochem Med (Zagreb) 2015;25(2):152-160 [FREE Full text] [CrossRef] [Medline]
    3. Liu CL, Xu YQ, Wu H, Chen SS, Guo JJ. Correlation and interaction visualization of altmetric indicators extracted from scholarly social network activities: dimensions and structure. J Med Internet Res 2013 Nov 25;15(11):e259 [FREE Full text] [CrossRef] [Medline]
    4. Fabry G, Fischer MR. Beyond the impact factor - what do alternative metrics have to offer? GMS J Med Educ 2017;34(2):Doc27 [FREE Full text] [CrossRef] [Medline]
    5. Dinsmore A, Allen L, Dolby K. Alternative perspectives on impact: the potential of ALMs and altmetrics to inform funders about research impact. PLoS Biol 2014 Nov;12(11):e1002003 [FREE Full text] [CrossRef] [Medline]
    6. Brigham TJ. An introduction to Altmetrics. Med Ref Serv Q 2014;33(4):438-447. [CrossRef] [Medline]
    7. Altmetric. Altmetric for Scopus Internet   URL: https://help.altmetric.com/support/solutions/articles/6000062186-altmetric-for-scopus [accessed 2018-02-13] [WebCite Cache]
    8. Rhee JS. High-impact articles-citations, downloads, and Altmetric score. JAMA Facial Plast Surg 2015;17(5):323-324. [CrossRef] [Medline]
    9. Smith DR, Watson R. Career development tips for today's nursing academic: bibliometrics, altmetrics and social media. J Adv Nurs 2016 Nov;72(11):2654-2661. [CrossRef] [Medline]
    10. Patthi B, Prasad M, Gupta R, Singla A, Kumar JK, Dhama K, et al. Altmetrics - a collated adjunct beyond citations for scholarly impact: a systematic review. J Clin Diagn Res 2017 Jun;11(6):ZE16-ZE20 [FREE Full text] [CrossRef] [Medline]
    11. Rosenkrantz AB, Ayoola A, Singh K, Duszak R. Alternative metrics (“Altmetrics”) for assessing article impact in popular general radiology journals. Acad Radiol 2017 Jul;24(7):891-897. [CrossRef] [Medline]
    12. Maggio LA, Meyer HS, Artino AR. Beyond citation rates: a real-time impact analysis of health professions education research using Altmetrics. Acad Med 2017 Oct;92(10):1449-1455. [CrossRef] [Medline]
    13. Pinho-Costa L, Yakubu K, Hoedebecke K, Laranjo L, Reichel CP, Colon-Gonzalez MDC, et al. Healthcare hashtag index development: identifying global impact in social media. J Biomed Inform 2016 Oct;63:390-399 [FREE Full text] [CrossRef] [Medline]
    14. Delli K, Livas C, Spijkervet FKL, Vissink A. Measuring the social impact of dental research: an insight into the most influential articles on the Web. Oral Dis 2017 Nov;23(8):1155-1161. [CrossRef] [Medline]
    15. Eysenbach G. Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J Med Internet Res 2011;13(4):e123 [FREE Full text] [CrossRef] [Medline]
    16. Finch T, O'Hanlon N, Dudley SP. Tweeting birds: online mentions predict future citations in ornithology. R Soc Open Sci 2017 Nov 01;4(11):171371. [CrossRef]
    17. Rinaldi A. Spinning the web of open science: social networks for scientists and data sharing, together with open access, promise to change the way research is conducted and communicated. EMBO Rep 2014 Apr;15(4):342-346 [FREE Full text] [CrossRef] [Medline]
    18. Yamato TP, Arora M, Stevens ML, Elkins MR, Moseley AM. Quality, language, subdiscipline and promotion were associated with article accesses on Physiotherapy Evidence Database (PEDro). Physiotherapy 2017 Aug 12:1-7. [CrossRef] [Medline]
    19. Murray CJL, Lopez AD. Measuring the global burden of disease. N Engl J Med 2013 Aug 01;369(5):448-457. [CrossRef] [Medline]
    20. Beattie PF, Silfies SP, Jordon M. The evolving role of physical therapists in the long-term management of chronic low back pain: longitudinal care using assisted self-management strategies. Braz J Phys Ther 2016;20(6):580-591 [FREE Full text] [CrossRef] [Medline]
    21. GBD 2015 DiseaseInjury Incidence and Prevalence Collaborators. Global, regional, and national incidence, prevalence, and years lived with disability for 310 diseases and injuries, 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015. Lancet 2016 Dec 08;388(10053):1545-1602 [FREE Full text] [CrossRef] [Medline]
    22. Hoy D, Bain C, Williams G, March L, Brooks P, Blyth F, et al. A systematic review of the global prevalence of low back pain. Arthritis Rheum 2012 Jun;64(6):2028-2037 [FREE Full text] [CrossRef] [Medline]
    23. Physiotherapy Evidence Database. 2018 Feb 05.   URL: http://www.pedro.org.au [accessed 2018-02-23] [WebCite Cache]
    24. Moseley AM, Sherrington C, Elkins MR, Herbert RD, Maher CG. Indexing of randomised controlled trials of physiotherapy interventions: a comparison of AMED, CENTRAL, CINAHL, EMBASE, hooked on evidence, PEDro, PsycINFO and PubMed. Physiotherapy 2009 Sep;95(3):151-156. [CrossRef] [Medline]
    25. Michaleff ZA, Costa LOP, Moseley AM, Maher CG, Elkins MR, Herbert RD, et al. CENTRAL, PEDro, PubMed, and EMBASE are the most comprehensive databases indexing randomized controlled trials of physical therapy interventions. Phys Ther 2011 Feb;91(2):190-197. [CrossRef] [Medline]
    26. Maher CG, Moseley AM, Sherrington C, Elkins MR, Herbert RD. A description of the trials, reviews, and practice guidelines indexed in the PEDro database. Phys Ther 2008 Sep;88(9):1068-1077. [CrossRef] [Medline]
    27. de Morton NA. The PEDro scale is a valid measure of the methodological quality of clinical trials: a demographic study. Aust J Physiother 2009;55(2):129-133 [FREE Full text] [Medline]
    28. Hopewel S, Clarke M, Moher D, Wager E, Middleton P, Altman DG, et al. [CONSORT for reporting randomized controlled trials in journal and conference abstracts: explanation and elaboration]. Zhong Xi Yi Jie He Xue Bao 2008 Mar;6(3):221-232. [Medline]
    29. Amath A, Ambacher K, Leddy JJ, Wood TJ, Ramnanan CJ. Comparing alternative and traditional dissemination metrics in medical education. Med Educ 2017 Sep;51(9):935-941. [CrossRef] [Medline]
    30. Letchford A, Moat HS, Preis T. The advantage of short paper titles. R Soc Open Sci 2015 Aug;2(8):150266 [FREE Full text] [CrossRef] [Medline]
    31. Deng B. Papers with shorter titles get more citations. Nature 2015 Aug 26;2(8):150266 [FREE Full text] [CrossRef]
    32. Jacques TS, Sebire NJ. The impact of article titles on citation hits: an analysis of general and specialist medical journals. JRSM Short Rep 2010 Jun 30;1(1):2 [FREE Full text] [CrossRef] [Medline]
    33. Jamali HR, Nikzad M. Article title type and its relation with the number of downloads and citations. Scientometrics 2011 May 19;88(2):653-661. [CrossRef]


    Abbreviations

    CONSORT: Consolidated Standards of Reporting Trials
    PEDro: Physiotherapy Evidence Database
    RCT: randomized controlled trial
    SD: standard deviation


    Edited by G Eysenbach; submitted 08.11.17; peer-reviewed by C Liu, D Sanchez; comments to author 14.12.17; revised version received 19.12.17; accepted 02.01.18; published 05.04.18

    ©Amanda Costa Araujo, Dafne Port Nascimento, Gabrielle Zoldan Gonzalez, Christopher G Maher, Leonardo Oliveira Pena Costa. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 05.04.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.