Published on 28.11.19 in Vol 21, No 11 (2019): November
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/14809, first published May 24, 2019.
Patient-Reported Outcomes in Online Communications on Statins, Memory, and Cognition: Qualitative Analysis Using Online Communities
Background: In drug development clinical trials, there is a need for balance between restricting variables by setting eligibility criteria and representing the broader patient population that may use a product once it is approved. Similarly, although recent policy initiatives focusing on the inclusion of historically underrepresented groups are being implemented, barriers still remain. These limitations of clinical trials may mask potential product benefits and side effects. To bridge these gaps, online communication in health communities may serve as an additional population signal for drug side effects.
Objective: The aim of this study was to employ a nontraditional dataset to identify drug side-effect signals. The study was designed to apply both natural language processing (NLP) technology and hands-on linguistic analysis to a set of online posts from known statin users to (1) identify any underlying crossover between the use of statins and impairment of memory or cognition and (2) obtain patient lexicon in their descriptions of experiences with statin medications and memory changes.
Methods: Researchers utilized user-generated content on Inspire, looking at over 11 million posts across Inspire. Posts were written by patients and caregivers belonging to a variety of communities on Inspire. After identifying these posts, researchers used NLP and hands-on linguistic analysis to draw and expand upon correlations among statin use, memory, and cognition.
Results: NLP analysis of posts identified statistical correlations between statin users and the discussion of memory impairment, which were not observed in control groups. NLP found that, out of all members on Inspire, 3.1% had posted about memory or cognition. In a control group of those who had posted about TNF inhibitors, 6.2% had also posted about memory and cognition. In comparison, of all those who had posted about a statin medication, 22.6% (P<.001) also posted about memory and cognition. Furthermore, linguistic analysis of a sample of posts provided themes and context to these statistical findings. By looking at posts from statin users about memory, four key themes were found and described in detail in the data: memory loss, aphasia, cognitive impairment, and emotional change.
Conclusions: Correlations from this study point to a need for further research on the impact of statins on memory and cognition. Furthermore, when using nontraditional datasets, such as online communities, NLP and linguistic methodologies broaden the population for identifying side-effect signals. For side effects such as those on memory and cognition, where self-reporting may be unreliable, these methods can provide another avenue to inform patients, providers, and the Food and Drug Administration.
J Med Internet Res 2019;21(11):e14809
Upon the implementation of the American College of Cardiology–American Heart Association’s guidelines published in 2013, it was estimated that 1 billion people worldwide would be eligible to take statins to prevent cardiovascular diseases [, ]. In fact, hydroxymethylglutaryl–coenzyme A reductase inhibitors, or statin medications, have become one of the most commonly prescribed classes of medications in the United States. Often positioned as safe and effective, their prevalence and relative acceptance do not mean that patients are without risk. Commonly cited statin-associated symptoms include muscular complaints (pain, cramps, and muscle weakness) [ - ], diabetes mellitus [ , ], and changes in memory or cognition [ - ].
Examples of research investigating the effects of statins on the central nervous system include case studies , observational studies [ ], and randomized clinical trials [ - ] with meta-analyses reporting contradictory results [ , ]. Unfortunately, a discrepancy lies between the reported degree of memory changes and the actual observed frequency in large-scale clinical trials.
However, in 2012, on the basis of a review of spontaneous reports of amnesia, confusion, and concentration complaints, among others, from the Food and Drug Administration’s (FDA) Adverse Event Reporting System, the FDA required a statin label change :
Memory loss and confusion have been reported with statin use. These reported events were generally not serious and went away once the drug was no longer being taken.
Many feel the evidence is inconclusive and the decision remains controversial.
In drug development clinical trials, there is a need for balance between restricting variables by setting eligibility criteria and representing the broader patient population that may use a product once it is approved. Similarly, although recent policy initiatives focusing on the inclusion of historically underrepresented groups are being implemented, barriers still remain. These limitations of clinical trials may mask potential product benefits and side effects. To bridge these gaps, online communication in health communities may serve as an additional population signal for drug side effects.
Using these novel data sources requires unique strategies. There is a wealth of patient experience data to be found in online patient forums. The data here are unstructured, allowing for naturally occurring themes and topics to be identified through organic patient and caregiver language.
Natural language processing (NLP) as well as hands-on linguistic analysis can be applied to online posts. In this study, posts were authored by patients and caregivers on Inspire, a company that creates and manages online support communities for more than 1 million patients and caregivers. Analysis of online patient and caregiver communications discussing statins utilized NLP methodology and technology to draw correlations among discussions of memory events by statin users and compared these events with discussions of memory events of patients using other classes of medication as well as discussions of memory events of all other patients on Inspire. The NLP system uses tokenization, lemmatization, stemming, edit distances, acronym dissection, and word and phrase boundaries to understand the content and meaning of the posts. These findings were then associated with Wikipedia entries and correlated with a dictionary of conditions and treatments from the National Institutes of Health (developed in partnership with Stanford University) to accurately extract the entities used in this study. In addition, manual curation of posts using linguistic analysis provided qualitative context to these statistical findings.
By applying these tools to posts created by members on Inspire who belonged to communities focused on heart health, the data could be used to look at statin medications and identify any underlying crossover between the use of statins and impairment of memory or cognition. Moreover, this combined strategy accessed the patient’s voice, providing a detailed guide to how community members describe their experiences with statin medications and memory changes.
The objective of this study was to employ a nontraditional dataset to identify drug side-effect signals. Specifically, the study was designed to apply both NLP technology and hands-on linguistic analysis to a set of online posts from known statin users (1) to identify any underlying crossover between the use of statins and impairment of memory or cognition and (2) to obtain patient lexicon in their descriptions of experiences with statin medications and memory changes.
Researchers utilized user-generated content (UGC) on Inspire, looking at over 11 million posts across it. Posts were written by patients and caregivers belonging to a variety of communities on Inspire. After identifying these posts, researchers used NLP and hands-on linguistic analysis to draw correlations between statin use and memory and cognition.
Natural Language Processing and Statistical Comparison
In the corpus of over 11 million unique posts and over 440,000 different posters (patients and caregivers—authors of posts) on Inspire, the researchers used an NLP system that extracted relevant entities from each post. Entities are words or phrases that relate to each concept under consideration. For example, if the concept under consideration is cognition, several words or phrases related to that concept result in the creation of an entity (see).
The first stage of the analysis was to extract every post that contained an entity that related to memory loss or cognitive decline. The following are the entities that were used, each one of which has a distinct entry on Wikipedia: cognition, cognitive disorder, memory, recall, short-term memory, memory disorder, working memory, memory span, severe cognitive impairment, mild cognitive impairment, and cognitive deficit.
Note that each entity is associated with dozens of variations, that is, different phrasings, colloquialisms, and misspellings. For instance, cognition is associated with cognitive processing, cognitive ability, and cognitive function, as well as 75 separate misspellings of these phrases.
The researchers then did the same with statins, using the following entities: ulinastatin, cilastatin, niacin, simvastatin, fluvastatin, migrastatin, follistatin, mevastatin, oncostatin M, cystatin, cerivastatin, somatostatin, simvastatin, lovastatin, combretastatin A4 phosphate, myostatin, nystatin, atorvastatin, amlodipine, angiostatin, Crestor, Lipitor, and Vytorin.
Finally, the team identified medication discussions that could serve as the control group. The researchers wanted to choose a baseline of posts that indicated people were writing about a specific medication to reduce the experimental variation to a single change, specifically the medication being discussed. In this case, the decision was made to use Tumor Necrosis Factor TNF inhibitors.
TNF inhibitors were chosen for this statistical comparison for several reasons. First, TNF inhibitors are a commonly prescribed medication, used by a significant portion of Inspire members. Second, TNF inhibitors are prescribed for conditions that do not overlap with the conditions that statins treat; therefore, there would be as little overlap as possible between people who take both types of medications. Third, TNF inhibitors have shown no association with memory loss or cognitive decline. Fourth, generally the same age cohort uses TNF inhibitors and statins, minimizing age-related effects of cognitive decline between the 2 groups.
The entities used for TNF inhibitors were the following: Humira, golimumab, Enbrel, certolizumab pegol, and Remicade.
By finding authors who had written about one or more entities from each set and authors who had written posts that contained overlap among entities in multiple sets, the researchers were able to perform significant statistical analysis among the sets. It was irrelevant to the analysis as to what was mentioned first. An author could mention cognitive issues at one point in time and then later mention statins, or it could happen in the reverse order.
Hands-On Linguistic Analysis
To provide qualitative context to statistical findings, manual curation using linguistic methodology was performed on a subset of 246 UGC posts that mentioned both statin use and memory events. Posts were extracted and placed within an analyzable Excel file containing the following information: anonymized user identifier, date of post, title of the post, link to the post, and content of the post. Researchers developed a data-driven codebook with code label, full definition, and an example. Following the process described by Boyatzi, the team first reviewed and reduced the raw information; second, identified subsample themes; third, compared themes; fourth, created codes; and fifth, determined the reliability of the codes . This comprised tags around top topics, challenges, and descriptors to identify patient themes around the type of memory impairment, experience of impairment, and the patient lexicon. Overall, 2 separate researchers applied the codebook to the data. Calculating reliability as the number of agreements divided by the total number of agreements plus disagreements resulted in intercoder reliability of 90%.
The study was reviewed internally by Mayo Clinic and found to be exempt from the review the of Institutional Review Board. All personally identifiable information was removed. All data were evaluated without the knowledge of the identity of those involved.
NLP analysis of posts identified statistical correlations between statin users and discussion of memory impairment, which were not seen in control groups. Furthermore, linguistic analysis of a sample of posts provided themes and context to these statistical findings.
Natural Language Processing and Statistical Comparison
Through the analysis described above, the researchers found the following results for the number of people who had posted about one of these topics on Inspire (see).
To test the statistical significance of the observed high proportion of members that had a post that included the statins entity and that had a post that included memory, Fisher exact test (calculated in R, version 3.4.1) was applied. The calculated odds ratio was 9.703 (P value <.001), with a 95% CI of 9.066 to 10.378. This is outlined in a 2-way contingency table () used to calculate the odds ratio and the associated significance measures.
|Inspire member numbers||Posts about anything||Posts about Tumor Necrosis Factor inhibitors||Posts about statin medication|
|Total members, n||440,835||14,323||5259|
|Subset of members who posted about memory, n (%)||13,878 (3.15)a||884 (6.17)a||1186 (22.55)a|
aPercentage of overlap.
Hands-On Linguistic Analysis
A total of 4 key themes related to the authors’ memory were found in the data: memory loss, aphasia, cognitive impairment, and emotional change. Summaries of these themes, patient lexicon, and specific examples can be found in. Within these themes, descriptions of memory loss and confusion often overlap, and authors may label moments of cognitive impairment, confusion, or difficulties with function as memory loss. Regardless of the labels, authors were passionate about memory and cognitive impairment. Many used qualifiers of severity and speed, such as catastrophic or instant, when describing effects on their memory. Furthermore, a subset described their experience as dementia or Alzheimer’s.
Memory difficulties were also attributed to aging, pumphead (postperfusion syndrome), surgery, or other medications. Authors who did attribute statin use to memory loss believed higher doses to be riskier. Authors also believed that their cognitive changes would resolve when statin use was discontinued. Nevertheless, during use, many memory events may have been written off by patients or may have gone unreported. Compounding the problem, authors reported having difficulty discussing their memory or cognitive issues with the doctors. Authors reported that many of their health care providers were dismissive of their claims, feeling that the benefits of statin medications far outweigh the risks.
|Key memory theme||Patient lexicon||Example|
|Memory loss: Though some patients have trouble with long-term memory, these patients are most passionate about and focused on the impact on short-term memory.|
|Aphasia: Authors describe difficulty communicating their thoughts. In particular, authors have difficulty remembering names of people that they recognize and know well.|
|Cognitive impairment: Authors describe a loss of ability to think, reason, or understand. Authors are also concerned with the loss of ability to function. Authors are particularly concerned about attention span and forgetting how to do basic tasks.|
|Emotional change: Authors describe their emotions as changed, struggling with feeling depressed or lack of emotion. Authors also describe becoming angry easily or being moody or disinterested. A small subset feels increased anxiety.|
This study uses online communities of self-identified patients and caregivers to evaluate the signals of self-reported memory impairment in statin users. A cohort of statin users was compared with a cohort of patients using TNF inhibitors and also to the patients on the site overall. Though discussions of memory impairment occurred 3.1% and 6.2% of the time in the overall population and the TNF inhibitor users, respectively, overlap between the statin user conversation and memory impairment posts was 22.6%, indicating a much higher and significantly different correlation between those on statin medications and the discussion of memory issues.
Furthermore, linguistic analysis was performed on a set of posts from statin users that discussed memory impairment to identify key themes. These patients and caregivers identified difficulty with memory loss, aphasia, cognitive function, and emotional change. Patients and caregivers indicated the speed and severity of these changes, likening their experience to instant Alzheimer’s.
Although there are clinical trials that have not found associations of cognitive impairment with statin use, those trials were specifically conducted to evaluate cardiovascular outcomes, not cognitive outcomes [- ]. During the postmarketing phase, a double-blind study comparing statins with placebo found small but significant differences in neuropsychological tests on attention and psychomotor speed [ ]. Using the Naranjo adverse drug reaction (ADR) probability scale in a survey of 171 patients on statins, 75% of the participants were found to have experienced cognitive ADRs [ ]. Another analysis involved examination of the FDA’s Adverse Event Reporting System. It revealed a significantly higher proportion of adverse reports for statins as compared with control medications [ ].
This study’s findings support the importance of using novel and nontraditional data sources as additional population signals for side effects. The nature of memory issues and cognitive impairment can lead to underreporting of these issues to doctors. Furthermore, in the online posts, patients report doctors discounting, ignoring, or not taking their concerns over memory issues seriously. Improved patient care and outcomes are seriously impaired when their concerns are discounted. Underreporting of this side effect to the FDA is inevitable when patient complaints and concerns are not addressed. This underscores the imperative of bringing the patient experience to light in nontraditional ways.
Drug development clinical trials should balance eligibility criteria, which allow for a defined population to be studied, against the limitations these criteria create. Specifically, eligibility criteria exclude and narrow, reducing the representativeness of the data. This subset of population may not represent the broader patient population that may use a product once it is approved. Although there have been recent policy initiatives that focus on the inclusion of historically underrepresented groups, barriers still remain that may limit or mask the potential benefits and side effects, respectively. Data provided from online communities may allow for an expansion of cohorts for study, potentially accessing patients who may not have traditionally been enrolled in other forms of research. Analyzing data from these patients and using mixed quantitative and qualitative methodologies allows corroboration of findings and provides a new avenue of side effect identification and investigation.
This study had some limitations. First, clinical diagnosis, medical history, and current treatment of individual authors are self-reported, cannot be confirmed, and may be missing some information. Second, demographic data of the authors are unknown, making it unclear how the data reflect the general population. In addition, there may be an element of detection bias that may be relevant, given the intrinsic virality of online communication. Moreover, qualitative analysis may also reflect a limitation of outcome misclassification predicated upon word selection on the part of participants. Search terms were selected based on free-text review to limit the impact of alternative term selection. Despite this, missed cases may be a problem if community participants chose not to discuss or share memory challenges owing to perceived associated stigma.
One strength of qualitative research, detailed information about the human experience, makes it a compelling tool when applied to health . The large datasets of online communication on social media sites, such as Inspire, can challenge the skills of individual researchers [ ]. Maintenance of rigor in analysis may be impacted. Using NLP can effectively deal with this limitation.
However, in a methodological research study comparing NLP-only analysis, qualitative text–only analysis, and combined NLP–qualitative text analysis, researchers concluded that NLP-only analysis was an effective tool to identify and quantify major themes but lacked the ability to capture contextual nuances necessary for clarity and understanding. Combining the 2 methodologies provided the most comprehensive and highest quality results .
Comparison With Prior Work
Stanford University Medical School in collaboration with Inspire conducted research on over 8 million posts on Inspire utilizing NLP to look for associations between chemotherapeutic agents and ADRs. The research specifically extracted mentions of common and rare cutaneous ADRs (eg, rashes, blisters, and psoriasis flares) from posts related to (1) the epidermal growth factor receptor inhibitor, erlotinib, and (2) the immune checkpoint programmed cell death–1 inhibitors, nivolumab and pembrolizumab. The team discovered that some patients receiving the chemotherapy drug erlotinib (Tarceva) reported hypohidrosis—the inability to sweat—a condition that can lead to heat exhaustion, heat stroke, or even death. This ADR had never been reported in the medical literature, but Inspire members had been discussing it for over 11 years. The team also found that Inspire members discussed, among themselves, ADRs for other checkpoint inhibitors much earlier than reported in the medical literature—an average of 7 months before any of these side effects had been reported . This research demonstrated the untapped resources and information to be uncovered in online health community postings.
Estimates of cessation of statin therapy within the first year are as high as 50%. This high degree of discontinuation is disconcerting as statins have been shown to reduce cardiovascular disease risk, accruing with each year of use and benefits persisting over the long term . Side effects play a role in medication cessation. Although the benefits of statin medications may outweigh the risks, it is still worth identifying adverse reactions to better and more fully inform patient and doctor decisions.
Using the deidentified communication occurring online can add to the knowledge base about medication usage and adverse reactions. Large online populations, comprising large cohorts of patients who may not traditionally participate in research, can serve as additional population signals for side effects.
As with the prior work with erlotinib, this type of research can be used to inform the FDA of the side effects or adverse reactions that are presently unreported for a number of reasons. Using the combination of NLP analysis with qualitative analysis broadens and deepens learnings, tests hypotheses, and uncovers insights into the patient experience. Regarding statin use, patients may not connect their medications to cognitive changes, may not feel comfortable informing their physicians, or may be unable to articulate these changes in the clinical setting. If patients do tell the doctors, doctors may not relay this information to the FDA or feel that benefits outweigh the risks.
Further research is needed to truly identify the extent of cognitive change that is occurring with the use of statins.
Conflicts of Interest
- Goff DC, Lloyd-Jones DM, Bennett G, Coady S, D'Agostino RB, Gibbons R, American College of Cardiology/American Heart Association Task Force on Practice Guidelines. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 2014 Jun 24;129(25 Suppl 2):S49-S73. [CrossRef] [Medline]
- Ioannidis J. More than a billion people taking statins?: Potential implications of the new cardiovascular guidelines. J Am Med Assoc 2014 Feb 5;311(5):463-464. [CrossRef] [Medline]
- Pasternak RC, Smith SC, Bairey-Merz CN, Grundy SM, Cleeman JI, Lenfant C, American College of Cardiology, American Heart Association, National Heart‚ LungBlood Institute. ACC/AHA/NHLBI clinical advisory on the use and safety of statins. J Am Coll Cardiol 2002 Aug 7;40(3):567-572 [FREE Full text] [CrossRef] [Medline]
- Thompson PD, Panza G, Zaleski A, Taylor B. Statin-associated side effects. J Am Coll Cardiol 2016 May 24;67(20):2395-2410 [FREE Full text] [CrossRef] [Medline]
- Bruckert E, Hayem G, Dejager S, Yau C, Bégaud B. Mild to moderate muscular symptoms with high-dosage statin therapy in hyperlipidemic patients--the PRIMO study. Cardiovasc Drugs Ther 2005 Dec;19(6):403-414. [CrossRef] [Medline]
- Ridker PM, Danielson E, Fonseca FA, Genest J, Gotto AM, Kastelein JJ, JUPITER Study Group. Rosuvastatin to prevent vascular events in men and women with elevated C-reactive protein. N Engl J Med 2008 Nov 20;359(21):2195-2207. [CrossRef] [Medline]
- Freeman DJ, Norrie J, Sattar N, Neely RD, Cobbe SM, Ford I, et al. Pravastatin and the development of diabetes mellitus: evidence for a protective treatment effect in the West of Scotland Coronary Prevention Study. Circulation 2001 Jan 23;103(3):357-362. [CrossRef] [Medline]
- Wagstaff LR, Mitton MW, Arvik BM, Doraiswamy PM. Statin-associated memory loss: analysis of 60 case reports and review of the literature. Pharmacotherapy 2003 Jul;23(7):871-880. [CrossRef] [Medline]
- Yaffe K, Barrett-Connor E, Lin F, Grady D. Serum lipoprotein levels, statin use, and cognitive function in older women. Arch Neurol 2002 Mar;59(3):378-384. [CrossRef] [Medline]
- Golomb BA, Dimsdale JE, Koslik HJ, Evans MA, Lu X, Rossi S, et al. Statin effects on aggression: results from the UCSD statin study, a randomized control trial. PLoS One 2015;10(7):e0124451 [FREE Full text] [CrossRef] [Medline]
- Muldoon MF, Barger SD, Ryan CM, Flory JD, Lehoczky JP, Matthews KA, et al. Effects of lovastatin on cognitive function and psychological well-being. Am J Med 2000 May;108(7):538-546. [CrossRef] [Medline]
- Muldoon MF, Ryan CM, Sereika SM, Flory JD, Manuck SB. Randomized trial of the effects of simvastatin on cognitive functioning in hypercholesterolemic adults. Am J Med 2004 Dec 1;117(11):823-829. [CrossRef] [Medline]
- McGuinness B, O'Hare J, Craig D, Bullock R, Malouf E, Passmore P. Statins for the treatment of dementia. Cochrane Database Syst Rev 2010 Aug 04(8):CD007514. [CrossRef] [Medline]
- Ott BR, Daiello LA, Dahabreh IJ, Springate BA, Bixby K, Murali M, et al. Do statins impair cognition? A systematic review and meta-analysis of randomized controlled trials. J Gen Intern Med 2015 Mar;30(3):348-358 [FREE Full text] [CrossRef] [Medline]
- The Food and Drug Administration. 2016. FDA Drug Safety Communication: Important Safety Label Changes to Cholesterol-lowering Statin Drugs URL: https://www.fda.gov/Drugs/DrugSafety/ucm293101.htm [accessed 2019-07-01]
- Boyatzis R. Transforming Qualitative Information: Thematic Analysis and Code Development. Thousand Oaks: Sage; 1998.
- Mora S, Ridker PM. Justification for the use of Statins in primary prevention: an Intervention Trial Evaluating Rosuvastatin (JUPITER)--can C-reactive protein be used to target statin therapy in primary prevention? Am J Cardiol 2006 Jan 16;97(2A):33A-41A. [CrossRef] [Medline]
- Heart Protection Study Collaborative Group. MRC/BHF Heart Protection Study of cholesterol lowering with simvastatin in 20,536 high-risk individuals: a randomised placebo-controlled trial. Lancet 2002 Jul 6;360(9326):7-22. [CrossRef] [Medline]
- Shepherd J, Blauw GJ, Murphy MB, Bollen EL, Buckley BM, Cobbe SM, PROSPER study group. PROspective Study of Pravastatin in the Elderly at Risk. Pravastatin in elderly individuals at risk of vascular disease (PROSPER): a randomised controlled trial. Lancet 2002 Nov 23;360(9346):1623-1630. [CrossRef] [Medline]
- Evans MA, Golomb BA. Statin-associated adverse cognitive effects: survey results from 171 patients. Pharmacotherapy 2009 Jul;29(7):800-811. [CrossRef] [Medline]
- Sahebzamani FM, Munro CL, Marroquin OC, Diamond DM, Keller E, Kip KE. Examination of the FDA warning for statins and cognitive dysfunction. J Pharmacovigil 2014;2:4. [CrossRef]
- Anderson C. Presenting and evaluating qualitative research. Am J Pharm Educ 2010 Oct 11;74(8):141 [FREE Full text] [CrossRef] [Medline]
- Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res 2007 Aug;42(4):1758-1772 [FREE Full text] [CrossRef] [Medline]
- Guetterman TC, Chang T, DeJonckheere M, Basu T, Scruggs E, Vydiswaran VV. Augmenting qualitative text analysis with natural language processing: methodological study. J Med Internet Res 2018 Jun 29;20(6):e231 [FREE Full text] [CrossRef] [Medline]
- Ransohoff JD, Nikfarjam A, Jones E, Loew B, Kwong BY, Sarin KY, et al. Detecting chemotherapeutic skin adverse reactions in social health networks using deep learning. JAMA Oncol 2018 Apr 1;4(4):581-583 [FREE Full text] [CrossRef] [Medline]
- Collins R, Reith C, Emberson J, Armitage J, Baigent C, Blackwell L, et al. Interpretation of the evidence for the efficacy and safety of statin therapy. Lancet 2016 Nov 19;388(10059):2532-2561. [CrossRef] [Medline]
|ADR: adverse drug reaction|
|FDA: Food and Drug Administration|
|NLP: natural language processing|
|TNF: Tumor Necrosis Factor|
|UGC: user-generated content|
Edited by G Eysenbach; submitted 24.05.19; peer-reviewed by R Booth, J Brixey; comments to author 18.07.19; revised version received 30.08.19; accepted 24.09.19; published 28.11.19
©Farris Timimi, Sara Ray, Erik Jones, Lee Aase, Kathleen Hoffman. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 28.11.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.