Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, December 24 through Wednesday, December 26 inclusive. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 20.11.18 in Vol 20, No 11 (2018): November

Preprints (earlier versions) of this paper are available at, first published Mar 23, 2018.

This paper is in the following e-collection/theme issue:

    Original Paper

    Web-Based Signal Detection Using Medical Forums Data in France: Comparative Analysis

    1Epidemiology and Benefit Risk Evaluation, Sanofi, Chilly-Mazarin, France

    2Kappa Santé, Paris, France

    3Kap Code, Paris, France

    4Information Technology and Solutions, Sanofi, Lyon, France

    5Global Pharmacovigilance, Sanofi, Bridgewater, NJ, United States

    6Epidemiology and Benefit Risk Evaluation, Sanofi, Bridgewater, NJ, United States

    Corresponding Author:

    Marie-Laure Kürzinger, MSc

    Epidemiology and Benefit Risk Evaluation


    1 avenue Pierre Brossolette

    Chilly-Mazarin, 91385


    Phone: 33 1 69 74 59 42



    Background: While traditional signal detection methods in pharmacovigilance are based on spontaneous reports, the use of social media is emerging. The potential strength of Web-based data relies on their volume and real-time availability, allowing early detection of signals of disproportionate reporting (SDRs).

    Objective: This study aimed (1) to assess the consistency of SDRs detected from patients’ medical forums in France compared with those detected from the traditional reporting systems and (2) to assess the ability of SDRs in identifying earlier than the traditional reporting systems.

    Methods: Messages posted on patients’ forums between 2005 and 2015 were used. We retained 8 disproportionality definitions. Comparison of SDRs from the forums with SDRs detected in VigiBase was done by describing the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy, receiver operating characteristics curve, and the area under the curve (AUC). The time difference in months between the detection dates of SDRs from the forums and VigiBase was provided.

    Results: The comparison analysis showed that the sensitivity ranged from 29% to 50.6%, the specificity from 86.1% to 95.5%, the PPV from 51.2% to 75.4%, the NPV from 68.5% to 91.6%, and the accuracy from 68% to 87.7%. The AUC reached 0.85 when using the metric empirical Bayes geometric mean. Up to 38% (12/32) of the SDRs were detected earlier in the forums than that in VigiBase.

    Conclusions: The specificity, PPV, and NPV were high. The overall performance was good, showing that data from medical forums may be a valuable source for signal detection. In total, up to 38% (12/32) of the SDRs could have been detected earlier, thus, ensuring the increased safety of patients. Further enhancements are needed to investigate the reliability and validation of patients’ medical forums worldwide, the extension of this analysis to all possible drugs or at least to a wider selection of drugs, as well as to further assess performance against established signals.

    J Med Internet Res 2018;20(11):e10466




    Adverse drug reactions (ADRs) are an important public health concern. Drug safety currently depends on postmarketing surveillance, which is conducted through spontaneous reporting systems based on voluntary reports. The reporting rate is low, and the time for getting access to reported data can be long. Thus, it is difficult to detect signals of disproportionate reporting (SDRs) in a timely manner, and some may not even be captured by those reporting systems. Alternative sources of data have already been used to detect drug-adverse event (AE) associations, including claims data [1], electronic medical records (EMRs) [2-4], and consumer search logs [5,6]. In addition, approaches have been developed to combine SDRs from data sources such as EMRs, claims, internet search logs with SDRs from the Food and Drug Administration Adverse Event Reporting System (FAERS) [5,7,8].

    With the development and popularity of social media and medical forums, many internet users are exchanging health-related information, which may involve ADRs, and the question of consistency and usefulness of these new data sources for Web-based signal detection is under scrutiny. Recent projects have aimed at investigating the quality of social media data, as well as investigating the most performant method for the Web-based signal detection. The use of Web-based data (such as query logs and social media) is emerging among regulators (Food and Drug Administration and European Medicines Agency), industry, and academia [5,6,8-10]. As an example, a public-private partnership between the European Commission and European Federation of Pharmaceutical Industries and Associations, called WEB-RADR: Recognising Adverse Drug Reactions has been launched in 2014; this consortium made of organizations, including European medicines regulators, academics, and the pharmaceutical industry, aims to develop new ways of gathering information on suspected ADRs. One of the objectives is to investigate the potential for publicly available social media data for identifying potential drug safety issues. A recently published study [11] has focused on the performance evaluation of established statistical signal detection algorithms in Twitter or Facebook for a broad range of drugs and adverse events. Another example is related to the recent collaboration between Sanofi and Microsoft [8]. In this study, a Web-based search query method, called a query log reaction score, was developed to detect whether AEs associated with certain drugs could be found from search engine query data. The results were compared with reference signal detection algorithms commonly used with the FAERS.

    The potential strength of Web-based data relies on their volume and real-time availability, allowing early SDR detection. This study aims to assess the consistency of SDRs detected from patients’ forums in France and the ability to identify SDRs earlier than the traditional reporting systems. Three products were selected for this study (insulin glargine, teriflunomide, and zolpidem). SDRs detected from this Web-based source were compared with those detected from World Health Organization (WHO) AEs reporting system (VigiBase) using traditional SDR detection methods.


    Data Sources

    The sources of data used were (1) patients’ medical forums in France and (2) VigiBase, the WHO individual case safety reports (ICSR) database; these sources of data are described later.

    Medical Forum’s Messages Database: Detec’t

    This was a retrospective study based on the secondary use of data from the Web—patients’ medical forums in France. The Detec’t database is a private database aggregating messages from social media, including safety information. We included 12 well-known medical forums in this study (see Multimedia Appendix 1). Every discussion publicly available at the time of the search and containing at least one message with the name of the drugs of interest (active substance or brand name) was extracted using a Web crawler (Detec’t Extractor). This messages scrapping was done by targeting patients' messages using the HTML structure of each forum. This Web crawler was then adapted for each data source. Messages were extracted with all metadata related to the message (date, author of the post ID, URL of the discussion, and name of the forum). Finally, data were cleaned (deletion of ads, quotation of other Web users, and signature) using the HTML structure of the posts.

    Retrieved messages went through several steps of processing so that the drug-event pairs necessary to perform the analysis were obtained. First, messages were deidentified. The data processing then automatically ensured that the messages contained the name of the drugs of interest and the concept of drug use or intake to constitute the corpora of messages related to the drugs under study. These corpora were formatted and screened to detect all references to medical events referring to potential AE (ie, not the indication of the drug, not an AE preceding the intake of the drug, not a question about adverse effects without having experienced it, etc).

    The deidentification of messages was performed using an in-house algorithm based on the regular expression to automatically identify specific sequences of characters (like proper names, phone numbers, postal codes, mail addresses, etc). Messages containing the names of the drugs were identified by automatically detecting, among the set of extracted messages, references to active substances, and brand names of the drugs of interest; this step included the detection of common spelling mistakes. Next, the drug intake notion was identified with a specific algorithm (on this set of messages) based on the detection of regular expressions (identifying first-person personal pronouns, for instance). This procedure ensured that the person experiencing the drug could be identified (this person could be the author of the post or one of their relative). These “intake messages” represented the set of messages on which the following steps were applied.

    As described earlier [12], the formatting of messages included the conversion of messages to lowercases, the removal of extra whitespaces, and the tokenization of messages. All words from messages were stemmed using Porter’s algorithm to associate inflected and derived words together with their root form.

    Figure 1. Extraction methodology and data preparation for analysis. AE: adverse events; AERS: Adverse Events Reporting System.
    View this figure

    Medical concepts were detected using an extended version of Medical Dictionary for Regulatory Activities (MedDRA) version 15.0 adjusted and supplemented by vernacular vocabulary. In MedDRA, Preferred Terms (PT) correspond to unique descriptors of medical concepts. Lower-Level Terms (LLT) correspond to variants of PT and the lowest level of the terminology (each LLT is associated with a single parent PT and several LLTs can be associated with one PT). Medical concepts were detected at the LLT level and encoded as PT. The supplementation of vocabulary by vernacular terms was performed by manually reviewing a sample of messages from Web forums previously automatically annotated with MedDRA. The review was performed by 4 computer science experts who were familiar with the MedDRA dictionary. All medical concepts manually identified and not detected by MedDRA were saved and manually associated with a PT as if they were new LLTs. The evaluation of medical concepts detection, after supplementation on a sample of 157 messages, conducted to a recall of 71% and an accuracy of 93%. Regarding the adjustments provided to MedDRA, some PTs out of the study scope were removed (ie, “poverty” or “married”). For example, terms including the mention NOS (Not Otherwise Specified), that is, “Allergy NOS,” were cleaned by removing the mention. Eventually, all terms were stemmed. A manual cleaning was performed to deduplicate the terms obtained after stemming. The detection of medical concepts was performed by looking for exact matches between the stemmed versions of MedDRA and messages.

    Potential AEs were identified by applying an algorithm [12] based on a Gaussian mixture model and then a support vector machine algorithm. This step identified drug-event pairs.

    Extracted messages were stored in the database with (1) metadata associated with the messages; (2) results of the annotation process—drug intake, medical concepts, and their MedDRA code; and (3) the result of the AE detection algorithm.

    The final step of the data preparation identified the number of messages for each drug-event pair (Figure 1).

    The Comparison Database: VigiBase

    The comparison database used as the gold standard was VigiBase, the WHO Global ICSR database. It consists of reports of adverse reactions received from member countries since 1968. VigiBase is updated with incoming ICSRs on a continuous basis. The VigiBase data resource is the largest and most comprehensive in the world, and it is developed and maintained by the Uppsala Monitoring Centre on behalf of the WHO. By May 2015, over 11 million reports were available in the database. The VigiBase database system includes linked databases containing medical and drug classifications—WHO Adverse Reactions Terminology or MedDRA, WHO International Classification of Diseases, and WHO Drug Dictionary; these classifications enable structured data entry, retrieval, and analysis at different levels of precision and aggregation.


    Three drugs (insulin glargine, teriflunomide, and zolpidem) were selected for the study to cover different therapeutic areas and different lengths of use since market authorization. Data corresponding to these selected drugs (active substance and brand name) were extracted from the Detec’t database. The following drugs were therefore searched on the database: insulin glargine AND glargine AND Lantus; teriflunomide AND Aubagio; zolpidem AND Stilnox. Synonyms of the drug name and spelling mistakes were considered (ie, detail on misspelling). Misspellings were added by identifying most common errors and adding them as researched forms of the drug.

    In order to carry out the analysis of disproportionate reporting, a background group of 327 drugs was randomly selected from the Detec’t database. This later included drugs that were randomly chosen from an exhaustive list of French drugs. The messages corresponding to these 327 drugs were selected from the same set of forums and time period; they went through the same analysis and encoding steps.

    Signals of Disproportionate Reporting Detection Metrics and Definition

    The disproportionality analysis of spontaneous reports (comparing the number of observed cases with that of expected cases) was used. The quantitative method in signal detection relies on the principle of disproportionality [7,13,14]. We used the Proportional Reporting Ratio (PRR), Reporting Odds Ratio (ROR), Reporting Fisher’s Exact Test (RFET), empirical Bayes geometric mean (EBGM), and the Information Component (IC). A total of 8 disproportionality definitions were considered for this study (Table 1).

    Statistical Analysis

    Descriptive Analysis

    A description of messages for the overall period (cumulative from 2005 to 2015) and across time within the study period was provided—numbers of messages with the drug name, numbers of messages containing the concept of drug use or intake, medical concepts, and potential AEs.

    Comparative Analysis

    The comparison of SDRs detected from patients’ forums in France to SDRs detected in VigiBase were described using sensitivity (true positive rate)=a/N1, specificity (true negative rate)=d/N2, positive predictive value (PPV)=a/M1, negative predictive value (NPV)=d/M2, and accuracy (a+d)/N (Tables 2 and 3). The receiver operating characteristics (ROC) curve and the area under the curve (AUC) were considered to measure the overall performance of the test to discriminate between positive and negative SDRs. The ROC curve represented the true positive rate (sensitivity) plotted in function of the false positive rate (100-specificity) for different thresholds of the metric.

    Table 1. Definition of disproportionate signals.
    View this table
    Table 2. Signals: two-by-two contingency table for a combination of positive and negative signals from medical forums and VigiBase to measure performance.
    View this table
    Table 3. Performance indicators.
    View this table
    Time Analysis

    For SDRs identified in both data sources, we performed an analysis of time difference in months between the date of detection of SDRs from French patients’ forums and the date of detection of SDRs from VigiBase.


    Descriptive Analysis

    The data from 8 medical forums were considered for analysis and corresponded to messages published between January 1, 2005 and December 31, 2015, for insulin glargine and zolpidem and between April 1, 2014 and December 31 2015, for teriflunomide. For teriflunomide, this time restriction was because the product received its first marketing authorization application in September 2013 and was first launched in March 2014. Figure 2 shows the messages flowchart. The extraction was conducted for the identification of 102 messages for teriflunomide, 3326 messages for insulin glargine, and 4584 messages for zolpidem, which were relative to the drug intake. Among those, 61 messages for teriflunomide, 2335 messages for insulin glargine, and 3732 messages for zolpidem contained medical concepts. Among those, 41 messages for teriflunomide, 1799 messages for insulin glargine, and 2998 messages for zolpidem contained potential AEs. This resulted in 33 unique drug-event pairs for teriflunomide, 194 for insulin glargine, and 318 for zolpidem. The number of SDRs detected varied across the definition of disproportionate SDRs. For teriflunomide, insulin glargine and zolpidem, the number of SDRs varied from 4 to 12, 21 to 48, and 23 to 95, respectively. These SDRs were compared with the SDRs detected in VigiBase.

    Comparative Analysis

    Between 2005 and 2015, the medical forums data contained 545 drug-event pairs. Overall, 7618 pairs were identified in VigiBase, of which 422 drug-event pairs combinations overlapped with the forums data (Figure 3). The overlap considered an exact match with the event terminology. When restricting to pairs with at least 2 messages, the overlap was 275 drug-event pairs.

    Among 545 drug-event pairs from the forums, only 123 were not identified in VigiBase; those 123 drug-event pairs corresponded each to only one message in the forums. The individual inspection showed that some PTs were not adequately specific. For example, some PTs identified in the forums were “weight,” but it was not possible to match them with the PT, such as “underweight” or “overweight” or “abnormal weight gain” or “weight abnormal,” which were identified in the drug-events pairs from VigiBase.

    Among the overlap of 422 drug-event pairs, the specificity was high (87.5%-95.5%) depending on the SDR definition (Table 4). On the other hand, the sensitivity was low (29%-50.6%), indicating that an important proportion of SDRs from VigiBase was not identified in the forums. The PPV (51.2%-75.4%), NPV (68.5%-91.6%), and accuracy (68%-87.7%) were high.

    Among 275 drug-event pairs (for which at least 2 reports or messages were considered; Table 5), the figures were slightly higher, improving the overall performance. The specificity varied between 86.9% and 93.4% and the sensitivity between 39.1% and 56.5%. The PPV (53.9%-76.6%), NPV (68.7%-91.5%), and accuracy (70.6%-86.2%) were high.

    Whatever definition of disproportionate SDR used, the ROC curves and the AUC showed an overall good performance. AUC varied around 0.8. The highest AUC was shown with the EBGM metric (AUC=0.85; Figure 4).

    Time Analysis

    For SDRs detected both in VigiBase and patients’ forums, we calculated the time difference in months between the date of detection of positive SDRs from the French patients’ forums and the date of detection of these SDRs in VigiBase (Table 6).

    Figure 2. Flowchart for the 3 drugs and the other 327 drugs (comparison group). AE: adverse events.
    View this figure
    Figure 3. Time periods covered by VigiBase and the forums database and the number of drug-event pairs overlap, as well as pairs overlap with at least 2 messages (smallest circle).
    View this figure
    Table 4. The sensitivity, specificity, positive predictive value, negative predictive value, and accuracy among 422 drug-event pairs.
    View this table
    Table 5. The sensitivity, specificity, positive predictive value, negative predictive value, and accuracy among 275 drug-event pairs.
    View this table
    Figure 4. The receiver operating characteristics (ROC) curves and area under the curve applying empirical Bayes geometric mean (EBGM)≥4 in the VigiBase and EBGM in the forums.
    View this figure
    Table 6. The time difference in months of signals detection dates (∆time) between patients’ forums and VigiBase.
    View this table

    Depending on the definition of SDRs, up to 38% (12/32) of common SDRs were detected earlier (up to 128 months earlier) in the forums than in VigiBase. In addition, up to 13% (4/32) were detected at the same date but were available earlier in the forums given the real-time availability of data on the Web. The qualitative exploration of SDRs detected earlier in the forums showed heterogeneity as some were related to serious medical events and other to patients-related symptoms (ie, stress and hunger).

    Most signals that were detected earlier in VigiBase were linked to serious medical events, which probably led to medical consultation and, thus, to an AE reporting done through a health care professional. In addition, most of those events were related to the System Organ Class “nervous system disorders” and “psychiatric disorders.”


    Principal Findings

    This study aimed at assessing the consistency of SDRs detected from patients’ forums in France over the last 11 years and the ability to identify SDRs earlier than that in VigiBase.

    The potential strength of Web-based data relies on their volume and real-time availability, allowing early signal detection. This pilot study showed a good performance and earlier detection of SDRs in the French medical forums compared with SDRs detected in traditional sources. In addition, these pilot results indicate that using patients’ medical forums may be considered as a complementary source of data to traditional sources, allowing SDRs to be detected earlier and, thus, facilitating the increased safety of patients.

    We first compared SDRs (by considering several definitions of disproportionate SDRs) detected in the forums data and the WHO AEs reporting system (VigiBase). The comparison of positive and negative SDRs showed that whatever the definition of disproportionate SDR, the sensitivity was low and the specificity was very high. In addition, the PPV and NPV were high. The overall performance was good, showing that data from the medical forum may be a valuable source to be considered for signal detection. In another study [8] using query log data, results showed that the method had moderate sensitivity and low specificity in detecting signals in Web query data compared with reference signal detection algorithms in FAERS. In another study [11] using Twitter and Facebook, the authors suggested that broad-ranging statistical signal detection in Twitter and Facebook, using currently available methods for adverse event recognition, performs poorly and cannot be recommended at the expense of other pharmacovigilance activities; this indicates that results in terms of performance might vary according to the Web data source used and the metric used for SDRs.

    Second, among SDRs from patients’ forums and VigiBase, we calculated time differences in detection of SDRs to measure the ability of forums data to detect earlier SDRs compared with VigiBase. Up to 38% (12/32) of common SDRs could be detected earlier when using the forums data, which is an important finding. The qualitative exploratory analysis of the SDRs detected earlier showed that events were related to serious as well as patient-related symptoms. This finding is consistent with recent studies [15] that addressed the question of earlier detection of drug-related AEs in the social media compared with FAERS. The findings highlighted some of the promises of social media data sources for detecting early AE reports patterns compared with conventional pharmacovigilance sources and showed that social media AE reports helped predict the occurrence of FAERS reports several months later for one of the two drugs that were studied. In a study [16], the objective was to examine whether specific product-AE pairs were reported through social media before being reported to FAERS. In one of the positive cases, the first report occurred in social media prior to the SDR detection from FAERS. Authors concluded that an efficient semiautomated approach to social media monitoring might provide earlier insights into certain AEs.

    Strengths and Limitations

    One of the strengths of this study was the quality of preprocessing and processing of the data extracted from the forums. Messages that were used for Web-signal detection in this study were not only containing the drug but also a medical event (cooccurrence) as this is done in other studies [17]. Several cleansing and validation steps were performed to ensure that the identified messages were related to an internet user who used the concept of use or intake of the drug and with potential AE.

    This study has several limitations. First, the results only apply to 3 drugs and for the French medical forums. Thus, results are not generalizable to all drugs and at a worldwide scale. However, we do not have a strong hypothesis to believe that the use of Web-based medical forums and interactions of French internet users would be different in other developed countries. Thus, further studies focusing on worldwide patients’ forums should be envisaged. Second, automatics algorithms have their limitations. The management of Web-based data needs continuous updates on modeling and data processing to ensure high quality and accuracy of the information retrieved. Although the data were processed, it is still possible that some drug names or medical concepts could be missed, that some AEs may be confused with drugs indications, or questions from Web users about AEs they have not experienced, or descriptions of symptoms that are not adverse reactions to drugs. Third, Web-based data rely on patients’ perspective and declaration but not on a true medical diagnosis. Web-based data are sensitive to increase in the media coverage, resulting in increased searches or posts and are prone to changes in people’s search or communication behavior. Finally, VigiBase is not a true gold standard, as it has its own limitations (such as lack of denominator and underreporting). VigiBase, however, has been used as a standard for signal detection by regulators and pharmaceutical companies, and our study showed that patients’ forum could be used as a complementary data source to detect SDR earlier. Although the choice of the reference data remains challenging [18], further studies using a refined gold standard, such as drug-event pairs shown in the labels, should be considered.


    This study shows a good performance and earlier detection of SDRs detected in patients’ medical forums compared with SDRs detected in traditional sources. Those SDRs relate to serious medical events as well as subjective patients-related symptoms (eg, stress and hunger). These results indicate that using patients’ medical forums may be considered as a complementary source of data to traditional sources, allowing SDRs to be detected earlier and, thus, ensuring the increased safety of patients. Further enhancements are needed to investigate the reliability and validation of patients’ medical forums worldwide, the extension of this analysis to all possible drugs or at least to a wider selection of drugs, as well as to further assess performance against established signals.


    This research was fully funded by Sanofi.

    Conflicts of Interest

    MLK, JP, STL, SL, and JJ are all employees of Sanofi, and this work was conducted as part of their employment. LZ also contributed to this paper as a former employee of Sanofi. SS and NT are employed by Kappa Santé and provided the data through funding from Sanofi. CF is employed by Kap Code, a Kappa Santé start-up that owns Detec’t. RA also contributed to this paper as a former employee of Kap Code.

    Multimedia Appendix 1

    List of medical forums included in the study.

    PDF File (Adobe PDF File), 21KB


    1. Schneeweiss S. A basic study design for expedited safety signal evaluation based on electronic healthcare data. Pharmacoepidemiol Drug Saf 2010 Aug;19(8):858-868 [FREE Full text] [CrossRef] [Medline]
    2. Coloma PM, Trifirò G, Schuemie MJ, Gini R, Herings R, Hippisley-Cox J, EU-ADR Consortium. Electronic healthcare databases for active drug safety surveillance: is there enough leverage? Pharmacoepidemiol Drug Saf 2012 Jun;21(6):611-621. [CrossRef] [Medline]
    3. LePendu P, Iyer SV, Bauer-Mehren A, Harpaz R, Mortensen JM, Podchiyska T, et al. Pharmacovigilance using clinical notes. Clin Pharmacol Ther 2013 Jun;93(6):547-555 [FREE Full text] [CrossRef] [Medline]
    4. Iyer SV, Harpaz R, LePendu P, Bauer-Mehren A, Shah NH. Mining clinical text for signals of adverse drug-drug interactions. J Am Med Inform Assoc 2014;21(2):353-362 [FREE Full text] [CrossRef] [Medline]
    5. Yom-Tov E, Gabrilovich E. Postmarket drug surveillance without trial costs: discovery of adverse drug reactions through large-scale analysis of web search queries. J Med Internet Res 2013;15(6):e124 [FREE Full text] [CrossRef] [Medline]
    6. White RW, Harpaz R, Shah NH, DuMouchel W, Horvitz E. Toward enhanced pharmacovigilance using patient-generated data on the internet. Clin Pharmacol Ther 2014 Aug;96(2):239-246 [FREE Full text] [CrossRef] [Medline]
    7. Harpaz R, DuMouchel W, LePendu P, Bauer-Mehren A, Ryan P, Shah NH. Performance of pharmacovigilance signal-detection algorithms for the FDA adverse event reporting system. Clin Pharmacol Ther 2013 Jun;93(6):539-546 [FREE Full text] [CrossRef] [Medline]
    8. Colilla S, Tov EY, Zhang L, Kurzinger M, Tcherny-Lessenot S, Penfornis C, et al. Validation of New Signal Detection Methods for Web Query Log Data Compared to Signal Detection Algorithms Used With FAERS. Drug Saf 2017 May;40(5):399-408. [CrossRef] [Medline]
    9. Harpaz R, Callahan A, Tamang S, Low Y, Odgers D, Finlayson S, et al. Text mining for adverse drug events: the promise, challenges, and state of the art. Drug Saf 2014 Oct;37(10):777-790 [FREE Full text] [CrossRef] [Medline]
    10. Sarker A, Ginn R, Nikfarjam A, O'Connor K, Smith K, Jayaraman S, et al. Utilizing social media data for pharmacovigilance: A review. J Biomed Inform 2015 Apr;54:202-212 [FREE Full text] [CrossRef] [Medline]
    11. Caster O, Dietrich J, Kürzinger ML, Lerch M, Maskell S, Norén GN, et al. Assessment of the Utility of Social Media for Broad-Ranging Statistical Signal Detection in Pharmacovigilance: Results from the WEB-RADR Project. Drug Saf 2018(24) [FREE Full text] [CrossRef] [Medline]
    12. Abdellaoui R, Schück S, Texier N, Burgun A. Filtering Entities to Optimize Identification of Adverse Drug Reaction From Social Media: How Can the Number of Words Between Entities in the Messages Help? JMIR Public Health Surveill 2017 Jun 22;3(2):e36 [FREE Full text] [CrossRef] [Medline]
    13. Hauben M, Zhou X. Quantitative methods in pharmacovigilance: focus on signal detection. Drug Saf 2003;26(3):159-186. [Medline]
    14. Berlin C, Blanch C, Lewis DJ, Maladorno DD, Michel C, Petrin M, et al. Are all quantitative postmarketing signal detection methods equal? Performance characteristics of logistic regression and Multi-item Gamma Poisson Shrinker. Pharmacoepidemiol Drug Saf 2012 Jun;21(6):622-630. [CrossRef] [Medline]
    15. Duh MS, Cremieux P, Audenrode MV, Vekeman F, Karner P, Zhang H, et al. Can social media data lead to earlier detection of drug-related adverse events? Pharmacoepidemiol Drug Saf 2016 Dec;25(12):1425-1433 [FREE Full text] [CrossRef] [Medline]
    16. Pierce CE, Bouri K, Pamer C, Proestel S, Rodriguez HW, Van Le H, et al. Evaluation of Facebook and Twitter Monitoring to Detect Safety Signals for Medical Products: An Analysis of Recent FDA Safety Alerts. Drug Saf 2017 Dec;40(4):317-331 [FREE Full text] [CrossRef] [Medline]
    17. Yang CC, Yang H, Jiang L. Postmarketing Drug Safety Surveillance Using Publicly Available Health-Consumer-Contributed Content in Social Media. ACM Trans. Manage. Inf. Syst 2014 Apr 01;5(1):1-21. [CrossRef]
    18. Hauben M, Aronson JK, Ferner RE. Evidence of Misclassification of Drug-Event Associations Classified as Gold Standard 'Negative Controls' by the Observational Medical Outcomes Partnership (OMOP). Drug Saf 2016 May;39(5):421-432. [CrossRef] [Medline]


    AE: adverse event
    AERS: Adverse Events Reporting System
    AUC: area under the curve
    EBGM: empirical Bayes geometric mean
    FAERS: Food and Drug Administration Adverse Event Reporting System
    IC: information component
    ICSR: individual case safety reports
    LLT: lower-level terms
    NOS: Not Otherwise Specified
    NPV: negative predictive value
    PPV: positive predictive value
    PRR: proportional reporting ratio
    PT: preferred terms
    RFET: Reporting Fisher’s Exact Test
    ROC: receiver operating characteristics
    ROR: reporting odds ratio
    SDR: signals of disproportionate reporting
    WHO: World Health Organization
    MedDRA: Medical Dictionary for Regulatory Activities

    Edited by G Eysenbach; submitted 23.03.18; peer-reviewed by C Freifeld, M Gilbert; comments to author 25.04.18; revised version received 29.06.18; accepted 29.06.18; published 20.11.18

    ©Marie-Laure Kürzinger, Stéphane Schück, Nathalie Texier, Redhouane Abdellaoui, Carole Faviez, Julie Pouget, Ling Zhang, Stéphanie Tcherny-Lessenot, Stephen Lin, Juhaeri Juhaeri. Originally published in the Journal of Medical Internet Research (, 20.11.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.