To evaluate the relationship between Twitter sentiment and marketplace enrollment, we retrospectively collected ACA-related tweets by state and then tested a correlation of Twitter sentiment with marketplace enrollment by state.

We collected ACA-related tweets from March 1-31, 2014, using the Twitter Search Application Programming Interface [19]. Specifically, we selected all tweets containing the terms “ACA”, “#ACA”, “Obamacare”, and “#Obamacare” (a total of 977,303 tweets) as well as those that were directed toward the Twitter account handles for HealthCare.gov (eg, @HealthCaregov) and the 17 state-based marketplace Twitter accounts (an additional 34,605 tweets; see Multimedia Appendix 1). Additionally, we collected a random sample of 977,303 tweets from March 2014 to use as a comparison group for the sentiment of ACA and Obamacare tweets.

Tweet content and geolocation data, when available, were extracted. Twitter provides latitude and longitude coordinates or the self-reported location of the Twitter user. For the self-reported location, we matched the location text with state names, state abbreviations, and the 60 most populated cities in the United States. To verify this automatic technique, we manually examined 400 randomly selected mappings and found 99% accuracy.

Twitter sentiment was measured using the National Research Council (NRC) sentiment lexicon [20]. The lexicon contains a list of 54,120 words along with sentiment weights ranging from positive values for positive sentiment to negative values for negative sentiment (eg, the word “excellent” has a positive sentiment weight, while “awful” is negative). The NRC lexicon was created in a data-driven fashion by analyzing tweets with positive and negative sentiment hashtags. Specifically, Mohammad et al use the point-wise mutual information metric to find an association between words and their being a part of a tweet with a positive or negative sentiment hashtag [20]. The lexicon was validated against a hand-annotated set of tweets as part of the SemEval-2012 sentiment task and was found to perform with an F1 value of 0.65. We are not aware of any lexicon achieving better accuracy over a standard set of Twitter data.

To apply the lexicon to a tweet, we computed the relative frequency for each word in the tweet (eg, the word frequency divided by total number of words in the tweet) and word sentiment scores per tweet by multiplying the sentiment weight by the relative frequency of each word. A single sentiment score for each tweet was produced by summing all word sentiment scores for the tweet. This is illustrated in the following equation where sentiment(word) is the NRC lexicon sentiment score for the word, frequency is the number of times the word occurred in the tweet, and frequency_all_words gives the number of (non unique) word instances in the tweet:

sentiment score(tweet) = Σ _wordεtweet sentiment(word) * frequency(word, tweet)/frequency_all_words(tweet)

These scores were then standardized by Z scores, and a state’s sentiment score was calculated as the average sentiment of tweets in the state.

The NRC sentiment lexicon has been used to produce state-of-the-art accuracies for general domain tweets [20]. However, because our ACA corpus of tweets is biased compared to general tweets, we validate the NRC sentiment lexicon over our tweets. We randomly sampled 300 tweets from our corpus and had 2 raters score the sentiment from -3 (extremely negative) to 0 (neutral) to 3 (extremely positive); interrater reliability was strong (intraclass correlation=.72). We found the NRC sentiment lexicon was significantly correlated with the mean of the human ratings (r=.26; P<.001).

We used health insurance marketplace enrollment data by state from the Office of the Assistant Secretary for Planning and Evaluation and the Kaiser Family Foundation, which provides the number of people who selected a new health insurance plan in the marketplaces through March 31 and the special extended enrollment period through April 19, 2014 [12,21]. The total number of consumers eligible to enroll in a marketplace plan included legally residing individuals who were uninsured or purchased non-group coverage, had incomes above Medicaid/CHIP eligibility levels, and who did not have access to employer-sponsored coverage. The estimate excluded uninsured individuals with incomes below the federal poverty level in states that did not expand Medicaid [12,22]. The percent eligible who selected a plan represented the number of participants who selected a marketplace plan (with or without receipt of the first premium payment) divided by the total number of consumers eligible to enroll in a marketplace plan.

Descriptive statistics and paired t tests were used to assess Twitter sentiment for ACA and Obamacare tweets, tweets directed to (eg @) the HealthCare.gov handle and state-based marketplace handles, and the random sample of comparator tweets. State Twitter sentiment for tweets containing ACA, #ACA, Obamacare, or #Obamacare were mapped using ArcGIS version 10.1.

The correlation between sentiment for ACA and Obamacare tweets and the percent of eligible individuals who selected a marketplace plan was assessed using a linear regression with robust standard errors. Adjusted analysis controlling for state Medicaid expansion [22] and if a state had a state-based or federally facilitated marketplace [2] were also conducted. Vermont was excluded as an outlier since their enrollment rate was over 2.5 standard deviations above the mean.

All analyses were conducted using STATA 13.1. The study was deemed exempt by the University of Pennsylvania Institutional Review Board.

In our study conducted during the first open enrollment period of the health insurance marketplaces, we identified a significant correlation at the US state level between the ACA-related Twitter sentiment and the percent of eligible individuals who enrolled in a marketplace plan under the ACA. This correlation indicates the potential for Twitter to be a real-time monitoring strategy for future health insurance marketplace enrollment periods. Marketplaces could leverage systematic tracking of Twitter sentiment through commercially available software packages to more rapidly identify signals that indicate changes in marketplace enrollment, including emerging issues [23]. For example, a down-trending Twitter sentiment may indicate a problem in the marketplace, which could be further delineated by examining the content of negatively skewed tweets.

Our study was limited by examination of a single month of Twitter and marketplace enrollment data, though March 2014 enrollment surpassed all prior months [21]. Further analyses are needed to assess if the correlation remains robust in the next open enrollment period and if content analyses could be used to address emerging issues in real time. Additionally, non-geocoded tweets that were excluded from analysis may have differed from those that were geocoded, though the absolute Twitter sentiment for each group was similar. Finally, further examination of states in which Twitter sentiment and marketplace enrollment were discordant are needed.

Of note, some studies using social media have recently come under scrutiny for manipulating people and violating their privacy [24,25]. Concern for such research was directed toward both the use of private data and the performance of an intervention without a clear informed consent. Neither of those properties apply to this study: we used public archival data and did not attempt to intervene on Twitter users’ lives. We believe studies like ours, which simply present statically grounded observations over public data, pose no risk to individuals and present no ethical concerns. Our study was approved by the University of Pennsylvania’s Institutional Review Board under “exempt” status.

Twitter is a repository of free and accessible consumer-generated opinions [26]. Our study adds to a body of work indicating that Twitter is an emerging part of the health and health policy landscape [15,27-30]. The novel methodology used in our study linking Twitter sentiment to ACA implementation data may be an innovative way to inform how to improve the health care system in real time and may be applicable to other settings as health policy is implemented [15]. As the public debate over the ACA continues and federal and state marketplaces deal with the November 2014 open enrollment session, those managing the marketplaces and monitoring the ACA rollout may consider adding Twitter to their data and evaluation toolkit.