We collected comments from the subreddit r/CysticFibrosis to represent the experiences and concerns of the cystic fibrosis community. On August 31, 2022, we collected all Reddit comments starting from the inception date of March 24, 2011, using the Pushshift Reddit application programming interface through Python (version 3.9.12; Python Software Foundation). The Pushshift Reddit application programming interface offers a convenient way to collect text data for all comments and submissions in a chosen subreddit. We made this data set, as well as the script we used to collect the data, available on our GitHub repository [21].

We checked for duplicate comments and converted everything to lowercase. We also removed links, tags, and mentions of other users. At this stage, we did not perform any further data cleaning to maintain the natural structure of the comments since the BERTopic library was developed with natural text and has its own way of dealing with noise and outliers.

BERTopic is a topic modeling technique that uses state-of-the-art language models and applies a class-based term frequency-inverse document, which calculates how relevant a word is to the class of documents and uses a frequency procedure for generating topics [22]. We trained a BERTopic model on the Reddit comments using the library bertopic (version 0.12.0 [22]). To keep our results consistent and reproducible with each iteration, we set the random state parameter of the Uniform Manifold Approximation and Projection model to 42 using the library umap (version 0.0.1 [23]). With these parameter specifications, BERTopic produced 824 topic categories: 823 usable topics and a topic for outliers, topic “–1.” Additional cleaning was done to remove stopwords by defining a vectorizer with sklearn (version 3.7 [24]) and setting the stopwords using the list from spacy (version 3.3.1 [25]). We also included “im,” “like,” “use,” “ive,” and “ill” to the list of stopwords before finally reducing the number of topics further.

We set BERTopic to reduce the number of topics from 823 to 30 and noticed that most of the comments from the dropped topics were classified as outliers rather than belonging to one of the 30 topics. Instructing BERTopic to produce 30 topics from the beginning similarly resulted in more outliers. Thus, in order to mitigate the loss of data and obtain more general topics, we set BERTopic to reduce the number of topics to 30 and then used hierarchical clustering to manually merge topics based on topic similarity metrics.

This study did not require participants to be involved in any physical or mental intervention. As this research did not use personally identifiable information, it was exempt from institutional review board approval in accordance with the Ethical Guidelines for Medical and Health Research Involving Human Subjects stipulated by the Japanese national government.

A total of 120,738 comments from 5827 unique user IDs were collected, with dates ranging from inception on March 24, 2011, until August 31, 2022. Figure 1 shows the number of comments and the number of users per month. Table 2 contains sample comments and their assigned topics.

The initial 823 topics were too much for us to analyze. We specified our BERTopic model to reduce the number of topics to 30 instead, as a more manageable starting point. Figure 2 shows the top 8 topics and the words with the highest class-based term frequency-inverse document scores representative of each topic, and Figure 3 shows the intertopical distance map between topics that can be used to create more general topics.

A dendrogram graph was generated to visualize the hierarchical clustering of the 30 topics to see which ones were similar enough to be merged (Figure 4). The clustering was based on the cosine distance matrix of the topic embeddings with default parameters in BERTopic. Changing these parameters will result in different, more, or fewer cluster suggestions, so we merely use this result as a guide to manual merging.

In each cluster, we merged similar topics based on our judgment and understanding of what these topics discussed. We treated the dendrogram from BERTopic as recommendations on which topics we could look into for merging. From these recommended clusters, we looked at representative comments and also the top 10 representative words for the topic to make our decisions regarding merging. In addition, we looked into symptoms, treatments, life expectancy, comorbidities, and the general lives of patients with cystic fibrosis in order to name each of the merged topics. We decided to merge the following sets of topics: {29, 2, 26}, {28, 10, 4}, {15, 19}, {8, 25}, and {5, 24, 12}. In contrast, we chose not to merge the set {3, 21} since one appears to talk about financing while the other was about some type of medication; similarly, we excluded 27 from {8, 25} since the keywords for topic 27 imply that it talks about children and hereditariness, while 8 and 25 talks about mutations; likewise, topics 11 and 1 both have keywords that talk about medical care and life in general; and lastly, topic 0 was not included in {5, 24, 12} because it looks like topic 0 talks about a specific drug, Trikafta, and the liver, while the others talked about things related to respiration or the respiratory tract. After this step, we were left with 42,060 comments categorized into 22 topics. In Table 3, we summarize these topics, their top 10 representative words, and how they were merged relative to the original 30 topics.

While it was not our main focus of the analysis, we provided a simple visualization of the evolution of topics within our time period using dynamic topic modeling. Figure 5 shows the time series of the top 5 topics. From the graph, we can see that there was little activity in the early years of the subreddit, and most topics seemed to be fairly uniform until around 2020 when more variations between topics became visible.

We first checked the cross-correlation of the prewhitened series for the number of active users and the number of comments and noticed that only lag 0 was significant. This means that both time series are correlated within the same time period rather than timesteps ahead of or behind each other. To simplify the inclusion of the number of active users as a regressor, we add it to the regression model in the same way a regressor would be added to a linear regression model without the use of a transfer function.

The series for each data set were nonstationary after observing the graphs of their ACFs. We determined that 1 order of differencing was enough to make each series stationary and confirmed this with the Augmented Dickey-Fuller unit root test. Observing the behaviors of their PACFs, which cuts off after a certain lag in addition to their ACF decaying to zero (see Table 1), leads us to conclude that each of the series follows an AR process. We determine the orders for the AR process by the lag where the PACF cuts off (see Table 4). The order of the AR process can be interpreted as how many lags are predictive of the current timestep; for example, an AR(3) process implies that the observation at the current timestep can be described using observations from the past 3 timesteps. We also note here that for the series representing the topic “IV access port,” the cutting-off behavior of the PACF was not as apparent as in the other series, but we still opted to fit an AR process for simplicity.

The results from fitting an ARIMA model for the number of comments with exogenous variables for the number of active users and the dummy variable to indicate the months of the COVID-19 pandemic can be seen in Table 4. For all series, all lags following the orders for the AR process were significant at an α level of .05.

The last column in Table 4 shows the P values for the COVID-19 dummy variable. Using an α level of .05, we say that the user activity for the data sets that show a P value less than .05 showed a significant difference in trend during the COVID-19 pandemic. There were 9 data sets: the denoised aggregate data set and 8 topic data sets that showed a significant change in trend. For the aggregate data set and topics “Trikafta and side effects,” “Gratitude,” “Lung transplants and respiration,” “Sweat testing and mutations,” “Medical facilities and professionals,” “Tobramycin side effects and alternatives,” and “CF inheritance and family planning,” there was a significant decrease in the number of comments during the COVID-19 period. In contrast, for the topic “Marijuana,” there was a significant increase in the number of comments during the period.

We followed the same steps to check the significance of COVID-19 on the number of active users without any exogenous variables except for the dummy variable representing COVID-19 and found that COVID-19 was not statistically significant in the model. This implies that the trend in activity during the pandemic period was not significantly different from the trend outside this period.

There is a mix of increased and decreased activity for topics that showed a significant difference in trend during the COVID-19 pandemic. This indicates a shift in attention or focus of discussion topics during the pandemic, despite an overall decrease in activity on the subreddit. In addition, our results also describe the time series processes that describe user activity for each topic. These models show how immediate or prolonged the predictive capacity of past activity is for future activity.

The topics from our BERTopic model showed a variety of discussion interests within the cystic fibrosis community, ranging from medications, treatments, symptoms, finances, or living with their disease. By incorporating a dummy variable for the pandemic period, we were able to check the statistical significance of this period on the amount of user activity for each topic.

Our study shows that there was an overall decrease in user activity during the pandemic. Considering that there was no statistically significant decrease in the number of active users during the pandemic, the decrease in user activity during this period can be attributed to other factors, such as other priorities that came about during the pandemic. Users could have been spending less time commenting on subreddits and instead focusing on gathering information about the pandemic, keeping up with changing health protocols, job security, or family members, to name a few. The decreased activity during this period did not automatically reflect in the individual topics, as evident in the increased activity for the topic “Marijuana.” In the succeeding subsections, we discuss the topics that underwent a significant change in trend during the pandemic and the possible reasons behind the change, as well as the varying ARIMA orders among the topics.

Column “P value” in Table 4 summarizes the significance testing for the COVID-19 dummy variable using an α level of .05. We interpret topics with a P value less than .05 for the dummy variable as showing a significant change in trend during the COVID-19 pandemic. In this section, we discuss the topics that showed a statistically significant difference in trends during the COVID-19 pandemic.

The topic “Marijuana” showed top keywords such as “smoking,” “smoke,” “edibles,” and “weed.” Looking at some sample comments on this topic led us to conclude that the discussion is about the use of cannabis and alternative means of using cannabis aside from smoking (see Table 2). The rise in activity during this period could be due to the additional stresses brought about by the pandemic and, thus, the curiosity in exploring the use of medical marijuana. In Stephen et al’s [28] study, they showed that side effects were rare and mild, should there be any at all, and that the use of medical marijuana was effective in relieving symptoms of stress and pain.

A trickle-down effect on the overall negative effect on the aggregate data set could explain topics with a negative coefficient for COVID-19. The topics “Trikafta and side effects” and “Tobramycin side effects and alternatives” both talk about medications and could have lower activity because of other newer and more pressing matters during the pandemic that require more of their attention. In contrast, these medications have a more established database of information. Trikafta is an approved medication for patients with cystic fibrosis with the most common type of mutation that targets the underlying cause, while Tobramycin is an antibiotic that targets Pseudomonas aeruginosa infection in patients with cystic fibrosis [29]. The decrease in activity for “Gratitude” could be explained by having fewer things for users to be grateful for during the period or due to fewer interactions overall. The topics “Lung transplants and respiration,” “Sweat testing and mutations,” and “Medical facilities and professionals” relate to hospital or clinic visitation; decreased activity could be explained by lower visitation rates due to limited hospital capacity or precautions in place for patients with cystic fibrosis to avoid getting COVID-19 [11]. Lastly, for the topic “CF inheritance and family planning,” the decreased activity could be due to higher uncertainties about the future in light of the long pandemic period and lower interest in discussing this topic.

In this section, we discuss the interpretations for the ARIMA models used in each topic data set and give possible insights into their time series behavior. The ARIMA orders for the different data sets mostly follow an ARIMA(1,1,0) process, including the aggregate data set. This means that these series require 1 order of differencing to make the series stationary and can be described using an AR(1) process. An AR(1) process implies that the observation of the current time period can be described using observations from the previous time period. We can interpret this as users having shorter terms for engagement since only the previous timestep is descriptive of current activities. In contrast, topics that were modeled with a higher AR order imply a longer engagement of users since longer durations of user activity describe the current timestep.

The topics “Dairy and calories” and “Diabetes” have the highest AR order, AR(4). The higher order can be explained by a longer duration of interest in discussions or a user following up on previous discussions. Compared to an AR(1) process, an AR(4) process means that activities from the previous 4 timesteps are descriptive of the activities in the current timestep. Both topics describe dietary concerns, which could explain the prolonged engagement since it takes time to observe changes or side effects from diet changes. The topic “Diabetes” is talking specifically about cystic fibrosis–related diabetes, which is the most common complication experienced by patients with cystic fibrosis, with around 50% of patients with cystic fibrosis developing cystic fibrosis–related diabetes by the age of 30; the high comorbidity rate could be another reason for the prolonged engagement. Additionally, the topic “Creon medication and diet” is another topic relating to the diets of patients with cystic fibrosis and is modeled using AR(2) and is also longer than most other topics. These topics relate to the nutrition and diet of patients with cystic fibrosis, which need to be carefully monitored to avoid complications and improve symptoms [30-33].

The topic “Tobramycin side effects and alternatives” has a high AR order as well, AR(3). Tobramycin is an antibiotic used to treat P aeruginosa infections in patients with cystic fibrosis. This infection is highly prevalent among patients with cystic fibrosis, often results in chronic infections after the first infection, and is resistant to antibiotics [34,35]. This infection can drastically reduce the quality of life and life span. The seriousness of this infection and its resistance to antibiotics could explain the longer durations in discussing this topic.

Interestingly, topics that were modeled using the AR(2) had a number of topics (5 out of 7) related to respiration and common symptoms: “Nebulizers,” “Medical facilities and professionals,” “Sinuses and breathing,” “Air quality,” and “Airway clearance vests.” The higher AR order could be explained by longer discussions about symptoms, procedures used to alleviate symptoms, and the time it takes to notice effects or side effects. Interestingly, the topic “Birthdays” also follows an AR(2) process, where the long process could be due to prolonged celebrations of reaching another milestone [36].

Many of the comments were not classified into any topics, as evident by the 78,678 comments categorized as outliers in topic –1. This can be expected in social media, especially for short texts and quick exchanges between users; thus, these comments do not have enough contextual information for the model to classify them. Although some comments, on their own, may not talk about a specific topic, they may still contribute to the sentiment of the topic discussed in the parent comments or the main post.

We chose BERTopic for its convenience and because it assigns each document to exactly 1 topic; however, we did not spend much time fine-tuning the model and tweaking its performance. We checked a few comments for each topic and saw that they were properly clustered with similar documents. While BERTopic has shown more robust performance in topic modeling of social media posts than other topic modeling methods [37], topic modeling is still a fairly subjective process. The results we presented demonstrate the possibility of using topic modeling with time series analysis to gain insight into the needs of patients with rare diseases rather than to evaluate the performance of BERTopic as a topic model.

Our definition of the dummy variable used to indicate the period of the pandemic was based on how long we believe the pandemic affected the number of comments, but the effects of the pandemic could have started later or could have ended sooner. We chose the year 2020 to capture the early influence of the pandemic when cases were starting to be reported and until the end of the year, when countries were starting to relax health protocols. Our time series modeling was also not as extensive to perform the same procedure for each data set, referring to our assumption that only the current lag for each topic’s data set is significant following the behavior of the aggregate data set. The purpose of the analysis was to fit a model to the time series data and check for the statistical significance of an event rather than forecasting future values; thus, we no longer performed residual analysis to evaluate our models and instead observed the graph of the fitted values superimposed on the raw data. This allowed us to observe the broader trends across all the discussion topics rather than being restricted to only a small data set of topics.

We focused on the cystic fibrosis community because of the active Reddit users in r/Cystic Fibrosis and because the symptoms of cystic fibrosis overlap with those of COVID-19. We found that during the events of the COVID-19 pandemic, there was an overall decrease in user activity. Our time series analysis accounting for the number of active users showed a mix of positive and negative effects from COVID-19, indicating a shift in focus in discussion topics during this period. Additionally, the varying orders of the ARIMA models among the topics indicate that the user activity for some topics has a longer predictive capacity than others. These topics do not necessarily reflect the needs or interests of patients with cystic fibrosis or patients with rare diseases more generally. Still, our results show what patients were concerned about in the dynamic interactions between patients on the subreddit, as well as how they were disrupted by the COVID-19 pandemic.

We anticipate that this methodology and these results can be further developed and studied to be used in providing better care for patients with rare diseases by gaining a better understanding of their needs and how they are affected by external factors. In contrast with the traditional survey- or clinical-based methods, analyzing social media is much quicker while still providing useful insights into patients with rare diseases. Future research may consider extending this methodology to other rare disease subreddits, or even comparisons across multiple subreddits, as a quick and efficient source of information for the lived experiences of patients and their daily struggles and concerns.