Use of Social Media Data to Diagnose and Monitor Psychotic Disorders: Systematic Review

Introduction

Background

Schizophrenia is a chronic mental disease affecting 20 million people worldwide [1]. Although treatment with medicine and psychosocial support is effective, people with schizophrenia are less likely to seek treatment. According to the World Health Organization, efforts to transfer care from mental health institutions to the community need to be accelerated [2]. Currently, schizophrenia is a disease associated with high burden [3,4], and efforts should be taken to reduce this burden.

Artificial intelligence (AI) has emerged as a way to improve several medical tasks [5,6]. AI algorithms can identify patterns in a data set to generate diagnostic tools. In other medical disciplines, AI has already shown good accuracy for diagnostic purposes. It can match current diagnostic capabilities in some specific fields. In psychiatry, AI could be used for diagnostic purposes to support daily patient assessment or drug prescription [7]. AI has also been studied to improve diagnostic and classification capabilities [8]. Additionally, it has been used in suicide risk detection [9] and mood disorder diagnoses [10,11].

Despite encouraging results, it is still unclear how AI should be implemented in clinical practice. The potential of this technology is not yet fully understood. AI could be used to bring completely new tools into health care. We believe that social media might be used to create new diagnostic or monitoring tools. Indeed, AI requires a large database to extract a patient’s profile [12], and social media platforms provide very broad sources of information. People can disclose personal information on social media, including health-related information. Studies have used these data to identify broad human traits (such as intelligence or personality traits) [13]. Subtle features of everyday language could be analyzed to predict mental diseases [14-16]. Prior works showed that social media data can be used for risk classification associated with mental health diseases, such as suicide risk [17]. AI can be used to detect posts associated with mental illness [18]. Therefore, we chose to study the use of AI applied to social media because we believe it could become a brand-new tool in the care of patients with psychotic disorders.

Focus on AI Technologies

AI can be used to perform several different tasks. Machine learning algorithms are generally classified as supervised or unsupervised learning. The main type of machine learning algorithms used in the included studies was supervised learning. Supervised learning algorithms allow patterns correlated to a result to be determined in a data set [12]. The supervised algorithms are separated into two categories: regression and classification. Classification algorithms allow data to be classified into separate categories. Patterns can be used to classify patients in a given group. Decision tree, support vector machine (SVM), and random forest can perform classification tasks. Regression algorithms are used to predict quantitative data. Logistic regression and LASSO (least absolute shrinkage and selection operator) regression are part of this class [6].

Artificial neural networks (NNs) are powerful AI tools built in reference to the cortical neural structure. They can perform supervised or unsupervised tasks. NNs are organized in a succession of layers, with each layer having its input on the output of the previous one. Information travels from the input neurons to the hidden layers before arriving at the outcome layer where the final decision is made [5]. In an NN, each layer functions differently. This type of AI requires significant computing power and large databases [19].

The most used machine learning algorithm in our study is the SVM [20]. The idea of this algorithm is to learn a linear separation (a hyperplane) of the data points to classify them. As there are infinite hyperplanes satisfying this condition, the SVM algorithm learns a hyperplane with the maximum margin, the maximum distance between the classes. However, as most real-life data sets cannot be linearly separated, the SVM uses what is called the “kernel trick.” This transformation projects the initial data points in a higher dimensional feature space where the new is linearly separable. However, there are many limitations to consider when using the SVM: (1) finding a good kernel function is difficult in practice, (2) training is time-consuming on large data sets, and (3) the model is very sensitive to the hyperparameters.

Overfitting means fitting an AI model on data noise or error instead of the actual relationship. It represents one of the limitations of AI. Overfitting is either due to having a small data sample or too many parameters compared to the data [12]. Cross-validation is one of the techniques used to reduce overfitting. With this technique, the data set is split into several groups that are themselves divided into training data and validation data. Therefore, for each group, the statistical model is trained and then validated by a different data set. This technique reduces the risk of having an overoptimistic estimate [21]. Other techniques such as the dropout rate are also used to reduce overfitting. Dropout is a regularization technique for NNs to reduce overfitting and improve generalization [22]. The idea is to randomly ignore neurons (and their connections) from the NN during training. Thus, as the NN architecture is changing at every inference, the same input data can produce a different output. The intuition is that it forces the units to be less codependent and more robust. The main difference between cross-validation and dropout lies in the source of randomness; in cross-validation, the data are randomly split into training and validation sets whereas in dropout, the neural units are randomly discarded.

We have explained the machine learning parameters used in this paper in Multimedia Appendix 1 [23-25].

Prior Work and Goal of This Study

Currently, the diagnosis of psychotic disorders can be subjected to delay. These delays can vary depending on where the patient lives. When the patient remains untreated with psychotic symptoms, there can be important social consequences, including a risk of violence in some cases [26]. The duration of untreated psychosis could have a significant impact on the patient’s psychosocial condition. Early detection and treatment could help improve the care of patients with psychotic disorders [27]. Later, during the evolution of the disease, being able to diagnose a relapse sooner could have significant impact on the patient’s quality of life and reduce caregiver burden.

AI has been studied for the diagnosis of several psychiatric conditions, including schizophrenia. A Korean team used machine learning to identify patterns on CT (computed tomography) scans and classify patients [28]. One of the most robust studies on the classification of psychotic disorders using machine learning comes from an American team [8] that used a clustering algorithm to build 3 biotypes using clinical data and laboratory measures. Neuroimaging and social functioning measures were used for validation. Beyond the main initial goal, this study showed that several new sources of data can be used to improve diagnostic capabilities. In our study, we examined social media data as a new source of data. The symptoms of schizophrenia are very broad, and some of these symptoms could impact patients’ social media activities. For example, we can hypothesize that delusion and disorganized speech or behavior could be seen on social media posts. Alterations in language are being increasingly studied in schizophrenia [15,29] and could be used for the detection of psychotic disorders on social media.

Several studies have shown the capabilities of AI in identifying mental health diseases on social media, with most of them published in the last few years. One brief report [30] reviewed the literature available until December 2020. We hypothesize that social media data could be used to follow patients with schizophrenia or patients at risk of psychosis to identify the first psychotic episode or a relapse of psychotic disorders sooner. Patients could have access to care before psychotic symptoms overtake their social functioning capabilities. Thus, we could reduce the burden of schizophrenia.

Objective of This Review

The objective of our study is to analyze prior works on the use of social media data with machine learning to diagnose a psychotic episode. The diagnostic capabilities would be studied in a broad sense, including the diagnosis of relapse. Therefore, we performed a systematic review of the literature and critically evaluated the included articles and their methodology.

Methods

Search Strategy

We used the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) criteria to identify, select, and critically assess relevant studies while minimizing bias. We searched the bibliographic databases PubMed, Embase, Cochrane, PsycInfo, and IEEE Xplore for studies published until November 12, 2021. We based the keywords list on 3 fields: schizophrenia, AI, and social networks. The search strategy is described in Multimedia Appendix 2. To limit the selection bias, we did not apply any restriction in terms of population. Papers found by any other means were included if they met the inclusion criteria. Studies that were not published in English were excluded.

Study Selection

We included clinical trials and observational studies. The primary objective was to include studies using AI to identify users with a psychotic disorder on social media. Given the low number of published studies on this subject, studies related to any psychotic disorder were included. Studies were selected by 2 independent authors. We excluded studies using social media posts as control data to study language alterations in schizophrenia.

Data Collection Process

Data were extracted from each paper independently using a standard form by 1 reviewer. The following information was collected: the main author’s name and country of origin, year of publication, population, social media and technology used, features, inclusion and exclusion criteria, main objective, method, main endpoint, results, and main limitations.

Synthesis Method

The results of the selected studies will be presented as graphs and tables. Machine learning studies often use different parameters. The graphs will group studies using similar parameters. No secondary analysis of statistical data will be performed. The results presented will comprise only those presented by the authors of the included studies. The results will be presented with the parameters used in the articles. The machine learning parameters used in this paper are explained in Multimedia Appendix 1.

Risk of Bias and Quality Assessment

The quality of the included studies will be assessed using the PRISMA certainty tool. This research will be conducted using the PRISMA checklist (Multimedia Appendix 3). The machine learning methodology of the articles will be evaluated by an author experienced in AI. Risk of bias will be critically assessed by all the authors.

Results

Flowchart

We developed a PRISMA flowchart summarizing the steps of the review (Figure 1). The initial search yielded 93 studies. Based on the titles and abstracts, we excluded 78 studies. We downloaded the remaining studies for full-text review and included the 7 studies that matched the inclusion criteria.

‎View this figure

Authors, Year of Publication, and Country of Origin

The included studies were conducted in the United States (5/7) and Korea (2/7) and were published between 2015 and 2021. Birnbaum and colleagues conducted 3 of the 7 included studies.

Study Design, PRISMA Quality Assessment, and Sample Size

The included studies had a retrospective design. The quality of the studies was assessed using the PRISMA criteria (Table 1 and Multimedia Appendix 4). The quality of the 7 studies was heterogeneous, with a mean PRISMA score of 32.3. Risk of bias varied across the studies. The main bias was a classification bias in 5 of the 7 studies not using a clinical diagnosis. The sample sizes were mostly small, with 6 of the 7 studies having a sample size smaller than 5392 participants. The samples size varied between 51 and 265,396 participants.

Social Media and Technologies Used

Several AI technologies have been used (Table 2). The 2 most commonly used algorithms were SVM and random forest. None of the included studies used data augmentation. Cross-validation techniques to prevent overfitting were used in 5 studies. One study used a dropout rate of 0.25 to prevent overfitting. The social media platforms the studies used were Facebook, Twitter, and Reddit. The studies used mainly linguistic features, as well as some activity-related features (Table 3).

Study Objectives and Algorithm Performance

Main Results

Most studies aimed to identify users with schizophrenia on social media. One study aimed to identify and predict early relapse after hospitalization for schizophrenia [33].

The results were informed by multiple parameters, including the area under the curve (AUC), accuracy, as well as sensitivity and specificity. The AUC of the included studies ranged from 0.76 to 0.97 (Table 4), which is considered to be good to excellent. However, only 1 of these studies [34] used data from clinically diagnosed patients, obtaining an AUC of 0.76. The studies whose results were informed by the accuracy parameter obtained an accuracy ranging from 81% to 96%. One study [33] reported results obtained using predictive models with a sensitivity/specificity couple (Table 5). This study sought to identify and predict relapse of schizophrenia. The authors collected Facebook data from a small sample of patients diagnosed with schizophrenia who had a relapse in the following months. They used these data to build a machine learning model that could be used to analyze the patients’ data in real time. They obtained several sensitivity/specificity couples. The 3-month ensemble model showed good sensitivity (90%) although the specificity was low (40%). This is an example of the unique tools that could be developed using AI. A high-sensitivity tool could allow physicians to detect a relapse earlier and offer timely care to their patients. The 1-month model had a high specificity (0.71) but low sensitivity (0.38).

Data Used

Most of the studies did not have access to clinical diagnostic or health data. Instead, they used evaluations of users’ profiles by psychiatrists to access which user could be classified as having schizophrenia. The included studies used the content of the posts to train and test the models. They also used activity-derived markers such as friending, check-ins, and the number of followers. Used features were chosen to represent the symptoms of schizophrenia described in the literature and were focused on identifying disorganized symptoms and cognitive abilities (Table 2).

Discussion

Principal Results

On a statistical basis, the included studies reported good to excellent performance. Indeed, many of the metrics they reported on are at the top of their respective ranges (AUC, accuracy, sensitivity, and specificity). However, accuracy has high chances of being biased, as most studies did not have access to clinical diagnostic data to train the models. Most studies used the evaluations of social media profiles by trained physicians to classify patients into different groups. Thus, we cannot reach any conclusions regarding the performance of AI in detecting patients with schizophrenia on social media. Moreover, 2 of the included studies that did have access to clinical diagnostic data [33,34] showed the most conservative results (Tables 3 and 4). The included studies were heterogeneous, and some of them introduced interesting new perspectives. After the first psychotic episode, AI and social media could be used to monitor the clinical state of the patients and detect a relapse sooner. This strategy has been studied by Birnbaum et al [33]. We also hypothesize that it could be useful with cohorts of ultra-high-risk patients. Social media provides a constant flow of data, which could in theory allow for the monitoring of large patient cohorts and detect early signs of a psychotic episode. This tool could be integrated in the care of these patients with their consent.

Critical Assessment of the Machine Learning Methodology in the Included Studies

The problem we are considering in this review is a binary classification problem (whether a user profile on social media indicates schizophrenia) mainly based on textual data (the user’s posts). In this section, we introduce the key challenges that we need to tackle given the collected data, analyze the methodology used in the literature, and present our machine learning methodology to solve this problem.

Key Challenges in Performing Statistical Studies

The first major obstacle statisticians need to tackle is the imbalanced data set [38]. Indeed, in the included studies, the number of included controls is several times higher than the included cases. Thus, a “naïve” binary classifier (random forest, SVM, logistic regression, etc) should not be used. It would tend to overestimate the dominant class over the minor one. Moreover, the small sample size in the included studies suggests that there is a chance that the probability distribution in clinical practice is different than the one in the training set used for the experiments.

The second challenge imposed by the data structure is the textual data. Indeed, unlike most problems, we must deal with unstructured data, as opposed to structured data where the features are well organized in a table. Here, the data we consider include text (posts of the users), and it is unstructured. Therefore, natural language processing (NLP) techniques are called for to extract relevant features to run a machine learning classifier. This is often a delicate stage, as researchers often introduce an inductive bias when they decide which features to extract from the data. State-of-the art NN algorithms like the BERT (Bidirectional Encoder Representations from Transformers) algorithm [39] can automatically extract features from textual data in a “pretrained stage.” The algorithm is then trained to perform the desired classification task during the “fine-tuned stage.”

Analysis of the Methodology Used in Previous Works

Most previous works introduced in the last section extract their features using NLP modules such as the natural language toolkit [32,35], the linguistic inquiry and word count package [36], or even older methods like the n-gram [31]. The major drawback of these “bag-of-words” techniques [40] or term frequency–inverse document frequency methodologies [41] is that they often vectorize the textual data only based on the words (and their statistics in the sentences) without accounting for grammar and semantic relations between them and their context. Thus, these feature extraction methods fall short when capturing semantic or syntactic information or the sentiments of words [42].

Perspectives for Future Studies: Our Recommendations for Machine Learning Methodology

Based on the aforementioned challenges and the limitations of previous works, we present in this section our approach to identify patients with schizophrenia based on social media activity.

First, we need to address the imbalanced data set problem. Among the numerous approaches, we selected one of the most used methods in practice called SMOTE (Synthetic Minority Oversampling Technique) [43]. The idea is to balance the data set by creating synthetic samples from the minority class so that both classes become more balanced. Specifically, SMOTE selects an example from the minority class and its k neighbors (typically k=5) and creates a synthetic example as the convex combination of these 2 data points. This procedure can produce as many synthetic samples for the minority class as needed, and it guarantees that these created examples are realistic, as they are close to the existing ones in the feature space.

However, for this data augmentation technique to work, the feature space needs to be continuous, which is not the case in textual data. To alleviate this issue, we need to use word embeddings. In NLP, word embedding is a continuous representation of a word that encodes the meaning of the word in a feature space, and it is usually a real-valued vector [44]. Thus, 2 words with close meanings like “ill” and “disease” will be closer in the feature space than 2 words like “ill” and “car.” Nevertheless, the remaining challenge is to create a relevant embedding space. A popular method in NLP is to use a pretrained language model like BERT [39]. This algorithm has been pretrained by Google on the concatenation of the two largest data sets: BookCorpus [45], gathering 11,038 unpublished books, and English Wikipedia, gathering 6,427,217 articles. These high-quality embeddings not only allow us to use the SMOTE method to augment the minority class, but they also allow us to represent the textual data in an informative feature space. The latter will be used as the input to our classification algorithm.

Finally, we need to define the binary classifier for our problem. As we have already used the BERT algorithm to create the word embeddings of the posts, the natural approach would be to use it as a binary classifier as well. To do so, we need to “fine-tune” it using our own data set by adding a linear layer to the existing NN architecture. This approach to text classification has demonstrated state-of-the-art results on 8 widely studied text classification data sets [46].

Limitations of This Study

We performed a systematic review of studies using machine learning to identify schizophrenia on social media. Based on our hypothesis, the main limitation of our review is the small number of included studies. When submitting this paper for publication, the published studies on this subject were limited and we were not aware of any new study that met the inclusion criteria. Our review aimed to evaluate the potential of this technology as a new tool for the care of patients with schizophrenia. Therefore, we used broad inclusion criteria to include more papers. The included studies do not have the same objective and thus, their results cannot be compared. However, they describe the various uses of this technology.

Most of the studies (5/7) did not use clinical diagnostic data. Instead, they used evaluations of the mental states of the included subjects based on their public profile history and the contents of their posts. It is unlikely that this method is efficient in accurately identifying patients with schizophrenia. Future studies should use health data including medical diagnostic data to develop an accurate model.

Furthermore, some of the included studies showed limitations in their methodology and choice of machine learning algorithms. We analyzed these limitations and used them to propose recommendations for future projects.

Ethical Reflection and Privacy

Machine learning tools could be useful in several ways to improve the care of patients. We could monitor the social media activity of patients to detect psychotic relapses sooner. These tools could also be used to detect the first psychotic episode sooner in patients monitored for high risk of psychosis.

The use of machine learning to predict mental health disease raises ethical questions. In what context should we use such tools? Patients would need to comply with the use of these tools and their data in their care. The use of machine learning would be appropriate only if patients consent to it. Furthermore, many countries are establishing a regulatory framework on AI usage [47,48]. AI tools will have to comply with regulation laws to be used in a clinical setting.

In particular, machine learning algorithms need to be trained on massive amounts of unbiased data. To prevent third parties from using these technologies for other purposes, ensuring the safety of medical data is essential.

Conclusions

AI brings new perspectives in research on schizophrenia. It could be used to monitor the clinical condition of patients at risk of psychosis or to detect relapses of schizophrenia by observing patients on social media. There are currently only a few studies published on this subject, and most of them do not accurately estimate the potential of this technology. However, this technology could become a new tool in the care for patients with schizophrenia, ultimately reducing the burden on caregivers. It should be developed and used in accordance with ethical and legal frameworks.