Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Currently submitted to: Journal of Medical Internet Research

Date Submitted: Jul 4, 2019
Open Peer Review Period: Jul 8, 2019 - Sep 2, 2019
(currently open for review and needs more reviewers - can you help?)

Application of Transfer Learning for Automatic Triage of Social Media Posts

  • Derek Howard; 
  • Marta Maslej; 
  • Justin Lee; 
  • Jacob Ritchie; 
  • Geoffrey Woollard; 
  • Leon French

ABSTRACT

Background:

Mental illness affects a significant portion of the worldwide population. Online mental health forums can provide a supportive environment for those afflicted and also generate a large amount of data which can be mined to predict mental health states using machine learning methods.

Objective:

We benchmark multiple methods of text feature representation for social media posts and compare their downstream use with automated machine learning (AutoML) tools to triage content for moderator attention.

Methods:

We used 1588 labeled posts from the CLPsych 2017 shared task collected from the Reachout.com f​ orum (Milne et al., 2019). Posts were represented using lexicon based tools including VADER, Empath, LIWC and also used pre-trained artificial neural network models including DeepMoji, Universal Sentence Encoder, and GPT-1. We used TPOT and auto-sklearn as AutoML tools to generate classifiers to triage the posts.

Results:

The top-performing system used features derived from the GPT-1 model, which was finetuned on over 150,000 unlabeled posts from ​Reachout.com​. Our top system had a macro averaged F1 score of 0.572, providing a new state-of-the-art result on the CLPsych 2017 task. This was achieved without additional information from meta-data or preceding posts. Error analyses revealed that this top system often misses expressions of hopelessness. We additionally present visualizations that aid understanding of the learned classifiers.

Conclusions:

We show that transfer learning is an effective strategy for predicting risk with relatively little labeled data. We note that finetuning of pretrained language models provides further gains when large amounts of unlabeled text is available.


 Citation

Please cite as:

Howard D, Maslej M, Lee J, Ritchie J, Woollard G, French L

Application of Transfer Learning for Automatic Triage of Social Media Posts

JMIR Preprints. 04/07/2019:15371

DOI: 10.2196/15371

URL: https://preprints.jmir.org/preprint/15371


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.