RDSMproj (Rare Diseases Social Media Project) for the National Center for Advancing Translational Sciences at the NIH. This project looks at mining information from social media (Reddit) and finding subreddits that are related to different rare diseases found in the GARD database. The project matches rare diseases to Reddit subreddits, downloads the post and comment data, and then analyzes the text data to find the different topics that people are talking about.
The project is split into four packages as part of rdsmproj:
- mapper is a python package that maps text to a rare disease(s) using nltk and spaCy. An alternate name for this package is NormMap V2.
- sm_reddit is a collection of scripts that utilizes pmaw to download Reddit post and comment text data for use in topic modeling or other text analyses.
- tm_t2v is a python package that creates topic models of text using Top2Vec.
- tm_lda is a (legacy) python package that creates topic models of text primarily using LDA as implemented by Gensim. This package was used in this paper.
Ensure that you have up to date copies of pip
, setuptools
, and wheel
prior to installation.
pip install --upgrade pip setuptools wheel
For now, each package above is installed separately. Installation can be done using pypi
pip install rdsmproj[mapper]
pip install rdsmproj[sm_reddit]
pip install rdsmproj[tm_t2v]
pip install rdsmproj[tm_tlda]
from rdsmproj import sm_reddit
pmaw_args = {'limit':1000}
# Example subreddit 'MachineLearning'.
# Passes pmaw arguments to search_submissions.
sm_reddit.GetPosts(name='MachineLearning', silence=False, pmaw_args=pmaw_args)
from rdsmproj import utils
from pathlib import Path
# Default path to where the post data is located.
path = utils.get_data_path('posts')
data = utils.load_json(Path(path,'MachineLearning_posts.json'))
# Example passes pmaw arguments to search_submission_comment_ids.
sm_reddit.GetRedditComments(data=data, silence=False, pmaw_args=pmaw_args)
from rdsmproj import preprocess as pp
# Example processes the comment data for use with tm_lda or tm_t2v.
data = pp.PreProcess(name='MachineLearning')
documents, tokenized_documents, id2word, corpus = data()
from rdsmproj import tm_t2v
embedding_model = 'doc2vec'
name = 'MachineLearning'
clustering_method = 'leaf'
i = 0
# Creates and saves a model.
model = tm_t2v.Top2VecModel(name,
f'{name}_{embedding_model}_{clustering_method}_{i}',documents=documents,
embedding_model=embedding_model,
speed='fast-learn'
).fit()
# Analyzes model and records the results.
tm_t2v.AnalyzeTopics(model=model,
model_name=f'{name}_{embedding_model}_{clustering_method}_{i}',
subreddit_name=name,
tokenized_docs=tokenized_documents,
id2word=id2word,
corpus=corpus,
model_type='Top2Vec')