Piecing social movements together by their junctures, demonstrating how individual social movements can be interpreted as components of an encompassing narrative.

View the project here.

data procuration preprocessing/...

• Wikipedia's List of social movements is used to seed the initial movement keywords in socialList.txt
• pytrends is used in googleTrends.py to retrieve Google Trends by keyword
• the New York Times API is used in nytimesTrends.py to retrieve NYT articles by keyword

data processing preprocessing/...

• Pandas is used to merge and read CSV files aggregated through mergegoogletrends.py
• TextBlob is used in nytimesfeatureExtraction.py for sentiment analysis of the aggregated NYT articles
• DuckDuckGo is scraped in nytimesfeatureExtraction.py to textual description of the aggregated NYT articles
• NLTK is used in nytimesfeatureExtraction.py and PageRankSummarizer.py to tokenize strings
• networkx is used in PageRankSummarizer.py for treating the sentences as a network
• sklearn's feature_extraction is used in PageRankSummarizer.py to extract the features from the text
• scikit's PCA is used in reduction.py to reduce trends into two dimensions

data visualization js/...

• queue.js is used to read the data from the data/*.json files
• d3.js is used to parse and render the data
• jquery.selection.js is used to select text from the descriptions