Hi Sabina,

Since I couldn't get gensim to run on my Windows, I decided to go with the lda package. It requires a document-term cooccurrence matrix as input and I spent all day today writing code to create that matrix. While it worked on a toy body of text I created, the computer apparently runs out of memory before creating such a matrix for two files from the corpus. Do you know how I can get around this issue? I added the code to the parsereddit file. I have also added the lda code (copied from the package, might need changes) as a comment at the end of the code.

Thanks!

To address the difference in "quality" between LDA models with different numbers of topics, calculate the topic coherence of models with 25, 50, 75 and 100 topics (example here).

• Monthly topic contributions
• Any topic passing a threshold of average 2% contribution over the years considered top topic
• Remove shared variables from the multicore function to speed up processing

A couple things I noticed/questions about the regex:

• It doesn't match the string r"[same-sex|same sex] marriage".
• Should we include patterns like "LGBTQ rights"?

For each top topic, run a linear regression model on the time-series of the topic's contribution over time. This will allow us to calculate:

• If the slope estimates are significantly different from 0
• If the slope estimates are significantly different from each other

Add a function to Utils.py that writes to file n random comments from write_original files for a set of given years (when year==2008, include data from 2006 and 2007, too).

From what I understand you're working on something that:

1. Reads JSON data from a file,
2. Creates a Python object that stores the line of the file that match a regex, and
3. Then lemmatizes/analyzes the Python object.
   I propose filtering posts before saving the response from the API to a file. To be more specific, we could filter the posts that get passed from json_obj (JSON-formatted API response) to data (JSON data to be written to a file) in lines 42-43 of get_all_posts.py.

Does that make sense? If so, I can definitely implement that. Also, did you tell me that you've already done some work writing the regex we'll use to filter posts?

• Rewrite the NN code to train it for regression on values-consequences-preferences human ratings
• The code should work with the ratings stored in a CSV file, formatted like the sampled comments resulting from the LDA analysis
• Retain the ability to use the network for classification, if only to compare performance with regression

I committed a skeleton in 3c56c27, but it's super rough and we're under no obligation to build on it.

Create a separate module to create n bootstrapped LDA models, each using a subset of fixed size of our data. These models can then be used to generate confidence intervals around the model as a whole (using metrics like model alignment), and around trends in the prevalence of any given topic over time.

Relatedly, think about other ways of aligning two topics from different models, e.g.

• The number of comments for which the topics are both considered the "most probable" in their respective models
• The MSE between the probability assigned to them by individual comments, i.e. sum([ (prob(model1, topic1, comment) - prob(model2, topic2, comment))**2 for comment in comments ])

I think it would be nice (i.e. more usable and readable) to break up Utils into a few separate modules, e.g. a Parser, an LDATrainer, an NNTrainer, a TopicContributionCalculator, etc. @BabakHemmatian, what do you think? I'd be more than happy to take this on if you think it's a good idea.

Edit: As of 1716955, this is done except for one outstanding thing that's still bothering me: Top topics are still calculated directly from Reddit_LDA_Analysis. I think this should be incorporated into the LDAModel class as a class method (and the report object incorporated as a LDAModel class attribute).

Also, I'd like to collect all filenames into a set of configurable class attributes, rather than having them be hard-coded in the bodies of various class methods.

Which topics predict the value vs. consequence vs. preference human judgments best? This can be done using linear models to confirm the results of LDA.

• Update list of stopwords, so that words with deontic value are not removed from the original comments
• Make sure removal of special characters does not hinder interpretability of preprocessed text
• Ensure that the hashsum check does not result in infinite loops
• Ensure that it is easy to continue parsing if the Parser is stopped after processing a specific file

See title.

Add support to:

• Download Reddit comments to disk as-needed
• Clear them once the cleaned text has been written as a separate file

See title.

The variable indices created in Define_Sets and passed to Create_New_Sets is read from RC_Count_List (which is a list of counts per month IIRC):

ipdb> indices
[0, 0, 3, 11, 4, 17, 30, 8, 12, 14, 38, 15, 14, 28, 26, 27, 23, 34, 57, 68, 99, 95, 69, 94, 76, 92, 71, 79, 178, 185, 104, 166, 177, 366, 1394, 371]

However, it seems to be being treated as though it's a list of all comment indices. In Create_New_Sets it's used to determine the comment indices to sample the train and eval sets from:

67    num_comm = indices[-1]
68    indices = range(num_comm)
...
126    LDA_sets['eval'] = sample(indices,num_eval)
127    LDA_sets['train'] = set(indices).difference(set(LDA_sets['eval']))

I just put two DataFrames in the repo that should give us a sense of how our data is distributed across platforms and years. You can open them like this:

import pandas as pd
pd.read_pickle('n_posts.pandas')

n_posts.pandas shows the number of posts we have grouped by source and year, and n_comments.pandas shows the number of comments we have grouped by source and year.

As I wrote in my email, the only caveat to keep in mind is that the program crashed while collecting data for 2015 posts from Fox News--because there was so much of it. So while looking at these numbers, assume that we could also have tens of thousands of comments from Fox News in both 2015 and 2016.

One thing I noticed is that we have no data from 2008, and very little from 2009.

So I have two questions (well, two categories of questions...):

1. I haven't really studied to what extent there's an unequal distribution if we just group by source partisanship (in other words, it's totally possible that, for example, the thousands of FoxNews comments are balanced out by thousands of nytimes comments). Do you notice anything about these numbers that we should (or shouldn't) correct for? What else should I be asking about the distribution of this data, other than how it groups by source/year?

2. Given the potential for an uneven distribution across sources and years, how and do you think we should re-weight or re-sample our data? Some vaguely-formulated proposals I have are to:

• Sample randomly and evenly across platforms (this would not account for the lack of data from '08).
• Almost equivalently, re-weight observations so comments from sources (and years?) with less data are more heavily weighted.
• If either of these approaches makes it so we don't have enough data, or are heavily weighting ridiculously small amounts of data, we could also look into retrieving data from additional news sources.

Edit: Here is the infographic that inspired my initial choice of platform: http://wilkins.law.harvard.edu/projects/2017-08_mediacloud/Graphics/Fig7_11.pdf. It's taken from this paper: Faris, Robert and Roberts, Hal and Etling, Bruce and Bourassa, Nikki and Zuckerman, Ethan and Benkler, Yochai, Partisanship, Propaganda, and Disinformation: Online Media and the 2016 U.S. Presidential Election (August 2017). Berkman Klein Center Research Publication 2017-6. Available at SSRN. Quoting from the paper, "[t]he size of each media source node is in proportion to the number of other media sources that link to that source at least once" and "[t]he colors on the map reflect the partisan pattern of attention to the media sources based on the sharing
behavior of Twitter users who have clear partisan allegiances".

• Function for similarity between top topics

Quantify the difference between the original "one-hot" topic contribution calculation, and a topic contribution calculation function that records entire probability distributions over topics.

***The dev module for this issue is dev/issue19.py.***

Just posting this for the sake of documentation: Sometimes raw comment files contain comments from other months, e.g. the file for 2007-12 contains two (relevant) comments from 2007-11. This explains discrepancies between RC_Count_List and RC_Count_Dict.

The random subsampling function seems to have a bug. Many of the numbers get written to file multiple times, so that the sets of unique indices comprising the training and evaluation sets are much smaller than they should be.
I made a lot of changes to different parts of the code. But I didn't touch your function. So I don't think this is my doing.
I will email you the correctly parsed dataset right away so that you won't have to run the parser again and jump through all the hoops I had to in the past few days.

• Random comment samples
• Probabilities of all top topics for each sample
• Different categories of top topics based on the values/consequences distinction

Fix the topic contribution function so that it uses comment-specific topic probabilities for calculations.