desmarais-lab / govwebsites Goto Github PK

We should be able to add many more cities to our data---casting a national web, if we can analyze local campaign contribution data to classify the partisanship of mayors. This would broaden the generalizability of our results.

Could provide national overview of partisan issues in terms of Democrats' focus

This data consists of 1,467,938 files (only txt, html, pdf, doc, and docx), totaling 857.2 GB

I am not sure what to do with this data right now, since (a) I always NEED TO (and there are good reasons for that) make backups before I do anything to the files, but in this case, I can't zip them because (at least) one website has file paths so long that the compression program can't handle them; and another 800GB backup seems really excessive (b) any analysis on this will take ages, and in some cases, probably won't be possible at all.

The main culprits are San Diego, New York and Chicago, which are around 30-60GB each

I am starting to have more and more doubts about this approach, recently fueled by the fact that I noticed that mayors later re-use their mayoral campaign websites as their campaign websites for running for governor/congress, etc. Basically, there is no guarantee that the current state of their campaign website reflects the way it was during the campaign. At the very least, a lot of mayors often seem to make a 'mission accomplished' post in which they brag about their victory - which obviously wasn't there during the campaign.

Also, the results, while admittedly a lot better when incorporating the mayors of the top 100 cities (see figure 6 (cosine corrplot) and table 13 (bootstrapped lower/upper bounds) in the manuscript), they still don't really prove anything.

As noted previously, there are 30 websites on our list of URLs that SHOULD be downloadable, but for some reason aren't. I've now explicitly tried to focus only on these, but wget just can't get them. Notably, this includes Philadelphia and, much more importantly, New Orleans. NOLA was still downloadable back in the fall and contained quite a lot of content, so the question is: do we still include our old version of the website in the paper?

This issue lists the reasons we might have difficulties turning this project into an R package. These aren't urgent problems that need to be responded to immediately, I just want to maintain a list of concerns that need to be kept in mind. Consequently I will update this post rather than adding additional posts, if anything else comes up.

Unix-only libraries:
wget
libmagic (determining file types) (works on Windows according to the Github page)

Python packages:
spaCy (lemmatization) (spacyR works well enough now)
Selenium (webscraping on interactive websites) (in theory there is an R version, but it is terrible)

R packages not on CRAN:
SpeedReader (fightin' words) (not used any more)
wand (R interface to libmagic)