phantominsights / subreddit-analyzer Goto Github PK

Hi @agentphantom ,

This is me again, sorry. I have another issue with the script.
I'm trying to download the submissions from r/goodanimemes from the last 2 weeks.

But after some time I get this error message :
Traceback (most recent call last): File "subreddit_submissions_alt.py", line 115, in <module> init() File "subreddit_submissions_alt.py", line 40, in init download_submissions(subreddit=subreddit) File "subreddit_submissions_alt.py", line 110, in download_submissions download_submissions(subreddit, latest_timestamp) File "subreddit_submissions_alt.py", line 110, in download_submissions download_submissions(subreddit, latest_timestamp) File "subreddit_submissions_alt.py", line 110, in download_submissions download_submissions(subreddit, latest_timestamp) [Previous line repeated 80 more times] File "subreddit_submissions_alt.py", line 72, in download_submissions json_data = response.json() File "C:\Python38\lib\site-packages\requests\models.py", line 898, in json return complexjson.loads(self.text, **kwargs) File "C:\Python38\lib\json\__init__.py", line 357, in loads return _default_decoder.decode(s) File "C:\Python38\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Python38\lib\json\decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

So I tried to launch the script again and I get the same error message but this line [Previous line repeated 80 more times] changed to [Previous line repeated 24 more times] (it's a random number everytime).

I'm no JS programmer but my wild guess is that at some point, response.json() does not receive the right parameters, program does not know how to handle it and it causes it to stop.

Do you have any fix for this ?
Thank you !

Has this code been tested lately?

Hey,
I was wondering if it was possible to add an option in the submission script to retrieve the posts tags ? I believe it's currently : datetime,author,title,url,domain.

Also, despite me changing those two variables : TARGET_DATE = "2019-08-31" and sys.setrecursionlimit(20000)
it always download the last 100 submissions.
Downloading: 100 submissions
No more results.

Thank you.

I haven't made any modifications other than moving stopwords from assets to scripts (some issue with file permissions when i don't)

I see this error when running step3:

➜  scripts git:(master) ✗ python3 step3.py
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 2897, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'text_lower'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "step3.py", line 589, in <module>
    generate_most_common_entities_word_cloud(comments_df)
  File "step3.py", line 541, in generate_most_common_entities_word_cloud
    df = df[~df["text_lower"].isin(stopwords)]
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pandas/core/frame.py", line 2995, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 2899, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'text_lower'

the main func looks like:

if __name__ == "__main__":

    submissions_df = pd.read_csv("mexico-submissions.csv",
                           parse_dates=["datetime"], index_col=0)

    comments_df = pd.read_csv("mexico-comments.csv",
                              parse_dates=["datetime"], index_col=0)

    tokens_df = pd.read_csv("tokens.csv")
    entities_df = pd.read_csv("entities.csv")
    generate_most_common_entities_word_cloud(comments_df)

First of all, thank you for making this public :) Have always wanted to do something like this but have been lazy.

It would be great if you could put some simple instructions on how to install / use it without reading the source too much :P (like if I could see what you're running in cmd line)

Merry Christmas :)