mcdallas / gert Goto Github PK

This looks like an awesome project! I'm gonna use it for wallpaper scraping and stuff.

Maybe I'm doing something wrong, but whenever I use the -l flag and add any kind of limit, it always limits it to 15 pictures. If I don't use any kind of limit, it downloads 43 images rather than the 25 that should be the default.

I'm having some problems getting this program to work (using 0.2.2, since I'm on windows). When I run a bunch of jobs without using the .env auth file, I eventually get this error: [2023-09-09T07:48:18Z INFO gert] Starting data gathering from Reddit. This might take some time. Hold on.... thread 'main' panicked at 'Failed to parse JSON: reqwest::Error { kind: Decode, source: Error("missing field kind", line: 1, column: 46) }', src/subreddit.rs:40:93 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace I'm assuming this is some kind of rate limiting, although it shouldn't be making that many requests (roughly 100 subs, top 10 of each). The error happens about half way through.

When I try to use my env file, I get this error: Error: ReqwestError(reqwest::Error { kind: Decode, source: Error("missing field access_token", line: 1, column: 26) }) I have double and triple checked that my gert.env file has all the correct information from the template provided. Any help would be greatly appreciated.

Downloading from redgifs is currently broken, as redgifs requires api authorization for all requests. Adding support for this would be appreciated.

https://github.com/Redgifs/api/wiki/API-access

The -H flag seems to occasionally fail to write a file, because the filename would have an illegal character for a windows file name. Some kind of check for any of these banned characters and replacing them with a placeholder would be appreciated, as it currently just skips that file.

Let's say I just want image media type but not videos, is this possible to implement?

Only download around 24 per subreddit

Unsure if this effects other hosts or not, but redgif videos do not have sound for me. They are being converted to mp4s with ffmpeg, unsure how they would lose sound from the gifs however. It looks like redgif now defaults to muted when watching in a browser, perhaps there was some api change as well? A fix would be greatly appreciated.

It would be nice if there was an option to supply a text file containing reddit links to the app to iterate over

Currently the only way is to loop over the contents of the text file, calling gert on each individual line, which is inefficient due to having to log in and initialise gert each time

Hello,
I was wondering if you could create a new release that includes your most recent changes?

Would it be possible to download all posts of a particular user (not a subreddit)?

From what I understand, {}.json is also available for u/ pages, and it seems to have a similar (the same?) format as the subreddits