response status 429 Too Many Requests: Rate limit exceeded about twitter-media-downloader HOT 12 OPEN

I am modifying it as follows.

	wg := sync.WaitGroup{}
	for tweet := range scraper.GetTweets(context.Background(), usr, nbrs) {
		time.Sleep(2 * time.Second) // Sleep after each tweet is fetched 
		if tweet.Error != nil {
			wg.Wait() //Wait to prevent program from exiting during image acquisition
			fmt.Println(tweet.Error)
			os.Exit(1)
		}
		if vidz {
			time.Sleep(5 * time.Second) // Extra long sleep when acquiring video 
			wg.Add(1)
			go videoUser(&wg, tweet, output, retweet)
		}
		if imgs {
			time.Sleep(1 * time.Second) //A little extra sleep during image acquisition
			wg.Add(1)
			go photoUser(&wg, tweet, output, retweet)
		}
	}
	wg.Wait()

from twitter-media-downloader.

@shunnag you can also use: scraper.WithDelay(2) instead of time.Sleep(2 * time.Second) // Sleep after each tweet is fetched .

but for:

		if tweet.Error != nil {
+     		wg.Wait() //Wait to prevent program from exiting during image acquisition
			fmt.Println(tweet.Error)
			os.Exit(1)
		}
		if vidz {
+     		time.Sleep(5 * time.Second) // Extra long sleep when acquiring video 
			wg.Add(1)
			go videoUser(&wg, tweet, output, retweet)
		}
		if imgs {
+      		time.Sleep(1 * time.Second) //A little extra sleep during image acquisition
			wg.Add(1)
			go photoUser(&wg, tweet, output, retweet)
		}
	}

did you tried ans Is it working?

from twitter-media-downloader.

I am also encountering this issue and was thinking of having a delay function. which version is working best for you guys ?
I'd wait for a new release if @mmpx12 will incorporate it otherwise I may try to add it. though i'm new to using github for software so i might struggle

from twitter-media-downloader.

@Volvalzark english plz

from twitter-media-downloader.

Sorry, I thought I sent it in English, but I sent it in Japanese. Is it possible to set a delay in seconds for API requests to solve this problem?

from twitter-media-downloader.

Same issue there, only happens once logged

from twitter-media-downloader.

Since i'm using go routine, i first need to set the number of go routines it used before setting up a delay (without adding delay will change nothing).

from twitter-media-downloader.

I don't know the appropriate number of go routines, so I can't really say, how many is appropriate?

from twitter-media-downloader.

“response status 429 Too Many Requests: Rate limit exceeded”
Yes, it's easy to see this!

from twitter-media-downloader.

@mmpx12
It appears to be functioning correctly. There is a 2-second delay applied to all API accesses, and an additional delay is added when fetching tweets with the "imgs, vids" option enabled, even if the tweet itself doesn't contain any images or videos. We could potentially adjust the numbers to align with the 429 rate limit in order to increase the acquisition speed, but for now, it's running smoothly without causing errors, albeit at a slower pace.

from twitter-media-downloader.

@shunnag This should be added with a flag like '-S|--slow'. I use a different proxy every time I run twmd (not for a single tweet, but for -a, -v, or -i) so I don't encounter 429 error.
Another option should be to add '-f|--fast' to enable goroutine. But I'm not sure if without goroutine and without sleep, 429 error would occur.

from twitter-media-downloader.

@mmpx12
like this?

-	var nbr, single, output string
+	var nbr, single, output, delay string
	var retweet, all, printversion, nologo, login, twofa bool

	op.On("-B", "--no-banner", "Don't print banner", &nologo)
+	op.On("-d", "--delay SECOND(S)", "Use specified seconds delay for API access", &delay)
	op.Exemple("twmd -u Spraytrains -o ~/Downlaods -a -r -n 300")

	wg := sync.WaitGroup{}

+	if delay == "" {
+		delay = "0"
+	}
+	delay64, _ := strconv.ParseInt(delay, 10, 64)
+
	for tweet := range scraper.GetTweets(context.Background(), usr, nbrs) {

and use
time.Sleep(time.Second * time.Duration(delay64)) in loop
or
scraper.WithDelay(delay64)

from twitter-media-downloader.