Comments (12)
I am modifying it as follows.
wg := sync.WaitGroup{}
for tweet := range scraper.GetTweets(context.Background(), usr, nbrs) {
time.Sleep(2 * time.Second) // Sleep after each tweet is fetched
if tweet.Error != nil {
wg.Wait() //Wait to prevent program from exiting during image acquisition
fmt.Println(tweet.Error)
os.Exit(1)
}
if vidz {
time.Sleep(5 * time.Second) // Extra long sleep when acquiring video
wg.Add(1)
go videoUser(&wg, tweet, output, retweet)
}
if imgs {
time.Sleep(1 * time.Second) //A little extra sleep during image acquisition
wg.Add(1)
go photoUser(&wg, tweet, output, retweet)
}
}
wg.Wait()
from twitter-media-downloader.
@shunnag you can also use: scraper.WithDelay(2)
instead of time.Sleep(2 * time.Second) // Sleep after each tweet is fetched
.
but for:
if tweet.Error != nil {
+ wg.Wait() //Wait to prevent program from exiting during image acquisition
fmt.Println(tweet.Error)
os.Exit(1)
}
if vidz {
+ time.Sleep(5 * time.Second) // Extra long sleep when acquiring video
wg.Add(1)
go videoUser(&wg, tweet, output, retweet)
}
if imgs {
+ time.Sleep(1 * time.Second) //A little extra sleep during image acquisition
wg.Add(1)
go photoUser(&wg, tweet, output, retweet)
}
}
did you tried ans Is it working?
from twitter-media-downloader.
I am also encountering this issue and was thinking of having a delay function. which version is working best for you guys ?
I'd wait for a new release if @mmpx12 will incorporate it otherwise I may try to add it. though i'm new to using github for software so i might struggle
from twitter-media-downloader.
@Volvalzark english plz
from twitter-media-downloader.
Sorry, I thought I sent it in English, but I sent it in Japanese. Is it possible to set a delay in seconds for API requests to solve this problem?
from twitter-media-downloader.
Same issue there, only happens once logged
from twitter-media-downloader.
Since i'm using go routine, i first need to set the number of go routines it used before setting up a delay (without adding delay will change nothing).
from twitter-media-downloader.
I don't know the appropriate number of go routines, so I can't really say, how many is appropriate?
from twitter-media-downloader.
“response status 429 Too Many Requests: Rate limit exceeded”
Yes, it's easy to see this!
from twitter-media-downloader.
@mmpx12
It appears to be functioning correctly. There is a 2-second delay applied to all API accesses, and an additional delay is added when fetching tweets with the "imgs, vids" option enabled, even if the tweet itself doesn't contain any images or videos. We could potentially adjust the numbers to align with the 429 rate limit in order to increase the acquisition speed, but for now, it's running smoothly without causing errors, albeit at a slower pace.
from twitter-media-downloader.
@shunnag This should be added with a flag like '-S|--slow'. I use a different proxy every time I run twmd (not for a single tweet, but for -a, -v, or -i) so I don't encounter 429 error.
Another option should be to add '-f|--fast' to enable goroutine. But I'm not sure if without goroutine and without sleep, 429 error would occur.
from twitter-media-downloader.
@mmpx12
like this?
- var nbr, single, output string
+ var nbr, single, output, delay string
var retweet, all, printversion, nologo, login, twofa bool
op.On("-B", "--no-banner", "Don't print banner", &nologo)
+ op.On("-d", "--delay SECOND(S)", "Use specified seconds delay for API access", &delay)
op.Exemple("twmd -u Spraytrains -o ~/Downlaods -a -r -n 300")
wg := sync.WaitGroup{}
+ if delay == "" {
+ delay = "0"
+ }
+ delay64, _ := strconv.ParseInt(delay, 10, 64)
+
for tweet := range scraper.GetTweets(context.Background(), usr, nbrs) {
and use
time.Sleep(time.Second * time.Duration(delay64))
in loop
or
scraper.WithDelay(delay64)
from twitter-media-downloader.
Related Issues (20)
- I can not log in HOT 2
- Download ends midway. HOT 1
- Multiple images and videos downloaded are 0KB or incomplete HOT 1
- Can't type in password at all HOT 12
- Any request results in "response status 404 Not Found:" HOT 4
- How to login HOT 10
- Not launching ? HOT 4
- can't download all video or image HOT 3
- Freezing on login HOT 1
- Despite user/pw being correct, I always get error HOT 2
- Can't login into account HOT 7
- Can’t download video HOT 3
- how to use login feature?
- Filename can't contain any colon character, while using the {TITLE} formater. #45
- Download text tweets? HOT 1
- can't download anything HOT 1
- Can download pictures from an user confirmed in any timeline,from XXX to XXX days,include pictures from other answers under the topic?
- doesn't support win7? HOT 1
- Not works HOT 2
- Does not work HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from twitter-media-downloader.