HTTPError: HTTP Error 429: Too Many Requests about sportsipy HOT 21 OPEN

I'm having the same issue with NHL and NFL, but NBA was working for me

from sportsipy.

I'm having the same issue with NHL and NFL, but NBA was working for me

Yeah Im using NFL. I tried NBA and it seemed to work fine.

from sportsipy.

Seems like SportsReference recently introduced a limit on requests to their site if I understand correctly? This may be the issue, but I had been using the same code past few days and after October 26th and was working fine

https://www.sports-reference.com/bot-traffic.html

from sportsipy.

Seems like SportsReference recently introduced a limit on requests to their site if I understand correctly? This may be the issue, but I had been using the same code past few days and after October 26th and was working fine

https://www.sports-reference.com/bot-traffic.html

Same here but that's probably what's happening. Pretty much renders sportsipy useless.

from sportsipy.

Going to try adding time.sleep(60) in between requests for roster/boxscore data, let me know if you have any success with workarounds as well

from sportsipy.

@roclark let us know if you have any ideas as well

from sportsipy.

@mattpfreer adding the timer worked!

from sportsipy.

awesome to hear, did you add it before each single roster request or into the actual sportsipy py files? still having trouble on my end but have used a counter to try getting 10 at a time before doing sleep, seems like i may need to add this before each individual roster request

from sportsipy.

awesome to hear, did you add it before each single roster request or into the actual sportsipy py files? still having trouble on my end but have used a counter to try getting 10 at a time before doing sleep, seems like i may need to add this before each individual roster request

I added it within a function I defined that calls Schedule and Team info several times for previous years and the current year for two separate teams. I think going forward I'm going to just save them as a csv file and just update the current year as needed.

from sportsipy.

got it, thank you! hopefully there will be a resolution in the future but makes sense.

from sportsipy.

Yeah this is a problem rendering the API useless for my use case in NCAAB. I can't even run a Teams() call.

Any recommendations? I see the time.sleep option, but wow this will take some time to run for 300+ teams.

from sportsipy.

Everything seemed to be working well the past few days, especially after adding in the timer in a few spots, but today I called Teams() and got the error. I hadn't run anything else.

from sportsipy.

The Team function was working for me previous weeks, but it stopped yesterday. I tried running it on different IP addresses and different online notebooks and it doesn't work.

I am a novice, but could it be that someone is spamming NFL requests through the API which is causing the error?

from sportsipy.

The Team function was working for me previous weeks, but it stopped yesterday. I tried running it on different IP addresses and different online notebooks and it doesn't work.

I am a novice, but could it be that someone is spamming NFL requests through the API which is causing the error?

Not sure. I experimented with NHL and CBB a little bit and they worked but at this point if the Teams function for NFL isn't going to work it's pretty much rendered the API useless for me.

@mattpfreer is it working for you?

from sportsipy.

I was able to get NCAAB to work by editing the teams.py file and commenting out the code that uses Conferences data, seems like that is the only problematic piece

from sportsipy.

I was able to get NCAAB to work by editing the teams.py file and commenting out the code that uses Conferences data, seems like that is the only problematic piece

Would you be able to share what lines you commented out to get it to work? Thanks

from sportsipy.

The problem with Teams is it quickly fires off as many requests as there are teams, which almost immediately violates the new request limit at sports reference. I fixed it on my fork. You can see the change here. I just created a new utility function that adds a time.sleep after each url request via pyquery.

As long as you don't run multiple sportsipy commands in parallel this fix should guarantee you won't exceed the new limit. Obviously it's not ideal for NCAAB since fetching 300 teams will take ~15 minutes, but at least it works.

from sportsipy.

I mean I'm also having an issue to boxscores, which is more concerning.

My teams fix is a time.sleep(13) in _retrieve_all_teams, around line 1130.

I'm trying to train a model with boxscores back to '07. Querying each game individually with a 13sec sleep would take ~9 days...

Does anyone here happen to have a cache / boxscore info that goes back from last season to some year? Any help is much appreciated!!

from sportsipy.

Does anybody have a good solution for boxscore.py?

from sportsipy.

Does anybody have a good solution for boxscore.py?

The best way I've found to fix the it is to intentionally limit the rate of all calls to the website like I mentioned in my post above (check out this PR for specific details). I don't think your use case of bulk downloading data for several years at once is really an option anymore unless you're willing to wait a really long time for the results. You'll be better off scheduling a long running job and caching all the historical data you intend to re-use.

from sportsipy.

Going to try adding time.sleep(60) in between requests for roster/boxscore data, let me know if you have any success with workarounds as well

Just ran into this problem as well. I created a model to predict the winning teams each week in the nfl and wanted to update my ultimate CSV and ran into this issue. The timesleep worked perfectly for me. I wish sports reference would just create an API directly

from sportsipy.