Utilities for scrapy
Make OAuth authentication with Github API token
Watch GitHub Rate Limit in GitHub API header and close the connection for crawling if Rate Limit is over the limit user set in Scrapy setting file.
Get Github API OAuth token. Please refer to Github Developer OAuth or Github Developer Getting Started
Create the middlewares
directory under your spider folder.
mkdir -p <your_spider>/middlewares
Copy github_oauth.py under the middlewares
directory.
git clone https://github.com/yu74n/scrapy_utils.git
cp scrapy/downloadermiddlewares/github_oauth.py <your_spider>/middlewares
Open your spider's setting.
vi <your_spider>/settings.py
Add Middleware name to DOWNLOADER_MIDDLEWARES
in settings.py like
DOWNLOADER_MIDDLEWARES = {
'<your_spider>.middlewares.github_oauth.GithubOAuthMiddleware': 290,
}
Set your Github API token to oauth_token
variable in your spider code
class YourSpider(scrapy.Spider):
...
oauth_token = "xxxxxxxxxxxxxxxxxxxxxxxxx"
...
def start_requests(self):
...
Create the extensions
directory under your spider folder.
mkdir -p <your_spider>/extensions
Copy github_oauth.py under the extensions
directory.
git clone https://github.com/yu74n/scrapy_utils.git
cp scrapy/extensions/github_rate_limit_watcher.py <your_spider>/extensions
Open your spider's setting.
vi <your_spider>/settings.py
Add Extension name to EXTENSIONS
in settings.py like
EXTENSIONS = {
'<your_spider>.extensions.github_rate_limit_watcher.GithubRateLimitWatcher': 500,
}
X-RateLimit-Remaining
(please refer to Rate Limit) describes the number of requests remaining you can send with GitHub API.
When X-RateLimit-Remaining
reaches GITHUB_RATE_LIMIT_WATCHER_LIMITCOUNT
, your crawler will be closed.