bmln / job_scraper Goto Github PK

why didn't we call parse with a param to begin with?

possible options to adjust main args:

• https://stackoverflow.com/questions/36166225/using-the-same-option-multiple-times-in-pythons-argparse
• https://gist.github.com/jirihnidek/3f5d36636198e852280f619847d22d9e (https://stackoverflow.com/questions/10448200/how-to-parse-multiple-nested-sub-commands-using-python-argparse)

dont set SCRAPY_SETTINGS_MODULE

its not working probably:

• setting the scrapy cfg correctly will load the module
• setting the ENV VAR with the same properties wont load the module

instead:
cd src && scrapy crawl <spider_name>

• API_PAGESIZE from settings
• requests currently only take non-mobile user agent headers
• parse_api is missing a stop indicator

group the spider arg params

need to parse all <script> tags?