A wrapper utility with few additions based off suhailidrees/autotrader_scraper
Clone the repo and run pip3 install -r requirements.txt
Place the configuration file under conf directory.
Refer to conf/sample.json
{
"make" : "Audi",
"model" : "A5",
"model_variant" : "",
"manufacturer_approved" : "",
"maximum-mileage":40000,
"max_price": 20000,
"postcode" : "SW1A 0AA",
"radius" : 1500,
"min_year" : 2016,
"max_year" : 2019,
"include_writeoff" : "exclude",
"max_attempts_per_page" : 5,
"verbose" : "False"
}
| Criteria / Argument | Type | Description | Values | Default Value |
|---|---|---|---|---|
| make | String | Make of the car | Get these from autotrader.co.uk*. Examples are "Audi", "BMW", "Jaguar" | "Audi" |
| model | String | Model of the car | Get these from autotrader.co.uk*. Examples are "A3", "A4", "A4" for Audi | "A5" |
| model_variant | String | Model Variant of the car | Get these from autotrader.co.uk*. Examples are "S line", "Black Edition", "A4" for Audi | "" |
| manufacturer_approved | String | Manufacturer Approved | Takes Y or blank | "" |
| maximum-mileage | String | Maximum Milage | Takes integer, eg. 40000 | "" |
| postcode | String | Postcode where you are searching | Example: "CB2 1TN", "NW1 2BH" | "SW1A 0AA" |
| radius | Integer | Radius of your search from the postcode | Can be any positive integer. Use 1500 for nation-wide search | 1500 (i.e. nation-wide) |
| min_year | Integer | Minimum year of the car's manufacture | Can be any positive integer | 1995 |
| max_year | Integer | Maximum year of the car's manufacture | Can be any positive integer | 1995 |
| include_writeoff | String | Whether or not to include insurance write-off categories (i.e. Cat S/C/D/N) | "include", "exclude", "writeoff-only" | "include" |
| max_attempts_per_page | Integer | Maximum times to attempt scraping a page. A request may fail due to connectivity issues, server response issues, etc. and so in the event of a failure, the request will be retried a number of times specified by this argument | Can be any positive integer | 5 |
| verbose | Boolean | Whether or not to print progress on the console. Good for debugging | True, False | False |
usage: scraper.py [-h] [--type TYPE] [--outfile OUTFILE] [--verbose VERBOSE] [--inputfile INPUTFILE]
optional arguments:
-h, --help show this help message and exit
--type TYPE output type csv or json
--outfile OUTFILE file prefix
--verbose VERBOSE verbose mode
--inputfile INPUTFILE
input file in json
./scraper.py --type csv --inputfile conf/bmw_1_series.json --outfile out/bmw_1.csv
or
./scraper.py --type json --inputfile conf/bmw_1_series.json --outfile out/bmw_1.json
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent