This project utilizes Scrapy to get data on used cars from auto.ria.com
and saves it into PostgreSQL database.
- The code responsible for parsing is in
autoria_parser/spider/car_parser.py
- All database actions are handled in
autoria_parser/pipelines.py
- In
autoria_parser/celery.py
Celery settings for scheduled tasks are stored - In
autoria_parser/log.py
Logger settings are stored
Overview:
POSTGRES_HOST=db
Hostname for PostgreSQL DatabasePOSTGRES_DB=cars
Name of the DatabasePOSTGRES_USER=postgres
UsernamePOSTGRES_PASSWORD=postgres
Password for the userPOSTGRES_PORT=5432
Database port, for PostgreSQL default is 5432RABBIT_URL=amqp://guest:guest@rabbitmq3:5672/
RabbitMQ broker urlPAGES=10
Number of pages to be scrapedPARSE_HOUR=12
For daily parsing: hourPARSE_MINUTE=0
For daily parsing: minutesDUMP_HOUR=12
For daily database dump: hourDUMP_MINUTE=0
For daily database dump: minutes
To run the project follow next steps:
-
Fork the repository
-
Clone it:
git clone <here goes the HTTPS link you could copy on github repositiry page>
-
Create a virtual environment:
python3 -m venv venv
-
Acivate venv:
- MAC
source venv/Scripts/activate
- Windows
cd venv/Scripts/activate
->. activate
- Create .env file:
- You can copy .env.sample if you are going to use Docker.
- Launch the project:
- With Docker:
docker-compose up --build
- Locally:
scrapy crawl car_parser