This is the part 1 of the series of Restaurant Recommender Story. This project aims to crawl the number of restaurants in the first page of each district
Source: www.openrice.com Automate the process to retrieve all restaurant information The restaurant information is organized by district There is pagination in each district page Challenge: The number of restaurants in the first page of each district is different
Bee-Queue (https://github.com/bee-queue/bee-queue) Redis Firebase
- queue server definiation (queue-processor.js)
- database (database.js)
- server instance (app-1.js)
- submit job client (count-restaurant-per-page.js)
- start the queue server
node ./restaurant-per-page/index.js
- submit the counting jobs
node ./restaurant-per-page/count-restaurants-per-page.js
Result is saved in Firebase
- start the queue server
node ./pages-per-district/index.js
- sumbit the jobs
node ./pages-per-district/solve-page-count-per-district.js