codeforafrica-scrapers / taxclock_ke_news Goto Github PK

Currently when we request for a page and the website is down or unreachable, the scraper fails. It should instead just log an error and move to the next website to scrape.

We are currently only scraping the Standard Newspaper's stories but we should also scrape data on other newsrooms that are doing financial, business, and budget stories.

• Business daily
• Nation.co.ke
• The Star
• Capital FM

The Standard scraper is currently being initialized and run different from the rest. We should instead strive for uniformity.

NOTE: This is in reference to the develop branch

Improve how we are doing logging including using our slack logger code - https://github.com/CodeForAfricaLabs/python-slack-logger

Related: #18

Currently we have each news outlet as it's own separate JSON.

For our purposes of having the stories displayed on https://taxclock.codeforkenya.org/ We should combine them into a single data/news.json that is sorted by time and limited to 7 stories.

Currently the scraper is only pushing data to S3. We should instead look at having the data saved locally when S3 isn't set for local development.

We should check on Nation scraper as there is no data being saved to the JSON.

Some of the stories we scrape might not have images so we should link through to the following placeholder - https://github.com/CodeForAfrica/TaxClock/blob/kenya/img/placeholder.png