gis_nicd_scraper about covid19za HOT 24 OPEN

And it's broken...

from covid19za.

We are unfortunately dealing with the fallout of digital vibes. Would NICD be better @lrossouw?

from covid19za.

It looked like they stopped doing them. But I see the 9th is available again. They skipped the 8th though.

from covid19za.

I also see the NICD page layout has changed. Captured all data manually until 9 June. Going to wait for the dust to settle before I update my scripts, but of course this week appears to be key in terms of stats that is coming out!

from covid19za.

I've created something new that would (hopefully!) be more stable. It collects from various sources so no longer posting the exact urls as the source. I'm flagging auto scraped results as source = "gis_nicd_scraper" as most of the data is scraped from public dashboard there, Don't have anything for vaccines yet.

from covid19za.

I scan two or three different dashboards for the figures. I typically can get the cases and tests the evening they are released from one dashboard (but they do break the dash from time to time). I usually seem to pickup the deaths and recoveries the next day (and the cases from this if the other dashboard is broken). Haven't solved the vaccines yet.

But these are more stable than scarping the pages of the media releases that change all the time.

from covid19za.

@lrossouw It seems the scraper has stopped working again? Before I do a manual update just want to check on it's status?

from covid19za.

Tx did not notice with all the other COVID-19 news out. Will have a look.

from covid19za.

Sometimes the Rt calculation is still running (someone else maintains that?) and it commits back and creates a conflict with my process. So my bot keeps updating my local repo but can't push until I manually resolve the conflict.

Not sure how to fix that.

Anyway it's resolved now.

from covid19za.

It might be this:
f8bfa83#diff-0e6e5c3c2330a562992a4157e9afb54fdea1938025dd074fec10a03e4e655aed

Can we make it pull before the push here as my bot might have made changes while this bot was running. That way it seems less likely to get into conflicts. @vukosim do you maintain that code?

from covid19za.

I will check late this evening. It runs after a change to the file.

from covid19za.

My bot posts new case data, Rt bot runs and then updates new data comes in and my bot update again while Rt is running. It creates a technical merge conflict but Rt bot uses --force so overwrites. Perhaps do a pull just before the push to bring the latest changes in. So you don't effectively reverse my or other changes. Rt bot has also reversed other data I captured manually before. I.e. I capture vaccine data while it's running and then it kind of reverse it.

from covid19za.

I did a manual update of Death and Recoveries today as they had not updated by mid-day.

#854

from covid19za.

Sorry just noticed now. Will sort it out.

from covid19za.

Fixed. @vukosim did you managed to update the bot?

from covid19za.

Good morning. The provincial data for confirmed cases for 2021-07-05 is missing. Should I add it manually or would you prefer to have the scraper make another pass and add it instead?

from covid19za.

Seems to be there (line 487):
https://github.com/dsfsi/covid19za/blob/master/data/covid19za_provincial_cumulative_timeline_confirmed.csv#L487

from covid19za.

Seems to be there (line 487):

https://github.com/dsfsi/covid19za/blob/master/data/covid19za_provincial_cumulative_timeline_confirmed.csv#L487

Louis, apologies if I missed something, will check my import again tonight and get back to you.

from covid19za.

Seems to be there (line 487):

https://github.com/dsfsi/covid19za/blob/master/data/covid19za_provincial_cumulative_timeline_confirmed.csv#L487

My mistake, sorry. Data for 05 July 2021 is indeed available in the confirmed cases file, but is missing from cumulative deaths and cumulative recoveries.

from covid19za.

Ah 5 July had issue I believe:https://www.nicd.ac.za/latest-confirmed-cases-of-covid-19-in-south-africa-05-july-2021/

They did not release provincial figures for deaths and recoveries on NICD site but I see now they are available here:
https://sacoronavirus.co.za/2021/07/05/update-on-covid-19-05th-july-2021/

Feel free to capture.

from covid19za.

My data source stopped providing deaths/recoveries in machine readable form on 30 July or so. What sources are people using?

from covid19za.

@lrossouw Does this problem apply to testing too? covid19za_timeline_testing.csv I don't know how I missed that this has not been updating since the end of July.

from covid19za.

Thanks @shaze yeah that needs to be updated.

from covid19za.

Also looping @krokkie seems we have a few more failures. So we might need again to sync between you and @lrossouw

from covid19za.