Cases and deaths still failing. about covid-rt-estimates HOT 20 CLOSED

No, just the specific location. I'll give a full batch run a go.

from covid-rt-estimates.

Mine looked to grow and shrink but I'll see once it runs a full suite.

from covid-rt-estimates.

I think that's because it wasn't returning the data. I'm currently running belgium with higher timeouts then those regional estimates are next on my hit list.

from covid-rt-estimates.

@joeHickson any ideas?

from covid-rt-estimates.

I have seen that message when the R worker thread gets killed off (only ever at my bidding with a kill 0000 to stop a runaway thread). Let me give it a spin. How long did the failure take?

from covid-rt-estimates.

info.log

Here is the log (cutting out the successful subnational estimates). Falls over after a few minutes for the first failure and then more rapidly. I agree that is the likely issue but as this was previously working I think the changes have introduced a problematic factor. You can see this sort of problem if you run out of RAM or disk space as well (the disk on this server seems fine).

from covid-rt-estimates.

Thanks @seabbs - that looks to be failing very quickly. Mine's still running so I'll see if I can reproduce it. Ram doesn't look like a problem on mine - currently 13.1 / 252G

from covid-rt-estimates.

Did you run starting with all the subnational estimates? It may be related to #25 which I think is due to RAM not being released properly when trying to sequentially run multiple processes in a single R script.

from covid-rt-estimates.

The server I have been running on has 126Gb of RAM so if each estimate uses around 10 ~ 15 Gb of RAM then it would make sense that issues appear after 10 or so estimates. If your box has 256Gb of RAM it may well be able to brute force it's way through this issue for now which would be good as the deaths Rt estimates are now a week out of date so could really do with an update.

from covid-rt-estimates.

Well done getting the automated update sorted @joeHickson. I am seeing some missing estimates for national/cases but nothing too bad. national/deaths and region/cases + region/deaths seem like they are being problematic though?

Let me know if there is something I can do to help with this. As not running regularly now I don't have access to the logs etc in order to diagnose problems.

from covid-rt-estimates.

National deaths should be coming through.
It looks like the region ones just haven't been added to the git repo yet.

from covid-rt-estimates.

I see what you mean about the national deaths - I'll have a deeper dig in the morning

from covid-rt-estimates.

it processed this run - I committed all the national data and left it running on the rest. I'll keep an eye on that.

from covid-rt-estimates.

So it looks like cases and deaths are now up to date but regional estimates are a problem (https://epiforecasts.io/covid/posts/global/) with only an estimate for Europe available.

I see the new issue flagging problem areas (#42) but can the logs be pushed somewhere (or some other summary metric) to allow me to keep track of your updates more easily than going through the GitHub to look at pushed locations?

from covid-rt-estimates.

you can get the summary from the runtimes.csv file in the root if that helps?

from covid-rt-estimates.

As far as I can tell it doesn't show anything for deaths, or regional estimates in general? Potentially due to some upstream failure?

from covid-rt-estimates.

region/deaths is there from this latest run but it's outputting in region/deaths/national which is definitely a bug (#43).
I have kicked off region/cases and will see what's there when I get back from lunch.
The deaths is now showing in the timings - I think there was an issue where the regional data sets were not requesting a return_timings from epinow2 which will hopefully resolve it (hotfix this week with they typo on the epinow2 call for return_timings and all_regions_summary )

from covid-rt-estimates.

Ah right all makes sense - good stuff!

from covid-rt-estimates.

Still seeing only Europe in the latest region outputs. Is this a timeout issue or something else? Checking runtimes.csv this looks like a timeout issue. Not sure what has changed in this data to introduce the issue as not seen this before to this extent.

Suggests to me that epiforecasts/covid-rt-estimates-infra#2 may be fairly urgent.

from covid-rt-estimates.

Resolved by updates from @joeHickson and epiforecasts/EpiNow2#73

from covid-rt-estimates.