Nice! (not a real issue) about nowcast-hub HOT 5 CLOSED

Ah I see from your metadata you have in fact explored the variation over time - clearly need to check out your paper!

from nowcast-hub.

Bit hasty again I see you have explored nthreads

df85afd#diff-7ebb9d589758e49654f96070e578d174808be2d6cbabfbbb51fddefc22ab27c2

from nowcast-hub.

Also interesting to see how much better your approach begins to look vs others over time as I think most are truncating the reporting triangle quite a bit more than this (I'm truncating at 40 days for example so expect to underpredict long term).

from nowcast-hub.

Wrapping up (😆) I'd be very interested to know how your method performs fit independently to each data stream (so to each age group and location).

In my work also been exploring hierarchical nowcasts (for age group) and annoyingly every permutation of model I have looked at so far has performed quite a lot worse than fitting independently. As coding myself I don't currently have access to splines so variation is via random effects and random walks and my findings could be down to a coding error.

Looking at my very preliminary evaluation of our methods I see some real outliers in certain states in terms of performance for your model. Obviously far far to early to tell and it could be you are just capturing very delayed reporting but potentially suggests your method may have the same issue as mine when it comes to independent vs joint fitting.

Evaluation of our methods
Evaluation of my model permuations (note the independent model outperforms all the others all of the time

from nowcast-hub.

Hi Sam, thanks! That's a lot to process. Here some replies:

1. mgcv vs Stan: I also have some crude implementations of P-splines in Stan in this context, but I certainly agree that mgcv is much faster and more transparant.
2. Yes, we considered calender time x delay interactions in our orignal paper (2019). However, the issue with such spline interactions is that extrapolations become very unstable. Hence, we had to apply several constraints. For the nowcast hub I try to keep it simple.
3. Indeed I used nthread in bam, but for largers values, say >=4, it gave different results across different machines. Therefore I set nthreads = 1. That gave the same outcomes everywhere. You can imagine I got a bit worried before I figured that out...
4. The raw reporting triangle data is truncated at 81 days. I am currently experimenting with truncating things to, say 40 - 50 days, just to see what is happening. We should not throw these delays away of course, but aggregate them in a >= 40 category. I am wondering if this would improve my fits, because now the RIVM model systematically overestimates the hospitalizations for 28 to ~7 days before the nowcast date.
5. Have not tried that yet, but of course I could fit everything independently for each age group and location. Currently it is done by random effects (hierarchically or "shared parameters" so to say).
6. So nice to see your evaluation in action. Yes, a few states do perform very badly! I will try to figure out what is going on there.

Again, thank you very much for you comments and suggestions!

Jan

from nowcast-hub.