Add Video Prediction models about satflow HOT 13 OPEN

Pix2Pix does okay trying to predict the 12 satellite channels for the next half hour from the past 20min of data. I tried just half an hour in the future as it was easier to visualize and see what is happening, but plan on going up to 4 hours in the future soonish. This implementation is simple in that it takes a single input image, where the timeseries of images is just stacked along the channel dimension as input, and outputs a multi-channel output image in the same format, so each 12 channels is a single timestep.

These are the generated images after 9 epochs of 2000*16 input image stacks.

And this is the ground truth for that same timestep

As can be seen, Pix2Pix does get some of the structure present, but is very grainy looking, and fails on the last 2 channels. It does a bit better with this one:

Generated:

Target:

from satflow.

Yeah, in the Zarr file that I've been using, all 12 SEVIRI channels have been down-sampled to something like 4 km resolution.

We have a bunch of code for spatially re-projecting the raw SEVIRI data, and converting to Zarr, if that's useful.

from satflow.

Yeah, I've been looking at that code and messing around with it currently, just working out the best way to integrate all the data together

from satflow.

Cool beans. I think the most recent reprojection code is in the satip project.

I'd be tempted to start simple, and start with just one (or a small number) of satellite channels. (And perhaps ignore the High Res Visible channel to start with. I've heard some people suggest that HRV might be one of the less informative channels when it comes to nowcasting clouds, and estimating irradiance at the Earth's surface! But I could be wrong!) That said, I can totally see the attraction of designing a data pipeline from the ground-up that can handle multiple, diverse datasets; so go for it if you want to start there!

from satflow.

Another one is https://github.com/FrontierDevelopmentLab/PyRain which does global rain forecasting, mostly seems to be just ConvLSTM, but could have some good stuff on data preparation, etc. Or this one, which the code doesn't seem to be public, but they use a GAN for thunderstorm prediction http://spaceml.org/repo/project/60002402f5647f00129f7287

from satflow.

https://github.com/CNALeon007/TrajGRU/blob/master/TrajGRU.py Trajectory GRU is also interesting, built specifically for precipitation forecasting, it could improve the tracking of clouds through time.

from satflow.

Perciever IO now is also available for predicting dense predictions: https://github.com/deepmind/deepmind-research/tree/master/perceiver

from satflow.

The perceiver IO is now also implemented in PyTorch here: https://github.com/lucidrains/perceiver-pytorch, openclimatefix/perceiver-pytorch#1

from satflow.

PyTorch Video has some neat built in ones, and it might also be helpful to try out https://github.com/facebookresearch/AVT and https://github.com/facebookresearch/SlowFast

from satflow.

MetNet's work with axial self-attention did really well with precipitation forecasting of up to 8 hours later, and use axial self-attention, which there is a PyTorch implementation here: https://github.com/lucidrains/axial-attention

Interestingly, MetNet is compared to optical flow methods as well in the paper, and does outperform it. Additionally, one important point is the need for a large spatial area around the target of interest to capture changed moving both into and out of the target area, which it does by having the center 64kmx64km crop, at full resolution, then downsampling the 1024x1024km area around the crop. It seems like trying out this approach, because of the similar type of problem makes the most sense as a first try. Trying out different amounts of historical data would also be interesting. MetNet uses the previous 90min at 15min intervals for prediction, since we have 5min observations, we could see if including different number of frames from the previous 90min, or focusing on more recent data (say previous 45min?) might matter more for the next 4 hours.

from satflow.

The 12 spectral bands from SEVRI are not all the same resolution, with the non-HRV being 3km resolution, and HRV being 1km, so need to figure out best way to resample that or potentially just put the non-HRV values at 9 pixels in the HRV image, easily enough.

from satflow.

After the chat today, seems like a good one to try first would be UNET, MetNet, then something like DVD-GAN. Ideally, UNET should get fairly decent results. Also, try out deeplabv3 .

from satflow.

According to https://www.climatechange.ai/papers/icml2021/54 GANs show promise for predicting future frames from current ones, the code isn't available, but they seem to use a Recurrent Unet + cGAN in their slide.

from satflow.