To make the models more useful potentially and easier to import/use in other contexts, these models, which are the best performing ones so far, should be split into their own repos. Then all model development can be done there, in a singular space, and just wrapped with SatFlow specific bits in this repo.

• Update https://github.com/openclimatefix/perceiver-pytorch with SatFlow Perceiver/encoder/decoders
• Move Skillful Nowcasting GAN to own repo
• Move MetNet implementation to own repo

One idea is to use just clouds, and not the ground for training and inference, so that the model can't get messed up by the ground. EUMETSAT does have binary cloud masks for the RSS, which should allow us to mask out the ground and only look at the movement of clouds. One potential issue is if new clouds develop or disappear, there could be useful information in the ground, or topology of the area that would be discarded, but we could try just using an elevation map as well to see. Related #1

Not sure if this is the right place to post this (!) (maybe we should use github discussions?!?) but here's an interesting critique of MetNet by @NiklasRiewald: https://niklasriewald.com/2021/06/29/paper-review-metnet-a-neural-weather-model-for-precipitation-forecasting/

Add creating video visualizations with the output from SatFlow models, possibly overlaying the cloud mask as well. The main idea is for the HuggingFace Spaces visualization (https://huggingface.co/spaces/openclimatefix/MetNet) so that we can have interactive widgets showing off the models we are training!

@peterdudfield Would the visualizations you've been making work for making videos?

While having a network predict super-resolution images from just the geo satellite images is probably not helpful, as it would be making up data that does not exist there, one option that could be kinda cool and useful could be to have it create a super-resolution image using data that we do have in higher resolution. For example, while the sat images are roughly ~3km-4km resolution, we can get radar data at 1km resolution, and probably NWP at 1km resolution as well, so combining that data with the lower resolution images, and higher resolution images from other satellites, such as Sentinel or Landsat or something, could work? And would not just be a network extrapolating entirely from just the low resolution satellite image.

PyTorch recently came out with this https://github.com/Distributed-AI/PipeTransformer that sped up training transformers and potentially using less GPU resources, or at least train faster.

Probably a very stupid issue, but there seems to be a problem when wrapping torch layers with the spectral_norm in that PyTorch complains of in-place operations and crashes the training. Spectral normalization seems quite important for the model, so definitely need to get it working.

The webdataset works decently well, but it still takes awhile for the timesteps to be loaded, cropped, normalized, etc. Prepreparing some cutouts could be quite useful, and make the dataloading process a lot quicker. Currently, nearly all my experiments use the last half hour of images to predict the next 1-4 hours, so just doing the processing offline would probably work better. Webdataset actually recommend that too, and recommends AIStore for ETL on clusters, etc. Not there yet in terms of need, but this could lead to a lot faster loading. Also can then be sure validation set is identical everytime.

Use Pixel Shuffle and other ones instead of the standard transpose convolutions, self-attention in the lowest downsample layer, add upsample with blur

After looking over pytorch lightning some more, and seeing they also have a DeepSpeed plugin, changing the training and models to be lightning models seems like a good idea. Should reduce work a bit, and hopefully fiddling with DeepSpeed too much to get the memory savings.

Just MAE or MSE or most normal performance metrics might show that optical flow performs better on average than an ML based approach, even if the ML model outperforms the optical flow in rarer situations or more complicated situations. So something like the MSE per hour, or something other than MSE over all examples might allow us to see where optical flow fails and ML models do well.

Want to switch to using the more complete and less ad-hoc nowcasting dataset for training SatFlow now. Additional bonus of then just having a single data pipeline for SatFlow and PV yield.

When using conda to install rasterio, cfgrib, cartopy, satpy and Pytorch, there are inconsistencies in requirements and, if Pytorch is then installed, torch.cuda.is_available() returns False. The workaround right now is that I have two conda environments, one to preprocess the raw satellite and DEM data to Geotiff and numpy arrays, then switch to a different conda environment, the one in environment.yml, for actual training/experimenting/etc.

Detailed Description

We want a model that can predict both satellite imagery and pv yield.

Context

While we ultimately just care about the PV output of the model, predicting where the clouds are moving in the satellite imagery should be a good auxiliary task that might help the model learn what is important vs not.

Possible Implementation

Add another version of the Perceiver that takes two queries, one for the satellite imagery and one for PV output.

https://ojs.aaai.org/index.php/AAAI/article/view/6819 is one example, which look promising for predicting furture images

Various ones to try include https://github.com/thuml/predrnn-pytorch and ConvLSTMs, and potentially GANs as shown in https://arxiv.org/abs/2104.00954

Get the base utils, like BaseModel and the HF integration, possibly registering models? Losses? Custom layers, like CoordConv? into a utils repo

This reduces duplication, and allows for both to predict_pv_yield and satflow to benefit from development on visualizations, etc.

Detailed Description

To know how changes are affecting model performance, it could be nice to be able to run a model with commands from PRs and record how it compares to the baseline/other models

GitHub has some actions that can be accomplished to do that: https://github.blog/2020-06-17-using-github-actions-for-mlops-data-science/

Context

We need to compare our models to other PV predictions, and so this could be a nice way to record some model performance, in addition to Neptune.ai

Possible Implementation

https://github.blog/2020-06-17-using-github-actions-for-mlops-data-science/
https://github.com/machine-learning-apps/actions-ml-cicd

Based off an idea in TimeGAN and TempoGAN, have two discriminators, one which focuses on discriminating between a single generated timestep and the real timestep, and a second one that compares encodings of all the generated and real timesteps, which should help with enforcing temporal consistency.

Hey Jacob,

I had a random idea about predicting satellite images... it's probably really silly!

Background

The idea is slightly inspired by watching Yannic Kilcher's video about Alpha Fold 2 last night (whilst drinking a couple of glasses of wine, so I might have gotten over-excited!). It turns out that, when predicting protein structure, it's really useful to compare the amino acid sequence you're interested in with amino acid sequences from other species. (DeepMind don't take credit for this idea - this was known before DeepMind began work on Alpha Fold). By looking at which amino acids have been conserved by evolution, you can make some guesses about the 3D structure of the protein. So, at inference time, before Alpha Fold does any clever ML, it searches through terabytes of genetic databases to find "similar" amino acid sequences, and then uses those "multiple sequence alignments" (MSAs) as an input to the ML model (along with the amino acid sequence you're interested in, of course). Alpha Fold 1 extracted hand-crafted features from those MSAs. Alpha Fold 2 instead uses self-attention to figure out what's most important from those other sequences (no hand-crafted features).

In forecasting, people use vaguely similar tricks. In the National Grid ESO control room today, they manually pick a couple of "similar" days from the recent past to help them predict national electricity demand (here's a screenshot from their "Demand Predictor" that National Grid ESO shared the day after the England vs Denmark Euro2020 football game:)

And the forecasting literature talks about "analog ensembles" (e.g. Monache et al. 2013) where they trawl through historical weather data to find "analogs" (similar events) to the weather you're trying to forecast.

The idea

Which got me thinking that maybe we could use a vaguely similar trick for predicting how clouds are going to move & change over the next few hours: Maybe there's a way - at inference time - to find "similar" image sequences from the entire historical dataset, and then extract useful information from those "similar" sequences to help the model predict the future.

But I have little idea how to actually do this!

The conceptually simplest way to do this might be to do self-attention over the entire historical dataset, but that's orders of magnitude more data than any attention mechanism can handle (100,000 timesteps x 128 pixels x 128 pixels x 12 channels)!

But maybe there's a middle-ground... Maybe we hand-code filters to select, say, 5 "vaguely similar" sequences from the historical dataset (maybe we only consider historical data from the same season and same geographical location, and roughly the same time of day, and maybe also consider numerical weather prediction variables like mean temperature, wind speeds, humidity, etc.). Or maybe there's a way to learn some or all of those filters. Or, like in self-attention, pre-compute a "key" (computed by a learnt neural network) for each image sequence, which we can compare to a "query" at inference time.

But, even once we've got a set of, say, 5 "vaguely similar" sequences, I'm not entirely sure how best to let the ML model extract useful info from those sequences (not least because these "similar" sequences won't be perfectly aligned in space or time; far from it, in fact). Maybe we could just use a timeseries model which gets one timestep at a time, and each timestep has a frame from each "similar" sequence concatenated as channels (but that's unlikely to work because the sequences won't be perfectly aligned in time). Or maybe something like The Perceiver could handle 5 "similar" image sequences, each image at 128x128 (for a total of 81,920 input dimensions per timestep!) But The Perceiver can't predict a full image (let alone a full image sequence) so maybe the output from The Perceiver would condition the generator in a GAN?!? (This is getting complicated!)

Anyway... sorry for this epic message! I'm just thinking out loud. And, of course, absolutely no obligation to actually try any of this! It could well be a dumb idea. And, of course, the models you've already implemented should already learn important stuff from historical data during training!

When looking at some of the outputs from the dataset when training, there seems like there could be an odd bug in getting examples. For example, in this training, while most inspected training examples look like a normal progression from frame to frame, considering they are 5 minutes apart, there are also intermittently changes like in the top example, where the cloud mask seems to change rapidly from one frame to the next.

https://github.com/adobe/antialiased-cnns

Since normal convolutional, pooling, etc. layers ignore the Nyquist sampling theorem, they can be very sensitive to slight changes in input. This adds an extra layer that can fix that. On previous work I've done, it helped get an improvement of a few percent on the task, so might help here.

Detailed Description

We plan to use pydocstyle to automatically check that our docstrings conform to a standard format. It would be amazing to have help tweaking our existing docstrings to conform!

pydocstyle's list of criteria is here: http://www.pydocstyle.org/en/stable/error_codes.html

Here's the Pull Request (and discussion) about enabling pydocstyle for nowcasting_utils: openclimatefix/nowcasting_utils#23

Classify each pixel as the output for if there is a cloud, or type of cloud with the #3 instead of regression. Attempting to predict all 12 spectral channels as the next images is probably too complicated and not as useful as just predicting the cloud masks, which is more suited to trying as a classification of each pixel vs the spectral channels.

DARDAR has some very detailed maps from CALIPSO and CloudSat satellites. Unfortunately, the data ends around 2016-2017. There could be some use in still training a model on historical data, then using it to predict labels on newer clouds? It includes detailed breakdowns of the types of clouds as well, compared to most cloud masks are binary cloud/no-cloud without considering the different between a thin cloud blocking only some irradiance, and a thick cloud blocking essentially all of it for example.

Atmospheric Motion Vectors for example could be useful to include where winds seem to go? Still need to look into it more, but there are a few other rapid scan outputs that could be promising. The motion vectors are averaged over 3 5 minute scans, so are only once every 15 minutes though, but still could be useful.

Detailed Description

Update the Perciever and MetNet Lightning modules to use the extra data available from Nowcasting Dataset, including NWPs, etc.

Context

Now that SatFlow supports working with Nowcasting Dataset (see #94) it would be nice to use all of the available data and see how performance changes.

Possible Implementation

See #98

For other nowcasting applications, such as precipitation, the training data is usually sampled so that rainfall exists or is above some threshold in every, or nearly every input sample. This ensures that, while most the time there is no rainfall in a given area, the model will learn what happens when there is. For forecasting clouds and cloud movement, something similar should probably be done.

We want to be able to easily swap out models and data sources both to make it simple to iterate on experiments, and easier for people to jump in and use/extend our models. Originally, I created a Model class for this, but then have been switching to PyTorch-Lightning, so it is kind of in a half/half state. So need to restandardize it and maybe design it a bit better.

Need to find where to get historical radar data, thinking that ground based radar would have better constant coverage of UK than satellites. Could add in as some more channels to the EUMETSAT image, or as an auxiliary image input, leaning more towards adding in as another channel(s).

As we primarily care about predicting where clouds are now and in the future, topographical information for the images could help, as clouds form differently around mountains or coasts potentially.

This is where, with some random chance, we give the RNN like ConvLSTM the ground truth label when its generating sequences in training. This can help with convergenc especially in the beginning, since if the model messes up in the second timestep, all later ones will be useless, but if we give it the GT image to continue the sequence it might still learn better representations.

Add logging to Neptune now that OCF has an account there. PR #56 added the configs, so it should be ready, but have to double check that.

Currently, if we are treating the time steps as layers, for example for Unet, and use auxiliary layers such as the lat lon, topographic data, etc. They will be added to each timestep before they are concatenated together, resulting in a lot of extra layers that give no more information.

Various ideas

Some from @JackKelly:

• Pretrain predicting next frame from past two
• Simulated clouds/optical flow
• Try AutoFlow like in Perceiver paper

Add a callback(s) to log the best model artifact, the model/datamodule config files used, and images to Neptune.

https://pytorch-lightning.readthedocs.io/en/latest/extensions/callbacks.html

Pretraining without labels, using contrastive learning for example, before then training on the actual predictions could help.

Detailed Description

https://www.iarai.ac.at/weather4cast/ (on GitHub here: https://github.com/iarai/weather4cast) has some nice weather data from around the world, including cloud masks, all at 4km resolution, so similar to what EUMETSAT gives. This could be useful for pretraining any of the models before finetuning more on our specific data.

Context

Pretraining has been proven to help quite a bit in large models, so this might help there.

Interestingly, they use MSE as their loss function for it all, even though its a video prediction task, and they want the next 8 hours of data predicted. So maybe MSE isn't the worst?

Possible Implementation

Download the data, and run some models on it.

Currently, validation is randomly sampled from the validation dataset. Removing the randomness, or resetting the dataloader each time? (PL has a function for this) could work best. Alternatively, creating a hand-picked validation set could also be helpful that covers the most challenging and diverse set. This would require a bit more work, but could use something like #42 to select where the models fail the most would be a much more informative set. Could also then change to store the validation samples as a much smaller webdataset, which would be faster to load.

Based off https://twitter.com/jack_kelly/status/1411615500371640320?s=19 might help with metnet training, should be easy to add and experiment with. Plan would be to add it to the generic model interface so we can try it on any of the models

Add in some sample training data for others to be able to download and run some quick testing or benchmarks. Also record benchmarks for models.

As mention in #85 one pre-training idea is to create a flow dataset to pre-train on using clouds. We would need simulated flow, and would want to have realistic clouds in all spectral channels. Easiest way to do that, I think, would be to take real clouds and then crop/paste them over the base ground image and move them to create it. So, to do this, there needs to be

• Get clean base image of ground for all timesteps
• Get set of real clouds that we can use for copy/paste, has to be available for all timesteps so its still the correct pixel values as the day goes on
• Create augmentation pipeline to generate images on fly and optical flow

This should help them load faster. Downsides would be less randomly cropped across the whole image, as no crops crossing boundaries would happen. Would split up the arrays from 892x1956 to 446x489 though, which should really help with loading speed.

Give the network future frames from optical flow as well as the GT past frames, have it try to predict the future frames, or corrections to the optical flow frames as output

See if the model does better or worse when predicting future frames dependent on the time of day and month. Related to openclimatefix/predict_pv_yield#54