Quentin Paletta et al.'s 2021 "ECLIPSE" paper show that it's a good idea to predict cloud masks, because these are concise representations of the sky (with the caveat that I'm afraid I've only skim-read the paper so far, so I may have misunderstood!).
Also, ideally, we'd probably prefer satellite image sequences of just the clouds, rather than the clouds plus the land. Sometimes optical flow messes up by moving the land around (!), so optical flow would probably perform better on 'pure cloud' images (with the land removed). And, for ML approaches for forecasting future satellite images, it's unfair to expect the ML model to reconstruct images of land as the cloud moves away from land, when that information may be entirely absent from the input image sequence (because the cloud has completely covered the land in the input image sequence).
On the other hand, I'm a little nervous about using binary cloud masks because clouds are so varied, and thin wispy clouds which might not be classified as 'cloud' by a binary mask can have a significant effect on irradiance.
Also, reliable cloud mask labels are hard to come by, I think? (Sure, there are algorithms for segmenting clouds, but none are perfect, right?)
So maybe we can automatically subtract 'cloud' from 'land' something like this: If we had a perfect 'cloud-free' satellite image for every time of day, then we could do a pixel-wise subtraction: just_cloud_image[t] = cloud_free_image[t] - actual_image[t]
. Then we could use the just_cloud_image
for our downstream models.
The question then becomes: How to generate the set of cloud_free_image[t]
for every time of day, and time of year? Could it be as simple as taking the median pixel value, per pixel, at a given time of day, over the last month or so of imagery? e.g. to get the cloud_free_image
for 12:00, look at all the images taken at 12:00 over the last month, and take the median pixel value?
Or maybe the median is the wrong statistic: Maybe instead we can assume the histogram of pixel values at a given time of day, over the last month, would have two peaks: one corresponding to 'cloud free', and the other corresponding to 'cloudy'. And we want the mode of the 'cloud free' peak, which I guess will always be the less-bright peak?
(also see the twitter discussion about this issue)
TODO: