Enable per pixel metadata/provenance tracking about datacube-core HOT 2 CLOSED

How often will knowledge of pixel level provenance be required? From my understanding, it must be available, but wouldn't be needed to be saved in high resolution on disk.

At the moment the database records indicate which datasets are used to produce a storage units, from which it's possible to compute per-pixel provenance.

The extra (probably crazy) step, would be storing this provenance information in the storage units as a variable, but I don't see any benefit for doing this, since it won't be necessary for general usage. And it increases both storage overhead and code complexity.

from datacube-core.

Suspect it would start to get very busy quickly. As an example, the seasonal ground cover composites use pretty much every landsat pixel (will all the masking and shadow and water...) to make a prediction for every date so the metadata soon blows out. We (JRSRP) currently have an issue raised looking and compression and truncation methods for these higher level products so the provenance can be recreated easily, just not necessarily stored within every file.

from datacube-core.