Data Access API interface as per GDF interface. about datacube-core HOT 4 CLOSED

get_descriptor

get_descriptor request
Storage type has been replaced with finer grained elements:

• satellite (eg 'LANDSAT_8')
• sensor (eg 'TM', 'OLI_TIRS')
• product (eg 'EODS_NBAR', 'EODS_PQ')

Should we still support a 'storage_type' identifier? Should the response be keyed with a concatenation of satellite and sensor?

• dimensions
  • range
  • crs

Support for CRS in dimension specifiers limited to latitude and longitude using EPSG codes.
Time range can be a datetime object or seconds since UNIX epoch. Further support to be added.

get_descriptor response
get_descriptor now returns storage_units with values for storage_max, storage_min, storage_shape, storage_path.

Not yet implemented:

• 'buffer_size'
• 'irregular_indices'
• 'grouping_function' - Use of xray may cause a rethink of how this is used and implemented.

Known issues:

• result_shape sometimes suffers from a rounding error when calculating the impact of the bounding box range specifier using an estimated nearest neighbor method than differs from the get_data call by a pixel. Currently looking to fix this.

get_data

get_data request
Currently supported:

• satellite
• sensor
• variables (mandatory - should it return all if not specified?)
• dimensions
  • range
  • array_range
  • coordinate_reference_system (see get_descriptor)

get_data response

• 'indices' currently returns numpy.datetime64 objects for time fields, rather than the stored coordinate labels.
• 'element_sizes' returns the "pixel" size. Behavior is undefined for irregular variables eg time.

Not yet implemented:

• 'coordinate_reference_systems'

Known Issues:

• Irregular variables (eg time) are currently causing issues when times do not line up over storage boundaries. (Eg Times stacked in a file) Currently fixing this, after discussing the issue with Alex.

from datacube-core.

Thanks Andrew.

Note that we can’t rely on satellite/sensor: not all datasets/units will have them, and are dynamic fields specific to the ‘eo’ dataset metadata type.

Given that this is storage-unit focused (?), the name/id of the storage mapping (essentially a "storage type" by the old definition*) may be an alternative. But it depends on the intended use-case or user of the API.

(*Note that we're currently looking at merging storage types and mappings into one concept, probably to be called "storage type" — Greg's started doing it in a branch, which would simplify this)

from datacube-core.

@andrewdhicks

Thanks Andrew for adding more information.

The GDF concept for the idea is as follows:
storage_type is a mapping to the type of storage unit, a unique handle to type of data if you'd like.
This handle is defined in the database.

This should really be more generic like in the GDF.

It should be able to define the names/attributes in the database. i.e. it shouldn't be hard coded to "Satellite", "Sensor", "Product".

(Satellite, Sensor, Product) should have more generic names to cater for other data sets.
As an example (family, sensor, product) or (family, product).

To answer your question about whether we should have a storage_type, Yes it should still have a storage_type.

One thing to keep in mind regarding the storage_type is how to handle composite datasets e.g. modis/landsat blending. derived datasets, datasets that are the results of analytics that is re-ingested, or other datasets that people want to ingest and use that are not acquired by a satellite.

My suggestion and is only a suggestion, is to keep it as a tag/id and keep information about the data about the storage_type in the the database. It simplifies the issue when you have composite datasets.

To answer your question about concatenation of satellite and sensor. This is how the storage_type pretty much worked in the GDF. It isn't supposed to be just a concatenation but a unique handle/tag to the type of data. Having said that, most of the storage_types were just concatenations. Just make sure you can do a get_data with this tag instead of (Satellite, Sensor, Product). Lets have a chat about this offline to prevent cluttering up GitHub.

@jeremyh

What do you mean when you say merging the storage types and mappings into one concept?
Are you planning on using the storage_type concept in the GDF or is it something different?

from datacube-core.

Closing this.

Thanks Andrew for completing this task in a timely manner.

I'll create new tasks for the next iteration of get_descriptor and get_data for HPC soon once the EDA milestone is complete.

from datacube-core.