https://registry.opendata.aws/copernicus-dem/

Hi,

I'm working on ingesting CBERS-4A images to AWS and extending rio-tiler-pds. Will open a WIP PR in a few moments to allow you to discuss, if desired, the subject while it being developed.

The Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) satellite is a publicly available dataset with 3 VNIR, 6 SWIR, and 5 thermal bands available both through Google Earth Engine and AWS. Data collected from this platform are extensively used in the geosciences and environmental monitoring. I would like to request this dataset be added to rio-tiler-pds.

AWS id = arn:aws:s3:::terrafusiondatasampler
GEE id = ASTER/AST_L1T_003

Web resources:
https://registry.opendata.aws/terrafusion/
https://developers.google.com/earth-engine/datasets/catalog/ASTER_AST_L1T_003#description

Hi Vincent,

I'm battle testing Rio-tiler-PDS and so far some issues,

Mainly ran into cloud detection issues, as i could not get the "B10" band from L2A.

I'm also looking for the "CLD" bands, do you know how to get that one?

The B10 is available for on AWS, however Rio-tiler-PDS cannot retrieve it.

aws s3 ls s3://sentinel-s2-l1c/tiles/44/P/LU/2020/10/12/0/ --request-payer requester
2020-10-12 11:25:38 3672535 B01.jp2
2020-10-12 11:25:38 98640868 B02.jp2
2020-10-12 11:25:38 99913324 B03.jp2
2020-10-12 11:25:38 106900680 B04.jp2
2020-10-12 11:25:38 30355344 B05.jp2
2020-10-12 11:25:38 32663001 B06.jp2
2020-10-12 11:25:38 33809595 B07.jp2
2020-10-12 11:25:38 117991600 B08.jp2
2020-10-12 11:25:38 3420599 B09.jp2
2020-10-12 11:25:38 2114876 B10.jp2
2020-10-12 11:25:38 31471597 B11.jp2
2020-10-12 11:25:38 30893542 B12.jp2
2020-10-12 11:25:38 33728721 B8A.jp2
2020-10-12 11:25:38 135421286 TCI.jp2

band definition between satellite is different. For Landsat it can be B1 while for Sentinel2 it should be B01. I think we can ease the task for the user and accept apply some magic within the _get_band_url method

cc @kylebarron

Hey all, I am trying to use S2COGREADER to read part of a raster according to a bbox..
here is my geojson courtesy of geojson.io

geojson = {
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {},
      "geometry": {
        "type": "Polygon",
        "coordinates": [
          [
            [
              35.013427734375,
              2.0869407308811065
            ],
            [
              37.46337890624999,
              2.0869407308811065
            ],
            [
              37.46337890624999,
              4.850154078505659
            ],
            [
              35.013427734375,
              4.850154078505659
            ],
            [
              35.013427734375,
              2.0869407308811065
            ]
          ]
        ]
      }
    }
  ]
}

I then use featureBounds to get the bbox and pass it to the reader with a band expression:

geom = geojson["features"][0]["geometry"]
with S2COGReader("S2A_29RKH_20200219_0_L2A") as sen:
    data = sen.part(bbox=featureBounds(geom), expression="(B03-B11)/(B03+B11)")

I get this error

ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 618 and the array at index 1 has size 555

I have tried different geometries and different ID numbers. Any help?

Hack: to get the bbox, I'm using a small GeoJSON https://gist.github.com/vincentsarago/ba3a2c026b47d52fd8a9b65ba9e1696d which could be provided with rio-tiler-pds module 🤷‍♂️

import attr
from typing import Any, Dict, Tuple, Type, Union

from rio_tiler.io import COGReader, MultiBandReader
from rasterio.features import bounds as featureBounds
from rio_tiler import constants
from morecantile import TileMatrixSet


default_bands = (
    "B02",
    "B03",
    "B04",
    "B08",
    "B11",
    "B12",
)


def get_grid_bbox(name: str) -> Tuple[float, float, float, float]:
    feat = list(filter(lambda x: x["properties"]["GZD"] == name , mgrs_grid["features"]))[0]
    return featureBounds(feat["geometry"])


@attr.s
class S2DigitalTwinReader(MultiBandReader):
    """Sentinel DigitalTwin Reader"""

    grid: str = attr.ib()
    year: int = attr.ib()
    month: int = attr.ib()
    day: int = attr.ib()
    reader: Type[COGReader] = attr.ib(default=COGReader)
    reader_options: Dict = attr.ib(factory=dict)
    tms: TileMatrixSet = attr.ib(default=constants.WEB_MERCATOR_TMS)
    minzoom: int = attr.ib(default=6)
    maxzoom: int = attr.ib(default=10)

    bands: tuple = attr.ib(init=False, default=default_bands)

    _scheme: str = "s3"
    _hostname: str = "sentinel-s2-l2a-mosaic-120"
    _prefix: str = "{year}/{month}/{day}/{grid}"

    def __attrs_post_init__(self):
        """Fetch item.json and get bounds and bands."""
        self.bounds = get_grid_bbox(self.grid)

    def _get_band_url(self, band: str) -> str:
        """Validate band name and return band's url."""
        band = band if len(band) == 3 else f"B0{band[-1]}"

        if band not in self.bands:
            raise InvalidBandName(f"{band} is not valid.\nValid bands: {self.bands}")

        prefix = self._prefix.format(year=self.year, month=self.month, day=self.day, grid=self.grid)
        return f"{self._scheme}://{self._hostname}/{prefix}/{band}.tif"

with S2DigitalTwinReader("57U", 2019, 1, 1) as s2Twin:
    print(s2Twin.bounds)

Hi,

As Sentinel-1 data requires pre-processing, e.g. radiometric calibration, thermal noise reduction, speckle filtering, we need to use the metadata files, delivered in the productinfo. As rio-tiler takes a part of the entire image, to derive the metadata, we need to know the Azimuth line and pixelcount of the extracted tile/point.

Does rio-tiler provide these, and if not, could you help me figure out how to get these in a convenient way. It would then good to add this feature to the tiler.

thanks,

Carst

more info on:
https://sentinel.esa.int/documents/247904/685163/S1-Radiometric-Calibration-V1.0.pdf

I'm trying to run the following snippet, as reported in the documentation

from rio_tiler_pds.sentinel.aws import S2COGReader

with S2COGReader("S2A_29RKH_20200219_0_L2A") as sentinel:
    print(type(sentinel))

I got the following error:

NoCredentialsError: Unable to locate credentials

In my understanding, sentinel2 COGs are accessible without any AWS credentials.

Moreover, recently, global sentinel 2 COGs was made publicly available. There is some ways to access them trough rio_tiler_pds? If yes, documentation still refer only to Africa related COG.

With the landsat tiler (and maybe others), it uses a ThreadPoolExecutor and you can't change the number of threads, it would be good to have that as an argument, similar to in mosaic_reader

as of 6c82b6e the code is just a pure copy of rio-tiler code.

I'd love to update each submodule to use ContextManager as rio_tiler_io.COGReader and rio_tiler_io.STACReader

TO DO

• update code to use rio_tiler.io.cogeo.multi_* modules (replacement of rio_tiler.reader_multi)
• decide and change the code architecture cogeotiff/rio-tiler#124 (comment)

The idea behind moving all code here was to ease addition of new PDS (e.g next Landsat release from USGS, Sentinel2 COG....), I'm not sure what's the best way to do that.

Hi Vincent, great work.

I tried to install a local version on windows, the Landsat tiler metadata/tile breaks because the parser adds control characters. here:
line 125: meta_file = "http://{bucket}.s3.amazonaws.com/{prefix}/{scene}_MTL.txt".format(
**scene_params
)

Parser results in:
http://landsat-pds.s3.amazonaws.com/c1\L8\197\024\LC08_L1TP_197024_20180703_20180717_01_T1/LC08_L1TP_197024_20180703_20180717_01_T1_MTL.txt

while should be:
http://landsat-pds.s3.amazonaws.com/c1/L8/197/024/LC08_L1TP_197024_20180703_20180717_01_T1/LC08_L1TP_197024_20180703_20180717_01_T1_MTL.txt

Thanks

I am trying to access the SCL product from Sentinel 2, but it seems like it is not listed in the list of available bands:

with S2COGReader("S2A_28QED_20230514_0_L2A") as sentinel:
    img = sentinel.preview("SCL")

gives

InvalidBandName: SCL is not valid.
Valid bands: ('B01', 'B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B8A', 'B09', 'B11', 'B12')

However if I modify the bands to add it as a valid band, everything seems to work fine:

with S2COGReader("S2A_28QED_20230514_0_L2A") as sentinel:
    sentinel.bands += ("SCL",)
    img = sentinel.preview("SCL")
imshow(img.data_as_image())

I wanted to check if there was a particular reason these products weren't available (e.g. a correctness issue and thus I should definitely avoid doing this) or if it was perhaps an oversight?

Right now, virtually nothing is exported at the top level:

It would make sense at a minimum to export other modules

From the tests, it looks like sentinel2_parser expects a string of the format S2A_L2A_20170729_19UDP_0:

But the STAC metadata included with sentinel-cogs is of a different format:

S2A_26PPC_20200514_0_L2A

Our titiler instance fails when requesting bounds for the following scene:
S1A_S3_GRDH_1SDV_20230807T171941_20230807T172010_049772_05FC2A_B3DA

The object itself is rather strange looking in Scihub. With some weird sort of value-threshold feathering happening beyond the edge of what I would call the bounds of the image.

ref #56 (comment)

for this new version I chose to use the word assets instead of bands to better align with STAC. This is in fact confusing and might also not help with migration.

I propose (with @kylebarron) to move back to bands.

Unsure whether this is something that needs to be fixed, or maybe just documented.

sceneid = 'LC08_L2SP_077010_20210616_20210623_02_A1'
sceneid_parser(sceneid)
# InvalidLandsatSceneId: Could not match LC08_L2SP_077010_20210616_20210623_02_A1

Though the data does exist:

> aws s3 ls s3://usgs-landsat/collection02/level-2/albers/oli-tirs/2021/077/010/LC08_L2SP_077010_20210616_20210623_02_A1/ --request-payer
2021-06-22 19:06:09     117266 LC08_L2SP_077010_20210616_20210623_02_A1_ANG.txt
2021-06-22 19:06:09   72555703 LC08_L2SP_077010_20210616_20210623_02_A1_BT_B10.TIF
2021-06-22 19:06:10   73056444 LC08_L2SP_077010_20210616_20210623_02_A1_BT_B11.TIF
2021-06-22 19:06:40       8458 LC08_L2SP_077010_20210616_20210623_02_A1_BT_stac.json
2021-06-22 19:06:34      24467 LC08_L2SP_077010_20210616_20210623_02_A1_MTL.json
2021-06-22 19:06:11      20153 LC08_L2SP_077010_20210616_20210623_02_A1_MTL.txt
2021-06-22 19:06:11      30060 LC08_L2SP_077010_20210616_20210623_02_A1_MTL.xml
2021-06-22 19:06:11    4555577 LC08_L2SP_077010_20210616_20210623_02_A1_QA_PIXEL.TIF
2021-06-22 19:06:11     285318 LC08_L2SP_077010_20210616_20210623_02_A1_QA_RADSAT.TIF
2021-06-22 19:06:11    1612709 LC08_L2SP_077010_20210616_20210623_02_A1_SAA.TIF
2021-06-22 19:06:11   98628574 LC08_L2SP_077010_20210616_20210623_02_A1_SR_B1.TIF
2021-06-22 19:06:13   98888810 LC08_L2SP_077010_20210616_20210623_02_A1_SR_B2.TIF
2021-06-22 19:06:14   99221882 LC08_L2SP_077010_20210616_20210623_02_A1_SR_B3.TIF
2021-06-22 19:06:15  100230500 LC08_L2SP_077010_20210616_20210623_02_A1_SR_B4.TIF
2021-06-22 19:06:16  101047724 LC08_L2SP_077010_20210616_20210623_02_A1_SR_B5.TIF
2021-06-22 19:06:17   82672180 LC08_L2SP_077010_20210616_20210623_02_A1_SR_B6.TIF
2021-06-22 19:06:17   79008495 LC08_L2SP_077010_20210616_20210623_02_A1_SR_B7.TIF
2021-06-22 19:06:18    6094460 LC08_L2SP_077010_20210616_20210623_02_A1_SR_QA_AEROSOL.TIF
2021-06-22 19:06:40      12491 LC08_L2SP_077010_20210616_20210623_02_A1_SR_stac.json
2021-06-22 19:06:19    2353540 LC08_L2SP_077010_20210616_20210623_02_A1_ST_ATRAN.TIF
2021-06-22 19:06:19   75025324 LC08_L2SP_077010_20210616_20210623_02_A1_ST_B10.TIF
2021-06-22 19:06:19   18508425 LC08_L2SP_077010_20210616_20210623_02_A1_ST_CDIST.TIF
2021-06-22 19:06:20    1467008 LC08_L2SP_077010_20210616_20210623_02_A1_ST_DRAD.TIF
2021-06-22 19:06:20   10647256 LC08_L2SP_077010_20210616_20210623_02_A1_ST_EMIS.TIF
2021-06-22 19:06:21   24674455 LC08_L2SP_077010_20210616_20210623_02_A1_ST_EMSD.TIF
2021-06-22 19:06:21   32356524 LC08_L2SP_077010_20210616_20210623_02_A1_ST_QA.TIF
2021-06-22 19:06:22   60622007 LC08_L2SP_077010_20210616_20210623_02_A1_ST_TRAD.TIF
2021-06-22 19:06:22    2018399 LC08_L2SP_077010_20210616_20210623_02_A1_ST_URAD.TIF
2021-06-22 19:06:40      11764 LC08_L2SP_077010_20210616_20210623_02_A1_ST_stac.json
2021-06-22 19:06:22    1047442 LC08_L2SP_077010_20210616_20210623_02_A1_SZA.TIF
2021-06-22 19:06:23   95533127 LC08_L2SP_077010_20210616_20210623_02_A1_TOA_B1.TIF
2021-06-22 19:06:23   96512869 LC08_L2SP_077010_20210616_20210623_02_A1_TOA_B2.TIF
2021-06-22 19:06:24   97422593 LC08_L2SP_077010_20210616_20210623_02_A1_TOA_B3.TIF
2021-06-22 19:06:25   98720561 LC08_L2SP_077010_20210616_20210623_02_A1_TOA_B4.TIF
2021-06-22 19:06:26  100035700 LC08_L2SP_077010_20210616_20210623_02_A1_TOA_B5.TIF
2021-06-22 19:06:27   80975371 LC08_L2SP_077010_20210616_20210623_02_A1_TOA_B6.TIF
2021-06-22 19:06:28   77750728 LC08_L2SP_077010_20210616_20210623_02_A1_TOA_B7.TIF
2021-06-22 19:06:29   46371301 LC08_L2SP_077010_20210616_20210623_02_A1_TOA_B9.TIF
2021-06-22 19:06:40      12862 LC08_L2SP_077010_20210616_20210623_02_A1_TOA_stac.json
2021-06-22 19:06:29    8090125 LC08_L2SP_077010_20210616_20210623_02_A1_VAA.TIF
2021-06-22 19:06:30    2393656 LC08_L2SP_077010_20210616_20210623_02_A1_VZA.TIF
2021-06-22 19:06:30      92210 LC08_L2SP_077010_20210616_20210623_02_A1_thumb_large.jpeg
2021-06-22 19:06:30      11016 LC08_L2SP_077010_20210616_20210623_02_A1_thumb_small.jpeg

Currently there's a significant amount of initialization done in __enter__ instead of __init__. This means that code will silently fail if not used with a context manager:

scene = S2COGReader('S2B_12SYJ_20200901_0_L2A')
# no error
scene.center
# errors:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-77-5a7975690aa4> in <module>
----> 1 self.center

<ipython-input-43-587f5a19de47> in center(self)
     41         """Dataset center + minzoom."""
     42         return (
---> 43             (self.bounds[0] + self.bounds[2]) / 2,
     44             (self.bounds[1] + self.bounds[3]) / 2,
     45             self.minzoom,

AttributeError: 'S2COGReader' object has no attribute 'bounds'

I believe __init__ should be used for any necessary initialization of the class. __enter__ should only instantiate context-specific attributes like a file descriptor or a database cursor. Such a context really doesn't apply here, and __enter__ is essentially on there for syntactical sugar. Hence initialization should be move entirely to __init__.

I noticed I was getting a lot of 404s on my Sentinel 2 tiler. It seems that something has changed in the datasets as the return of the band names is inconsistent between scenes:

Expected:

from rio_tiler_pds.sentinel.aws import S2COGReader

with S2COGReader("S2A_31NBJ_20190524_0_L2A") as sentinel:
   print(type(sentinel))
    > <class 'rio_tiler_pds.sentinel.aws.sentinel2.S2L2ACOGReader'>
    
    print(sentinel.bands)
    > ('B01', 'B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B8A', 'B09', 'B11', 'B12')
    
    print([i for i in sentinel.stac_item["assets"]])
    > ['thumbnail', 'overview', 'info', 'metadata', 'visual', 'B01', 'B02', 'B03', 'B04', 'B05', 'B06', 'B07', 'B08', 'B8A', 'B09', 'B11', 'B12', 'AOT', 'WVP', 'SCL']

When running scene from documentation:

with S2COGReader("S2A_29RKH_20200219_0_L2A") as sentinel:
   print(type(sentinel))
    > <class 'rio_tiler_pds.sentinel.aws.sentinel2.S2L2ACOGReader'>
    
    print(sentinel.bands)
    > ()
    
    print([i for i in sentinel.stac_item["assets"]])
    > ['aot', 'blue', 'coastal', 'granule_metadata', 'green', 'nir', 'nir08', 'nir09', 'red', 'rededge1', 'rededge2', 'rededge3', 'scl', 'swir16', 'swir22', 'thumbnail', 'tileinfo_metadata', 'visual', 'wvp', 'aot-jp2', 'blue-jp2', 'coastal-jp2', 'green-jp2', 'nir-jp2', 'nir08-jp2', 'nir09-jp2', 'red-jp2', 'rededge1-jp2', 'rededge2-jp2', 'rededge3-jp2', 'scl-jp2', 'swir16-jp2', 'swir22-jp2', 'visual-jp2', 'wvp-jp2']

I ran several tests with scenes from different areas and date ranges, but almost all scenes seem to return the wrong band names. The S2COGReader class expects band names to be in the "B[0-9A]{2}" format and therefore returns an empty tuple. It seems that name and common_name are the same for the eo:bands metadata in these scenes.

The scene id parser parses both scenes as expected:

from rio_tiler_pds.sentinel import s2_sceneid_parser

print(s2_sceneid_parser("S2A_31NBJ_20190524_0_L2A"))
> {'sensor': '2', 'satellite': 'A', 'utm': '31', 'lat': 'N', 'sq': 'BJ', 'acquisitionYear': '2019', 'acquisitionMonth': '05', 'acquisitionDay': '24', 'num': '0', 'processingLevel': 'L2A', 'scene': 'S2A_31NBJ_20190524_0_L2A', 'date': '2019-05-24', '_utm': '31', '_month': '5', '_day': '24', '_levelLow': 'l2a'}

print(s2_sceneid_parser("S2A_29RKH_20200219_0_L2A"))
> {'sensor': '2', 'satellite': 'A', 'utm': '29', 'lat': 'R', 'sq': 'KH', 'acquisitionYear': '2020', 'acquisitionMonth': '02', 'acquisitionDay': '19', 'num': '0', 'processingLevel': 'L2A', 'scene': 'S2A_29RKH_20200219_0_L2A', 'date': '2020-02-19', '_utm': '29', '_month': '2', '_day': '19', '_levelLow': 'l2a'}

I'm running rio-tiler-pds version 0.7.0

Hi, love your work.

Is it possible to add MODIS? With it, we can build regression models to transpose Sen2 images for example NDVI.

https://docs.opendata.aws/modis-pds/readme.html

Typically objects are compact single polygons that return reasonable values [minx, miny, maxx, maxy] when you get their bounding box.

Case of interest:

When an object crosses the antimeridian (i.e. ±180º longitude), the typical behavior is splitting it into a multipolygon with disconnected elements on opposite sides of the map.

Current behavior:

The current implementation of the /bounds endpoint does not treat this correctly... It returns a degenerate bbox that looks like [-180, miny, 180, maxy], spanning the whole globe

Preferred behavior:

Instead, according to the GeoJSON spec: https://datatracker.ietf.org/doc/html/rfc7946#section-5.2 , the bbox should have the "wrong" order for the longitude coordinates (i.e. minx > maxx).
Note that it is not simply enough to return[180, miny, -180, maxy]. The endpoint should actually find the leftmost bound of the rightmost polygon as minx, and the rightmost point of the leftmost polygon as maxx.

A frontend application may then choose to handle this case as best they see fit, for instance see: https://github.com/developmentseed/rio-viz/blob/main/rio_viz/templates/index.html#L1202-L1210

Open question:

I'm not sure how this behavior should be defined in more complex situations, where the object is not convex. For instance, a U-shaped polygon that crosses the antimeridian could result in multipolygon with 3 components.

Example:

/bounds?sceneid=S1A_IW_GRDH_1SDV_20230726T183302_20230726T183327_049598_05F6CA_31E7
returns
[-180.0, 61.06949078480844, 180.0, 62.88226850489882]
because it's productInfo.json looks similar to this:

"footprint" : {
    "type" : "MultiPolygon",
    "coordinates" : [ 
        [ [ [ 180.0, 62.52138872791733 ], [ 179.94979152463551, 62.52658106629492 ], [ 179.46990657679865, 62.57485506167223 ], ..., [ 179.97829453382462, 61.06949078480844 ], [ 180.0, 61.115074335603865 ], [ 180.0, 62.52138872791733 ] ] ], 
        [ [ [ -180.0, 62.52138872791733 ], [ -180.0, 61.115074335603865 ], [ -179.95528668344312, 61.208976568342585 ], ..., [ -179.34123633603335, 62.451941133348306 ], [ -179.81084543337622, 62.50182719954094 ], [ -180.0, 62.52138872791733 ] ] ] ]
  },

Ref https://lists.osgeo.org/mailman/listinfo/landsat-pds

All, in case you have not see it, you should look over the recent announcement from USGS related to Collection 2 and its availability on AWS.

https://www.usgs.gov/news/usgs-releases-most-advanced-landsat-archive-date
https://www.usgs.gov/core-science-systems/nli/landsat/landsat-commercial-cloud-data-access

First of all, I just want to say how awesome this is! Planet has been managing the dataset on AWS from the beginning and through their work and all of your demonstrated use cases, USGS saw the value of making the data available in a new format via a fundamentally new distribution mechanism (though their existing distribution mechanisms still remain if you’re looking for that).

Since there is now an S3 bucket with Collection 2 data, the full archive and owned directly by USGS, I am proposing that we deprecate the existing landsat-pds bucket. I know that a number of you have come to depend on the landsat-pds bucket and I want to be sensitive to that, so I am proposing a 6 month deprecation period which would put us at an end date (bucket deleted) of July 1, 2021.

The USGS is making the data available in COG format with STAC metadata. There are some prefix changes from landsat-pds and of course the science updates for Collection 2 and the data is being made available in a Requester Pays S3 bucket (https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html). There is currently no SNS topic available for new data, but I believe a few of us have reached out to USGS about that.

I want to note that this data is owned and managed by USGS, AWS has no control over the data or the bucket. However, I am happy to help anyone work through the implications of switching their workloads to the USGS bucket rather than the landsat-pds bucket.

Also, I want to give a huge thank you to Amit who has managed this resource for the community for the past several years!

I’ll be out for the new two weeks so you might see an OOO response if you reach out to me, but I will look to answer when I am back in the office.

Thanks everyone and hope you are staying safe.

Joe Flasher