cs-si / eodag-sentinelsat Goto Github PK

View Code? Open in Web Editor NEW

16.0 7.0 3.0 77 KB

eodag-sentinelsat, an eodag plugin for searching and downloading products from Copernicus Scihub

License: GNU General Public License v3.0

Python 99.50% Shell 0.50%

eodag scihub sentinel copernicus download search

eodag-sentinelsat's People

Contributors

Stargazers

Watchers

Forkers

jpv-costa aoyono sebastienpeillet

eodag-sentinelsat's Issues

api endoint needs to be updated

See sentinelsat/sentinelsat#444

and https://scihub.copernicus.eu/news/News00868

Open Access Hub users are informed that the API Hub URL has changed from https://scihub.copernicus.eu/apihub/ to https://apihub.copernicus.eu/apihub/ as of 28 April 2021.
The former URL will not be dismissed. An automatic redirection to the new URL is performed transparently.

Error searching on scihub using eodag 2.8

Describe the bug
Error when searching on scihub using eodag 2.8

Code To Reproduce

from eodag import EODataAccessGateway
dag= EODataAccessGateway()
dag.set_preferred_provider("scihub")
dag.search(
    productType='S2_MSI_L1C',
    geom={'lonmin': 1, 'latmin': 43.5, 'lonmax': 2, 'latmax': 44},
    start='2021-01-01',
    end='2021-01-15'
)

Output

Error while searching on provider scihub (ignored):
Traceback (most recent call last):
  File "/home/thomas/.local/lib/python3.10/site-packages/eodag/api/core.py", line 1389, in _do_search
    res, nb_res = search_plugin.query(count=count, **kwargs)
  File "/home/thomas/.local/lib/python3.10/site-packages/eodag_sentinelsat/eodag_sentinelsat.py", line 107, in query
    query_params, provider_product_type = self._update_keyword(**kwargs)
  File "/home/thomas/.local/lib/python3.10/site-packages/eodag_sentinelsat/eodag_sentinelsat.py", line 445, in _update_keyword
    provider_product_type = self.map_product_type(product_type, **kwargs)
TypeError: QueryStringSearch.map_product_type() got an unexpected keyword argument 'auth'
([], 0)

Environment:

Python version: 3.10.6
EODAG version: 2.8
eodag-sentinelsat version: 0.4.1

Additional context
eodag-sentinelsat plugin works fine with eodag 2.6.2 and 2.7 (did not check with other versions)

Inherit from QueryStringSearch instead of ODataV4Search

Not a bug per say, but I noticed that this API plugin inherits from ODataV4Search. I couldn't figure out why exactly. ODataV4Search itself inherits from QueryStringSearch and has 2 specialized methods: do_search and get_metadata_search_url. None of them seems to be used by the plugin itself.

It uses however normalize_results, map_product_type, get_product_type_def_params and build_query_string, which are all methods which all belong to QueryStringSearch.

I guess it could just inherit from QueryStringSearch.

Get the total number of products that match the search criteria

query should return a tuple with a list of EOProduct and the total number of products that match the search criteria (this can be bypassed with count=False).

The log messages with a search made with this plugin contained:

2021-04-16 09:01:31,015-15s sentinelsat.SentinelAPI          [INFO    ] (sentinel         ) Found 314 products

I don't know whether this number can be extracted or not.

If not, sentinelsat has a count method that could be used instead.

LTA products handling

https://sentinelsat.readthedocs.io/en/stable/api_overview.html#lta-products

Copernicus Open Access Hub no longer stores all products online for immediate retrieval. Offline products can be requested from the Long Term Archive (LTA)

Trying to download LTA products results in:

from eodag import EODataAccessGateway, setup_logging
setup_logging(verbose=3)

dag = EODataAccessGateway()

ds = '2020-05-01T00:00:00'
de = '2020-08-15T00:00:00'
extent_wkt = "POLYGON ((1.23 43.42, 1.23 43.76, 1.68 43.76, 1.68 43.42, 1.23 43.42))"
product_type = 'S2_MSI_L1C'
dag.set_preferred_provider("scihub")
dags_L1C, _ = dag.search(
    productType=product_type, 
    geom=extent_wkt, 
    start=ds, end=de,
)
path = dag.download(dags_L1C[0])

2021-06-11 17:47:55,290-15s eodag.plugins.download.base      [INFO    ] (base             ) Download url: https://apihub.copernicus.eu/apihub/odata/v1/Products('dba84f1c-deaf-4fee-97ac-e98ddf67b280')/$value
2021-06-11 17:47:55,297-15s sentinelsat.SentinelAPI          [INFO    ] (sentinel         ) Will download 1 products using 2 workers
Fetching archival status:   0%|                                                                                                                                                                  | 0.00/1.00 [00:00<?, ? products/s]
2021-06-11 17:47:56,635-15s sentinelsat.SentinelAPI          [DEBUG   ] (sentinel         ) Accepted for retrieval
Fetching archival status: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.00/1.00 [00:01<00:00, 1.34s/ products]
2021-06-11 17:47:56,863-15s sentinelsat.SentinelAPI          [INFO    ] (sentinel         ) Product dba84f1c-deaf-4fee-97ac-e98ddf67b280 is in LTA.
2021-06-11 17:47:57,384-15s sentinelsat.SentinelAPI          [DEBUG   ] (sentinel         ) Accepted for retrieval
2021-06-11 17:47:57,384-15s sentinelsat.SentinelAPI          [INFO    ] (sentinel         ) dba84f1c-deaf-4fee-97ac-e98ddf67b280 accepted for retrieval
2021-06-11 17:47:57,656-15s sentinelsat.SentinelAPI          [DEBUG   ] (sentinel         ) Accepted for retrieval
2021-06-11 17:47:57,656-15s sentinelsat.SentinelAPI          [INFO    ] (sentinel         ) dba84f1c-deaf-4fee-97ac-e98ddf67b280 is not online.
dba84f1c-deaf-4fee-97ac-e98ddf67b280: 0.00B [00:02, ?B/s]
2021-06-11 17:47:57,666-15s eodag.api.product                [DEBUG   ] (_product         ) Product location updated from 'https://apihub.copernicus.eu/apihub/odata/v1/Products('dba84f1c-deaf-4fee-97ac-e98ddf67b280')/$value' to 'https://apihub.copernicus.eu/apihub/odata/v1/Products('dba84f1c-deaf-4fee-97ac-e98ddf67b280')/$value'
2021-06-11 17:47:57,667-15s eodag.api.product                [INFO    ] (_product         ) Remote location of the product is still available through its 'remote_location' property: https://apihub.copernicus.eu/apihub/odata/v1/Products('dba84f1c-deaf-4fee-97ac-e98ddf67b280')/$value

And the product is not downloaded.

Handle LTA products the same way it is done on eodag for products ordering

Download doesn't start: "waiting for it to come online..."

Hi,
After having installed and configured the credentials properly, eodag can find the products in scihub, but it always freeze the "waiting for it to come online" message.
I have already raised the logging to level 3 to check for other insights, but couldn't find a fix.
Here is the output:

Any ideas?
Thanks,
Mauricio

ProdType not found for scihub

Describe the bug
When searching products on scihub for prodtype 'S1_SAR_GRD' or 'S2_MSI_L2A', Eodag warns that product type is not available

Code To Reproduce

from eodag.utils.logging import setup_logging
setup_logging(verbose=3)
from eodag import EODataAccessGateway
dag= EODataAccessGateway()
dag.set_preferred_provider("scihub")
dag.search(
    productType='S1_SAR_GRD',
    geom='POLYGON ((-3.000272 48.753013, -1.506731 48.743337, -1.535213 47.755819, -3.000267 47.765167, -3.000272 48.753013))'),
    start='2022-05-01',
    end='2022-08-31'
)

Output

2023-03-29 10:26:04,517 eodag.core                       [DEBUG   ] (core             ) Searching for all the products with provider peps and a maximum of 500 items per page.
2023-03-29 10:26:04,517 eodag.core                       [WARNING ] (core             ) Product type 'S1_SAR_GRD' is not available with provider 'scihub'. Searching it on provider 'peps' instead.
2023-03-29 10:26:04,518 eodag.core                       [DEBUG   ] (core             ) Using plugin class for search: QueryStringSearch
2023-03-29 10:26:04,518 eodag.core                       [INFO    ] (core             ) Iterate search over multiple pages: page #1
2023-03-29 10:26:04,518 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Mapping eodag product type to provider product type
2023-03-29 10:26:04,518 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Getting provider product type definition parameters for S1_SAR_GRD
2023-03-29 10:26:04,518 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Getting provider product type definition parameters for S1_SAR_GRD
2023-03-29 10:26:04,519 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Building the query string that will be used for search
2023-03-29 10:26:04,519 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Retrieving queryable metadata from metadata_mapping
2023-03-29 10:26:04,519 eodag.plugins.search.qssearch    [INFO    ] (qssearch         ) Sending search request: https://peps.cnes.fr/resto/api/collections/S1/search.json?startDate=2022-05-01&completionDate=2022-08-31&geometry=POLYGON ((-3.0003 48.7530, -1.5067 48.7433, -1.5352 47.7558, -3.0003 47.7652, -3.0003 48.7530))&productType=GRD&maxRecords=500&page=1
2023-03-29 10:26:05,209 eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Adapting 72 plugin results to eodag product representation
2023-03-29 10:26:05,301 eodag.core                       [DEBUG   ] (core             ) Iterate over pages: last products found on page 1
2023-03-29 10:26:05,301 eodag.core                       [INFO    ] (core             ) Found 72 result(s) on provider 'peps'

Environment:

Python version: 3.8.10
EODAG version: eodag 2.9.1
eodag-sentinelsat 0.4.1

Paths returned by download_all don't point to an existing path

CS-SI/eodag#244 will update the paths returned by the download methods of EODAG. It'd be nice to update this plugin too.

I have observed that right now the paths returned by download_all don't actually point to any file or directory. For example:

Returned path: '/tmp/S2B_MSIL1C_20210321T104639_N0209_R051_T31TEJ_20210321T130616'
Path of the archive: '/tmp/S2B_MSIL1C_20210301T104859_N0209_R051_T31TEJ_20210301T115849.zip'
Path of the extracted product: '/tmp/S2B_MSIL1C_20210301T104859_N0209_R051_T31TEJ_20210301T115849.SAFE'

In the core plugins, when a product is extracted, a directory is first created in outputs_prefix whose name is product.properties["title"], then the product is extracted in there and the returned path points to that subfolder, e.g. /tmp/S2B_MSIL1C_20210321T104639_N0209_R051_T31TEJ_20210321T130616/S2B_MSIL1C_20210321T104639_N0209_R051_T31TEJ_20210321T130616.SAFE

@remi-braun since we've just talked about that, would it be OK for you if this plugin was updated to be more in sync with the core? I have the feeling that it would overall benefit from behaving more closely to the core, which means the core has to provide some utilities to make it easier ;)

Copernicus POD Hub

The Copernicus Open access hub offers another endpoint for downloading products.
Namely this is the following: https://scihub.copernicus.eu/gnss/odata/.. and from what I can understand it has the same functionality as the https://apihub.copernicus.eu/apihub/.

Maybe it would be good to have both endpoints as differents providers that can be used by EODAG in order to download products.

For example scihub and scihub-gnss.

Pinging @sbrunato , @geonux

Update how the progress bar is handled

Once this is fixed CS-SI/eodag#239 eodag-sentinelsat should be updated to reflect the changes made in eodag

Improve the README

To explain how to setup the package (credentials)
With a small getting started example

Apihub URL has changed

For a while I was unable to download Sentinel data from scihub because of invalid credentials (although my credentials worked perfectly on the DHUS)
Then I saw that: https://scihub.copernicus.eu/news/News00868

Open Access Hub users are informed that the API Hub URL has changed from https://scihub.copernicus.eu/apihub/ to https://apihub.copernicus.eu/apihub/ as of 28 April 2021.

My issue was resolved when changing the old URL to the new :)

Change PEPS provider to ESA scihub copernicus

Feature request
PEPS does not distribute Sentinel-2 images since the beginning of 2020. These images are distributed by the ESA Copernicus scihub. Changing or adding that provider would allow users to get the S2_MSI_L2A data directly from the source that distributes it.

extract by default

Extraction should be enabled by default even if extract is not specified in the user configuration, see CS-SI/eodag#238

Error writing search results to GeoJson using eodag-sentinelsat provider

Serialization of search results does not take care of Date/Datetime objects in metadata of Products fetched.
It might be an edge case but it might be interesting to fix it.

Error
TypeError: Object of type datetime is not JSON serializable

Install sentinelsat plugin

python -m pip install eodag
python -m pip install eodag-sentinelsat

Configure 'scihub' provider with highest priority.

Code To Reproduce
CLI command:

eodag -vvv search --conf "eodag-user_conf.yml" --box .... --all --storage "my_search.geojson"

Output

Environment:

Python version: 3.7.4 on Windows 10
EODAG version: 2.2.0

Additional context:
Using CS-SI/eodag#235 to configure the settings of this provider in an user configuration file.

Fix in my own code:

Impossible to connect to scihub: lacks valid authentication credentials

Describe the bug
Impossible to log to scihub, although I was able to (with the precedent version), my configuration hasn't changed and my account is valid.
Something has changed somewhere ? in the configuration file ?

Note that it works for other providers such as peps.

2021-05-05 11:35:50 - [INFO] - Querying S1_SAR_GRD products.
2021-05-05 11:35:50 - [INFO] - Searching product type 'S1_SAR_GRD' on provider: scihub
2021-05-05 11:35:50 - [DEBUG] - Using plugin class for search: SentinelsatAPI
2021-05-05 11:35:50 - [DEBUG] - Initializing Sentinelsat API
2021-05-05 11:35:50 - [DEBUG] - Mapping eodag product type to provider product type
2021-05-05 11:35:50 - [DEBUG] - Getting provider product type definition parameters for S1_SAR_GRD
2021-05-05 11:35:50 - [DEBUG] - Building the query string that will be used for search
2021-05-05 11:35:50 - [DEBUG] - Retrieving queryable metadata from metadata_mapping
2021-05-05 11:35:50 - [DEBUG] - Running query: order_by=None, limit=500, offset=0, query=beginPosition:[2020-05-14T00:00:00Z TO 2020-07-25T00:00:00Z] platformname:Sentinel-1 polarisationmode:VV\ VH producttype:GRD footprint:"Intersects(POLYGON ((-0.9570 43.6241,
 -1.2143 43.6305, -1.2127 43.6866, -0.8368 43.8658, -0.7765 43.8667, -0.7765 43.7476, -0.9570 43.6241)))"
2021-05-05 11:35:50 - [DEBUG] - Sub-query: offset=0, limit=500
2021-05-05 11:35:50 - [DEBUG] - Starting new HTTPS connection (1): scihub.copernicus.eu:443
2021-05-05 11:35:50 - [DEBUG] - https://scihub.copernicus.eu:443 "POST /apihub/search?format=json&rows=100&start=0 HTTP/1.1" 307 0
2021-05-05 11:35:50 - [DEBUG] - Starting new HTTPS connection (1): apihub.copernicus.eu:443
2021-05-05 11:35:51 - [DEBUG] - https://apihub.copernicus.eu:443 "POST /apihub/search?format=json&rows=100&start=0 HTTP/1.1" 401 None

The exception:

# HTTP Status 401 – Unauthorized

* * *

 **Type** Status Report

 **Message** Unauthorized

 **Description** The request has not been applied because it lacks valid
authentication credentials for the target resource.

* * *

### Apache Tomcat/8.0.53.

My configuration file:

scihub:
    priority: 100  # Lower value means lower priority (Default: 0)
    api:
        extract: false # whether to extract the downloaded products (true or false).
        outputs_prefix: "" # where to store downloaded products.
        dl_url_params:
        product_location_scheme:
        credentials:
            username: yyyy
            password: xxxx

Code To Reproduce
Download any product with scihub

Output
Compete output obtained with maximal verbosity.

Environment:

Python version: 3.7
EODAG version: 2.2.0
eodag-sentinelsat version: 0.3.0

Type object 'datetime.datetime' has no attribute 'fromisoformat'

Describe the bug
When doing a eodag search by scihub provider, throws an AttributeError: type object 'datetime.datetime' has no attribute 'fromisoformat'.

Code To Reproduce

from eodag.utils.logging import setup_logging
from eodag.api.core import EODataAccessGateway

import os
import traceback

def search(providerPriorityList):
    try:
        setup_logging(verbose=1)
        workspace = os.getcwd()
        print(workspace)
        dag = EODataAccessGateway(user_conf_file_path=os.path.join(workspace, 'user_conf_template.yml'))
        
        extent = { 'latmin': 37.249188, 'latmax': 37.299456, 'lonmin': -8.29536, 'lonmax': -8.404541 }
        start='2016-08-01'
        end='2018-12-01'

        defaultSatellite = 'Sentinel2'
        defaultLevel = 'Level2A'
        d ={
            'Sentinel1':{
                'GRD': {
                    'peps':'S1_SAR_GRD',
                    'scihub':'S1_SAR_GRD'
                },
                'OCN': {
                    'peps':'S1_SAR_OCN',
                    'scihub':'S1_SAR_OCN',
                },
                'SLC': {
                    'peps':'S1_SAR_SLC',
                    'scihub':'S1_SAR_SLC',
                },
            },
            'Sentinel2':{
                'Level1C':{
                    'peps':'S2_MSI_L1C',
                    'scihub':'S2_MSI_L1C',
                },
                'Level2A':{
                    'peps':'S2_MSI_L2A', 
                    'theia':'S2_MSI_L2A_MAJA',
                    'scihub':'S2_MSI_L2A',
                },
            }
        }

        providersAndProductTypes = d[defaultSatellite][defaultLevel]
        products = []
        for ppl in providerPriorityList:
            provider = ppl
            print("======== "+provider+" ============")
            if ppl in providersAndProductTypes:
                productType = providersAndProductTypes[ppl]
                print("========OK! Search "+provider+" in "+productType+"========")
                dag.set_preferred_provider(provider)
                products += dag.search_all(
                    productType=productType,
                    start=start, end=end,
                    geom=extent
                )                
            else:
                print("========ERROR! "+provider+" not found for "+defaultSatellite+" "+defaultLevel+"========")
                #exit(1)
        
        #generate json with the needed information
        print(len(products))
        for p in products:
            pjson = p.as_dict()
            print('------')
            #online, staging, offline
            customOutput = {
                'uid': pjson['properties']['uid'], 
                'filename': pjson['properties']['title'],
                'download': pjson['properties']['downloadLink'],
                'quicklook': pjson['properties']['quicklook'],
                'thumbnail': pjson['properties']['thumbnail'],
                'date': pjson['properties']['startTimeFromAscendingNode'],
                'storageStatus': pjson['properties']['storageStatus']
            }
            print(customOutput)
        
        print('Finished!')
        exit(0)
    except Exception as e:
        print(e)
        print(traceback.format_exc())
        exit(1)

try:
    providerPriorityList = ['scihub'] 
    search(providerPriorityList)
    exit(0)
except Exception as e:
    print(e)
    print(traceback.format_exc())
    exit(1)

Output

2021-05-05 17:01:28,674-15s eodag.config                     [INFO    ] Loading user configuration from: /home/ricardo/Worsica/user_conf_template.yml
2021-05-05 17:01:28,711-15s eodag.config                     [INFO    ] scihub: unknown provider found in user conf, trying to use provided configuration
2021-05-05 17:01:33,053-15s eodag.core                       [INFO    ] Locations configuration loaded from /home/ricardo/.config/eodag/locations.yml
======== scihub ============
========OK! Search scihub in S2_MSI_L2A========
2021-05-05 17:01:39,765-15s eodag.core                       [INFO    ] Searching product type 'S2_MSI_L2A' on provider: scihub
2021-05-05 17:01:39,766-15s eodag.core                       [INFO    ] Searching product type 'S2_MSI_L2A' on provider: scihub
2021-05-05 17:01:39,771-15s eodag.core                       [INFO    ] No result from provider 'scihub' due to an error during search. Raise verbosity of log messages for details
type object 'datetime.datetime' has no attribute 'fromisoformat'
Traceback (most recent call last):
  File "./eodag_search.py", line 60, in search
    geom=extent
  File "/home/foobar/py363_venv/lib/python3.6/site-packages/eodag/api/core.py", line 759, in search_all
    **kwargs,
  File "/home/foobar/py363_venv/lib/python3.6/site-packages/eodag/api/core.py", line 632, in search_iter_page
    search_plugin, count=False, raise_errors=True, **search_kwargs
  File "/home/foobar/py363_venv/lib/python3.6/site-packages/eodag/api/core.py", line 938, in _do_search
    res, nb_res = search_plugin.query(count=count, **kwargs)
  File "/home/foobar/py363_venv/lib/python3.6/site-packages/eodag_sentinelsat/eodag_sentinelsat.py", line 69, in query
    query_params, provider_product_type = self.update_keyword(**kwargs)
  File "/home/foobar/py363_venv/lib/python3.6/site-packages/eodag_sentinelsat/eodag_sentinelsat.py", line 256, in update_keyword
    datetime.fromisoformat(qp.pop("start")),
AttributeError: type object 'datetime.datetime' has no attribute 'fromisoformat'

Environment:

Python version: 3.6.3
EODAG version: 2.2.0
eodag-sentinelsat version: 0.3.0

Additional context
Doing a search for this error, this occurs because this function fromisoformat is only available from Python 3.7.
In Python 3.6.3, this function does not exist.
Although, there are some backports to fix this (https://github.com/movermeyer/backports.datetime_fromisoformat), but may not fix the problem, since this error occurs inside of the eodag_sentinelsat.py.
Thus, a workaround/fix for this is required for the people who run Python below 3.7. Maybe replace this function with strptime, like this fix? (precice/EventTimings@7cf5b06)

cs-si / eodag-sentinelsat Goto Github PK

eodag-sentinelsat's People

Contributors

Stargazers

Watchers

Forkers

eodag-sentinelsat's Issues

Recommend Projects

Recommend Topics

Recommend Org