Code Monkey home page Code Monkey logo

lpdaac-data-resources's Introduction

NASA Land Processes Distributed Active Archive Center (LP DAAC) Data Resources

This repository is a place to find data user resources that demonstrate how to use LP DAAC tools, services, and data.

Content include in this repository are listed below.

Repository Contents Type Summary
Data_Discovery_CMR_API_Request.ipynb Jupyter Notebook Demonstrates how to search for Earthdata data collections and granules using CMR API and Request Python package
Data_Discovery_CMR_API_Bulk_Query.ipynb Jupyter Notebook Demonstrates how to search and extract data URLs for an entire collection using Python's asyncio package
bulk_download_using_curl.md Markdown Demonstrates how to bulk download LP DAAC data using Curl from command line
bulk_download_using_wget.md Markdown Demonstrates how to bulk download LP DAAC data using Wget from command line

The other guides, tutorials, how-tos and scripts can be accessed in our mission specific repositories.

Resource Repository Summary Services and Tools
AppEEARS Data Resources How to use the Application for Extracting and Exploring Analysis Ready Samples (AppEEARS) Tutorials, AppEEARS API, Direct S3 Access
EMIT Data How to find, access, and work with EMIT data (Earth Surface Mineral Dust Source Investigation) Tutorials, Scripts, Direct S3 Access
GEDI Data How to find, access, and work with GEDI data (Global Ecosystem Dynamics Investigation) Tutorials
HLS Data How to find, access, and work with HLS data (Harmonized Landsat Sentinel-2) Tutorials, Scripts, Direct S3 Access
ECOSTRESS Data How to find, access, and work with ECOSTRESS data (The ECOsystem Spaceborne Thermal Radiometer Experiment on Space Station) Tutorials, Scripts, Direct S3 Access
VITALS How to find and work with EMIT and ECOSTRESS data together Tutorials

lpdaac-data-resources's People

Contributors

mjami00 avatar briannalind avatar amfriesz avatar ebolch avatar tkantz avatar cpkrehbiel avatar

Stargazers

Ivann avatar  BobDaGithubAccount avatar  avatar John Arban Lewis avatar Jeronimo Rodriguez-Escobar avatar Parth Parimalbhai Bhatt avatar Monika Anna Tomaszewska avatar Liam Irwin avatar Tiago de Conto avatar Joel M avatar  avatar Ezequiel Cimadevilla Alvarez avatar Xin Zhang avatar Scott Henderson avatar Michael Sumner avatar  avatar Tobias Augspurger avatar  avatar  avatar  avatar  avatar

Watchers

James Cloos avatar  avatar  avatar Tiago de Conto avatar  avatar  avatar  avatar Jeronimo Rodriguez-Escobar avatar Kyra Degen avatar Heather Grams avatar SREEDARSH T M avatar

lpdaac-data-resources's Issues

Authorization problem for AWS s3

I am trying to execute the download with boto3 notebook:

https://github.com/nasa/LPDAAC-Data-Resources/blob/main/python/how-tos/Earthdata_Cloud__Download_file_from_S3.ipynb

I successfuly login with my earthdata username and password and I also get the temporary credentials for aws. But when I execute client.download_file I get the following error:

what am I missing?

---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
Cell In[31], line 3
      1 key = "HLSS30.020[/HLS.S30.T56QPM.2023001T002959.v2.0/HLS.S30.T56QPM.2023001T002959.v2.0.B03.tif](http://localhost:8888/HLS.S30.T56QPM.2023001T002959.v2.0/HLS.S30.T56QPM.2023001T002959.v2.0.B03.tif)"
      2 filename = 'temp_download_example.tif'
----> 3 client.download_file(Bucket=bucket, Key=key, Filename=filename)

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/boto3/s3/inject.py:192](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/boto3/s3/inject.py#line=191), in download_file(self, Bucket, Key, Filename, ExtraArgs, Callback, Config)
    157 """Download an S3 object to a file.
    158 
    159 Usage::
   (...)
    189     transfer.
    190 """
    191 with S3Transfer(self, Config) as transfer:
--> 192     return transfer.download_file(
    193         bucket=Bucket,
    194         key=Key,
    195         filename=Filename,
    196         extra_args=ExtraArgs,
    197         callback=Callback,
    198     )

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/boto3/s3/transfer.py:405](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/boto3/s3/transfer.py#line=404), in S3Transfer.download_file(self, bucket, key, filename, extra_args, callback)
    401 future = self._manager.download(
    402     bucket, key, filename, extra_args, subscribers
    403 )
    404 try:
--> 405     future.result()
    406 # This is for backwards compatibility where when retries are
    407 # exceeded we need to throw the same error from boto3 instead of
    408 # s3transfer's built in RetriesExceededError as current users are
    409 # catching the boto3 one instead of the s3transfer exception to do
    410 # their own retries.
    411 except S3TransferRetriesExceededError as e:

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/futures.py:103](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/futures.py#line=102), in TransferFuture.result(self)
     98 def result(self):
     99     try:
    100         # Usually the result() method blocks until the transfer is done,
    101         # however if a KeyboardInterrupt is raised we want want to exit
    102         # out of this and propagate the exception.
--> 103         return self._coordinator.result()
    104     except KeyboardInterrupt as e:
    105         self.cancel()

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/futures.py:266](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/futures.py#line=265), in TransferCoordinator.result(self)
    263 # Once done waiting, raise an exception if present or return the
    264 # final result.
    265 if self._exception:
--> 266     raise self._exception
    267 return self._result

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/tasks.py:269](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/tasks.py#line=268), in SubmissionTask._main(self, transfer_future, **kwargs)
    265     self._transfer_coordinator.set_status_to_running()
    267     # Call the submit method to start submitting tasks to execute the
    268     # transfer.
--> 269     self._submit(transfer_future=transfer_future, **kwargs)
    270 except BaseException as e:
    271     # If there was an exception raised during the submission of task
    272     # there is a chance that the final task that signals if a transfer
   (...)
    281 
    282     # Set the exception, that caused the process to fail.
    283     self._log_and_set_exception(e)

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/download.py:354](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/s3transfer/download.py#line=353), in DownloadSubmissionTask._submit(self, client, config, osutil, request_executor, io_executor, transfer_future, bandwidth_limiter)
    325 """
    326 :param client: The client associated with the transfer manager
    327 
   (...)
    349     downloading streams
    350 """
    351 if transfer_future.meta.size is None:
    352     # If a size was not provided figure out the size for the
    353     # user.
--> 354     response = client.head_object(
    355         Bucket=transfer_future.meta.call_args.bucket,
    356         Key=transfer_future.meta.call_args.key,
    357         **transfer_future.meta.call_args.extra_args,
    358     )
    359     transfer_future.meta.provide_transfer_size(
    360         response['ContentLength']
    361     )
    363 download_output_manager = self._get_download_output_manager_cls(
    364     transfer_future, osutil
    365 )(osutil, self._transfer_coordinator, io_executor)

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/botocore/client.py:565](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/botocore/client.py#line=564), in ClientCreator._create_api_method.<locals>._api_call(self, *args, **kwargs)
    561     raise TypeError(
    562         f"{py_operation_name}() only accepts keyword arguments."
    563     )
    564 # The "self" in this scope is referring to the BaseClient.
--> 565 return self._make_api_call(operation_name, kwargs)

File [~/miniconda3/envs/hls/lib/python3.11/site-packages/botocore/client.py:1021](http://localhost:8888/home/pauravent/miniconda3/envs/hls/lib/python3.11/site-packages/botocore/client.py#line=1020), in BaseClient._make_api_call(self, operation_name, api_params)
   1017     error_code = error_info.get("QueryErrorCode") or error_info.get(
   1018         "Code"
   1019     )
   1020     error_class = self.exceptions.from_code(error_code)
-> 1021     raise error_class(parsed_response, operation_name)
   1022 else:
   1023     return parsed_response

ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.