leftfield-geospatial / geedim Goto Github PK

Thank you for this excellent tool! I was playing around with the new export function/types and ran into a minor issue with export to google cloud bucket. When using export() with ExportType "cloud" and filename e.g. "test" I get:

ee.ee_exception.EEException: Unknown configuration options: {'assetId': 'test'}

It appears that assetId is not an argument of Export.image.toCloudStorage(). I think it should work to set just description here:

https://github.com/dugalh/geedim/blob/7d8ecfcb2e4505192a3e19364fc0afcfda7452ae/geedim/download.py#L805-L808

First, thank you for this awesome lib!
In the current implementation, the value used to represent nodata is defined automatically based on the data type. Offering the user the possibility to define it manually can be very useful. For example, if you try to export an image (uint8) with valid zero values, it will result in a bad behavior since 0 is used as nodata value for uint8. I faced a similar problem trying to export Google Dynamic World classes. To overcome this limitation, I had to unmask the image using the value 255 (value to represent nodata) and pass set_nodata = False to geedim. However, I lost the possibility to set 255 as nodata in the geotiff file metadata.

See the uploaded image. Other methods look fine (ish) with the same settings.

Hello,
I am downloading an imagecollection as a single image using toBands() method. It exceeds the max number of bands. I was wodnering if this lib, given it already takes care of image size, could also take care of number of bands to decide for a split.
Thank you

The download function at the following link does not consider crs transformation parameter, which is causing shift in downloaded data.
https://github.com/dugalh/geedim/blob/main/geedim/download.py#L686

• Searching an unbounded collection without a region is not possible due to ee.Image.reduceRegion() in MaskedImage._set_region_stats(). The ee.EEException that is raised should be communicated to the user in a more helpful way.
• MODIS NBAR images are unbounded but not recognised as such by the checks in BaseImage._prepare_for_export(), resulting in a cryptic EE error. The unbounded check should also check for a footprint that is global.

Very useful package! I am trying to incorporate it into geemap, but it is seems the BaseImage.download() function cannot be placed within a function. It throws an error 'BaseImage' object has no attribute 'dowwnload'. It works fine when it is not placed within a function. It is probably because multi-thread processes can only be run at the top level. Do you have a solution for this?

def download_ee_image(
    image,
    filename,
    scale=None,
    **kwargs,
):

    import geedim as gd
    
    if scale is not None:
        kwargs['scale'] = scale
    
    img = gd.download.BaseImage(image)
    img.dowwnload(filename, **kwargs)

I'm interested in this approach but whenever I try it it always runs out of memory (even though other compositing methods work fine with the same settings)

I was Successfully installed geedim-0.1.4, but This error occurs when I use geemap to download images

Hi, First of all, thank you very much for this wonderful repo. I am happy that, I am able to download the GEE image from my area of interest.

When I trying to download the sentinel2 imagery using below code, it downloads the image but the image doesn't have correct scale.

dataset = ee.ImageCollection('COPERNICUS/S2').filterDate('2021-11-01', '2021-11-30').filterBounds(ee_fc).filter(ee.Filter.lt('CLOUDY_PIXEL_PERCENTAGE',5)).map(maskS2clouds)
dataset = dataset.median().select(['B4', 'B3', 'B2', 'B8', 'B11', 'B12']).clip(ee_fc)

img = geedim.download.BaseImage(dataset)
img.download(scale = 10, filename="aoi_s2.tif", dtype='float32', region=ee_fc.gemetry(), crs='EPSG:43426')

From this code, I got the image having following pixel size, which is kind of strange for me.

Dimensions	X: 441 Y: 188 Bands: 6
Origin	80.4700950046142083,29.1321850009392591
Pixel Size	8.983152841195391012e-05,-8.983152841195391012e-05

Am I doing anything wrong here?

Issue : Downloading the clipped tile by passing the region fails with the below error message

User limit exceeded, decrease the max_tile_size or max_tile_dim limit

When I set the region to None there is no issue.

Below is the code i use to download the data

  gee_img = gd.MaskedImage.from_id(fc_df["ID")
  download_bands_from_gee(gee_img, geom, band_names=['B4','B8','SCL'])

def download_all_bands_from_gee(collection,geom=None, ofilename=None):
    if ofilename is None:
        ofilename = pathlib.Path(collection.id).stem + ".tif"

    collection.download(ofilename, region=geom,overwrite = True)

I search and get the list of tiles and loop through the tiles.

def download_bands_from_gee(collection, geom, band_names=0):
    
    coll = collection.ee_image
    img = coll.select(band_names)
    info = img.getInfo()
    band_name = info["bands"][0]["id"]
    proj_target = img.select(0).projection()
    crs_target = proj_target.getInfo()['crs']
    ofilename = pathlib.Path(collection.id).stem + "_" + band_name + ".tif"
    collection_new = gd.MaskedImage(img)
    
    download_all_bands_from_gee(collection_new, None,  ofilename)

The follow code will throw an error: EEException: Can't convert a computed geometry to GeoJSON. Use getInfo() instead. However, if the scale is changed to a value larger than 30 (e.g., 60), it works fine.

import geedim as gd
image = ee.Image('LANDSAT/LC09/C02/T1_L2/LC09_023031_20220617')
region = image.geometry()
image = image.multiply(0.0000275).add(-0.2).set(image.toDictionary())
img = gd.download.BaseImage(image)
img.download('landsat.tif', region=region, scale=30)

I am having trouble using geedim with a specific image from the harmonized sentinel 2 collection, namely COPERNICUS/S2_SR_HARMONIZED/20230809T201859_20230809T201853_T10WEE. Whenever this images comes up in a search collection, i cannot process the collection for example to extract data with error: Image.rename: Parameter 'input' is required.

I do not know whats specifically different with this image. I drilled that down to a problem with the processing geedim applies to the image when using the region parameter but i cannot find the problem itself.

i am using a conda environment with

python                    3.11.9               he1021f5_0
earthengine-api           0.1.408            pyhd8ed1ab_0    conda-forge
geedim                    1.8.0              pyhd8ed1ab_0    conda-forge

I tested it with a bunch of images in a specific region and different region polygons. I made an example using getInfo() on the resulting image to trigger the processing, but it applies to any other client-side GEE method including geedims download():

import ee
import geedim

ee.Initialize()

poly_coords1 = [
    [-121.30224921178123,71.77255565243381],
    [-121.69226386021873,71.58260352221238],
    [-120.87378241490623,71.52698680217173],
    [-120.86828925084373,71.70370209220792],
    [-121.30224921178123,71.77255565243381],
]
poly_coords2 = [
    [-121.65930487584373,71.92795376863671],
    [-121.70874335240623,71.82200371558541],
    [-121.16492011021873,71.84769170326668],
    [-121.45056464146873,71.90237555398673],
    [-121.65930487584373,71.92795376863671],
]
poly_coords3 = [
    [-120.97265936803123,71.98070472145938],
    [-120.98913886021873,71.73784463314388],
    [-120.08276678990623,71.74472884706717],
    [-120.28052069615623,71.98750044350669],
    [-120.97265936803123,71.98070472145938],
]

bounds = [
    ee.Geometry.Polygon(list(poly_coords1), proj=None, evenOdd=False),
    ee.Geometry.Polygon(list(poly_coords2), proj=None, evenOdd=False),
    ee.Geometry.Polygon(list(poly_coords3), proj=None, evenOdd=False),
]

images = [
    "20230801T200901_20230801T200857_T10WEE",
    "20230802T202849_20230802T202847_T10WEE",
    "20230803T195859_20230803T200054_T10WEE",
    "20230804T201851_20230804T201853_T10WEE",
    "20230804T201851_20230804T202447_T10WEE",
    "20230806T200859_20230806T201056_T10WEE",
    "20230807T202851_20230807T203151_T10WEE",
    "20230808T195901_20230808T195945_T10WEE",
    "20230811T200901_20230811T200857_T10WEE",
    "20230812T202849_20230812T202934_T10WEE",
    "20230813T195859_20230813T195900_T10WEE",
    "20230814T201851_20230814T202428_T10WEE",
    "20230809T201859_20230809T201853_T10WEE", # <- this one is broken
]

for i,poly in enumerate(bounds):
    print(f"Polygon {i}:")
    for image_id in images:
        try:
            ( geedim.MaskedImage
                .from_id("COPERNICUS/S2_SR_HARMONIZED/" + image_id, region=poly)
                .ee_image.getInfo()
            )
            print(f" {image_id} -> √")
        except ee.ee_exception.EEException as e:
            print(f" {image_id} -> error: {e}")

which produces

Polygon 0:
 20230801T200901_20230801T200857_T10WEE -> √
 20230802T202849_20230802T202847_T10WEE -> √
 20230803T195859_20230803T200054_T10WEE -> √
 20230804T201851_20230804T201853_T10WEE -> √
 20230804T201851_20230804T202447_T10WEE -> √
 20230806T200859_20230806T201056_T10WEE -> √
 20230807T202851_20230807T203151_T10WEE -> √
 20230808T195901_20230808T195945_T10WEE -> √
 20230811T200901_20230811T200857_T10WEE -> √
 20230812T202849_20230812T202934_T10WEE -> √
 20230813T195859_20230813T195900_T10WEE -> √
 20230814T201851_20230814T202428_T10WEE -> √
 20230809T201859_20230809T201853_T10WEE -> error: Image.rename: Parameter 'input' is required.
Polygon 1:
 20230801T200901_20230801T200857_T10WEE -> √
 20230802T202849_20230802T202847_T10WEE -> √
 20230803T195859_20230803T200054_T10WEE -> √
 20230804T201851_20230804T201853_T10WEE -> √
 20230804T201851_20230804T202447_T10WEE -> √
 20230806T200859_20230806T201056_T10WEE -> √
 20230807T202851_20230807T203151_T10WEE -> √
 20230808T195901_20230808T195945_T10WEE -> √
 20230811T200901_20230811T200857_T10WEE -> √
 20230812T202849_20230812T202934_T10WEE -> √
 20230813T195859_20230813T195900_T10WEE -> √
 20230814T201851_20230814T202428_T10WEE -> √
 20230809T201859_20230809T201853_T10WEE -> error: Image.rename: Parameter 'input' is required.
Polygon 2:
 20230801T200901_20230801T200857_T10WEE -> √
 20230802T202849_20230802T202847_T10WEE -> √
 20230803T195859_20230803T200054_T10WEE -> √
 20230804T201851_20230804T201853_T10WEE -> √
 20230804T201851_20230804T202447_T10WEE -> √
 20230806T200859_20230806T201056_T10WEE -> √
 20230807T202851_20230807T203151_T10WEE -> √
 20230808T195901_20230808T195945_T10WEE -> √
 20230811T200901_20230811T200857_T10WEE -> √
 20230812T202849_20230812T202934_T10WEE -> √
 20230813T195859_20230813T195900_T10WEE -> √
 20230814T201851_20230814T202428_T10WEE -> √
 20230809T201859_20230809T201853_T10WEE -> error: Image.rename: Parameter 'input' is required.

Hello. I have a problem with uploading an annual composite to a certain territory. When part of the image is loaded, it gives me an error:

Error downloading tile: Expected a homogeneous image collection, but an image with incompatible bands was encountered: First image type: 23 bands ([B1, B2, B3, B4, B5, B6, B7, B8, B8A, B9, B11, B12, AOT, WVP, SCL, TCI_R, TCI_G, TCI_B, MSK_CLDPRB, MSK_SNWPRB, QA10, QA20, QA60]). Current image type: 21 bands ([B1, B2, B3, B4, B5, B6, B7, B8, B8A, B9, B11, B12, AOT, WVP, SCL, TCI_R, TCI_G, TCI_B, QA10, QA20, QA60]).

Is there a way around this error?

I used a library called geemap. One of the functions in this library, geemap.download_ee_image, uses the geedim library to download image results calculated through the Earth Engine API.

However, when I use this function to download the image, I can see the progress bar increasing, but when it reaches a certain download progress, it throws an error.

BadZipFile Traceback (most recent call last)
Cell In[23], line 37
34 temp_image.select(["SR_B1", "SR_B2", "SR_B3", "SR_B4","SR_B5", "SR_B7"])
36 y1=1990+i
---> 37 geemap.download_ee_image(temp_image, filename=f'landsat_image_{y1}.tif', scale=30, region=boundary.geometry(),crs='EPSG:4326')
40 for i in range(0,11):
41 start_time=ee.Date('2013-06-01').advance(i,'year')

File d:\anaconda3\envs\myenv\lib\site-packages\geemap\common.py:12925, in download_ee_image(image, filename, region, crs, crs_transform, scale, resampling, dtype, overwrite, num_threads, max_tile_size, max_tile_dim, shape, scale_offset, unmask_value, **kwargs)
12922 kwargs["scale_offset"] = scale_offset
12924 img = gd.download.BaseImage(image)

12925 img.download(filename, overwrite=overwrite, num_threads=num_threads, **kwargs)

File d:\anaconda3\envs\myenv\lib\site-packages\geedim\download.py:955, in BaseImage.download(self, filename, overwrite, num_threads, max_tile_size, max_tile_dim, **kwargs)
953 logger.info(f'Exception: {str(ex)}\nCancelling...')
954 executor.shutdown(wait=False)
--> 955 raise ex
957 bar.update(bar.total - bar.n) # ensure the bar reaches 100%
958 # populate GeoTIFF metadata

File d:\anaconda3\envs\myenv\lib\site-packages\geedim\download.py:951, in BaseImage.download(self, filename, overwrite, num_threads, max_tile_size, max_tile_dim, **kwargs)
949 try:
950 for future in as_completed(futures):
--> 951 future.result()
952 except Exception as ex:
953 logger.info(f'Exception: {str(ex)}\nCancelling...')

File d:\anaconda3\envs\myenv\lib\concurrent\futures_base.py:437, in Future.result(self, timeout)
435 raise CancelledError()
436 elif self._state == FINISHED:
--> 437 return self.__get_result()
439 self._condition.wait(timeout)
441 if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:

File d:\anaconda3\envs\myenv\lib\concurrent\futures_base.py:389, in Future.__get_result(self)
387 if self._exception:
388 try:
--> 389 raise self._exception
390 finally:
391 # Break a reference cycle with the exception in self._exception
392 self = None

File d:\anaconda3\envs\myenv\lib\concurrent\futures\thread.py:57, in _WorkItem.run(self)
54 return
56 try:
---> 57 result = self.fn(*self.args, **self.kwargs)
58 except BaseException as exc:
59 self.future.set_exception(exc)

File d:\anaconda3\envs\myenv\lib\site-packages\geedim\download.py:941, in BaseImage.download..download_tile(tile)
939 def download_tile(tile):
940 """Download a tile and write into the destination GeoTIFF. """
--> 941 tile_array = tile.download(session=session, bar=bar)
942 with out_lock:
943 out_ds.write(tile_array, window=tile.window)

File d:\anaconda3\envs\myenv\lib\site-packages\geedim\tile.py:121, in Tile.download(self, session, response, bar)
118 zip_buffer.flush()
120 # extract geotiff from zipped buffer into another buffer
--> 121 zip_file = zipfile.ZipFile(zip_buffer)
122 ext_buffer = BytesIO(zip_file.read(zip_file.filelist[0]))
124 # read the geotiff with a rasterio memory file

File d:\anaconda3\envs\myenv\lib\zipfile.py:1269, in ZipFile.init(self, file, mode, compression, allowZip64, compresslevel, strict_timestamps)
1267 try:
1268 if mode == 'r':
-> 1269 self._RealGetContents()
1270 elif mode in ('w', 'x'):
1271 # set the modified flag so central directory gets written
1272 # even if no files are added to the archive
1273 self._didModify = True

File d:\anaconda3\envs\myenv\lib\zipfile.py:1336, in ZipFile._RealGetContents(self)
1334 raise BadZipFile("File is not a zip file")
1335 if not endrec:
-> 1336 raise BadZipFile("File is not a zip file")
1337 if self.debug > 1:
1338 print(endrec)

BadZipFile: File is not a zip file

How can I solve this problem? Is there a better way to directly download Earth Engine image objects?

GEE reports the MODIS CRS as SR-ORG:6974 but downloads it as SR-ORG:6842 resulting in incorrect image position.

Thanks for the excellent package! I've been putting it to good use lately.

I noticed that if a download results in a combined GeoTIFF >4GB (after 'deflate' compression), writing to the file with rasterio will result in this GDAL error:

ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.

From reading https://gdal.org/drivers/raster/gtiff.html#creation-options, I see GDAL defaults to BIGTIFF=IF_NEEDED, which incidentally never results in BIGTIFF format file when a compression method is being used.

Since geedim specifies "deflate" compression and other GDAL dataset write options, but not BIGTIFF, lengthy downloads will fail part way through if/when the file reaches ~4GB. I think there probably would be little risk to just setting BIGTIFF='YES' in profile from _prepare_for_download() but perhaps IF_SAFER would be better. It is my understanding that some non-GDAL software still does not support libtiff > 4 / BIGTIFF.

More generally, if the user could decide which (if any) compression method they want to use, or have access to set/modify other GDAL options, then setting BIGTIFF for some data sets/extents would be up to the user