Creating COG out of big rasters about rio-cogeo HOT 8 CLOSED

Apparently add_mask=True is a culprit. With add_mask=False it takes same amount of time as for gdal_translate.

from rio-cogeo.

I don't know the root cause of the issue but I've found a workaround.

In my case blockxsize / blockysize values of the source raster are 512px. When I set blockxsize=512 and blockxsize=512 for the output raster then everything works fine (1024px is also ok). But if change it to 256px then performance becomes very poor. Closing for now.

from rio-cogeo.

@drnextgis

• what happens if you create the COG using gdal_translate?
• can you check the memory consumption?
• are you using --allow-intermediate-compression

from rio-cogeo.

• memory consumption is quite low (2G out of 15G) but CPU is under 100% load
• --allow-intermediate-compression is not being used

I'm going to check how it behaves with gdal_translate.

from rio-cogeo.

@drnextgis try with --allow-intermediate-compression. The problem is that you are creating a really big temporary file. using intermediate compression might do the trick (by default it will use deflate but this is configurable when using the API)

from rio-cogeo.

gdal-translate: 30 minutes

$ gdal_translate \
    -of COG -co COMPRESS=DEFLATE \
    -co BLOCKSIZE=256 \
    -co BIGTIFF=IF_SAFER \
    -co TILING_SCHEME=GoogleMapsCompatible \
    -co RESAMPLING=CUBIC \
    --config GDAL_TIFF_OVR_BLOCKSIZE 256 \
    --config GDAL_TIFF_INTERNAL_MASK TRUE \
    --config GDAL_NUM_THREADS ALL_CPUS \
    20211207_023956_SN8_L3_SR_MS.tif cog.tif

rio-cogeo (even with allow_intermediate_compression enabled) : 8 hours

from rio_cogeo.cogeo import cog_translate
from rio_cogeo.profiles import cog_profiles


dst_profile = cog_profiles.get("deflate")
dst_profile.update({"blockxsize": 256, "blockysize": 256, "BIGTIFF": "IF_SAFER"})
gdal_config = dict(
    GDAL_NUM_THREADS="ALL_CPUS",
    GDAL_TIFF_INTERNAL_MASK=True,
    GDAL_TIFF_OVR_BLOCKSIZE="256",
)

cog_translate(
    "20211207_023956_SN8_L3_SR_MS.tif",
    "cogeo.tif",
    dst_profile,
    add_mask=True,
    resampling="cubic",
    config=gdal_config,
    web_optimized=True,
    allow_intermediate_compression=True,
)

from rio-cogeo.

thanks for the comparison @drnextgis

rio-cogeo (even with allow_intermediate_compression enabled) : 8 hours

can you tell which step took most of the time?

from rio-cogeo.

Writing mask slows down the whole process.

from rio-cogeo.