Support rasters which does not fit in memory about rio-cogeo HOT 11 CLOSED

So @dionhaefner did this in https://github.com/DHI-GRAS/terracotta/blob/master/terracotta/scripts/optimize_rasters.py#L229-L269 so we could bring the solution over

from rio-cogeo.

Good job @vincentsarago! Can we use contextlib2.ExitStack?

from rio-cogeo.

Yes, flake8 is very critical of lambdas (IMO to a fault, but who am I to judge). This should work though:

- dst_ctx = lambda: memfile.open(**meta)
+ dst_ctx = functools.partial(memfile.open, **meta)

If you're not too paranoid about unclosed file descriptors in some edge cases I think it would also be perfectly fine to initialize the contexts first and enter them later:

if in_memory:
    file_ctx = MemoryFile()
    dst_ctx = memfile.open(**meta)
else:
    file_ctx = ...
    dst_ctx = ...

with file_ctx, dst_ctx as dst:
    ...

from rio-cogeo.

Hi @drnextgis, Thanks for opening the issue.

Does it mean that this plugin is not suit for cases when source raster is not fit in memory?

Yes sadly I don't think it could work other ways or it will need to many refactor. The main problem I can think of is that overviews need to be happened after the high resolution data (inside the tiff structure) which is kind of tricky to get.

I don't have a perfect solution for you I'm sorry.

from rio-cogeo.

Works fine from my experience (raster files > 20GB, script staying below 500MB memory usage).

from rio-cogeo.

Well I gave a try in #75 but sadly the contextlib.ExitStack() method is a python 3 only method 😢.
I don't have python2 in my heart but dropping it might be a big deal for some users.

cc @dionhaefner

from rio-cogeo.

That's a bummer because it means some additional nesting, but shouldn't be a showstopper. You can e.g. do something like this instead:

if in_memory:
    file_ctx = MemoryFile
    dst_ctx = lambda: memfile.open(**meta)
    # (or functools.partial if you like that better)
else:
    file_ctx = ...
    dst_ctx = ...

with file_ctx(), dst_ctx() as dst:
    ...

from rio-cogeo.

@dionhaefner I'm not sure to understand, did you check the last version I pushed. Basically I did the same you did in terracota

from rio-cogeo.

I was still at the ExitStack issue :) But if it's solved through contextlib2 all the better. I'll have a closer look at your PR during the weekend.

from rio-cogeo.

Excellent I was going to ask you and @drnextgis if you could review it and see if it fit your needs 😄

from rio-cogeo.

@dionhaefner BTW the solution you proposed seems kinda of smart, I'll see if we can do this instead. Right now flake8 is not happy on dst_ctx = lambda: memfile.open(**meta) with this error do not assign a lambda expression, use a defflake8(E731)

from rio-cogeo.