occasional lockups during dask reads about stackstac HOT 5 OPEN

Yeah, a timeout on the read would be reasonable. Then we could have that timeout trigger a retry via whatever logic we implement for #18.

@mukhery I'm curious if you have a reproducer for this, or have noticed cases/datasets/patterns that tend to cause it more often?

For now, you might try playing with setting GDAL_HTTP_MAX_RETRY and GDAL_HTTP_RETRY_DELAY via LayeredEnv. See https://gdal.org/user/virtual_file_systems.html#vsicurl-http-https-ftp-files-random-access and https://trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_HTTP_TIMEOUT.

Maybe something like:

retry_env = stackstac.DEFAULT_GDAL_ENV.updated(dict(
    GDAL_HTTP_TIMEOUT=45,
    GDAL_HTTP_MAX_RETRY=5,
    GDAL_HTTP_RETRY_DELAY=0.5
))
stackstac.stack(..., gdal_env=retry_env)

from stackstac.

I tried to come up with something to reproduce but haven't been able to. We've also been seeing several other network-related/comms issues, so it's possible that our specific workload and how we've implemented the processing is causing some of these issues. I ended up just adding timeouts to the task futures and then cancelling and/or restarting the cluster if needed to meet our current need. Feel free to close this issue if you'd like and I can reopen later if I'm able to reliably reproduce.

from stackstac.

I'll keep it open, since I think it's a reasonable thing to implement.

I ended up just adding timeouts to the task futures

Curious how you implemented this?

from stackstac.

Sounds good, thanks!

I did something like this:

try:
    fut = cluster.client.compute(<task_involving_stackstac_data>)
    dask.distributed.wait(fut, timeout=600)
except dask.distributed.TimeoutError as curr_exception:
    error_text = f'{curr_exception}'[:100] #sometimes the error messages are crazy long
    print(f'task failed with exception: {error_text}')

from stackstac.

Nice! That makes sense.

from stackstac.