Connection broken: error 104, ECONNRESET about gcsfs HOT 11 CLOSED

797/800 isn't bad :)
Interesting that this seems to happen during ls. Yes, agree that there could be some retry logic for anything within _call. Would you be interested in implementing this and submitting a PR?

from gcsfs.

I agree it's not that bad - especially since I think I opened a few more files, so it's closer to 1000.

Since you're open to it, yes, I can try my hand at a PR. Can't guarantee it'll come today though.

from gcsfs.

The try/retry should go around this line: https://github.com/dask/gcsfs/blob/master/gcsfs/core.py#L298

from gcsfs.

OK to close?

from gcsfs.

Absolutely!

from gcsfs.

@martindurant can you reopen the issue ?

I was misled in thinking that error 104 was not caught as retriable but it is the logging of it that is confusing, especially when one looks at Stackdriver logging - we use gcsfs in Cloud Functions.

In particular:

        logger.exception("_call exception: %s", e)
            if retry == self.retries - 1:
                raise e
            if is_retriable(e):
                # retry
                continue
            raise e

If retriable, would we want to log as an exception ? Warning no good for us as this is mapped to Error in SD logging too.

I would remove the logging.exception and add a logging.info in the is_retriable block instead.

from gcsfs.

How about:

--- a/gcsfs/core.py
+++ b/gcsfs/core.py
@@ -467,12 +467,13 @@ class GCSFileSystem(object):
                 validate_response(r, path)
                 break
             except (HtmlError, RequestException, GoogleAuthError) as e:
-                logger.exception("_call exception: %s", e)
                 if retry == self.retries - 1:
+                    logger.exception("_call out of retries on exception: %s", e)
                     raise e
                 if is_retriable(e):
-                    # retry
+                    logger.debug("_call retrying after exception: %s", e)
                     continue
+                logger.exception("_call non-retriable exception: %s", e)
                 raise e
         try:

from gcsfs.

Very good ! The more - "switchable" - info the better !

from gcsfs.

@martindurant Can you reopen the issue to include your latest code proposal ?

from gcsfs.

This change was made in 323ac55 ( https://github.com/dask/gcsfs/blob/master/gcsfs/core.py#L474 )

from gcsfs.

Thx ! Had missed that.

from gcsfs.