Quamash crashes when no network is available about quamash HOT 5 OPEN

So after analyzing the problem, it seems to be something deep.

This happens when aiohttp uses loop.getaddrinfo which runs in an executor : https://github.com/python/asyncio/blob/217da1db65a1eca0ea1e14362e199f048a99ec84/asyncio/base_events.py#L536

Sometimes, the future is deleted, and the exception was not catched. The exception is then handled in the exception_handler and crashes my application. It seems like the QThreadExecutor might have a wrong behaviour somehow. But I'm really bad with threads so I have trouble analyzing it more...

from quamash.

things to try:

set the default executor to a concurrent.futures.ThreadPoolExecutor

loop.set_default_executor(concurrent.futures.ThreadPoolExecutor(10))

Modify getaddrinfo on the event loop (say by modifying QEventLoop)

def getaddrinfo(self, host, port, *, family=0, type=0, proto=0,

flags=0):
future = asyncio.Future()
if self._debug:
result = self._getaddrinfo_debug(host, port, family, type,
proto, flags)
else:
result = socket.getaddrinfo(host, port, family, type, proto,
flags)
future.set_result(result)
return future

(that is to make it synchronous, but keep the same signature)

that should give you more info.

On Thu, Dec 10, 2015 at 2:16 PM, Insoleet [email protected] wrote:

So after analyzing the problem, it seems to be something deeper.

This happens when aiohttp uses loop.getaddrinfo which runs in an executor
:
https://github.com/python/asyncio/blob/217da1db65a1eca0ea1e14362e199f048a99ec84/asyncio/base_events.py#L536

Sometimes, the future is deleted, and the exception was not catched. The
exception is then handled in the exception_handler and crashes my
application. It seems like the QThreadExecutor might have a wrong behaviour
somehow. But I'm really bad with threads so I have trouble analyzing it
more...

—
Reply to this email directly or view it on GitHub
#42 (comment).

from quamash.

Thanks.

Here is what I could find :

• With the concurrent ThreadPoolExecutor, the same crash happens
• With a synchronized getaddrinfo, I can get a bigger stack, but there is still nothing linked to a call in my code :

DEBUG:main:async_exception_handler:Exception handler executing
ERROR:main:async_exception_handler:Task exception was never retrieved
future: <Task finished coro=<__iter__() done, defined at /home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py:454> exception=ClientOSError(None, 'Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]')>
Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 557, in _create_connection
    server_hostname=hinfo['hostname'] if sslcontext else None)
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/base_events.py", line 569, in create_connection
    type=socket.SOCK_STREAM, proto=proto, flags=flags)
  File "/home/inso/code/quamash/quamash/__init__.py", line 481, in getaddrinfo
    result = socket.getaddrinfo(host, port, family, type, proto, flags)
socket.gaierror: [Errno -2] Nom ou service inconnu

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 289, in connect
    transport, proto = yield from self._create_connection(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 580, in _create_connection
    (req.host, req.port, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Can not connect to moul.re:8999 [None]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/tasks.py", line 238, in _step
    result = next(coro)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 456, in __iter__
    resp = yield from self._coro
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 173, in _request
    conn = yield from self._connector.connect(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 299, in connect
    .format(key, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]

Some questions :

• Why in the asynchronous case the gaierror does not raise the aiohttp specific exceptions ?
• Why aren't they raised from my code, but they are from nowhere in aiohttp code ?

from quamash.

the sync version of getaddrinfo I sent you doesn't actually handle
exceptions correctly:

try:

def getaddrinfo(self, host, port, *, family=0, type=0, proto=0,

flags=0):
future = asyncio.Future()
try:
if self._debug:
result = self._getaddrinfo_debug(host, port, family, type,
proto, flags)
else:
result = socket.getaddrinfo(host, port, family, type,
proto, flags)
future.set_result(result)
except Exception as e:
future.set_exception(result)
return future

On Thu, Dec 10, 2015 at 4:46 PM, Insoleet [email protected] wrote:

Thanks.

Here is what I could find :

• With the concurrent ThreadPoolExecutor, the same crash happens
• With a synchronized getaddrinfo, I can get a bigger stack, but there
  is still nothing linked to a call in my code :

DEBUG:main:async_exception_handler:Exception handler executing
ERROR:main:async_exception_handler:Task exception was never retrieved
future: <Task finished coro=<iter() done, defined at /home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py:454> exception=ClientOSError(None, 'Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]')>
Traceback (most recent call last):
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 557, in _create_connection
server_hostname=hinfo['hostname'] if sslcontext else None)
File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/base_events.py", line 569, in create_connection
type=socket.SOCK_STREAM, proto=proto, flags=flags)
File "/home/inso/code/quamash/quamash/init.py", line 481, in getaddrinfo
result = socket.getaddrinfo(host, port, family, type, proto, flags)
socket.gaierror: [Errno -2] Nom ou service inconnu

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 289, in connect
transport, proto = yield from self._create_connection(req)
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 580, in _create_connection
(req.host, req.port, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Can not connect to moul.re:8999 [None]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/tasks.py", line 238, in _step
result = next(coro)
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 456, in iter
resp = yield from self._coro
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 173, in _request
conn = yield from self._connector.connect(req)
File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 299, in connect
.format(key, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]

Some questions :

• Why in the asynchronous case the gaierror does not raise the aiohttp
  specific exceptions ?
• Why aren't they raised from my code, but they are from nowhere in
  aiohttp code ?

—
Reply to this email directly or view it on GitHub
#42 (comment).

from quamash.

When catching the exception and using set_exception, here is what I get :

DEBUG:main:async_exception_handler:Exception handler executing
ERROR:main:async_exception_handler:Task exception was never retrieved
future: <Task finished coro=<__iter__() done, defined at /home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py:454> exception=ClientOSError(None, 'Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]')>
Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 557, in _create_connection
    server_hostname=hinfo['hostname'] if sslcontext else None)
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/base_events.py", line 581, in create_connection
    infos = f1.result()
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/futures.py", line 275, in result
    raise self._exception
  File "/home/inso/code/quamash/quamash/__init__.py", line 482, in getaddrinfo
    result = socket.getaddrinfo(host, port, family, type, proto, flags)
socket.gaierror: [Errno -2] Nom ou service inconnu

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 289, in connect
    transport, proto = yield from self._create_connection(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 580, in _create_connection
    (req.host, req.port, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Can not connect to moul.re:8999 [None]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inso/.pyenv/versions/3.4.3/lib/python3.4/asyncio/tasks.py", line 238, in _step
    result = next(coro)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 456, in __iter__
    resp = yield from self._coro
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/client.py", line 173, in _request
    conn = yield from self._connector.connect(req)
  File "/home/inso/.pyenv/versions/cutecoin/lib/python3.4/site-packages/aiohttp/connector.py", line 299, in connect
    .format(key, exc.strerror)) from exc
aiohttp.errors.ClientOSError: [Errno None] Cannot connect to host moul.re:8999 ssl:False [Can not connect to moul.re:8999 [None]]

As you can see, the stacktrace is still out of the code. The exception is raised after being read in f1.result().

from quamash.