Thread Exception when running Chaussette with gevent backend on a Django1.6 wsgi app about chaussette HOT 18 CLOSED

Try to use the gevent aware db adapter in django
https://pypi.python.org/pypi/gevent-psycopg2

And probably apply monkey_patch() as early as possible.
see: https://github.com/zacharyvoase/gevent-psycopg2

import gevent_psycopg2; gevent_psycopg2.monkey_patch()

I'm also not sure if you are using the right WSGIHandler. Can you post your circus configuration?

from chaussette.

Hi
The cmd in my circus watcher is:

cmd = chaussette --backend gevent --fd $(circus.sockets.frontend) --pre-hook myproject.prehook.pre_hook myproject.wsgi.application

Full Circus config: http://pastebin.com/7cdkeSwV

The prehook is where I already use psycopg patch (https://bitbucket.org/dvarrazzo/psycogreen/)

def pre_hook(args):
    from psycogreen.gevent import patch_psycopg
    patch_psycopg()

and myproject.wsgi.application is the default Django wsgi application:

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

Regards,

from chaussette.

Thanks for the hook!

from chaussette.

Closing since it seems to be resolved

from chaussette.

shouldn't this be documented somewhere?

from chaussette.

I have the exact same problem with django 1.6.5, chaussette and circus, any fix ?

from chaussette.

The hook has improved the stability of circus and chaussette when I applied it but this error was still occurring sometimes. So, as I wanted to keep using circus for different reasons, I tried all the different backends. The only one which has really been fully stable is "waitress". Now, it's 7 months I'm using it in a my staging and production environment and I never had any issues. My only regret is about performance, "waitress" a quite good but "fastgevent" was really amazing! Maybe later I'll try again :-)

from chaussette.

But you still use waitress with a hook ?

from chaussette.

Reopening so we address the doc issue

from chaussette.

I'm using waitress without the hook.

from chaussette.

The hook indeed doesn't solve the thread exception problem. So far what I have reached is that the problem is related to how chaussette uses gevent monkey patch. Gevent monkey.patch_all should be called in the early stage before using patched modules, like socket, threading etc...
Socket is imported in server.py, while gevent monkey patching is done in the backend. When I modified server.py to patch_all first thing, I couldn't reproduce the issue (tested with django==1.6.5)

here is the diff

diff --git a/chaussette/server.py b/chaussette/server.py
index 9666181..64c8d0d 100644
--- a/chaussette/server.py
+++ b/chaussette/server.py
@@ -1,3 +1,6 @@
+from gevent import monkey
+monkey.patch_all()
+
 import sys
 import os
 import argparse

Although this solves the issue, but it is not a generic solution. It is more like a gevent specific. So it needs to be implemented in a cleaner way.

It is also worth mentioning, that I faced Celery related instability issues when load testing my Django app. While load testing, some celery tasks "went missing" (using RabbitMQ & Postgresql as result backend). The above patch solved the instability problem as well.

I haven't tested the above solution heavily, but if someone can test it we can confirm the cause of the issue.

from chaussette.

Thanks for your tests.

Although this solves the issue, but it is not a generic solution. It is more like a gevent specific. So it needs to be implemented in a cleaner way.

maybe an option like --gevent-monkey-patch ?

from chaussette.

or by checking if --backend is gevent related backend.

from chaussette.

yes that too. But do we want to force the monkey patching everytime in that case ? I am not sure

from chaussette.

The monkey patch is already applied on gevent backends (_gevent, _fastgevent, _geventwebsocket), but it is applied late.

from chaussette.

ah right!

It's weird because we only read constants out of the socket module, and the patching seems to be applied before the socket creation: https://github.com/mozilla-services/chaussette/blob/master/chaussette/backend/_gevent.py#L32

could be interesting to follow with pdb the initialization process and see why we don't get the patched version. Maybe it's just a matter of moving that import line after the monkey patch : https://github.com/mozilla-services/chaussette/blob/master/chaussette/backend/_gevent.py#L4

from chaussette.

Did a quick test with _gevent backend monkey patched at the start of the file.
https://github.com/mozilla-services/chaussette/blob/master/chaussette/backend/_gevent.py#L1
Added the patching before importing any modules:

from gevent import monkey
monkey.patch_all()
import socket
from gevent.pywsgi import WSGIServer, WSGIHandler
# from gevent import monkey
from chaussette.util import create_socket

I couldn't reproduce the thread exception, and the env seemed to be stable. So I guess this can be the cleaner fix.

from chaussette.

Fixed by #62

from chaussette.