dieseldev / diesel Goto Github PK
View Code? Open in Web Editor NEWGreenlet-based event I/O Framework for Python
Home Page: http://diesel.io
License: Other
Greenlet-based event I/O Framework for Python
Home Page: http://diesel.io
License: Other
I couldn't find the flask example on diesel.io homepage in git or I'd have diffed it and issued a pull request. At any rate, it's missing:
from diesel.util.queue import deque
at the beginning
and
app.run(debug=True)
at the end
A dieselweb.org website would be great, actually. There's a lot of competition in the async Python library space. I would feel better about diesel if it were more ... presentable.
Hi there,
Diesel seems a really interesting project, such a shame it is not available for python3 !
So I've forked the repo (https://github.com/touilleMan/diesel/tree/py3k) and started to port the code ;-)
So far, all the tests are passing for Python 2.6, 2.7, 3.3 and 3.4 using the same codebase, but it seems those tests are not really exhaustive and there is still some work to do !
Now I'm trying to make work all the examples available.
My main point of concern is how to handle the strings:
str
is an array of bytes (bytes
type is an alias of str
), to use unicode we must is unicode
typestr
is an array of unicodes characters and only bytes
an array of bytesConsidering Diesel, all the code is using classical str
types (i.e. array of bytes), this seems pretty logical to me given there is no standard way to determine the encoding of data coming from a socket.
The downside of this is to make the code compatible with Python3, all the strings needs to be explicitly converted to bytes, leading to a lot of small changes, an code more complex (numerous conversions unicode => bytes).
Example:
# Python2 only
send('AUTH', self.password)
send("HTTP/%s %s %s\r\n" % (('%s.%s' % version), resp.status_code, resp.status))
# Python2&3
send(b'AUTH', self.password.encode())
send(("HTTP/%s %s %s\r\n" % (('%s.%s' % version), resp.status_code, resp.status)).encode())
I've started to see if the core.send/core.receive function could be used to handle encoding, but - although it would be a nice solution - I don't believe this could lead to a easy drop-in replacement.
def send(data, priority=5, encoding='ascii'):
if isinstance(data, builtins.str):
data = data.encode(encoding)
return current_loop.send(data, priority=priority)
def receive(spec=None, encoding='ascii'):
data = current_loop.input_op(spec)
if isinstance(data, builtins.bytes):
data = data.decode(encoding)
return data
Do you have any advice/idea about the best way to handle this ?
Hi Diesel,
How can I have client user's information on Diesel and ofcourse in server side.
I want informations like: ip,port,user,...
I want to write user's presence for chat application.
the http examples don't work with pyparser, you must have had Cython when you install http_parser for them to work
Might actually be more than the example ...
$ python examples/http_client.py
[2013/03/20 06:22:32] {diesel} WARNING:Starting diesel <hand-rolled select.epoll>
[2013/03/20 06:22:32] {http-client} INFO:<Response 34645 bytes [200 OK]>
[2013/03/20 06:22:32] {diesel} ERROR:-- Unhandled Exception in local loop <<function req_loop at 0x2e0fb18>> --
Traceback (most recent call last):
File "/home/christian/src/bump/server/contrib/diesel/diesel/core.py", line 189, in run
self.loop_callable(*self.args, **self.kw)
File "examples/http_client.py", line 15, in req_loop
log.info(str(client.request('GET', '/', heads)))
File "/home/christian/src/bump/server/contrib/diesel/diesel/core.py", line 141, in __call__
raise ClientConnectionClosed(str(e), addr=self.client.addr, port=self.client.port)
ClientConnectionClosed: Connection closed by remote host (addr=www.jamwt.com, port=80)
^C[2013/03/20 06:22:34] {diesel} WARNING:-- KeyboardInterrupt raised.. exiting main loop --
[2013/03/20 06:22:34] {diesel} INFO:Ending diesel application
hi I use this code
https://bitbucket.org/sjl/diesel/src/4f9dff1bb14679d55f96d46dc58c7543d39b5174/examples/chat.py?at=default
but when I try telnet the Connection closed by foreign host. I think It happen because of yield . when I remove It from code telnet work fine . but the aplication dosen't work right!
can you give me a simple example of chat server with diesel?
Where can I find the documentation of this project? dieselweb.org redirects to github.com/jamwt/diesel/
Diesel website is not opening.
pip installation log:
Downloading/unpacking diesel==3.0.16 (from -r /opt/nagios-api/requirements.txt (line 5))
Could not fetch URL https://pypi.python.org/simple/diesel/3.0.16: HTTP Error 404: Not Found
Will skip URL https://pypi.python.org/simple/diesel/3.0.16 when looking for download links for diesel==3.0.16 (from -r /opt/nagios-api/requirements.txt (line 5))
Ignoring link https://pypi.python.org/packages/source/d/diesel/diesel-2.1.0.tar.gz#md5=bf52eb9c2d51fedc667348af613f3250 (from https://pypi.python.org/simple/diesel/), version 2.1.0 doesn't match ==3.0.16
Ignoring link https://pypi.python.org/packages/source/d/diesel/diesel-2.1.1.tar.gz#md5=0184b336fea8f5d4482befde5716ce14 (from https://pypi.python.org/simple/diesel/), version 2.1.1 doesn't match ==3.0.16
Ignoring link https://pypi.python.org/packages/source/d/diesel/diesel-3.0.5.tar.gz#md5=c70f7e096e1722b9b4eecd60af1fdf1a (from https://pypi.python.org/simple/diesel/), version 3.0.5 doesn't match ==3.0.16
Ignoring link http://download.dieselweb.org/diesel-0.9.0b.tar.gz (from https://pypi.python.org/simple/diesel/), version 0.9.0b doesn't match ==3.0.16
Ignoring link http://download.dieselweb.org/diesel-0.9.1b.tar.gz (from https://pypi.python.org/simple/diesel/), version 0.9.1b doesn't match ==3.0.16
Ignoring link http://jamwt.com/diesel/diesel-2.0.0.tar.gz (from https://pypi.python.org/simple/diesel/), version 2.0.0 doesn't match ==3.0.16
Ignoring link http://jamwt.com/diesel/diesel-2.1.0.tar.gz (from https://pypi.python.org/simple/diesel/), version 2.1.0 doesn't match ==3.0.16
Ignoring link http://jamwt.com/diesel/diesel-2.1.1.tar.gz (from https://pypi.python.org/simple/diesel/), version 2.1.1 doesn't match ==3.0.16
Ignoring link http://jamwt.com/diesel/diesel-3.0.5.tar.gz (from https://pypi.python.org/simple/diesel/), version 3.0.5 doesn't match ==3.0.16
HTTP error 403 while getting http://jamwt.com/diesel/diesel-3.0.16.tar.gz (from https://pypi.python.org/simple/diesel/)
Could not install requirement diesel==3.0.16 (from -r /opt/nagios-api/requirements.txt (line 5)) because of error HTTP Error 403: Forbidden
Could not install requirement diesel==3.0.16 (from -r /opt/nagios-api/requirements.txt (line 5)) because of HTTP error HTTP Error 403: Forbidden for URL http://jamwt.com/diesel/diesel-3.0.16.tar.gz (from https://pypi.python.org/simple/diesel/)
Looks like problem is caused by user agent set by urlib.
Next request works:
curl -L -o /dev/null -v http://jamwt.com/diesel/diesel-3.0.16.tar.gz
This one doesn't work:
curl -H "User-Agent: Python-urllib" -v http://jamwt.com/diesel/diesel-3.0.16.tar.gz
<title>Access denied | jamwt.com used CloudFlare to restrict access</title>
...
<h2 data-translate="what_happened">What happened?</h2>
<p>The owner of this website (jamwt.com) has banned your access based on your browser's signature (171b9664133c03d6-ua48).</p>
The reddit group is not used/useful and everything that happens on irc is lost in the ether... hard for a community to build up when there's limited means of communication.
(Using Ubuntu 12.10 and packages from the official apt repositories)
When using WSGIResource/Site in order to run a DieselFlask app (in these examples, "app"), you'll always end up with getting some exception from Diesel and Twisted's internal code.
from twisted.internet import reactor
from twisted.web.wsgi import WSGIResource
from twisted.web.server import Site
resource = WSGIResource(reactor, reactor.getThreadPool(), app)
site = Site(resource)
reactor.listenTCP(8080, site)
reactor.run()
With this method (app is a simple "hello world" route on /), going to / throws this exception:
WSGI application error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 167, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 332, in run
self.reactor.callFromThread(wsgiError, self.started, *exc_info())
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 315, in run
appIterator = self.application(self.environ, self.startResponse)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1518, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1502, in wsgi_app
with self.request_context(environ):
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1463, in request_context
return RequestContext(self, environ)
File "/usr/lib/python2.7/dist-packages/flask/ctx.py", line 87, in __init__
self.url_adapter = app.create_url_adapter(self.request)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1348, in create_url_adapter
return self.url_map.bind_to_environ(request.environ,
exceptions.AttributeError: 'dict' object has no attribute 'environ'
After several tries, @dontcare4free managed to make a WSGI middleware that fixes that case. More specifically:
from twisted.internet import reactor
from twisted.web.wsgi import WSGIResource
from twisted.web.server import Site
from flask.wrappers import Request
from UserDict import UserDict
class _RequestDict(UserDict, Request):
def __init__(self, environ):
Request.__init__(self, environ)
UserDict.__init__(self)
@property
def data(self):
return self.environ
@data.setter
def data(self, data):
pass # Bad, but required for this
def environFixerMiddleware(f):
def wrapper(environ, start_response):
return f(_RequestDict(environ), start_response)
return wrapper
resource = WSGIResource(reactor, reactor.getThreadPool(), environFixerMiddleware(app))
site = Site(resource)
reactor.listenTCP(8080, site)
reactor.run()
This works for a simple "hello world" example. However, when throwing more complicated stuff like WebSockets and such (taken from examples) into the cocktail, another exception arises, which ends up in the actual code that handles exceptions:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 167, in _worker
result = context.call(ctx, function, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 332, in run
self.reactor.callFromThread(wsgiError, self.started, *exc_info())
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 315, in run
appIterator = self.application(self.environ, self.startResponse)
File "avenir-api/hack.py", line 19, in wrapper
return f(_RequestDict(environ), start_response)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1518, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1506, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1194, in handle_exception
self.log_exception((exc_type, exc_value, tb))
File "/usr/local/lib/python2.7/dist-packages/diesel/web.py", line 61, in log_exception
self._logger.trace().error('Exception on {0} [{1}]',
exceptions.AttributeError: 'NoneType' object has no attribute 'trace
Not really sure whether this is an issue with Diesel or Twisted, though.
$ dpython
[2013/03/19 05:40:35] {diesel} WARNING:Starting diesel <hand-rolled select.epoll>
Python 2.7.1+ (r271:86832, Sep 27 2012, 21:12:17)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import diesel
>>> diesel.wait('for-something')
^C[2013/03/19 05:40:57] {diesel} WARNING:-- KeyboardInterrupt raised.. exiting main loop --
[2013/03/19 05:40:57] {diesel} INFO:Ending diesel application
$
This doesn't happen in the standard Python interpreter when a blocking operation is in progress. It just interrupts the operation and you can continue on your merry way.
$ python
Python 2.7.1+ (r271:86832, Sep 27 2012, 21:12:17)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import time
>>> time.sleep(10)
^CTraceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyboardInterrupt
>>>
This line in hub.py raises an AttributeError.
AttributeError: 'module' object has no attribute 'version'
I am using pyev==0.9.0 which was released this year (after 2 years of being stale) so I imagine this is a breaking change on their part. Quickest fix was to just hard code the version on my local install.
It's trying to typecast to int, e.g.
E:>python test.py
Traceback (most recent call last):
File "test.py", line 1, in
from diesel import Application, Service, until_eol, send
File "E:\Python\python-2.7.6\lib\site-packages\diesel__init__.py",
line 7, in
from app import Application, Service, UDPService, quickstart, quickstop, Thu
nk
File "E:\Python\python-2.7.6\lib\site-packages\diesel\app.py", line
10, in
from diesel.hub import EventHub
File "E:\Python\python-2.7.6\lib\site-packages\diesel\hub.py", line
18, in
import fcntl
ImportError: No module named fcntl
It would be very useful if you could use the same style for both connection methods.
Locally I'm emulating this by using a queue for every "connection" and a udp datagram send/receive loop that will create a new loop if a datagram is received from an unknown host, and then queue messages to that loop when further messages are received. Those loops then can send back data using a queue that the send/receive loop checks, and a modified diesel.send that allows sending already constructed datagrams (ie: not setting the address to self.parent.remote_addr)
it would be great if the framework abstracted that away for me so that a loop is created when an unknown host sends a udp datagram and within that loop I could use something along the lines of diesel.receive(diesel.datagram) and diesel.send(message), and just know the datagram is from that current "connection" and that diesel.send will respond to the correct addr/socket.
This would be especially useful when the server may be sending messages that are not in direct response to a received message.
Just a note for upstream fix. On mac os x 10.7.3 with python 2.7.1,
pip install pyev
fails with a permissions error on configure. The fix is:
chmod 755 configure
This will prevent pyev from installing cleanly, and subsequently diesel from running.
The title is self explanatory... I think it would be cool to mention the github repo for issues and people that want to follow the development/download a dev version.
I am trying to install diesel in widows xp but I faced following error:
cc1.exe : error : unrecognized command line option ' -mon-cygwin'
error: setup script exited with error : command 'gcc' failed with exit status 1
my friends have already installed it easily in linux and believed that it is because of windows. Have I to go to linux?
# cd "$(mktemp -d)"
# virtualenv fooo
# python --version
Python 3.6.6
# source fooo/bin/activate
# pip install https://github.com/dieseldev/diesel/archive/master.tar.gz
[..]
Successfully installed Jinja2-2.10 MarkupSafe-1.1.0 Werkzeug-0.14.1 asn1crypto-0.24.0 cffi-1.11.5 click-7.0 cryptography-2.4.2 diesel-3.0.24 dnspython-1.16.0 flask-1.0.2 greenlet-0.4.15 http-parser-0.8.3 idna-2.8 itsdangerous-1.1.0 pycparser-2.19 pyopenssl-18.0.0 six-1.12.0 twiggy-0.4.7
# python -c 'import diesel'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/tmp.3TqpDbqaeU/fooo/lib/python3.6/site-packages/diesel/__init__.py", line 2, in <module>
from logmod import log, levels as loglevels, set_log_level
ModuleNotFoundError: No module named 'logmod'
The from logmod
would ne to be from diesel.logmod
I believe.
Related: #95
It would be nice if MongoClient aliased getattr to getitem to better support dynamic db/collection naming.
with mongo[database][collection].find() as cursor:
vs.
with getattr(getattr(mongo, database), collection).find() as cursor:
Hi,
from diesel import response
ImportError: cannot import name response
I am getting an exception while working through the examples. Has request moved or renamed?
Regards,
Rahul
Would be awesome :)
Love the look of the framework, coming from other languages & frameworks, and I know a bit of python, but I'm starting my first full python project & learning the diesel framework at the same time and more framework documentation would be great!
The examples are definitely helping though
Traceback:
File ".../diesel/protocols/mongodb.py", line 302, in __exit__
raise RuntimeError("need to cleanup cursor!")
RuntimeError: need to cleanup cursor!
I'm running a local mirror of several python packages, and the latest diesel tarball appears to cause our mirror to fail with the following error:
ValueError(u'No PKG-INFO in archive: /opt/packages/private/diesel-3.0.24.tar.gz',)
It seems all the other packages we mirror have this file, including an older version -- diesel-3.0.16.tar.gz
Seems this manifest is missing from the current tarball?
The PKG-INFO is present in the python.org zip, but not the github tarball
$ wget https://pypi.python.org/packages/source/d/diesel/diesel-3.0.24.zip
$ unzip -l diesel-3.0.24.zip | grep PKG
1274 2015-03-14 20:11 diesel-3.0.24/PKG-INFO
1274 2015-03-14 20:11 diesel-3.0.24/diesel.egg-info/PKG-INFO
$ wget https://github.com/dieseldev/diesel/archive/diesel-3.0.24.tar.gz
$ tar -tzf 3.0.23.tar.gz | grep PKG
$
And I can't turn it off.
http://dieselweb.org is not working and everywhere that mention diesel points to it.
In the tutorial the Queues section has the following code in each consumer before the while loop.
item = queue.get()
This item doesn't get processed and confused me. After running the code, realized it was just a typo.
Steps:
extquery_prepare("SELECT count(*) FROM foo WHERE bar IN $1", "foo")
extquery(([1,2,3],), "foo")
Traceback:
File ".../diesel/protocols/pgsql.py", line 276, in <genexpr>
params = ''.join(struct.pack('!I%ss' % len(p), len(p), p) for p in self.params)
error: argument for 's' must be a string
from twiggy import log as olog, addEmitters, levels, outputs, formats, emitters
ImportError: cannot import name addEmitters
in file /usr/local/lib/python2.7/dist-packages/diesel/logmod.py
need to be changed to add_emitters to match logmod namespaces
works fine.
[x] Bug (Typo)
specfied
, however expect to see specified
.respones
, however expect to see response
.othewise
, however expect to see otherwise
.explictly
, however expect to see explicitly
.delimitted
, however expect to see delimited
.concurreny
, however expect to see concurrency
.bootsraps
, however expect to see bootstraps
.avaiable
, however expect to see available
.Semi-automated issue generated by
https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md
To avoid wasting CI processing resources a branch with the fix has been
prepared but a pull request has not yet been created. A pull request fixing
the issue can be prepared from the link below, feel free to create it or
request @timgates42 create the PR. Alternatively if the fix is undesired please
close the issue with a small comment about the reasoning.
https://github.com/timgates42/diesel/pull/new/bugfix_typos
Thanks.
Title says it all. I couldn't think of other ways to contact you so
Currently the TCP transport supports the concept of priority when sending. This bubbles up into the diesel.send function. Other transports have a dummy/unused priority kwarg to comply with diesel.send/the TCP implementation.
Instead of each transport being forced to support options that don't map directly to it, each should support receiving a dictionary of kwargs from diesel.send and acting on values that may exist there.
at least that I can see.
The PyPI entry has no PyPI-hosted downloads of Diesel -- https://pypi.python.org/pypi/diesel/3.0.16
Currently it's using a download URL to an external site, but pip seems to be deprecating that.
vagrant@ubuntu:~/dev/memcacheliked$ .tox/py26/bin/pip install diesel
Downloading/unpacking diesel
You are installing an externally hosted file. Future versions of pip will default to disallowing externally hosted files.
You are installing a potentially insecure and unverifiable file. Future versions of pip will default to disallowing insecure files.
Error <urlopen error [Errno -3] Temporary failure in name resolution> while getting http://jamwt.com/diesel/diesel-3.0.16.tar.gz (from https://pypi.python.org/simple/diesel/)
...
Bonus points for uploading wheels and/or eggs.
Waiting on background events doesn't work in dpython
or idpython
. I'm pretty sure that both are calling raw_input()
and blocking the event loop.
Need to investigate using diesel.util.stream
and fitting that into the diesel.interactive
code (which dpython
and idpython
use).
Unless it's going to be built out further, we should remove diesel.convoy. It's confusing for it to be there in sort of a half-baked state.
The constructor makes it a defaultdict, but parse makes it a dict. Should that latter be a defaultdict as well?
https://github.com/jamwt/diesel/blob/master/diesel/protocols/http.py#L99
The PostgresClient constructor doesn't require an explicit port. It'd be nice if the MongoClient constructor did the same.
I am seeing a problem where if I do a lot of reads and updates at the same time, a read will fail with an empty deque error in MongoIter.next.
Ok, I can add a new service and loop into Application, but and if I want to remove just one service?
If I pass a unicode object to simplequery, I get an error from line 111 of pgsql.py. Traceback:
File ".../diesel/protocols/pgsql.py", line 111, in send
tosend = self.idbyte + struct.pack('!I', len(allbytes)+4) + allbytes
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa1 in position 4: ordinal not in range(128)
This is because the struct.pack() output is being implicitly coerced to ascii by the concatenation with the unicode allbytes. Presumably the rest of the module should be checked for this bug as well. The workaround is to encode sql to str before passing to simplequery, like so:
sql = sql.encode('ascii')
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.