Code Monkey home page Code Monkey logo

vanilla's Introduction

Vanilla Welcome to Vanilla!

If Go and ZeroMQ had a baby, and that baby grew up and started dating PyPy, and they had a baby, it might look like Vanilla.

Overview

Vanilla allows you to build concurrent software in Python. Vanilla programs are structured around independent coroutines (greenlets) which communicate with each other via Pipes. Pipes are similar to channels in Go programming.

There's no callback crazyness and no monkey patching. Vanilla strives to be as explict and straightforward as possible.

Documentation

Read the Docs

Here's how it looks:

You spawn coroutines:

h = vanilla.Hub()

def beat(message):
    while True:
        print(message)
        h.sleep(1000)

h.spawn(beat, 'Tick')
h.spawn_later(500, beat, 'Tock')
# Tick / Tock / Tick / Tock

Coroutines communicate via Pipes:

h = vanilla.Hub()
sender, recver = h.pipe()
h.spawn(sender.send, 'Hello World')
recver.recv()
# 'Hello World'

Pipe-fu; inspired by reactive functional patterns, Pipes can be chained:

h = vanilla.Hub()
p = h.pipe().map(lambda x: x*2)
h.spawn(p.send, 4)
p.recv()
# 8

In Vanilla, everything is a Pipe. Here's how TCP looks:

h = vanilla.Hub()

server = h.tcp.listen(port=9000)
# server is a Recver which dispenses new TCP connections

conn = server.recv()
# conn is a Pipe you can recv and send on

message = conn.recv()
conn.send("Echo: " + message)

Installation

Vanilla works with Python 2.6 - 2.9 and PyPy.

pip install vanilla

Status

Build StatusCoverage Status

vanilla's People

Contributors

cablehead avatar davidbirdsong avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

vanilla's Issues

benchmarking, profiling

Hello again, Andy!

Just to call your attention to this, I put together a benchmarking network server in a couple of different greenlet I/O libraries, including vanilla: https://github.com/teepark/puffin/blob/master/bench_server.py (and the bench shell script).

You've inspired me to try and tighten up greenhouse's scheduler core, and in that "puffin" experiment I'm managing to (just) beat gevent on networking code with pure-python-to-epoll.

The bench_server.py server is intended to be as simple a networking test as possible, but still be usable against an http-based benchmarking tool (so for instance it goes straight to the socket and blindly reads until '\r\n\r\n', not wasting time doing anything but networking).

It can also activate cProfile during the bench run, but that's less illuminating than I'd hoped. Total time per call shows up as 0.000 for all functions in all libraries. It basically just comes down to python interpreter overhead, so the least python code wins.

Hope you find it helpful.
--travis

sys/signalfd.h not found

This is my first time using pypy and my cffi installation was weird (see #5) so I'm probably just doing something wrong. I get this when I import vanilla from pypy.

.../dev/vanilla/__pycache__/_cffi__ge2c3b10fx5ad6ec95.c:27:18: fatal error:
      'sys/signalfd.h' file not found
        #include <sys/signalfd.h>

cffi seems fine, import cffi works in pypy. Any ideas?

Exception in HTTP server


  File "/usr/local/lib/python2.7/dist-packages/vanilla/core.py", line 1274, in main
    self.registered[fd].send((fd, event))
  File "/usr/local/lib/python2.7/dist-packages/vanilla/core.py", line 405, in send
    return self.hub.switch_to(self.other.peak, self.other, item)
  File "/usr/local/lib/python2.7/dist-packages/vanilla/core.py", line 1108, in switch_to
    return target.switch(*a)
  File "/usr/local/lib/python2.7/dist-packages/vanilla/core.py", line 1637, in server
    HTTPServer(self.hub, socket, request_timeout, serve)
  File "/usr/local/lib/python2.7/dist-packages/vanilla/core.py", line 1927, in __init__
    request = self.read_request()
  File "/usr/local/lib/python2.7/dist-packages/vanilla/core.py", line 1974, in read_request
    method, path, version = self.socket.read_line().split(' ', 2)
ValueError: need more than 2 values to unpack

HTTP Server has no close method

    self.signal.subscribe(signal.SIGINT, signal.SIGTERM).recv()
  File "/home/andy/git/vanilla/vanilla/message.py", line 324, in recv
    return self.pause(timeout=timeout)
  File "/home/andy/git/vanilla/vanilla/message.py", line 203, in pause
    _, ret = self.hub.pause(timeout=timeout)
  File "/home/andy/git/vanilla/vanilla/core.py", line 305, in pause
    resume = self.loop.switch()
  File "/home/andy/git/vanilla/vanilla/core.py", line 496, in main
    masks[mask].send(True)
  File "/home/andy/git/vanilla/vanilla/message.py", line 260, in send
    return self.hub.switch_to(self.other.peak, self.other, item)
  File "/home/andy/git/vanilla/vanilla/core.py", line 322, in switch_to
    return target.switch(*a)
  File "bin/w", line 20, in _
    conn.close()
AttributeError: 'HTTPServer' object has no attribute 'close'

Couldn't install cffi

Setup.py was failing due to cffi failing (ffi.h not found). Easiest fix I found was to xcode-select --install to get libffi back on Mavericks. Unless there's an easier solution, this might warrant a readme update.

Test reclose off reused fd

Test: have a fd, close. get new fd from os, os hands us the previous fd closed, as its the current lowest fd available. The old user of fd calls close again - which closes the newly open fd.

Traceback in ticker example

Traceback (most recent call last):
  File "examples/ticker/ticker", line 162, in <module>
    sys.exit(main())
  File "examples/ticker/ticker", line 157, in main
    h.stop_on_term()
  File "/home/andy/git/vanilla/vanilla.py", line 974, in stop_on_term
    self.stop()
  File "/home/andy/git/vanilla/vanilla.py", line 960, in stop
    ch.send(Stop('stop'))
  File "/home/andy/git/vanilla/vanilla.py", line 488, in send
    self.hub.throw_to(getter, item)
  File "/home/andy/git/vanilla/vanilla.py", line 930, in throw_to
    return target.throw(*a)
  File "/home/andy/git/vanilla/vanilla.py", line 1001, in main
    run_task(task, *a)
  File "/home/andy/git/vanilla/vanilla.py", line 994, in run_task
    task.switch(*a)
  File "examples/ticker/ticker", line 52, in collect
    headers=headers).recv()
  File "/home/andy/git/vanilla/vanilla.py", line 511, in recv
    item = self.hub.pause(timeout=timeout)
  File "/home/andy/git/vanilla/vanilla.py", line 907, in pause
    resume = self.loop.switch()
vanilla.Closed: closed

unexpected behavior when iterating over a pipe

import vanilla
import signal
h = vanilla.Hub()

p = h.process.execv(('/bin/sleep', '30000'))

done = h.router()
h.signal.subscribe(signal.SIGINT, signal.SIGTERM).pipe(done)
p.done.pipe(done)

svc_name = 'test_service'
cclient = h.consul()
for x in done.recver:
    if type(x) == int:
        print 'killing child'
        p.terminate()
    else:
        cclient.agent.service.deregister(svc_name).recv()
        done.close()

throws:

[david@foulplay]$ python /tmp/t.py  
killing child
Traceback (most recent call last):
  File "/tmp/t.py", line 22, in <module>
    done.close()
  File "/usr/lib/python2.7/dist-packages/vanilla/message.py", line 104, in close
    self.recver.close()
  File "/usr/lib/python2.7/dist-packages/vanilla/message.py", line 222, in close
    self.hub.throw_to(self.other.current, exception)
  File "/usr/lib/python2.7/dist-packages/vanilla/core.py", line 335, in throw_to
    return target.throw(*a)
AttributeError: 'collections.deque' object has no attribute 'throw'

Traceback in ticker example

Traceback (most recent call last):
  File "examples/ticker/ticker", line 165, in <module>

  File "examples/ticker/ticker", line 160, in main

  File "/home/andy/git/vanilla/vanilla.py", line 973, in stop_on_term
    done.recv()
  File "/home/andy/git/vanilla/vanilla.py", line 511, in recv
    item = self.hub.pause(timeout=timeout)
  File "/home/andy/git/vanilla/vanilla.py", line 907, in pause
    resume = self.loop.switch()
  File "/home/andy/git/vanilla/vanilla.py", line 1043, in main
    self.registered[fd].send((fd, event))
  File "/home/andy/git/vanilla/vanilla.py", line 490, in send
    self.hub.switch_to(getter, (self, item))
  File "/home/andy/git/vanilla/vanilla.py", line 924, in switch_to
    return target.switch(*a)
  File "examples/ticker/ticker", line 56, in collect

  File "/home/andy/git/vanilla/vanilla.py", line 1583, in consume
    return ''.join(self.body)
  File "/home/andy/git/vanilla/vanilla.py", line 524, in __iter__
    yield self.recv()
  File "/home/andy/git/vanilla/vanilla.py", line 511, in recv
    item = self.hub.pause(timeout=timeout)
  File "/home/andy/git/vanilla/vanilla.py", line 907, in pause
    resume = self.loop.switch()
vanilla.Interrupted

Exception in HTTP Client

  File "/Users/andy/git/market/market/api.py", line 275, in body
    return self.get(target, params={'query': 'income-statement'})
  File "/Users/andy/git/market/market/api.py", line 174, in get
    body = ch.recv().consume()
  File "/Users/andy/git/vanilla/vanilla/http.py", line 179, in consume
    return ''.join(self.body)
  File "/Users/andy/git/vanilla/vanilla/message.py", line 329, in __iter__
    yield self.recv()
  File "/Users/andy/git/vanilla/vanilla/message.py", line 324, in recv
    return self.pause(timeout=timeout)
  File "/Users/andy/git/vanilla/vanilla/message.py", line 203, in pause
    _, ret = self.hub.pause(timeout=timeout)
  File "/Users/andy/git/vanilla/vanilla/core.py", line 305, in pause
    resume = self.loop.switch()
  File "/Users/andy/git/vanilla/vanilla/core.py", line 496, in main
    masks[mask].send(True)
  File "/Users/andy/git/vanilla/vanilla/message.py", line 255, in send
    self.pause(timeout=timeout)
  File "/Users/andy/git/vanilla/vanilla/message.py", line 203, in pause
    _, ret = self.hub.pause(timeout=timeout)
  File "/Users/andy/git/vanilla/vanilla/core.py", line 301, in pause
    assert getcurrent() != self.loop, "cannot pause the main loop"
AssertionError: cannot pause the main loop

Hang establishing TCP connection

return HTTPClient(self.hub, url)

File "/home/ubuntu/git/vanilla/vanilla/http.py", line 194, in init
self.socket = self.hub.tcp.connect(host=host, port=port)
File "/home/ubuntu/git/vanilla/vanilla/tcp.py", line 41, in connect
conn.connect((host, port))
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)

My experience installing on Ubuntu (precise32 then trusty32 then trusty64)

I know you're not in the business of Linux training but...

I had to apt-get update and then apt-get install python-dev from a fresh precise32 vagrant box. Then I had to apt-get install libffi-dev before python setup.py install would work. Of course, I also had to apt-get install pypy to get pypy. I guess because of the order I did things, I then needed to pip install --upgrade cffi because pypy couldn't find it for some reason (even after messing with my PYTHONPATH and redoing python setup.py install).

After all that, I realized apt-get had installed pypy 1.8 which is incompatible with cffi...so I started from scratch with a trusty32 box. Hooray. Installed pypy right away to confirm it is 2.0+ and did an import cffi to make sure I'm good, and then I had to install both pip and python-dev. Finally python setup.py install was mostly working except for the stupid ffi again. But since pypy 2.0+ comes with cffi I was finally able to import vanilla and get started. I went back and did apt-get install libffi-dev just in case and that made python setup.py install happy again.

Unfortunately I now have a pypy/routes problem (I can import routes in python but not in pypy) and I have to give up for tonight.

And all this makes me think...you might consider writing a short walkthrough for a standard Ubuntu distro on a vagrant box to make it easier for super newbs to get started.

HTTP Client Exception

  File "/Users/andy/git/market/market/api.py", line 412, in body
    body = conn.get(self.PATH, params=params).recv().consume()
  File "/Library/Python/2.7/site-packages/vanilla/http.py", line 179, in consume
    return ''.join(self.body)
  File "/Library/Python/2.7/site-packages/vanilla/message.py", line 329, in __iter__
    yield self.recv()
  File "/Library/Python/2.7/site-packages/vanilla/message.py", line 324, in recv
    return self.pause(timeout=timeout)
  File "/Library/Python/2.7/site-packages/vanilla/message.py", line 203, in pause
    _, ret = self.hub.pause(timeout=timeout)
  File "/Library/Python/2.7/site-packages/vanilla/core.py", line 305, in pause
    resume = self.loop.switch()
  File "/Library/Python/2.7/site-packages/vanilla/core.py", line 496, in main
    masks[mask].send(True)
  File "/Library/Python/2.7/site-packages/vanilla/message.py", line 255, in send
    self.pause(timeout=timeout)
  File "/Library/Python/2.7/site-packages/vanilla/message.py", line 203, in pause
    _, ret = self.hub.pause(timeout=timeout)
  File "/Library/Python/2.7/site-packages/vanilla/core.py", line 301, in pause
    assert getcurrent() != self.loop, "cannot pause the main loop"
AssertionError: cannot pause the main loop

Add SSL support to bean, HTTPClient, and TCP

Sadly I can't use the bean client to do what I need without support for SSL. I tried to hack in port 443 and socket.ssl() but I wasn't able to get it working in the short amount of time that I had.

I need SSL to connect to various data sources that require it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.