circuit breakers support about hug HOT 10 OPEN

I am very interested in this as well. I have looked at this issue before, and it just seems really hard to implement correctly in Python. A good circuit breaker would keep track of all requests across all threads/processes/greenlets and keep a global state.

Netflix has made Hystrix (https://github.com/Netflix/Hystrix/wiki/How-it-Works) and this wiki seems to explain how they solved some of the harder problems. The circuit breaker would almost certainly need to run in its own thread and manage worker thread pools as well. But since Hystrix is written for the JVM, I feel a Python implementation would run into a lot of performance issues from threading.

If you want to maybe help settle on a road map for this, I wouldn't mind taking a stab at the overall structure.

from hug.

@ryananguiano, as an alternative approach, I think you could get most of the benefits simply by taking advantage of a cache store that is available to all thread / processes (such as Redis / Memecached). I think conservatively in 99.9999% of the time, you really don't care if a race condition causes two requests to fail at the same time, and then cause the circuit breaker to take effect, so a non-locking pattern could be used for very good performance. An option could be added to the circuit breaker to enable locks for those that really need it.

What do you think about this approach?

from hug.

Hmm... I think requiring an external store for this feature would make it a lot harder to implement in production, but I think it should be pluggable if you would like to choose one of those stores. The other thing about using an external store, it has to keep a lot of stateful data and this in itself will generate a lot of overhead. And if that store happens to be across the network, then it misses the whole point.

Even if it suffers from performance, I think a pure Python implementation would fit the best into the Falcon ecosystem. I think multiprocessing will be the way to do it. If the system does not support it, we could of course always use multiprocessing.dummy which is just a wrapper around threading. But then again, this might be overkill and async/await might be able to handle it just fine.

from hug.

@ryananguiano, 100% agree that the data-store should be pluggable if used. I think the big problem I see with a pure Python implementation is that when using a standard like UWSGI you don't really have 100% control over how processes / threads are separated, these settings are defined by the actual server (a separate entity) and the end-users configuration.

from hug.

Also, as a counter point: circuit breakers are defined to handle all sorts of issues, I think the rarest of those is loss of network connectivity to everything (including all internal services) it's probably is perfectly reasonable to use a central, over internal network, cache and then define special logic when even that is down.

from hug.

I did find this which is pretty simple - https://github.com/danielfm/pybreaker/blob/master/src/pybreaker.py

I would still add a way to specify a cache and a fallback per each circuit breaker. Also, asynchronous responses are a huge requirement for me.

Can you think of any other features that should be present for a first draft?

from hug.

@ryananguiano I think that would be more then sufficient for an initial draft, and provide a tremendous amount of value. BTW thanks for adding your input and researching this issue it's very appreciated.

from hug.

So I was looking at a couple ways to do this, and I see valid use cases for different people needing either multiprocessing, threading, or concurrent tasks. So I think the best way to handle this is to make an interface that passes the necessary process info to whichever engine the user selects in the settings. The default should just be async concurrency so it only uses one process unless someone specifically configures it otherwise.

I have a couple questions to see which direction you want to go.

• I think this feature alone is going to add a lot of weight to the package. Do you think we should make it a subpackage that is an optional install? We could make it a separate package like hug-circuitbreaker and you can add it to your setup.py as an alternate install so it can be installed like hug[circuitbreaker].
• How do you feel about using external packages like pebble or gevent? We would use these packages to create the different "engines" that each connect our interface.

I'm going to be pulling a lot of influence from Hystrix and will make the most basic unit of work a CircuitCommand.

We can also have generic commands that will cover most use cases like a CircuitURLRequest or CircuitSocketRequest. We would then implement these in hug/use.py and also allow people to use in their code.

from hug.

Have you considered pybreaker? I don't know anything about it. I was
just curious about the topic.
On Feb 26, 2016 3:07 AM, "Ryan Anguiano" [email protected] wrote:

So I was looking at a couple ways to do this, and I see valid use cases
for different people needing either multiprocessing, threading, or
concurrent tasks. So I think the best way to handle this is to make an
interface that passes the necessary process info to whichever engine the
user selects in the settings. The default should just be async concurrency
so it only uses one process unless someone specifically configures it
otherwise.

I have a couple questions to see which direction you want to go.

I think this feature alone is going to add a lot of weight to the
package. Do you think we should make it a subpackage that is an optional
install? We could make it a separate package like hug-circuitbreaker and
you can add it to your setup.py as an alternate install so it can be
installed like hug[circuitbreaker].

How do you feel about using external packages like pebble
https://pypi.python.org/pypi/Pebble or gevent http://www.gevent.org?
We would use these packages to create the different "engines" that each
connect our interface.

I'm going to be pulling a lot of influence from Hystrix
https://github.com/Netflix/Hystrix and will make the most basic unit of
work a CircuitCommand.

We can also have generic commands that will cover most use cases like a
CircuitURLRequest or CircuitSocketRequest. We would then implement
these in hug/use.py and also allow people to use in their code.

—
Reply to this email directly or view it on GitHub
https://github.com/timothycrosley/hug/issues/146#issuecomment-189175260.

from hug.

Ya. I looked at that already. It seems to be a good start but seems to be missing some functionality that I require. I've been looking at this problem in Python for quite a while and I feel that I can architect a solid foundation that will be scalable in production.

from hug.