Comments (5)
In issue #11, I tried to extend the lib with an Enqueue method, but there is a problem with the pool management:
I cannot guess when I can close it easily, because the original code close the pool when there is no more worker.
In fact I had to disable this but I cannot sleep anymore since then :-)
Sharing a pool is a very important feature I think because you want to be able to define the number of available connections for the whole application, not per library.
The problem should be solved by doing the opposite, there should be a way to pass a connection pool to Goworker. That the role of the application to manage it, not from the library.
What do you think?
from goworker.
@oliamb injecting your own redis pool sound great as long as it implements an interface that guarantees the operations goworker expects, exist in the custom worker pool.
There is however an inherent risk here sharing the pool between goworker and the client app/workers in that either one could in theory monopolise the redis connections and DoS the other. For this reason it may be better to keep them separate.
from goworker.
First, I should state that the actual behavior should be kept for the sake of simplicity. Passing the pool should be considered optional. From there, that's up to the developer to decide whether he trust his workers enough to share a common pool or if he wants to have different pools for different parts.
- DoS: yes, true. I had this concern about the Enqueue func. It seems that at least goworker do it right by pushing the connection back to the pool before processing the job, so even a pool containing one connection will not run into some kind of dead lock ; as long as the job also push the connection back as well.
- Implementing the right interface: true again, this is a somewhat hard dependency on redigo and code.google.com/p/vitess/go/pools, but probably it is ok?
from goworker.
@oliamb so you'd only be able to inject a vitess/redigo pool for now?
That's fine as it changes less code.
If someone needs to use another pool/redis client the option still remains to insert an interface into goworker that all pools need to conform to. This can be built when the need arises though.
from goworker.
That makes sense to me. I'd be fine with making my own redigo pool and then giving that to goworker to use.
from goworker.
Related Issues (20)
- How to panic handling on Job? HOT 2
- 100% cpu usage HOT 17
- Any plans to support Sidekiq backend? HOT 4
- Refactor the Enqueue API
- RabbitMQ Support HOT 1
- Example from Getting Started section doesn't run
- New version HOT 3
- How does goworker work? HOT 4
- Timeout for stuck workers HOT 6
- Any plan to replace garyburd/redigo with gomodule/redigo? HOT 2
- redis please use go-redis
- Supporting Resque Scheduler
- package broken
- missing the prefillParallelism argument in redis.go
- etcd support
- It's graceful, but how to optimize cpu usage? HOT 2
- pools.NewResourcePool request param has changed
- fail to install HOT 4
- There is atomicity problem when worker key is flushed by redis pool
- [Critical] Error on closing worker ip-xxx-xx-xx-xx:xxxxxx-4:xxx: redis: client is closed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goworker.