Comments (5)
Doesn't Resque do this though?
Or do a rescue in Go?
from goworker.
Hi,
I am experimenting with the goworker library.
I have a requirement of stopping and starting jobs.
Is it possible with the current version? Can anyone tell any workaround for it?
from goworker.
@rohit4813 The current implementation listens for few signals and if it receives them, it stops enqueuing new jobs but lets the running jobs finish.
I'm not sure I understand exactly what you're trying to do, but here's our use: We have a scenario where each job is potentially pretty long but has a natural stopping point. For this, we create a channel in the main function that gets written into when a signal is received (basically he same code that is in goworker already). Then we create another channel, this time buffered (capacity = number of workers) and pass that channel to each worker. Workers then select
from that channel at natural stopping points. In a separate goroutine (kicked off from the main function), we read from the signals channel and write N times (= number of workers) to the channel.
The whole flow looks like:
- The process receives a signal
- Both our signals channel and goworker's signals channels are written into
- Goworker stops enqueuing new jobs
- We copy the event N times
- When any worker finishes, it's handled normally
- When a worker gets to a checkpoint where it checks the channel, it
return
s and goworker takes care of it
from goworker.
@mingan Thanks for the great explanation.
If I understand correctly, all the workers will read from the workers channel(which gets populated from the signals channel)?
And I have a use case where I want to stop single/multiple worker(s), say which are running for a very long time and if that is the case all the workers will stop on passing the signal to the channel.
I can identify the worker on which the job is running for a very long time. How can I send the signal to this particular worker?
Hope this is not confusing, or am I missing something.
from goworker.
@rohit4813 Our use case just creates breakpoints in long-running jobs so that when we need to restart the process, we don't have to wait (tens of) minutes for the whole job to finish.
If you needed to discriminate between workers, I guess you could do that by sending some meaningful value through the channel and then the worker would decide "this msg is meant for me, I'll stop" or "this is meant for the slow one over there, I can keep running". Though, I can't imagine the use case for such behaviour.
from goworker.
Related Issues (20)
- How to panic handling on Job? HOT 2
- 100% cpu usage HOT 17
- Any plans to support Sidekiq backend? HOT 4
- Refactor the Enqueue API
- RabbitMQ Support HOT 1
- Example from Getting Started section doesn't run
- New version HOT 3
- How does goworker work? HOT 4
- Timeout for stuck workers HOT 6
- Any plan to replace garyburd/redigo with gomodule/redigo? HOT 2
- redis please use go-redis
- Supporting Resque Scheduler
- package broken
- missing the prefillParallelism argument in redis.go
- etcd support
- It's graceful, but how to optimize cpu usage? HOT 2
- pools.NewResourcePool request param has changed
- fail to install HOT 4
- There is atomicity problem when worker key is flushed by redis pool
- [Critical] Error on closing worker ip-xxx-xx-xx-xx:xxxxxx-4:xxx: redis: client is closed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goworker.