Comments (4)
Thanks for filing this bug report!
I ran the same code on my machine with 4 worker processes reading from the same redis instance but could not reproduce the bug.
It could be that you've run the client code multiple times and there were duplicate tasks in Redis. Would you mind trying this again with clean Redis DB?
You can asynqmon stats
to make sure that there's no tasks in Redis.
You can flush redis by running redis-cli flushdb
from asynq.
@hibiken I prepared an environment where this can be reproduced (uses docker, docker-compose):
https://github.com/hibiken/asynq/compare/master...gunnsth:issue90-reproduce?expand=1
to reproduce it:
$ cd testdata
$ docker-compose up
It will not always give exactly the same results, I guess there's some stochastic/random factor that determines which taskrunner catches tasks.
Example outputs:
taskrunner_1 | Tasks processed
taskrunner_1 | (map[int]int) (len=5) {
taskrunner_1 | (int) 10: (int) 1,
taskrunner_1 | (int) 2: (int) 1,
taskrunner_1 | (int) 4: (int) 1,
taskrunner_1 | (int) 6: (int) 1,
taskrunner_1 | (int) 8: (int) 1
taskrunner_1 | }
taskrunner3_1 | Tasks processed
taskrunner3_1 | (map[int]int) (len=3) {
taskrunner3_1 | (int) 1: (int) 1,
taskrunner3_1 | (int) 2: (int) 1,
taskrunner3_1 | (int) 20: (int) 1
taskrunner3_1 | }
taskrunner2_1 | Tasks processed
taskrunner2_1 | (map[int]int) (len=7) {
taskrunner2_1 | (int) 1: (int) 1,
taskrunner2_1 | (int) 3: (int) 1,
taskrunner2_1 | (int) 5: (int) 1,
taskrunner2_1 | (int) 7: (int) 1,
taskrunner2_1 | (int) 9: (int) 1,
taskrunner2_1 | (int) 11: (int) 1,
taskrunner2_1 | (int) 14: (int) 1
taskrunner2_1 | }
We see that both taskrunner1 and 3 have processed task 2. And taskrunner3 and 2 have both processed task1.
Is there any way we can ensure that this does not happen?
There should not be any duplicate tasks since this creates a fresh redis instance.
(to be sure, it's easy to clean all the docker images).
Would it be possible to clear all tasks prior to start creating tasks (programmatically)? Just to make sure?
from asynq.
@gunnsth
Could you try adding this to your taskqueue.go
to flush DB to start from clean slate?
import "github.com/go-redis/redis/v7"
func main() {
// Flush DB first to start from a clean slate.
rdb := redis.NewClient(&redis.Options{
Addr: "redis:6379",
Password: "xxxx",
})
if err := rdb.FlushDB().Err(); err != nil {
log.Fatalln(err)
}
// ... create asynq.Client and schedule tasks (your existing code)
}
Let me know if you are still seeing duplicate tasks.
Otherwise, we can close this issue 👍
from asynq.
@gunnsth I finally figured out what's causing this. Fix is in #170
from asynq.
Related Issues (20)
- [BUG] The related methods of Inspector basically lack the transmission of ctx HOT 2
- [FEATURE REQUEST] customize serializer
- Discussion/Suggestion: Decoupling `x` and `tools` packages.
- [BUG] Unable to schedule a task HOT 1
- Timing issues when running certain tests
- [FEATURE REQUEST] Settable maxArchiveSize HOT 1
- [BUG] Delete archived task fail, when task has been evicted from archived zset. HOT 2
- [FEATURE REQUEST] Unique option for run time of task
- [FEATURE REQUEST] aggregation task optimization
- handler not found for task "ping:worker"
- [FEATURE REQUEST] Custom ticker value for heartbeater duration HOT 4
- Scheduler: How to get the schedule time of the task in the task handler? HOT 1
- [FEATURE REQUEST] Exposing MinIdleConns redis config HOT 3
- [BUG] Description of the bug HOT 1
- [FEATURE REQUEST] can add opt for cron to set second HOT 2
- [FEATURE REQUEST] Process task with ratelimit HOT 2
- [BUG] After manually deleted redis keys, some of the features like metrics, inspector aren't working properly
- [FEATURE REQUEST] Wrap the payload contents in the CLI's Task Info modal
- [FEATURE REQUEST] How can I specific worker concurrency for each queue
- [FEATURE REQUEST] pause server
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from asynq.