Unwanted duplication of jobs in a queue about android-priority-jobqueue HOT 17 CLOSED

From @ckozak on February 7, 2015 6:41

it looks like a race condition. I checked that .addJobInBackground is called only once but a new consumer is added just after a job runs and the new consumer picks up the same job and executes it too.

02-06 22:35:08.375  15253-15328/com.package.staging D/JOBS﹕ added job id: 46 class: DeleteDictionaryJob priority: 1000 delay: 0 group : d persistent: true requires network: true
02-06 22:35:08.395  15253-16065/com.package.staging D/JOBS﹕ running job DeleteDictionaryJob
02-06 22:35:08.396  15253-15328/com.package.staging D/JOBS﹕ pool-2-thread-1: load factor check. false = (1 < 1)|| (1 * 1 < 0 + 1). consumer thread: false
02-06 22:35:08.438  15253-15253/com.package.staging D/JOBS﹕ main: load factor check. true = (0 < 1)|| (0 * 1 < 1 + 0). consumer thread: false
02-06 22:35:08.438  15253-15253/com.package.staging D/JOBS﹕ adding another consumer
02-06 22:35:08.443  15253-16164/com.package.staging D/JOBS﹕ starting consumer Thread-3229
02-06 22:35:08.511  15253-16164/com.package.staging D/JOBS﹕ running job DeleteDictionaryJob
02-06 22:35:08.520  15253-16065/com.package.staging D/JOBS﹕ finished job DeleteDictionaryJob
02-06 22:35:08.598  15253-16164/com.package.staging D/JOBS﹕ finished job DeleteDictionaryJob

from android-priority-jobqueue.

From @mecid on February 7, 2015 11:0

@ckozak i have no problems like this. I am using default consumer configuration. Just using grouping if don't want to run certain jobs in parallel.

from android-priority-jobqueue.

From @ckozak on February 7, 2015 20:59

Unfortunately even default configuration with grouping doesn't work for me. Nothing changes at all, some jobs are duplicated on every call, some are not. But every job is executed even more, 3-5 times when I call jobManager.start/stop. All my jobs extend the same base class and are essentially the same, with a difference in a network call they make.

from android-priority-jobqueue.

From @mecid on February 8, 2015 21:56

@ckozak i thinks this is bad idea using start/stop method.

from android-priority-jobqueue.

From @ckozak on February 9, 2015 21:44

@mecid Originally I thought that start/stop is a source of problem, but I saw duplication even without it. Start/Stop just made if much more visible.

from android-priority-jobqueue.

From @mecid on February 10, 2015 11:16

@ckozak very interesting, but i didn't see any duplicate.

from android-priority-jobqueue.

I face this too , priority is set , some delay and require network jobs are duplicated

from android-priority-jobqueue.

is it possible to create a reproducible case? This seems very serious but there isn't enough information to debug it. For instance, are you sure the job is not added twice?

Or will I see the error if I create a (require network + 1 min delay) job ?

from android-priority-jobqueue.

Hey i will create a small project and test this and get back to you in sometime with the code.. it seems that if i use priority 100 , 99 , 98 ( own logic) [ onRun i subtract it] and use a delay of a 1min or more i start seeing this : Jobs to complete - msg string

1. test 1
2. test2
3. test 3 and so on , on run will execute "test 3" twice for me

Using it in a messaging app , requiresNetwork is used in the consstructor

from android-priority-jobqueue.

Any updates on this issue?

I'm having the same problem.

from android-priority-jobqueue.

@tuliohmendes I cannot reproduce this so if you have a sample app that I can look at, that would be great.

from android-priority-jobqueue.

I am also experiencing duplication of jobs with some users sometimes. It seems to be pretty frequent. Unfortunately I am not able to reproduce the issue. But we need to fix this because it impacts the behaviour of the app.

from android-priority-jobqueue.

@kaciula any clues on when it happens, how? I would really like to fix this but w/o a way to reproduce, I cannot do anything.

from android-priority-jobqueue.

@yigit The app is used by a big NGO in areas where it's common not to have Internet connection most of the time. I am using exponential back-off (1, 2, 4, 8 minutes ...) with a upper limit of 1 hour.

If I am running a job that creates a resource on the server-side, in onRun() I first check if the local resource already has a remote id (which means a job was already run for that resource) but still I get duplicates sometimes on the server. I also use groupBy() local id to make sure that, even if I have duplicate jobs, they won't run in parallel and the later one would not run to completion because of the remote id check.

Maybe I am doing something wrong somewhere but I've run through the code logic at least 5 times and it seems bulletproof :) Maybe the issue is with the server that although it creates the resource, it does not return successfully.

If you have any ideas let me know. I'm still struggling to reproduce it.

from android-priority-jobqueue.

Hmm if the network is not good, there is a potential case where your web request reaches your web server (so resource is created) but the response does not arrive to the client. So in other words, resource is created on the backend but your job receives exception from the network stack, thus retries.

This is a common synchronization problem. A potential solution is to generate a UUID on the client side per resource and save it on the server side as well. In that case, if server receives the same request from the same user with the same UUID it simply returns success w/o actually saving the anything (or returns the existing one).

I recently gave a talk in Android Dev Summit where the demo application has a similar situation.
You can see how duplication is avoided here:
https://github.com/yigit/dev-summit-architecture-demo#avoiding-duplicate-posts

And also the source exists (for both server and client side) which may give you enough idea to fix the issue.

Lmk if this works.

from android-priority-jobqueue.

v2 is single threaded so this issue should be completely gone.

from android-priority-jobqueue.

v2 is single threaded so this issue should be completely gone.

but I still have this issue in v2. how to fix it?

from android-priority-jobqueue.