Call for comments and testimonials about que HOT 15 CLOSED

http://michael.otacoo.com/postgresql-2/postgres-9-5-feature-highlight-skip-locked-row-level/ might be interesting to talk about.

Tanga.com's been using que since March this year without any issues, we've processed around 100 million jobs. We love how the jobs are transactional with our other data -- if the data gets saved, the related jobs are also saved. We also love being able to manage the job queue via SQL.

Biggest con so far was no web-ui for monitoring jobs, but I think that's been solved by the project in the README, haven't looked at it yet.

from que.

Thanks, Joe!

from que.

I'm just now looking at Que to replace sidekiq because I can create most my jobs using INSERT SELECT semantics and save having to iterate 1 million rows in ruby and invoking a single job at a time.

One thing I'm curious is why you do not use stored procedures instead of sql for high-frequency statements? Would you be interested in a pull request? Versioning can be handled via comments on the sproc or md5 of the function contents.

from que.

Que uses prepared statements, should be just as fast as stored procedures (right?).

from que.

@joevandyk Yes if it does, there is a case statement at https://github.com/chanks/que/blob/f677acf34683a281c0915757e2f9566a07c903ad/lib/que/adapters/base.rb#L41 so cursory glance says its not guaranteed. Also not sure if it respects prepared_statements: false ActiveRecord configuration setting. We are planning to use pgpool to multiplex connections as we add nodes, requiring no prepared statements.

Also, using stored procs enables a pure sql api to be used from within postgres.

from que.

Prepared statements are used for all common queries unless the connection is currently in a transaction. This was done because ActiveRecord connection failures would occasionally result in the same connection objects referring to different backends, so a statement that was thought to be prepared wouldn't be and an error would be thrown. This is fine outside of a transaction, since we can just prepare it and retry, but inside a transaction the error would force a rollback, which would be unpleasant for the user. So, inside a transaction, plain non-prepared SQL is always used for safety. This isn't a huge concern, because the only significantly complex statement (the job lock query) is never executed inside a transaction.

Que doesn't respect ActiveRecord's configuration settings, because it only interacts with ActiveRecord the bare minimum that it needs to in order to access its pool of PG connection objects. It seems like the usage of prepared statements would be a problem for remote connection pools like pg_bouncer (or pgpool, I'm guessing), but nobody's lodged a complaint, so perhaps it isn't an issue for some reason I'm unaware of? If you run into problems I'd be happy to add a configuration option for disallowing prepared statements.

The prospect of using functions instead of prepared statements is interesting, but I think I'd only be convinced to move Que in that direction if there were evidence that they're significantly faster than prepared statements (prepared statements with a local connection pool being the way to go for the vast majority of apps), so much so that it'd be worth the added complexity of potentially migrating them between versions.

from que.

Pros:

• Extremely easy to set up. Makes us more likely to use background workers.
• Few external dependencies (no Redis).
• Very lightweight.
• Works in development just as well as production.
• Runs great on a single Heroku web dyno without requiring a worker dyno.

Cons:

• Dumb/confusing name. Seriously, at my shop we use Que every time but complain about the name every time.
• Requires a relatively recent PostgresQL, which some Rails hosts don't support (looking at you, WebFaction).
• Can't use rake db:load:schema.

from que.

Que uses prepared statements, should be just as fast as stored procedures (right?).

Stored procedures excel at reducing round-trips to the database, which can provide substantial improvements in a number of applications. I think in practice, in Que's case this could be an improvement since the algorithm for grabbing an item does at least two round-trips for every item, but even four or six (!) wouldn't be out of the question on a high concurrency queue.

However, they come at a price as well, since you need to handle updates to them, integrate them into your source control management etc.

from que.

I may be wrong, but my understanding is that in PG, the same MVCC snapshot is held throughout the entire execution of the function, so it wouldn't help us with the race condition that necessitates the second query to make sure the job still exists.

from que.

I may be wrong, but my understanding is that in PG, the same MVCC snapshot is held throughout the entire execution of the function, so it wouldn't help us with the race condition that necessitates the second query to make sure the job still exists.

Not in READ COMMITTED (which is the default) mode for VOLATILE functions.

from que.

Ah! Interesting. I'll have to play with that at some point, thanks!

from que.

This should probably be in a different thread, but I implemented something similar just now; I'm running the tests as we speak to see what happens.

from que.

We switched from Sidekiq to Que.

Where Que Rocks

• Cutting out the redis dependency eased setting up both development and production environments. If you've dealt with the pain of using Redis over SSL, you understand.
• Being able to queue in the same transaction as a web request is a killer feature. Transaction success + failed to enqueue or transaction failed + enqueue success were two occasional oddities we had with Redis.
• Runs plenty fast for our use case.

Where Que lacks

• Hooks. Having access to a middleware style pipeline for extensions like transaction wrappers and error reporting would be helpful.
• Stats. There's no historical statistic information. Being able to see jobs/sec, slow jobs, and 24 hour count would help give insight to the state of Que.
• Full fledged web interface (I'm working on that 😉).

Overall the experience has been positive. I dig the project and intend on continuing to use it.

from que.

Thanks, everyone. One last request, as I'm putting together my presentation for tomorrow - would people be willing/able to share the names of sites where they're using Que in production? And if you want to include the number of jobs you've put through it, that'd be cool, too, but don't feel obligated. I'd like to be able to show some concrete evidence that Que is serious business :)

And if you want to include a one-line description of what your site does, I'll be happy to include it. Free publicity and all. :)

from que.

Didn't get anyone in time for my talk, but it was alright, I asked kind of late. Thanks for your input everyone - I think it was recorded, so if it's posted somewhere I'll link to it in the README.

from que.