Comments (5)
This is fixed in the soon-to-be-released oban_met
v0.1.6
from oban.
@sorentwo FYI, we updated oban_met
to v0.1.6 on prod and it seems that problem is solved on OTP 26.2.5 🙌🏻
from oban.
@up2jj That's unpleasant. This is the first I've heard of an excessive message queue for the recorder process.
Was this a one off experience, or is the message queue remaining high after a reboot? A few other questions to get a better idea of your setup:
- How many nodes?
- How many queues?
- How many workers?
- How many jobs per hour (roughly)?
from oban.
Somebody else reported this on Slack. sThe relevant change in both cases is upgrading to OTP 26.2.5, and in both situations the current function was :ets.select/1
.
In reviewing the OTP 26.2.5 release notes, I noticed this change:
OTP-19070 Application(s): erts
Related Id(s): GH-8385
ETS functions did not properly handle keys containing
maps, sometimes matching too many or too few objects.
That traces back to this issue from @tsloughter, and this patch by @jhogberg.
A simple benchmark that loosely recreates how oban_met
is using :ets.select
highlights an extreme performance regression for select
, and oddly enough a massive boost for lookup
. Details are in this gist, but here's the relevant part:
Name ips average deviation median 99th %
lookup 1.62 M 616.07 ns ±1798.74% 584 ns 791 ns
select_reverse 1.21 M 823.61 ns ±1503.76% 667 ns 958 ns
select 0.00100 M 995660.57 ns ±2.84% 982750 ns 1058731.74 ns
Comparison:
lookup 1.62 M
select_reverse 1.21 M - 1.34x slower +207.54 ns
select 0.00100 M - 1616.16x slower +995044.50 ns
Removing the map portion of the key alleviates the performance issue entirely (and it's much faster for the other operations):
Name ips average deviation median 99th %
lookup 3.61 M 276.76 ns ±9550.30% 250 ns 333 ns
select_reverse 2.21 M 452.09 ns ±2259.09% 458 ns 583 ns
select 2.20 M 454.81 ns ±2213.31% 458 ns 583 ns
Comparison:
lookup 3.61 M
select_reverse 2.21 M - 1.63x slower +175.33 ns
select 2.20 M - 1.64x slower +178.05 ns
We'll look at revising the table structure to avoid using a map as an immediate fix for this, but this is bound to impact other applications and I'll report this upstream to the OTP team.
from oban.
Hi @sorentwo
Thank you for your quick and detailed response. Ultimately we have decided to downgrade the OTP to the previous version, however we will keep an eye out for any immediate updates to Oban itself tackling this issue.
from oban.
Related Issues (20)
- Recommended approach to enqueue a job with a guaranteed minimum delay HOT 1
- Race condition when inserting unique jobs HOT 3
- Support `cancelled` callback in `Oban.Testing.Pro.perform_callback/4` HOT 1
- Support Stream in `Oban.Pro.Workers.Batch.new_batch/2` HOT 3
- Potential issues regarding unique jobs HOT 2
- "Handler "oban.batch" has failed and has been detached" when job is cancelled via oban web HOT 3
- Single DynamicCron with missing worker will crash GenServer and hang all DynamicCrons HOT 1
- Getting oban_pro in github ci stopped working HOT 2
- Support `before_process/1` HOT 2
- Execution Deadline is ignoring the `scheduled_at` offset HOT 1
- Oban Web - support bulk actions on filtered jobs HOT 3
- Oban worker backoff algorithm HOT 2
- Possible regression when handling unique advisory lock HOT 4
- Oban Web: Completed Jobs list gets stuck in an empty state HOT 3
- Job executing but stuck forever HOT 4
- Oban.Pro.Workers.Workflow documentation typos HOT 1
- Oban Met - Configure Reporter Interval HOT 2
- Oban Pro Batch Workers stuck in "executing" forever after upgrade from Aurora PostgreSQL 14 -> 16 HOT 1
- Jobs that exit with `{:error, ...` are logged at `info` level. HOT 1
- Patch to support providing Oban name at runtime for Oban Web router
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from oban.