Comments (7)
@philss Thanks for reporting the issue. I admit, I haven't used the new release task yet and hadn't ran into this.
I'll work on instructions for avoiding this issue, and possibly make some changes to prevent it from happening entirely. My goal is to have as few manual steps for users as possible—the project should feel like it "Just Works", and crashing on initial boot definitely isn't "just working".
from oban.
@sorentwo Thank you for the clarification. The commit you mentioned seems to mitigate the problem. I will try this soon.
from oban.
@philss That was only a partial solution, I plan on writing some release specific deployment docs as well.
Currently I'm debating between two alternative solutions:
- Catch failing queries from the producer and log them as errors rather than crashing the process. This would prevent the queue supervisor from crashing, which would prevent the cascade that can bring down the system.
- Document that for the initial deployment Oban should not be started, or at the very least queues must be disabled. That seems like a bit of a hack and puts the burden of sequential deploys on developers.
I'm leaning heavily toward the first option and calling it "resilient" mode. It would be the default mode, which could be disabled if you'd prefer to have a crash.
Thoughts?
from oban.
@sorentwo the 1st option seems to be a good approach - oban could check if the expected table exists, log that the database needs to be migrated and sleep for a few minutes, so the system can recover if the table is created right away. This mechanism could help future upgrades that require database changes and people might not noticed it when updating the dependency.
from oban.
@sorentwo I like the approach of the 1st option, and like @lucasmazza suggested, it could check if the database is ready to run jobs and log the errors if not.
from oban.
The first option is in master and will be in v0.4 that I'm about to release. There is a small blurb about it in the README and a bit more detail in the CHANGELOG under [Oban.Queue.Producer]
.
Thanks again for the report. Hopefully this improves the deployment process for everybody as more and more people move to use releases.
from oban.
@sorentwo awesome! Thank you!
from oban.
Related Issues (20)
- Oban Web - support bulk actions on filtered jobs HOT 3
- Oban worker backoff algorithm HOT 2
- Possible regression when handling unique advisory lock HOT 4
- Oban Web: Completed Jobs list gets stuck in an empty state HOT 3
- Job executing but stuck forever HOT 4
- Oban.Pro.Workers.Workflow documentation typos HOT 1
- Oban Met - Configure Reporter Interval HOT 2
- Oban Pro Batch Workers stuck in "executing" forever after upgrade from Aurora PostgreSQL 14 -> 16 HOT 4
- Jobs that exit with `{:error, ...` are logged at `info` level. HOT 1
- Patch to support providing Oban name at runtime for Oban Web router HOT 1
- Oban OSS available jobs are not being executed HOT 17
- Browser console errors when using Oban Web UI HOT 4
- Postgres serialization failure error when fetching jobs using `serializable` transaction isolation level HOT 5
- Documentation request: What happens if a job times out? HOT 1
- Crash when upgrading to Ecto 3.12 HOT 2
- Extremely Slow `check_available` in Basic Engine (also used for pro smart engine) HOT 3
- Lots of debug log noise on 2.18.1 HOT 1
- Oban Web - Very slow query on Jobs page HOT 3
- Oban spawns duplicated workers for the same JobID HOT 2
- Missing `oban: true` Repo opts in some queries? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from oban.