Comments (3)
With liveness probes in mind, we may need to fine-tune Deployment
.spec.minReadySeconds
(https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#min-ready-seconds) to make sure it fits specific st2 service startup profile.
Min Ready Seconds
.spec.minReadySeconds is an optional field that specifies the minimum number of seconds for which a newly created Pod should be ready without any of its containers crashing, for it to be considered available. This defaults to 0 (the Pod will be considered available as soon as it is ready). To learn more about when a Pod is considered ready, see Container Probes.
from stackstorm-k8s.
st2web
liveness/readiness probe could be first implemented via exec.command
doing several curl && curl && curl
requests to /api/auth/stream endpoints.
from stackstorm-k8s.
Thinking about readinessProbe
we could probably approximate something with lifecycle.postStart.exec.command
that waits for some condition like some log line being printed out (if it has access to the logs) or for the main process to create a stable rabbitmq connection (as sniffed by ss
or netstat
). This would work because the pod is not marked as ready until after postStart
exits successfully.
The same could in readinessProbe.exec.command
as well, as long as it wasn't watching for a log line.
These use an eventlet.wsgi
loop, so using a tcp probe might be appropriate:
- st2auth
- st2api
- st2stream
These pods use st2common.transport.consumers.MessageHandler.start()
and so they're ready shortly after connecting to RabbitMQ:
- st2rulesengine
- st2actionrunner
- st2notifier
- st2workflowengine
- st2scheduler (this has a couple other eventlet threads that need to start, but MQ connection would be a good indicator)
This pod does not use MessageHandler, but it should still be ready shortly after MQ connection
- st2timersengine (MQ connection & apscheduler have to start)
For st2sensorcontainer pods defined in st2.packs.sensor
, we don't need to worry about the probes.
If we did, we a MQ connection would still be a good indicator as that should happen after the sensor processes get spanwed.
When we implement this in st2 itself - maybe we'll end up with a MQ based heartbeat for each service to say "I'm up". I'm not sure how that could interact with k8s, but it's a thought.
Hmm. or maybe we have a mini script on each pod, that sends a ping message to the MQ and all that has to happen for it to be live is for the service to respond back with a pong. That would allow k8s to check the liveness with command.exec
without exposing an http endpoint on services that don't already have endpoints.
from stackstorm-k8s.
Related Issues (20)
- Allow setting requests/limits on init containers HOT 2
- st2clientConfig issue with jobs.extra_hooks HOT 1
- Pack defaults do not get overridden when a pack is reinstalled HOT 2
- sendmail configuration is impossible inside a container HOT 6
- There is an issue with the StackStorm Kafka pack in stackstorm 3.8.0 HOT 1
- rabbitmq 'failed to merge schema: bad cookie' HOT 1
- Not able to create new action in stackstorm (which is deployed on K8s in docker desktop) HOT 2
- proper graceful shutdown settings HOT 6
- Release a new version HOT 11
- Secret DataStore Crypto Key should not be created when existing provided
- Error while installing multiple packs via configs HOT 1
- job-ensure-packs-volumes-are-writable missing policy and syntax error HOT 5
- Deployments have labels that are redundant when existing secrets are provided. HOT 1
- ST2 migration from standalone install to K8s?
- More detailed "Releasing information" HOT 1
- /bin/bash: /etc/nginx/conf.d/st2.conf: Permission denied HOT 2
- Unable to run st2client , st2web, st2actionrunner as stanley user after adding securityContext HOT 5
- Use Secret for `st2.docker.conf` instead of ConfigMap HOT 1
- st2api pod need pack images? HOT 1
- Upgrade `helm-unittest` HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stackstorm-k8s.