st2 pods stuck in loop about stackstorm-k8s HOT 9 CLOSED

Let's continue here.

@nagrigore Did you figure it out?

It looks like it's related to recent service registry changes I worked on. Which coordination backed do you have configured in st2.conf?

from stackstorm-k8s.

Alright, I found there issue. Workaround is here - StackStorm/st2#4607

Once that PR is merged and new v3.0.0dev packages built, redeploying stackstorm-ha should work.

Thanks again for reporting this issue.

from stackstorm-k8s.

Thanks @Kami for a quick reaction & fix.
We'll check again in a few days when both packages + docker images are available and close this issue.

from stackstorm-k8s.

Hello guys,
We're still can not deploy stackstorm-ha from helm.
helm install --name stackstormtest stackstorm/stackstorm-ha
Here is the log from 1 pod
019-03-21 20:12:48,348 ERROR [-] (PID=1) Worker quit due to exception.
Traceback (most recent call last):
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/actionrunner.py", line 80, in main
_setup()
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/actionrunner.py", line 41, in _setup
register_signal_handlers=True, service_registry=True, capabilities=capabilities)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/service_setup.py", line 187, in setup
start_heart=True)
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/service_setup.py", line 224, in register_service_in_service_registry
coordinator.create_group(group_id).get()
File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/tooz/coordination.py", line 476, in create_group
raise tooz.NotImplemented
NotImplemented

testcluster-st2actionrunner-77489f94c8-btbt4 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2actionrunner-77489f94c8-rj2v5 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2actionrunner-77489f94c8-tnklq 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2actionrunner-77489f94c8-tr242 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2actionrunner-77489f94c8-vr6r2 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2api-646fc6bfd7-hjjh5 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2api-646fc6bfd7-z2h5w 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2auth-5855d9d5bd-2ktlx 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2auth-5855d9d5bd-sn4mf 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2client-5c55669f6-k7v6j 1/1 Running 0 4m34s
testcluster-st2garbagecollector-8674bc649c-ns8hp 0/1 CrashLoopBackOff 5 4m33s
testcluster-st2notifier-c5b7d59b9-kg8l4 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2notifier-c5b7d59b9-s2pkn 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2rulesengine-fbff5c8dd-bzl5k 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2rulesengine-fbff5c8dd-mkgpg 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2scheduler-77db66bc5b-87crf 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2scheduler-77db66bc5b-9fskj 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2sensorcontainer-65c45f9445-9t9cz 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2stream-599cb6b7b4-4dsmt 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2stream-599cb6b7b4-s7zzn 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2timersengine-6c54464b5f-bns59 0/1 CrashLoopBackOff 5 4m34s
testcluster-st2web-6dd96cf9f7-l5l28 1/1 Running 0 4m33s
testcluster-st2web-6dd96cf9f7-qmhjm 1/1 Running 0 4m33s
testcluster-st2workflowengine-7499f6d446-m9t8r 0/1 CrashLoopBackOff 5 4m33s
testcluster-st2workflowengine-7499f6d446-vmxm2 0/1 CrashLoopBackOff 5 4m33s

from stackstorm-k8s.

@nagrigore Just a heads up - PR has been merged and new packages built.

If you redeployed the cluster with the latest version of StackStorm packages, it should work now.

from stackstorm-k8s.

We'll check again in a few days when both packages + docker images are available and close this issue.

Still need some time for packages + Docker images to build + deploy based on new code via nightly builds. That will happen likely tomorrow.

from stackstorm-k8s.

For me personally, the deployments have started to work again. Thanks a bunch @Kami @armab

from stackstorm-k8s.

Thanks for letting us know that it's working for you. I'll go ahead and close this issue now, please feel free to re-open if necessary.

from stackstorm-k8s.

Hello,

Seems like the the deploy is working now.
Thank you!

from stackstorm-k8s.