Kubernetes deployments, yelb-ui fails to stay up about yelb HOT 16 CLOSED

Hi there. Sorry for the problem. Which K8s are you testing with? I think the K8s yaml files in the repo right now would need a bit of hammering if you are using K8s 1.23 or above (maybe even 1.22).

It's something I would need to check (it was on my to-do but I did not get to it yet).

I also think the problem in #36 is unrelated (probably similar outcome but different cause).

from yelb.

I'm closing this issue, as the problem is clearly in my dev env.

from yelb.

I realized this issue is the same, or similar, to #36

But over there, no solution either. So I'll keep poking at this problem.

from yelb.

Using the most recent version on my Alpine, which is 1.26. I'll go and see if maybe an older version would be more cooperative. Still, most of the setup works, it's just that the Yelb-ui pod can't find the yelb-appserver pod. I'm trying to figure out why.

Plus, I'm trying with the Minikube Ingress config -> https://github.com/mreferre/yelb/blob/master/deployments/platformdeployment/Kubernetes/yaml/yelb-k8s-minikube-ingress.yaml

But, instead of using Minikube, I'm using vanilla kubelet + kubectl etc on Alpine... it's for the purpose of a lab env for my students. Just to demo a basic Kubernetes build.

from yelb.

In general the reason why the yelb-ui container would not stay up is because it can't reach the yelb-appserver service on port 4567. This is the part of the code involved (by default the NGINX proxy is configured to ... proxy to http://yelb-appserver:4567 and if it can't reach it it will fail to start and it will bring down the container with it).

I haven't tested minikube in a long time with that yaml (I should build more automation to test the various yaml files on their respective backends) and it could be possible that I'd need to fix them (I created those around the 1.1x K8s timeframe IIRC).

from yelb.

I read some article earlier which suggested using a different config for NGinx, where instead of defining with an upstream, you'd instead set it to a "dumber" proxy mode where it doesn't test for the upstream system... Alternatively, the yelb-ui pod could be adjusted to include an initContainer which waits for the appserver to be available.

I'm currently trying to get the latter to work.

from yelb.

While I want to attack the testing/automation problem alluded above (at some point), I am indeed interested in your first option to find a way to make NGINX behave less aggressively when the endpoint is not available. The current scenario is not ideal for proper troubleshooting and I would rather too that the container stayed up when the yelb-appserver endpoint is not reachable. If you find anything useful I'd love to hear.

The sidecar option is less palatable for me (but if it works for you great!).

from yelb.

I don't fully understand the setup, as I have next-to-none NGinx experience, but this is what I was looking at -> https://www.prasanna.dev/posts/start-nginx-when-upstream-unavailable

I also agree with you that the initContainer option is a kludge, whereas properly configuring NGinx is the right solution.

from yelb.

I am no NGINX expert either. My understanding is that the resolver directive can help in that NGINX fails when it can't resolve the endpoint (i.e. if you use an IP you won't have this problem) but this makes the deployment very dependent of the underneath infrastructure. I would need to research more. More background in this SO thread.

from yelb.

In the mean time, it turns out it's also not about startup timing. Because even if I force the yelb-ui pod to wait for the yelb-appserver and its service to be ready, it still says it can't find it.

This is frustrating. I feel like I should just understand why the yelb-ui pod cannot resolve the name for the yelb-appserver service. But... I don't. :D

from yelb.

Sorry you are going through this pain. I have just tested the yelb-k8s-minikube-ingress.yaml and the yelb-k8s-minikube-nodeport.yaml on a Docker Desktop running on a Mac and they work just fine as-is. DD deploys a local K8s 1.25 cluster (I thought I had to hammer the yamls a bit but they just worked).

I am wondering if the problem you see with the yelb-ui container not stabilizing is a symptom of a larger issue you have. Does it have coredns setup?

For example would the following work for you? Basically you could exec into the appserver and curl itself to see if it can reach out to the backend to read the votes. If this doesn't work name resolution just doesn't work IMO.

% kubectl exec -it yelb-appserver-5d8db45799-z56zn -- bash
root@yelb-appserver-5d8db45799-z56zn:/# curl yelb-appserver:4567/api/getvotes
[{"name": "outback", "value": 2},{"name": "bucadibeppo", "value": 0},{"name": "ihop", "value": 0}, {"name": "chipotle", "value": 0}]

from yelb.

I will test this tonight, when I'm at home. I agree with you, this very much appears to be a "me"-problem, with my environment. Thank you for helping with the troubleshooting, I really appreciate it.

from yelb.

Pleasure. I really want you to make it work :)

Let me know.

from yelb.

Yeah, the yelb-appserver can't curl itelf either, can't resolve the name.

My K8S setup is suspicious here. CoreDNS is running, but I'm sure there's an issue with it. I'll go poke around more.

from yelb.

This is interesting: I loaded the dnsutils container as per the K8S documentation -> https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

From THAT pod I can properly resolve the Yelb DNS names. But not from the Yelb pods themselves!

And yes, they're all getting the same resolv.conf.

EDIT:

Yeah, it's my core-dns / kube-dns. The dnsutils pod can resolve the names sometimes and then not. It keeps coming and going, so either the networking or the stability of the dns pods is busted. This is a problem with my K8S setup.

EDIT:

The more I research this situation, the more I believe that the problem is caused either by A) too little resources for my K8S nodes, or B) using Vagrant to build my VMs, which adds a NAT interface to each node.

EDIT:

It was not the resources. That only leaves Vagrant and its eth0-is-always-NAT tomfoolery.

from yelb.

Woot. Now I am hooked. Keep me updated.

from yelb.