Nginx not updating upstream ip addresses when bosh changes them about prometheus-boshrelease HOT 5 CLOSED

Hi @corlettb ,
thanks for pointing this out.
I will try to reproduce the issue on our side to see if the mentioned fix that solve it, but in general we are open for Pull Requests to fix such an issue.

from prometheus-boshrelease.

Hi @corlettb I tried to reproduce the issue but was not yet able to do so.
What I tried to do was to adjust the static ip used in grafana to a new ip and re-deploy. According to BOSH the nginx VM has not been touched but when I try to access the Grafana Dashboard via Nginx it worked perfectly fine. Do you have any instruction on how to reproduce that issue.

from prometheus-boshrelease.

Hmmmm.

To reproduce the issue in our dev environment, I simulated ip changes by manually editing the /etc/hosts file on the nginx box (changing the ip address of grafana). I then hit the grafana endpoint going via nginx and it still worked (expected a 502).

I executed a 'killall -s SIGHUP nginx' and nginx then started returning 502's.

Are you setting the resolver line in nginx.conf ?

from prometheus-boshrelease.

We have witnessed this issue and I made this PR which seems to resolve it.

from prometheus-boshrelease.

@corlettb we merged the PR of @dark5un which hopefully improves the situation for you as well. That PR is included in v28.0.0 which has been released yesterday. Feel free to re-open this ticket if your issue is not solved by this.

from prometheus-boshrelease.