bionic stemcell. grafana startup freeze about prometheus-boshrelease HOT 5 CLOSED

Hi there,

seeing same issues with bionic stemcell 0.24,

so it basically looks like Grafana is not starting up:

tail -f post-start.stdout.log
[Mon Apr 19 14:05:48 UTC 2021] Resetting grafana admin password ...
Waiting for grafana to listen on port 3000 (0/60)

from prometheus-boshrelease.

Seems the nc command checking the open grafana port is the culprit:

broker/801ac959-c4b5-4251-a4be-fd823f32355e:/var/vcap/sys/log/grafana# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root      5880   704  0 Apr19 ?        00:00:00 bash /var/vcap/jobs/grafana/bin/post-start
root      5887  5880  0 Apr19 ?        00:00:00 bash /var/vcap/jobs/grafana/bin/grafana-admin-password
root      5891  5887  0 Apr19 ?        00:00:00 nc 127.0.0.1 3000

broker/801ac959-c4b5-4251-a4be-fd823f32355e:/var/vcap/sys/log/grafana# nc -vz 127.0.0.1 3000
Connection to 127.0.0.1 3000 port [tcp/*] succeeded!

broker/801ac959-c4b5-4251-a4be-fd823f32355e:/var/vcap/sys/log/grafana# nc  127.0.0.1 3000
=> freeze

from prometheus-boshrelease.

Yes, that is was I also just found out... currently trying to figure out why this is happening because a curl on http://localhost:3000 is working fine

from prometheus-boshrelease.

So it seems like

nc 127.0.0.1 3000 < /dev/null

is behaving differently on Xenial and Bionic. For Xenial it immediately returns, for Bionic it does not. This approach was apparently applied to fix issues with cent os.

from prometheus-boshrelease.

We could adjust the following line

https://github.com/bosh-prometheus/prometheus-boshrelease/blob/9cccf9494a86b0b2420b5d9dd086397cbecf7c72/jobs/grafana/templates/bin/grafana-admin-password#L8

to look like the following

if nc -w 5 127.0.0.1 <%= p('grafana.server.http_port') %> < /dev/null; then

from prometheus-boshrelease.