Quickstart Multi-node fails on nova (nova-db-sync hangs) about atmosphere HOT 6 CLOSED

@schnidrig that's strange, so in multimode, the sync is actually running against the Ceph cluster that lives within the multinode cloud itself, which means that if you have an underlying Ceph cluster, it's potentially doing a 9 writes fore very write (1 write into Ceph => 3 Atmosphere OSDs => each write to 3 underlying cloud OSDs) which can be pretty slow.

This is why we started having AIO using local path provisioner instead to speed things up. If you leave that job running for sometime, does it eventually progress or is it fully stuck for a long time?

from atmosphere.

I let the job run. 14h hours now and no change.

nova-db-sync-kl9pn                                      1/1     Running     0          14h

I had changed the ceph pool from HDD to SSD. Otherwise keycloak would time out as well. With SSD keycloak is ok.

I had tried to run nova-manage api_db version. Even that would hang. So I'm not sure that the problem is really the speed of the underlying virtual disk.

from atmosphere.

after 14 hours, the db is still empty:

mysql> use nova_api;
Database changed
mysql> show tables;
Empty set (0.01 sec)

from atmosphere.

@mnaser

I tried replacing the nova image. I tested with the following images:

• docker.io/openstackhelm/nova:zed-ubuntu_jammy
• docker.io/openstackhelm/nova:yoga-ubuntu_focal

With them the nova-db-sync job hung as well

But with the image docker.io/openstackhelm/nova:xena-ubuntu_focal the job nova-db-sync finally ran through. The job took 4m to complete.

root@ctl1:~# kubectl -n openstack get jobs -l application=nova
NAME                COMPLETIONS   DURATION   AGE
nova-bootstrap      0/1           13m        13m
nova-cell-setup     0/1           13m        13m
nova-db-init        1/1           19s        13m
nova-db-sync        1/1           4m         13m
nova-ks-endpoints   1/1           78s        8m13s
nova-ks-service     1/1           22s        8m35s
nova-ks-user        1/1           5m4s       6m55s
nova-rabbit-init    1/1           27s        9m2s

That is of course not a viable work around, but it demonstrates that the problem is not the slow disk.

from atmosphere.

That's so strange. I've never really seen this behaviour.

Can you run an strace on the process by any chance to see what it's doing?

from atmosphere.

I found many lines like the following in the coredns logs

[ERROR] plugin/errors: 2 percona-xtradb-haproxy.openstack.svc.cluster.local.zh.switchengines.ch. A: read udp 10.0.1.189:43412->8.8.8.8:53: i/o timeout

So I replaced the DNS and now the nova-db-sync job succeeded.
Super wiered that nova behaves so differently than other services.

But I think I can close the issue.

Thanks for the help.

from atmosphere.