Comments (6)
@schnidrig that's strange, so in multimode, the sync is actually running against the Ceph cluster that lives within the multinode cloud itself, which means that if you have an underlying Ceph cluster, it's potentially doing a 9 writes fore very write (1 write into Ceph => 3 Atmosphere OSDs => each write to 3 underlying cloud OSDs) which can be pretty slow.
This is why we started having AIO using local path provisioner instead to speed things up. If you leave that job running for sometime, does it eventually progress or is it fully stuck for a long time?
from atmosphere.
I let the job run. 14h hours now and no change.
nova-db-sync-kl9pn 1/1 Running 0 14h
I had changed the ceph pool from HDD to SSD. Otherwise keycloak would time out as well. With SSD keycloak is ok.
I had tried to run nova-manage api_db version
. Even that would hang. So I'm not sure that the problem is really the speed of the underlying virtual disk.
from atmosphere.
after 14 hours, the db is still empty:
mysql> use nova_api;
Database changed
mysql> show tables;
Empty set (0.01 sec)
from atmosphere.
I tried replacing the nova image. I tested with the following images:
- docker.io/openstackhelm/nova:zed-ubuntu_jammy
- docker.io/openstackhelm/nova:yoga-ubuntu_focal
With them the nova-db-sync job hung as well
But with the image docker.io/openstackhelm/nova:xena-ubuntu_focal
the job nova-db-sync finally ran through. The job took 4m to complete.
root@ctl1:~# kubectl -n openstack get jobs -l application=nova
NAME COMPLETIONS DURATION AGE
nova-bootstrap 0/1 13m 13m
nova-cell-setup 0/1 13m 13m
nova-db-init 1/1 19s 13m
nova-db-sync 1/1 4m 13m
nova-ks-endpoints 1/1 78s 8m13s
nova-ks-service 1/1 22s 8m35s
nova-ks-user 1/1 5m4s 6m55s
nova-rabbit-init 1/1 27s 9m2s
That is of course not a viable work around, but it demonstrates that the problem is not the slow disk.
from atmosphere.
That's so strange. I've never really seen this behaviour.
Can you run an strace on the process by any chance to see what it's doing?
from atmosphere.
I found many lines like the following in the coredns logs
[ERROR] plugin/errors: 2 percona-xtradb-haproxy.openstack.svc.cluster.local.zh.switchengines.ch. A: read udp 10.0.1.189:43412->8.8.8.8:53: i/o timeout
So I replaced the DNS and now the nova-db-sync job succeeded.
Super wiered that nova behaves so differently than other services.
But I think I can close the issue.
Thanks for the help.
from atmosphere.
Related Issues (20)
- [stable/zed] Add missing deps to Molecule venv
- SRIOV and vGPU devices can't be discover by nova-compute HOT 1
- [stable/2024.1] fix(horizon): apply a patch to fix resource provider panel
- [stable/2023.2] fix(horizon): apply a patch to fix resource provider panel
- [stable/2023.1] fix(horizon): apply a patch to fix resource provider panel
- [stable/zed] fix(horizon): apply a patch to fix resource provider panel HOT 1
- [stable/2024.1] [ATMOSPHERE-302] fix: set variables for cluster issuer name for keycloak and kube-prom-stack
- [stable/2023.2] [ATMOSPHERE-302] fix: set variables for cluster issuer name for keycloak and kube-prom-stack
- [stable/2023.1] [ATMOSPHERE-302] fix: set variables for cluster issuer name for keycloak and kube-prom-stack
- [stable/zed] [ATMOSPHERE-302] fix: set variables for cluster issuer name for keycloak and kube-prom-stack
- [stable/zed] [ATMOSPHERE-307] Add Ironic
- [stable/2023.1] [ATMOSPHERE-307] Add Ironic
- [stable/2023.1] [ATMOSPHERE-327] images(python-openstackclient): Add python-ironicclient
- [stable/zed] [ATMOSPHERE-327] images(python-openstackclient): Add python-ironicclient
- Unable to install atmosphere HOT 6
- [stable/2023.1] Add TLS to node exporter
- [stable/zed] Add TLS to node exporter
- Issue with encrypted volumes be create from encrypted volume types HOT 1
- pxc_strict_mode HOT 1
- Keycloak ingress annotations not being used HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from atmosphere.