Comments (4)
jschmid1
commented on September 27, 2024
The error is raised due to an empty return from disks.deploy which is evaluated (unconditionally) in rebuild.py.
The fix for this is two-fold:
-
Allow to have empty return (but raise an error) in
_check_deploy -
Make
osd.remove $idactually zap the disk (unmount/zap/clean)
Due to a yet unknown behavior in the osd.remove runner, which no longer completely destroys the LVs on the OSDs it destroys:
after a zap:
data1:~ # lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 254:0 0 20G 0 disk
└─vda1 254:1 0 20G 0 part /
vdb 254:16 0 20G 0 disk
└─ceph--051e6039--51aa--4216--92b1--15d97b25a1f0-osd--data--141e0a0c--86f8--4082--99eb--6e526010d7f7
253:0 0 19G 0 lvm
vdc 254:32 0 20G 0 disk
└─ceph--e6cd697b--029b--4b78--8838--e1af0aa9e1df-osd--data--c1cd2c10--ee8b--4f90--bd0a--f26b4bcfc078
253:1 0 19G 0 lvm
vdd 254:48 0 20G 0 disk
└─ceph--29305dd3--c305--4f87--8e6d--ab3507d1c70b-osd--data--a634f37e--4a1c--483c--bd8a--312361bd779c
253:2 0 19G 0 lvm
vde 254:64 0 20G 0 disk
└─ceph--8ddf59a4--e7bd--4d9c--b06f--11e24aec4e52-osd--data--0ec7d57e--96dc--4a6f--a9aa--d2acbb68293d
253:3 0 19G 0 lvm
vdf 254:80 0 20G 0 disk
└─ceph--bf65731b--15d4--4c62--b04c--46a796786b29-osd--data--5a37b17c--5f56--4cb3--b6cf--9d3153c7032b
253:4 0 19G 0 lvm
vdg 254:96 0 10G 0 disk
└─ceph--226cf2f2--5f77--41fd--917c--f84601ba9d4d-osd--data--494a7156--7c50--4c59--b750--876cd38438ce
253:5 0 9G 0 lvm
vdh 254:112 0 10G 0 disk
└─ceph--49cb4914--6852--4ada--af04--7f58ff9ac38d-osd--data--7227cdd6--87d9--4ef1--b8e0--787129bef068
253:6 0 9G 0 lvm
it's expected to have clean/unmounted disks after this.
The command that is used is ceph-volume lvm zap --osd-id $id --destroy
I suspect that ceph-volume handles drives differently when passed with --osd-id and when passed a raw device (/dev/sdx). The --destroy parameter shouldn't change it's behavior based on the input.
This needs to be verified though.
from deepsea.
smithfarm
commented on September 27, 2024
Note that this only started happening when we switched the ceph package from 14.2.2 to 14.2.3.
from deepsea.
smithfarm
commented on September 27, 2024
Also, upstream has released (or, rather, is in the process of releasing) 14.2.4 to fix this. See ceph/ceph#30429
from deepsea.
smithfarm
commented on September 27, 2024
Confirmed - the failure doesn't happen with 14.2.4, so this is another symptom of the ceph-volume regression that found its way into 14.2.3.
from deepsea.
Related Issues (20)
- make install fails on archlinux HOT 1
- cephdisks.unused falsely returns mounted disks
- Typo - should be c_v_commands instead of d_v_commands
- Example code for return structures of modules HOT 6
- rebuild runner needs to read error messages from osd.py(runner) HOT 2
- osd.remove fails to zap devices on ceph version 14.2.3-349 HOT 10
- How to deploy /dev/sdb ,/dev/sdc only use /dev/nvme0n1 as db device ? HOT 5
- Can not remove cluster node HOT 2
- [SES5] remove.osd functest fails HOT 9
- No role for rbd-client (e.g. mapping images) HOT 1
- Stage.3 Hang in Disks.Deploy HOT 6
- IGW lrbd support
- Device discovery not working in KVM environment HOT 3
- SES5: "time_init: ntp" does not work as expected HOT 1
- SES5: osd.redeploy fails for filestore -> bluestore
- deepsea hangs in stage 3 @ ceph.mgr.auth on
- deepsea monitor not working with gitfs
- stage.0 Exception: 'getpwuid(): uid not found:
- Deepsea development status after cephadm release HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepsea.