[SES5] remove.osd functest fails about deepsea HOT 9 CLOSED

@kshtsk tells me that this failure has been keeping the devel-storage-5.0 CI red for a month now. Can we raise the priority?

from deepsea.

I just ran salt-run state.orch ceph.functests.1node.remove successfully. Is this environment with the failure accessible?

from deepsea.

@swiftgist The CI always destroys the environment. Reproducing would involve using the same command (salt-run state.orch ceph.functests.1node) in the same environment (SLE-15-SP1, deepsea 0.8.12, single-node cluster). Is that what you did?

from deepsea.

I ran salt-run state.orch ceph.functests.1node.remove. I do have a multinode cluster. How many OSDs do you have in your single node cluster? The multiple OSD removal test removes both OSD 0 and 1.

from deepsea.

we actually try and deploy the system

from deepsea.

@swiftgist Possibly you are not only running on a multinode cluster, but you might also be running on a newer version of DeepSea - I see the same test succeeds in the CI when run on the tip of the SES5 branch (v0.8.12-12-g162e1139).

@jschmid1 Can we get an incremental release pushed to Devel:Storage:5.0 to fix the breakage there?

from deepsea.

found the fix after experimenting on the teuthology VM. It's already in #1724.

from deepsea.

#1724 merge commit is 6dff6ca

$ git describe 6dff6ca
v0.8.12-2-g6dff6ca9

@swiftgist Right. Devel:Storage:5.0 still has DeepSea version v0.8.12-0-e885d513e, hence the CI breakage and this bug report. Thanks for fixing it and looking into it.

from deepsea.

The moral of this story, I guess, is to make sure the CI is passing on the tip of the release branch before cutting a release.

Also, after pushing to Devel:Storage:* I do not submit to maintenance immediately, but rather wait for a CI run in the devel CI and only submit once the CI is green with the new release.

from deepsea.