cinder LVM volume fails to delete about ansible-lxc-rpc HOT 16 CLOSED

Reproduced in Lab 02. Instances are stuck in 'creating' and 'deleting'.

https://gist.github.com/jcourtois/dd165a93f1ac5bd3310e

from ansible-lxc-rpc.

this issue is related to Issue: #99 and should be resolved in PR: #101.

from ansible-lxc-rpc.

Testing latest deployment in IAD lab 1. The suite was cleaning up about 8 volumes very rapidly (and perhaps a minute or two after creating) and it triggered another freezing. :|

Seeing very similar issue, with an additional detail that I don't remember noticing before. If I try to manually delete any of my volumes using lvremove inside the cinder container, I get this:

root@573972-cinder01_cinder_volumes_container-7454dcdb:~# lvremove /dev/mapper/cinder--volumes-volume--73584646--91f4--4651--b3a6--f46ee352fe50
Do you really want to remove and DISCARD active logical volume volume-73584646-91f4-4651-b3a6-f46ee352fe50? [y/n]: y
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  device-mapper: remove ioctl on  failed: Device or resource busy
  Unable to deactivate cinder--volumes-volume--73584646--91f4--4651--b3a6--f46ee352fe50 (252:5)
  Unable to deactivate logical volume "volume-73584646-91f4-4651-b3a6-f46ee352fe50"

Here are some logs from cinder-volumes.

https://gist.github.com/jcourtois/dd49918a88e4d99cb323

from ansible-lxc-rpc.

Couple questions:

• Is this a new install, or an upgraded one to the new code base?
• How long has the volume been in the deleting state?
• Is or was there a VM attached to the volume?
• Are there any processes within the container locking the volume? (dd, snapshot, other...)
• if you restart the container can you delete the volume?

from ansible-lxc-rpc.

Alright, so the issue did resolve itself; whatever was locking up LVM let go. I added a few more lines to https://gist.github.com/jcourtois/dd49918a88e4d99cb323.
As for your questions:
-This is a new install with latest code branch
-The deleting state for the seven or so volumes affected lasted about 25 minutes, after which they were all deleted within about a 1 minute period of time (about 5-10 seconds per volume to delete)
-These were compute integration tests, so there were probably VMs attached, but I can't say.
-Since the issue resolved, I can no longer say.

from ansible-lxc-rpc.

Testing is still underway. Since this resolved itself in a reasonable amount of time, I'll close this issue again. If it happens again I'll reopen.

from ansible-lxc-rpc.

This is likely simply a result of the volume having zeros written over it once the delete is executed. A process that does take time and creates a lock while zero'ing.

Let us know if this crops up again.

from ansible-lxc-rpc.

Of course it figures that when I stopped testing for the weekend, my last few cinder volumes would exhibit this behavior. I have 3 volumes that have been "deleting" since Saturday night.

Bonus: cinder-volumes has a stacktrace.

https://gist.github.com/jcourtois/49358546b9e4bdeb9242

from ansible-lxc-rpc.

Can you execute another delete to the same volume and let us know if it succeeds. It seems that the volume was in a locked state.

from ansible-lxc-rpc.

Which volume/snapshot and using the cinder api or lvremove?

from ansible-lxc-rpc.

Root problem? From the kernel logs.

Sep 22 19:31:28 569058-cinder01 kernel: [   12.570914] type=1400 audit(1411414288.192:137): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-openstack" name="/run/cgmanager/fs/none,name=systemd/" pid=6385 comm="cgmanager" fstype="cgroup" srcname="none,name=systemd" flags="rw"

from ansible-lxc-rpc.

The issue appears to be reproduced again in the lab where we changed the change_profile parameter in /etc/apparmor.d/abstractions/lxc/start-container to 'unconfined'. 🍂

from ansible-lxc-rpc.

Seeing this again in SAT6. In particular, after taking a snapshot of an LVM volume and deleting the snapshot, deleting the volume results in it getting stuck in the deleting stage.

from ansible-lxc-rpc.

@git-harry mentioned that this was a known issue in cinder. @git-harry does the gist above help you tracking down this issue?

from ansible-lxc-rpc.

Some additional info: https://gist.github.com/jameswthorne/62453bc79b9a9342acaf

from ansible-lxc-rpc.

This is going to be an upstream issue fix and is being tracked here: https://bugs.launchpad.net/cinder/+bug/1191960

@mancdaz @claco

from ansible-lxc-rpc.