VDO - Create Error Device excluded by a filter. about vdo HOT 10 CLOSED

Hi penguinpages!

Can you please try running pvcreate --config devices/scan_lvs=1 -qq --test /dev/disk/by-id/ata-SAMSUNG_SSD_PM851_mSATA_512GB_S1EWNYAF609306 -vvvv and post the output? (It will be rather verbose.)

The point of the 'pvcreate' stage is to make sure that the device can be opened exclusively (by a later VDO) and to also make sure nothing else has a superblock on it (the latter is basically what could be accomplished by blkid -p instead). However, it does mean that if the device can't be used by LVM because of LVM filter reasons, it also can't be used by VDO.

I expect that some LVM filter is excluding this device -- perhaps sysfs_scan if I am wildly guessing. Hopefully the pvcreate -vvvv output will help diagnose that. (do you have sysfs mounted? does /sys/block/ have a folder for your device?)

from vdo.

[root@odin ~]# vcreate --config devices/scan_lvs=1 -qq --test /dev/disk/by-id/ata-SAMSUNG_SSD_PM851_mSATA_512GB_S1EWNYAF609306 -vvvv
bash: vcreate: command not found...
[root@odin ~]# pvcreate --config devices/scan_lvs=1 -qq --test /dev/disk/by-id/ata-SAMSUNG_SSD_PM851_mSATA_512GB_S1EWNYAF609306 -vvvv
TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
Device /dev/disk/by-id/ata-SAMSUNG_SSD_PM851_mSATA_512GB_S1EWNYAF609306 excluded by a filter.

And no output in /var/log/messages

Ya.. this is why it has been a pain to root cause.

from vdo.

Does anything show up if you do "sudo blkid -p /dev/disk/by-id/ata-SAMSUNG_SSD_PM851_mSATA_512GB_S1EWNYAF609306" ?

from vdo.

[root@odin ~]# blkid -p "/dev/disk/by-id/ata-SAMSUNG_SSD_PM851_mSATA_512GB_S1EWNYAF609306"
[root@odin ~]#

from vdo.

Hi @penguinpages,

This is a strange one, because typically pvcreate will filter out the device if it meets certain criteria, but that doesn't seem to be happening here. Things like nested LV's, partitioned devices, filesystems, etc.

Can you try to run wipefs /dev/disk/by-id/ata-SAMSUNG_SSD_PM851_mSATA_512GB_S1EWNYAF609306? That should output if there are any signatures found (though blkid should as well). I would also have expected to have more output from pvcreate with -vvvv on it, so we're going against all expectations at the moment. :)

Can you also provide the output of grep filter /etc/lvm/lvm.conf?

What OS are you running this on? This may help us try to recreate this issue if needed.

from vdo.

I agree this is weird. I tried wipe and dd and just to test... if I could format with regular file system but it won't work.

[root@odin ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xbcff3e20.

Command (m for help): p

Disk /dev/sdb: 512.1 GB, 512110190592 bytes, 1000215216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xbcff3e20

Device Boot Start End Blocks Id System

Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-1000215215, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-1000215215, default 1000215215):
Using default value 1000215215
Partition 1 of type Linux and of size 477 GiB is set

Command (m for help): p

Disk /dev/sdb: 512.1 GB, 512110190592 bytes, 1000215216 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xbcff3e20

Device Boot Start End Blocks Id System
/dev/sdb1 2048 1000215215 500106584 83 Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@odin ~]# sync
[root@odin ~]# mke2fs -L test /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
/dev/sdb1 is apparently in use by the system; will not make a filesystem here!
[root@odin ~]# lsblk -a
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 232.9G 0 disk
├─ST9250827AS_5RG02WHL 253:5 0 232.9G 0 mpath
└─odin_vdo_bay5 253:11 0 7.2T 0 vdo
sdb 8:16 0 477G 0 disk
├─sdb1 8:17 0 477G 0 part
└─SAMSUNG_SSD_PM851_mSATA_512GB_S1EWNYAF609306 253:4 0 477G 0 mpath
sdc 8:32 0 931.5G 0 disk
├─WDC_WDS100T2B0B-00YS70_183533804564 253:3 0 931.5G 0 mpath
└─odin_vdo_bay2 253:8 0 3T 0 vdo /data/odin_bay2
sdd 8:48 0 931.5G 0 disk
├─TOSHIBA_MK1059GSM_126HP19VT 253:6 0 931.5G 0 mpath
└─odin_vdo_bay3 253:9 0 3T 0 vdo
sde 8:64 0 465.8G 0 disk
├─WDC_WD5000BEVT-22ZAT0_WD-WXE0A89C8089 253:2 0 465.8G 0 mpath
└─odin_vdo_bay4 253:10 0 14.3T 0 vdo
sdf 8:80 1 57.3G 0 disk
├─sdf1 8:81 1 1G 0 part /boot
└─sdf2 8:82 1 56.3G 0 part
├─centos_odin-root 253:0 0 35.2G 0 lvm /
├─centos_odin-swap 253:1 0 4G 0 lvm
└─centos_odin-home 253:7 0 17.2G 0 lvm /home
[root@odin ~]# wipefs /dev/sdb
offset type

0x1fe dos [partition table]

[root@odin ~]# wipefs -a /dev/sdb
wipefs: error: /dev/sdb: probing initialization failed: Device or resource busy
[root@odin ~]# wipefs -a /dev/sdb --force
/dev/sdb: 2 bytes were erased at offset 0x000001fe (dos): 55 aa
/dev/sdb: calling ioclt to re-read partition table: Success
[root@odin ~]# sync
[root@odin ~]# lsblk -a
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 232.9G 0 disk
├─ST9250827AS_5RG02WHL 253:5 0 232.9G 0 mpath
└─odin_vdo_bay5 253:11 0 7.2T 0 vdo
sdb 8:16 0 477G 0 disk
└─SAMSUNG_SSD_PM851_mSATA_512GB_S1EWNYAF609306 253:4 0 477G 0 mpath
sdc 8:32 0 931.5G 0 disk
├─WDC_WDS100T2B0B-00YS70_183533804564 253:3 0 931.5G 0 mpath
└─odin_vdo_bay2 253:8 0 3T 0 vdo /data/odin_bay2
sdd 8:48 0 931.5G 0 disk
├─TOSHIBA_MK1059GSM_126HP19VT 253:6 0 931.5G 0 mpath
└─odin_vdo_bay3 253:9 0 3T 0 vdo
sde 8:64 0 465.8G 0 disk
├─WDC_WD5000BEVT-22ZAT0_WD-WXE0A89C8089 253:2 0 465.8G 0 mpath
└─odin_vdo_bay4 253:10 0 14.3T 0 vdo
sdf 8:80 1 57.3G 0 disk
├─sdf1 8:81 1 1G 0 part /boot
└─sdf2 8:82 1 56.3G 0 part
├─centos_odin-root 253:0 0 35.2G 0 lvm /
├─centos_odin-swap 253:1 0 4G 0 lvm
└─centos_odin-home 253:7 0 17.2G 0 lvm /home
[root@odin ~]# shred /dev/sdb

^C
[root@odin ~]# pvcreate --config devices/scan_lvs=1 -qq --test /dev/disk/by-id/ata-SAMSUNG_SSD_PM851_mSATA_512GB_S1EWNYAF609306 -vvvv
TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.
Device /dev/disk/by-id/ata-SAMSUNG_SSD_PM851_mSATA_512GB_S1EWNYAF609306 excluded by a filter.
[root@odin ~]# grep filter /etc/lvm/lvm.conf
# is used to drive LVM filtering like MD component detection, multipath
# Configuration option devices/filter.
# Run vgscan after changing the filter to regenerate the cache.
# See the use_lvmetad comment for a special case regarding filters.
# filter = [ "a|./|" ]
# filter = [ "r|/dev/cdrom|" ]
# filter = [ "a|loop|", "r|.|" ]
# filter = [ "a|loop|", "r|/dev/hdc|", "a|/dev/ide|", "r|.|" ]
# filter = [ "a|^/dev/hda8$|", "r|./|" ]
# filter = [ "a|./|" ]
# Configuration option devices/global_filter.
# Because devices/filter may be overridden from the command line, it is
# not suitable for system-wide device filtering, e.g. udev and lvmetad.
# Use global_filter to hide devices from these LVM system components.
# The syntax is the same as devices/filter. Devices rejected by
# global_filter are not opened by LVM.
# global_filter = [ "a|./|" ]
# The results of filtering are cached on disk to avoid rescanning dud
# This is a quick way of filtering out block devices that are not
# the LVs with device filters. Also, when this setting is enabled,
# filtered), which can cause performance problems on systems with
# an LV. The LVs are ignored using a built in device filter that
# by pvscan --cache), devices/filter is ignored and all devices are
# scanned by default. lvmetad always keeps unfiltered information
# which is provided to LVM commands. Each LVM command then filters
# based on devices/filter. This does not apply to other, non-regexp,
# filtering settings: component filters such as multipath and MD
# are checked during pvscan --cache. To filter a device and prevent
# devices/global_filter.
# Configuration option activation/mlock_filter.
# mlock_filter = [ "locale/locale-archive", "gconv/gconv-modules.cache" ]
[root@odin ~]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)
[root@odin ~]# uname -a
Linux odin.penguinpages.local 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@odin ~]#

Put this drive back into Windows Laptop.. and it works (well.. I started install to it).

I struggle to say this is VDO now in that I can't create even ext file system on it, but I assumed it was VDO issue (or prep command VDO calls of pvcreate). Now.. I am just annoyed :(

from vdo.

Hi @penguinpages,

Ah, but I just noticed that this device you're attempting to put VDO on top of is an mpath device?

I also notice that the lsblk output is showing that your VDO devices odin_vdo_bay[2345] are on the same disks? That output looks really odd to me, and I'm not sure what to take away from that.

Check out this [0] post and see if it helps. This would tell me that the filter approach was probably the culprit.

[0] https://access.redhat.com/discussions/2423461

from vdo.

this device you're attempting to put VDO on top of is an mpath device?

Really not sure what that is about. The drive came out of a Windows laptop .. and after every possible way I can think of to wipe it.. I have...

I have no RAID. VDO+Gluster. So this is a replica brick. This server just has disk bays so I try to make it logical that I label the block device to "server_bay#" so errors correlate more for me.

This server I attempted to teach myself how to replace a drive "failure" where I pulled the drive hot so it meant I get to document full replacement steps. So this failure to format VDO on top of it has been a real head scratcher.

Maybe that background will help where the VDO alias is still around... though I did do remove command and it took so I did not think that was the issue.. Plus... I even gave up "hot repair" and reboot to rule out some kid of stale table of the device issue.

Per the website / discussion posted I don't see mSATA_512GB_S1EWNYAF609306 or odin_vdo_bay1 mpath devices.

[root@odin ~]# ls -al /dev/mapper/
centos_odin-home odin_vdo_bay2 TOSHIBA_MK1059GSM_126HP19VT
centos_odin-root odin_vdo_bay3 WDC_WD5000BEVT-22ZAT0_WD-WXE0A89C8089
centos_odin-swap odin_vdo_bay4 WDC_WDS100T2B0B-00YS70_183533804564
control ST9250827AS_5RG02WHL
[root@odin ~]#

I hooked it back up to my laptop.. formated it with NTFS and volume .. now when I plug it in I don't see any referance in dmesg or var/logs etc... Nothing even shows in /dev/disk/

Let me get better baseline and not waste your time. Sorry for banter posting.. I assumed this was just a VDO remove step / cycle I was doing wrong. I will go down some other ratholes.

from vdo.

Hi @penguinpages,

This is really odd, and I'm sure it's going to turn out to be something really simple to fix, as it usually is. :)

Perhaps there's a firmware update for the disk that you could apply and maybe that would resolve?

from vdo.

Quick question: In the system where the drive appears to be excluded by a filter, what controller is this drive connected to? Is it a conventional onboard SATA controller, or is it a special controller (i.e.: SAS HBA, RAID controller, etc.)?

If you run "ls -l" on the "/sys/block/..." entry for the block device, you may be able to trace down which PCI ID it's hosted on, since the kernel will create a symlink to it. Here's an example:

$ ls -l /sys/block/sde
lrwxrwxrwx. 1 root root 0 May 10 19:30 /sys/block/sde -> ../devices/pci0000:00/0000:00:03.0/0000:05:00.0/host6/target6:2:0/6:2:0:0/block/sde

...and then I can see that PCI ID "05:00.0" is a hardware SATA/SAS RAID controller running a multi-drive RAID-0 volume.

from vdo.