Network issues possibly causing ceph rbd cmd timeouts about ceph-iscsi HOT 14 OPEN

For iscsi we do not support what you want to do. You would normally give each NIC port a IP, then create a iscsi target portal for each NIC port. Then you would have the initiator side do round robin over those iscsi portals. We do not support that with the upstream kernel and this project right now. SUSE ships a product that they support this type of config on. If you want to do that, you would have to contact them for details on how to get the kernel or do some googling, because I am not sure. I think it might just be in the free distro though.

For the ceph side, in this doc:

https://docs.ceph.com/docs/master/rados/configuration/network-config-ref/

it says:

For this reason, we generally recommend that dual-NIC systems either be configured with two IPs on the same network, or bonded.

Posting to the ceph users list is probably the best option for this type of questions, because I do not have any experience with this type of config in a real world setup.

from ceph-iscsi.

bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 9000
inet x.x.x.x netmask 255.255.255.0 broadcast x.x.x.x
inet6 fe80::82f1:ba05:ad23:a878 prefixlen 64 scopeid 0x20
ether x❌x❌x txqueuelen 1000 (Ethernet)
RX packets 19094018 bytes 85766194981 (79.8 GiB)
RX errors 0 dropped 34014 overruns 1794 frame 0
TX packets 20622318 bytes 86347023115 (80.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Nov 25 11:28:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.cdm-03: Timing out cmd.

This error means the iscsi target is sending commands to the ceph cluster, but it is taking over 30 seconds.

That bonding interface seems to have a lot of drops. Are you using that for ceph? If so then it is probably related to why we are getting timeouts when trying to access the ceph image.

The ext4 errors are a little odd though. Are you using ext4 for the root partition?

from ceph-iscsi.

hi
thanks for response
sorry that I'm late
i was sick and was resting

The ext4 errors are a little odd though. Are you using ext4 for the root partition?
yes but I think something in kernel is causing this problem and i suspect driver issue for dl380 G7

I just removed all bondings in my cluster and will report its behavior today

from ceph-iscsi.

hi again
sorry for the delay
I have been tweaking network section of my cluster
I have a question about network and it may not be related to iscsi but
how can I ditch bounding and use all my network adapters for higher throughput?
I have 2 nexus 3k switches and the funny thing is when I bound my adapters lcap won't go up except when I chose src-dst-ip-gre for my hashing policy and when it goes up I don't have a good experience
I have used teaming and bound and disabled NetworkManager but the same problem occurs
this is my network card https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c03572669
in general, how can I use multiple interfaces to maximize throughput?
should I forget about the age of my hardware and driver compatibility issues?

also, I can't thank you enough for your great work

from ceph-iscsi.

thank you so much for your time and help

sorry for the trouble but im trying to add new target to the second ip address
can i set HA to not show Failed: Clients can not be defined until a HA configuration has been defined(>2 gateways) ?
i don't want to add second gateway
have a great time till eternity

from ceph-iscsi.

hi
i did add new target to iscsi gateways
ill appreciate if you could comment on this
o- cluster ......................................................................................................... [Clusters: 1]
| o- ceph ............................................................................................................ [HEALTH_OK]
| o- pools .......................................................................................................... [Pools: 6]
| | o- .rgw.root ............................................................. [(x3), Commit: 0.00Y/613656640M (0%), Used: 768K]
| | o- default.rgw.control .................................................. [(x3), Commit: 0.00Y/613656640M (0%), Used: 0.00Y]
| | o- default.rgw.log ...................................................... [(x3), Commit: 0.00Y/613656640M (0%), Used: 0.00Y]
| | o- default.rgw.meta ...................................................... [(x3), Commit: 0.00Y/613656640M (0%), Used: 384K]
| | o- rbd ......................................................... [(x3), Commit: 16.0T/613656640M (2%), Used: 5776955081532b]
| | o- rbd-cache .................................................. [(x3), Commit: 0.00Y/6265060864K (0%), Used: 2381801488935b]
| o- topology .............................................................................................. [OSDs: 216,MONs: 3]
o- disks ....................................................................................................... [16.0T, Disks: 3]
| o- rbd ........................................................................................................... [rbd (16.0T)]
| o- cdm-01 ............................................................................................... [rbd/cdm-01 (10.0T)]
| o- cdm-02 ................................................................................................ [rbd/cdm-02 (1.0T)]
| o- cdm-03 ................................................................................................ [rbd/cdm-03 (5.0T)]
o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 2]
o- iqn.2019-10.com.datamanagement.iscsi-gw:iscsi-igw ................................................. [Auth: None, Gateways: 2]
| o- disks .......................................................................................................... [Disks: 0]
| o- gateways ............................................................................................ [Up: 2/2, Portals: 2]
| | o- ceph-client01 ....................................................................................... [10.40.81.103 (UP)]
| | o- ceph-client02 ....................................................................................... [10.40.81.104 (UP)]
| o- host-groups .................................................................................................. [Groups : 0]
| o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 1]
| o- iqn.1998-01.com.vmware:mdcesx040-6755d8eb ................................................. [Auth: CHAP, Disks: 0(0.00Y)]
o- iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw ......................................................... [Auth: None, Gateways: 2]
o- disks .......................................................................................................... [Disks: 3]
| o- rbd/cdm-01 ............................................................................... [Owner: ceph-client02, Lun: 0]
| o- rbd/cdm-02 ............................................................................... [Owner: ceph-client02, Lun: 1]
| o- rbd/cdm-03 ............................................................................... [Owner: ceph-client02, Lun: 2]
o- gateways ............................................................................................ [Up: 2/2, Portals: 2]
| o- ceph-client01 ....................................................................................... [10.40.81.101 (UP)]
| o- ceph-client02 ....................................................................................... [10.40.81.102 (UP)]
o- host-groups .................................................................................................. [Groups : 0]
o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 1]
o- iqn.1998-01.com.vmware:mdcesx040-6755d8eb ................................................. [Auth: CHAP, Disks: 3(16.0T)]
o- lun 0 ....................................................................... [rbd/cdm-01(10.0T), Owner: ceph-client02]
o- lun 1 ........................................................................ [rbd/cdm-02(1.0T), Owner: ceph-client02]
o- lun 2 ........................................................................ [rbd/cdm-03(5.0T), Owner: ceph-client02]

and i get this error in adding same disks
Failed - Disk rbd/cdm-01 cannot be used because it is already mapped on target iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
disk auto-map failed(1), try using the /iscsi-target//disks add command

from ceph-iscsi.

hi
i decided to add the second gateway to the ISCSI and second interface, the above log is resolved
and without creating iscsi portal VMWare has discovered second interface and now has active io on it
but the strange thing is the tcmu_rbd_handle_timedout_cmd still occurs
regards
Ahmadi

from ceph-iscsi.

For iscsi we do not support what you want to do. You would normally give each NIC port a IP, then create a iscsi target portal for each NIC port. Then you would have the initiator side do round robin over those iscsi portals. We do not support that with the upstream kernel and this project right now. SUSE ships a product that they support this type of config on. If you want to do that, you would have to contact them for details on how to get the kernel or do some googling, because I am not sure. I think it might just be in the free distro though.

For the ceph side, in this doc:

https://docs.ceph.com/docs/master/rados/configuration/network-config-ref/

it says:

For this reason, we generally recommend that dual-NIC systems either be configured with two IPs on the same network, or bonded.

Posting to the ceph users list is probably the best option for this type of questions, because I do not have any experience with this type of config in a real world setup.

are you saying i can't use 2 interfaces and get my throughput up for 1 lun?
as I googled ceph-iscsi cant add 2 ip addresses to the same hostname
bound is not good for iscsi and is not recommended

from ceph-iscsi.

can i set HA to not show Failed: Clients can not be defined until a HA configuration has been defined(>2 gateways) ?

In /etc/ceph/iscsi-gateway.cfg

add:

[config]
minimum_gateways = 1

from ceph-iscsi.

but the strange thing is the tcmu_rbd_handle_timedout_cmd still occurs

If you are seeing those errors, it's normally because commands are taking too long on the OSD/network. You should check the ceph OSD and MON logs for errors, timeouts etc, and check the network.

from ceph-iscsi.

Could you tell me a little more about what you are trying to do?

Exporting a image through 2 different targets is not a common case, so I am not sure if it works. Are you just waning to export an image to 2 different ESXi hosts/initiators? If so, then you would want to use a hostgroup.

Could you tell me what type of workload you are doing when you see the command timeouts?

from ceph-iscsi.

what I'm trying is as follows
I have 2 bare-metal dl380g7 as iscsi gateway, each has 2x 10Gb FC/ethernet interface
I want to use both interfaces to get maximum throughput (20Gb)
uname -a
Linux ceph-client02 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

what I have done so far is
I have enabled jumbo frame 9000 cluster-wide
bounding (same result)
teaming (same result)
modes that I have tried are LACP Load balance and pretty much every possible scenario
I get timeouts every time
tested on Windows Linux and Vmware ESXI
tried on 3 different servers dl380g7 and g6 s

my network technician says I have no errors on switch interfaces and I should mention switches are brand new and part numbers are N3K-C3172-BA-L3

this is a trace happend on my iscsi server
it was using single ethernet with one ip address

Dec 9 14:45:32 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:45:32 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:45:32 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:45:32 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:45:32 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:45:32 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:45:32 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:45:36 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.veeamh20T: Starting lock acquisition operation.
Dec 9 14:45:36 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.veeamh20T: Acquired exclusive lock.
Dec 9 14:45:36 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.veeamh20T: Lock acquisition successful
Dec 9 14:46:07 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:07 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.veeamh20T: Handler connection lost (lock state 1)
Dec 9 14:46:07 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:07 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:07 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:07 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:07 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:07 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:46:07 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:46:07 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:46:07 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:46:11 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.veeamh20T: Starting lock acquisition operation.
Dec 9 14:46:11 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.veeamh20T: Acquired exclusive lock.
Dec 9 14:46:11 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.veeamh20T: Lock acquisition successful
Dec 9 14:46:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:42 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.veeamh20T: Handler connection lost (lock state 1)
Dec 9 14:46:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:42 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:46:42 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:46:42 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:46:42 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:46:42 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:46:47 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.veeamh20T: Starting lock acquisition operation.
Dec 9 14:46:47 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.veeamh20T: Acquired exclusive lock.
Dec 9 14:46:47 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.veeamh20T: Lock acquisition successful
Dec 9 14:47:18 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:18 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.veeamh20T: Handler connection lost (lock state 1)
Dec 9 14:47:18 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:18 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:18 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:18 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:18 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:18 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:18 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:47:18 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:47:18 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:47:18 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:47:22 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.veeamh20T: Starting lock acquisition operation.
Dec 9 14:47:22 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.veeamh20T: Acquired exclusive lock.
Dec 9 14:47:22 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.veeamh20T: Lock acquisition successful
Dec 9 14:47:53 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:53 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.veeamh20T: Handler connection lost (lock state 1)
Dec 9 14:47:53 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:53 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:53 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:53 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:53 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:53 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:47:53 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:47:53 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:47:53 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:47:53 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:47:57 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.veeamh20T: Starting lock acquisition operation.
Dec 9 14:47:57 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.veeamh20T: Acquired exclusive lock.
Dec 9 14:47:57 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.veeamh20T: Lock acquisition successful
Dec 9 14:52:38 ceph-client02 kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 5131
Dec 9 14:54:19 ceph-client02 kernel: ABORT_TASK: Sending TMR_TASK_DOES_NOT_EXIST for ref_tag: 5155
Dec 9 14:54:22 ceph-client02 kernel: perf: interrupt took too long (2515 > 2500), lowering kernel.perf_event_max_sample_rate to 79000
Dec 9 14:55:52 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:55:52 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.veeamh20T: Handler connection lost (lock state 1)
Dec 9 14:55:52 ceph-client02 kernel: rx_data returned -512, expecting 48.
Dec 9 14:55:52 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:55:53 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:55:53 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:55:53 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:55:53 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:55:53 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:55:53 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:55:53 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:55:54 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:55:57 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.veeamh20T: Starting lock acquisition operation.
Dec 9 14:55:57 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.veeamh20T: Acquired exclusive lock.
Dec 9 14:55:57 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.veeamh20T: Lock acquisition successful
Dec 9 14:56:28 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:56:28 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.veeamh20T: Handler connection lost (lock state 1)
Dec 9 14:56:28 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:56:28 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:56:28 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:56:28 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:56:28 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:56:28 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:56:28 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:56:28 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:56:32 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.veeamh20T: Starting lock acquisition operation.
Dec 9 14:56:32 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.veeamh20T: Acquired exclusive lock.
Dec 9 14:56:32 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.veeamh20T: Lock acquisition successful
Dec 9 14:57:03 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:57:03 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.veeamh20T: Handler connection lost (lock state 1)
Dec 9 14:57:03 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:57:03 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:57:03 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:57:03 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:57:03 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:57:03 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:03 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:03 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:57:07 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.veeamh20T: Starting lock acquisition operation.
Dec 9 14:57:07 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.veeamh20T: Acquired exclusive lock.
Dec 9 14:57:07 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.veeamh20T: Lock acquisition successful
Dec 9 14:57:38 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:57:38 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.veeamh20T: Handler connection lost (lock state 1)
Dec 9 14:57:38 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:57:38 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:57:38 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:57:38 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 14:57:38 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:57:38 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:38 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:39 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:57:39] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 14:57:40 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:57:40] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 14:57:41 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:41 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:42 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:57:42] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 14:57:42 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:42 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:44 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:44 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:47 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:47 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:47 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:47 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:49 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:57:49] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 14:57:50 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:50 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:52 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:52 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:53 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:53 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:54 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:57:54] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 14:57:56 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:56 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:57 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:57 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:57:59 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:57:59] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 14:57:59 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:57:59 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:58:02 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:58:02 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:58:04 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:58:04] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 14:58:05 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:58:05 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:58:08 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:58:08 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:58:09 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:58:09] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 14:58:11 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:58:11 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:58:14 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:58:14] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 14:58:14 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:58:14 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:58:17 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:58:17 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:58:19 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:58:19] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 14:58:20 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 14:58:20 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 14:58:22 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 14:58:24 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:58:24] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 14:58:27 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 14:58:27] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 15:00:46 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.veeamh20T: Starting lock acquisition operation.
Dec 9 15:00:47 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.veeamh20T: Acquired exclusive lock.
Dec 9 15:00:47 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.veeamh20T: Lock acquisition successful
Dec 9 15:01:01 ceph-client02 systemd: Started Session 30 of user root.
Dec 9 15:01:17 ceph-client02 tcmu-runner: tcmu_rbd_handle_timedout_cmd:995 rbd/rbd.veeamh20T: Timing out cmd.
Dec 9 15:01:17 ceph-client02 tcmu-runner: tcmu_notify_conn_lost:201 rbd/rbd.veeamh20T: Handler connection lost (lock state 1)
Dec 9 15:01:17 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:245: Disabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 15:01:17 ceph-client02 kernel: Unable to locate Target Portal Group on iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw
Dec 9 15:01:17 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 15:01:17 ceph-client02 tcmu-runner: tgt_port_grp_recovery_thread_fn:271: Enabled iscsi/iqn.2019-10.com.redhat.iscsi-gw:iscsi-igw/tpgt_1.
Dec 9 15:01:21 ceph-client02 tcmu-runner: alua_implicit_transition:570 rbd/rbd.veeamh20T: Starting lock acquisition operation.
Dec 9 15:01:21 ceph-client02 tcmu-runner: tcmu_rbd_lock:762 rbd/rbd.veeamh20T: Acquired exclusive lock.
Dec 9 15:01:21 ceph-client02 tcmu-runner: tcmu_acquire_dev_lock:441 rbd/rbd.veeamh20T: Lock acquisition successful
Dec 9 15:03:17 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:03:17 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:03:17 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:03:17 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:03:17 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:03:17 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: Call Trace:
Dec 9 15:03:17 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:03:17 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:03:17 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:03:17 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:03:17 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:03:17 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:03:17 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:03:17 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:03:17 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:03:17 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:03:17 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f1a ]---
Dec 9 15:03:17 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:03:17 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:03:17 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:03:17 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:03:17 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:03:17 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: Call Trace:
Dec 9 15:03:17 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:03:17 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:03:17 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:03:17 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:03:17 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:03:17 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:03:17 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:03:17 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:03:17 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:03:17 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:03:17 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f1b ]---
Dec 9 15:03:17 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:03:17 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:03:17 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:03:17 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:03:17 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:03:17 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: Call Trace:
Dec 9 15:03:17 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:03:17 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:03:17 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:03:17 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:03:17 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:03:17 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:03:17 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:03:17 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:03:17 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:03:17 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:03:17 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f1c ]---
Dec 9 15:03:17 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:03:17 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:03:17 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:03:17 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:03:17 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:03:17 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: Call Trace:
Dec 9 15:03:17 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:03:17 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:03:17 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:03:17 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:03:17 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:03:17 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:03:17 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:03:17 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:03:17 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:03:17 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:03:17 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f1d ]---
Dec 9 15:03:17 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:03:17 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:03:17 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:03:17 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:03:17 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:03:17 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: Call Trace:
Dec 9 15:03:17 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:03:17 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:03:17 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:03:17 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:03:17 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:03:17 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:03:17 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:03:17 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:03:17 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:03:17 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:03:17 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:03:17 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f1e ]---
Dec 9 15:03:17 ceph-client02 kernel: Unexpected ret: -104 send data 48
Dec 9 15:03:17 ceph-client02 kernel: tx_data returned -32, expecting 48.
Dec 9 15:03:17 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 15:07:29 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 15:07:29] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 15:07:29 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 15:07:29] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 15:07:32 ceph-client02 journal: ::ffff:10.47.89.13 - - [09/Dec/2019 15:07:32] "GET /api/sysinfo/hostname HTTP/1.1" 200 -
Dec 9 15:10:08 ceph-client02 kernel: iSCSI Login timeout on Network Portal 10.47.89.102:3260
Dec 9 15:10:08 ceph-client02 kernel: Unexpected ret: -32 send data 48
Dec 9 15:10:08 ceph-client02 kernel: tx_data returned -32, expecting 48.
Dec 9 15:10:08 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 15:10:23 ceph-client02 kernel: iSCSI Login timeout on Network Portal 10.47.89.102:3260
Dec 9 15:10:23 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 15:10:38 ceph-client02 kernel: iSCSI Login timeout on Network Portal 10.47.89.102:3260
Dec 9 15:10:38 ceph-client02 kernel: Unexpected ret: -32 send data 48
Dec 9 15:10:38 ceph-client02 kernel: tx_data returned -32, expecting 48.
Dec 9 15:10:38 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 15:10:53 ceph-client02 kernel: iSCSI Login timeout on Network Portal 10.47.89.102:3260
Dec 9 15:10:53 ceph-client02 kernel: Unexpected ret: -32 send data 48
Dec 9 15:10:53 ceph-client02 kernel: tx_data returned -32, expecting 48.
Dec 9 15:10:53 ceph-client02 kernel: iSCSI Login negotiation failed.
Dec 9 15:11:05 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:11:05 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:11:05 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:11:05 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:11:05 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:11:05 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: Call Trace:
Dec 9 15:11:05 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:11:05 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:11:05 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:11:05 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:11:05 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:11:05 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:11:05 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f1f ]---
Dec 9 15:11:05 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:11:05 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:11:05 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:11:05 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:11:05 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:11:05 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: Call Trace:
Dec 9 15:11:05 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:11:05 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:11:05 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:11:05 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:11:05 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:11:05 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:11:05 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f20 ]---
Dec 9 15:11:05 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:11:05 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:11:05 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:11:05 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:11:05 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:11:05 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: Call Trace:
Dec 9 15:11:05 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:11:05 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:11:05 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:11:05 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:11:05 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:11:05 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:11:05 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f21 ]---
Dec 9 15:11:05 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:11:05 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:11:05 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:11:05 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:11:05 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:11:05 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: Call Trace:
Dec 9 15:11:05 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:11:05 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:11:05 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:11:05 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:11:05 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:11:05 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:11:05 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f22 ]---
Dec 9 15:11:05 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:11:05 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:11:05 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:11:05 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:11:05 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:11:05 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: Call Trace:
Dec 9 15:11:05 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:11:05 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:11:05 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:11:05 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:11:05 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:11:05 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:11:05 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f23 ]---
Dec 9 15:11:05 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:11:05 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:11:05 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:11:05 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:11:05 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:11:05 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: Call Trace:
Dec 9 15:11:05 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:11:05 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:11:05 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:11:05 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:11:05 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:11:05 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:11:05 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f24 ]---
Dec 9 15:11:05 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:11:05 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:11:05 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:11:05 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:11:05 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:11:05 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: Call Trace:
Dec 9 15:11:05 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:11:05 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:11:05 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:11:05 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:11:05 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:11:05 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:11:05 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f25 ]---
Dec 9 15:11:05 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:11:05 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:11:05 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:11:05 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:11:05 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:11:05 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: Call Trace:
Dec 9 15:11:05 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:11:05 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:11:05 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:11:05 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:11:05 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:11:05 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:11:05 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f26 ]---
Dec 9 15:11:05 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:11:05 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:11:05 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:11:05 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:11:05 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:11:05 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: Call Trace:
Dec 9 15:11:05 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:11:05 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:11:05 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:11:05 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:11:05 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:11:05 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:11:05 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f27 ]---
Dec 9 15:11:05 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:11:05 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:11:05 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:11:05 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:11:05 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:11:05 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: Call Trace:
Dec 9 15:11:05 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:11:05 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:11:05 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:11:05 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:11:05 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:11:05 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:11:05 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: ---[ end trace 5ec10e6c08909f28 ]---
Dec 9 15:11:05 ceph-client02 kernel: ------------[ cut here ]------------
Dec 9 15:11:05 ceph-client02 kernel: WARNING: CPU: 3 PID: 18 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: Modules linked in: target_core_pscsi target_core_file target_core_iblock target_core_user uio rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt gpio_ich iTCO_vendor_support ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ocrdma(T) ghash_clmulni_intel aesni_intel ib_core lrw gf128mul ipmi_si glue_helper joydev ablk_helper cryptd ipmi_devintf pcspkr ipmi_msghandler sg hpilo hpwdt acpi_power_meter i7core_edac lpc_ich pcc_cpufreq ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod pata_acpi radeon lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nvmet_fc nvmet libata
Dec 9 15:11:05 ceph-client02 kernel: crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core serio_raw hpsa be2net scsi_transport_fc bnx2 drm_panel_orientation_quirks scsi_tgt scsi_transport_sas crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
Dec 9 15:11:05 ceph-client02 kernel: CPU: 3 PID: 18 Comm: kworker/u66:0 Kdump: loaded Tainted: G W I ------------ T 3.10.0-957.el7.x86_64 #1
Dec 9 15:11:05 ceph-client02 kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Dec 9 15:11:05 ceph-client02 kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: Call Trace:
Dec 9 15:11:05 ceph-client02 kernel: [] dump_stack+0x19/0x1b
Dec 9 15:11:05 ceph-client02 kernel: [] __warn+0xd8/0x100
Dec 9 15:11:05 ceph-client02 kernel: [] warn_slowpath_null+0x1d/0x20
Dec 9 15:11:05 ceph-client02 kernel: [] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] core_tmr_lun_reset+0x4c1/0x780 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] target_tmr_work+0x107/0x170 [target_core_mod]
Dec 9 15:11:05 ceph-client02 kernel: [] process_one_work+0x17f/0x440
Dec 9 15:11:05 ceph-client02 kernel: [] worker_thread+0x126/0x3c0
Dec 9 15:11:05 ceph-client02 kernel: [] ? manage_workers.isra.25+0x2a0/0x2a0
Dec 9 15:11:05 ceph-client02 kernel: [] kthread+0xd1/0xe0
Dec 9 15:11:05 ceph-client02 kernel: [] ? insert_kthread_work+0x40/0x40
Dec 9 15:11:05 ceph-client02 kernel: [

from ceph-iscsi.

Ah sorry, I just realized you were the same user as in

open-iscsi/tcmu-runner#586

So basically the same thing where something is causing commands to timeout, then you hit a kernel bug which seems slightly different due to the different kernels (Bart's patch did not cause it, but just fired a warn in a different place due to adding the new warn call).

• For the root issue for what is causing the command timeouts, did you check the ceph OSD logs (normally in /var/log/ceph) like I mentioned in the previous comment? See the section "OLD REQUESTS OR SLOW REQUESTS":

https://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/

• Can you also confirm that on the current setup you are not seeing errors/overruns/drops in the ifconfig output for the interfaces you are using for ceph or vmware?

from ceph-iscsi.

I'm thinking that I have found the cause and wanted to wait till I'm sure
I think its due to low ram available on OSD hosts that causing this behavior
I have checked logs on OSD nodes and saw random slow-ops on disks furthermore checked the ram usage and saw when data is being written/read ram usage goes up and stops at a certain margin
so my monitoring station didn't warn me of it but I think bluestore do did not have enough ram to operate and disks show slow ops I don't have enough rams now so I'm thinking of adding new SSD for swap to check te behavior

from ceph-iscsi.