open-power-host-os / linux Goto Github PK

This project forked from torvalds/linux

Linux kernel source tree

License: Other

Makefile 0.27% C 96.98% Assembly 1.85% C++ 0.61% Objective-C 0.10% Shell 0.03% Perl 0.11% Awk 0.01% SourcePawn 0.01% Python 0.03% Yacc 0.01% Lex 0.01% UnrealScript 0.01%

linux's Introduction

OpenPOWER Host OS web page

This repository uses Jekyll to produce a static web page with information about OpenPOWER Host OS, its components and release tags.

The posts in _posts are created in an automated way each time a tag is built and tested by the build script.

Every post contains information about the release tag, the specific commit IDs from the build script and packages metadata that were used to build it and the branches and commit IDs of the packages themselves.

Regular and stable release tags show up under /tags/ and /tags/stable/ in the web page, respectively.

linux's People

Contributors

Stargazers

Watchers

Forkers

lisibann gyribeiro nxpower liyi-ibm

linux's Issues

PCI Passthrough : Starting the guest with CX5 adapter leads to Call Trace EEH

Guest failed to start with CX5 passthrough, same time host hit with EEH :

[ 1315.195786] mlx5_0:wait_for_async_commands:732:(pid 3719): done with all pending requests
[ 1317.267625] mlx5_1:wait_for_async_commands:732:(pid 3719): done with all pending requests
[ 1317.762382] EEH: PHB#0 failure detected, location: N/A
[ 1317.762435] CPU: 0 PID: 91 Comm: kworker/0:1 Not tainted 4.13.0-4.dev.git49564cb.el7.centos.ppc64le #1
[ 1317.762590] Workqueue: events work_for_cpu_fn
[ 1317.762710] Call Trace:
[ 1317.762764] [c000003fe7f87930] [c000000000ac906c] dump_stack+0xb0/0xf4 (unreliable)
[ 1317.762889] [c000003fe7f87970] [c00000000003b3b0] eeh_dev_check_failure+0x1f0/0x590
[ 1317.763016] [c000003fe7f87a10] [c0000000000a1204] pnv_pci_read_config+0xc4/0x140
[ 1317.763136] [c000003fe7f87a50] [c0000000005ccff4] pci_bus_read_config_word+0xb4/0x100
[ 1317.763265] [c000003fe7f87ab0] [c0000000005d653c] pci_raw_set_power_state+0xfc/0x280
[ 1317.763388] [c000003fe7f87b40] [c0000000005da310] pci_set_power_state+0xd0/0x1d0
[ 1317.763518] [c000003fe7f87b80] [c000000000785e2c] vfio_pci_probe+0x13c/0x250
[ 1317.763634] [c000003fe7f87bd0] [c0000000005dfe3c] local_pci_probe+0x6c/0x130
[ 1317.763751] [c000003fe7f87c60] [c0000000001169e8] work_for_cpu_fn+0x38/0x60
[ 1317.763862] [c000003fe7f87c90] [c00000000011bc00] process_one_work+0x1a0/0x490
[ 1317.763985] [c000003fe7f87d30] [c00000000011c168] worker_thread+0x278/0x520
[ 1317.764089] [c000003fe7f87dc0] [c0000000001244a8] kthread+0x168/0x1b0
[ 1317.764201] [c000003fe7f87e30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 1317.764320] EEH: Detected error on PHB#0
[ 1317.764363] EEH: This PCI device has failed 1 times in the last hour
[ 1317.764516] EEH: Notify device drivers to shutdown
[ 1317.764615] EEH: Collect temporary log
[ 1317.764681] PHB4 PHB#0 Diag-data (Version: 1)
[ 1317.764762] brdgCtl: 0000ffff
[ 1317.764821] RootSts: ffffffff ffffffff ffffffff ffffffff 0000ffff
[ 1317.764924] RootErrSts: ffffffff ffffffff ffffffff
[ 1317.765009] RootErrLog: ffffffff ffffffff ffffffff ffffffff
[ 1317.765112] sourceId: ffffffff
[ 1317.765167] nFir: 0000800000000000 0030001c00000000 0000800000000000
[ 1317.765271] PhbSts: 0000001800000000 0000001800000000
[ 1317.765358] Lem: 0000000100300100 ffffffffffffffff 0000000000000000
[ 1317.765467] PhbErr: 00000c8000000000 0000008000000000 2148000098000240 a008400000000000
[ 1317.765598] RxeArbErr: 4000000020000000 4000000000000000 0050000400000000 0000000000000000
[ 1317.765727] RxeMrgErr: 0000000000000001 0000000000000001 0000000000000000 0000000000000000
[ 1317.765859] PblErr: 0000000001000000 0000000001000000 0000000000000000 0000000000000000
[ 1317.765991] PcieDlp: 0000000000000000 0000000000000000 0fd5000000000000
[ 1317.766099] RegbErr: 0040004e54200800 0040000000000000 62000a1018000000 1800000000000000
[ 1317.766233] EEH: Reset without hotplug activity
[ 1317.792802] vfio-pci 0000:01:00.0: Refused to change power state, currently in D3
[ 1317.792918] iommu: Removing device 0000:01:00.0 from group 0
[ 1317.822800] mlx5_core 0000:01:00.0: Refused to change power state, currently in D3
[ 1317.822903] mlx5_core 0000:01:00.0: Using 64-bit DMA iommu bypass
[ 1317.823786] mlx5_core 0000:01:00.0: firmware version: 65535.65535.65535
[ 1327.853266] mlx5_core 0000:01:00.0: Firmware over 10000 MS in pre-initializing state, aborting
[ 1327.853363] mlx5_core 0000:01:00.0: mlx5_load_one failed with error code -16
[ 1327.853588] mlx5_core: probe of 0000:01:00.0 failed with error -16
[ 1327.887193] Unable to handle kernel paging request for data at address 0x00000008
[ 1327.887349] Faulting instruction address: 0xc000000000ad6c98
[ 1327.887462] Oops: Kernel access of bad area, sig: 11 [#1]
[ 1327.887553] SMP NR_CPUS=1024
[ 1327.887554] NUMA
[ 1327.887610] PowerNV
[ 1327.887688] Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core at24 opal_prd ofpart ipmi_powernv ipmi_devintf powernv_flash ipmi_msghandler i2c_opal mtd kvm_hv nfsd auth_rpcgss oid_registry nfs_acl lockd kvm grace
[ 1327.888930] sunrpc ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect tg3 sysimgblt fb_sys_fops ttm drm mlx5_core i2c_core ptp pps_core
[ 1327.889168] CPU: 14 PID: 492 Comm: systemd-journal Not tainted 4.13.0-4.dev.git49564cb.el7.centos.ppc64le #1
[ 1327.889334] task: c000003fe7a14a00 task.stack: c000003fdf660000
[ 1327.889432] NIP: c000000000ad6c98 LR: c0000000003c63c8 CTR: c0000000003c4430
[ 1327.889551] REGS: c000003fdf663ac0 TRAP: 0300 Not tainted (4.13.0-4.dev.git49564cb.el7.centos.ppc64le)
[ 1327.889704] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>
[ 1327.889718] CR: 28002828 XER: 00000000
[ 1327.889874] CFAR: c0000000003c63c4 DAR: 0000000000000008 DSISR: 40000000 SOFTE: 1
GPR00: c0000000003c6318 c000003fdf663d40 c000000001397a00 c00020397b4e9e00
GPR04: c000000002330368 c00020397552ded0 c00020397552de00 c00020397d455f00
GPR08: 0000000000000000 0000000000000000 c00020397d455f00 c000000000af0f48
GPR12: c0000000003c4430 c00000000fd88c00 000000007552de00 0000000000000000
GPR16: 0000000000000000 0000000000000019 0000000000000000 c00020397b4e9e58
GPR20: c00020397b4e9e18 0000000000000000 0000000000000000 c000000002330308
GPR24: c000000002330300 00000000de045700 c00020397552de30 c000003fde045700
GPR28: c00020397b4e9e00 c00020397552de00 c000000002330368 c00020397552ded0
[ 1327.891071] NIP [c000000000ad6c98] rb_insert_color+0x18/0x1a0
[ 1327.891183] LR [c0000000003c63c8] SyS_epoll_ctl+0x828/0xc10
[ 1327.891273] Call Trace:
[ 1327.891315] [c000003fdf663d40] [c0000000003c6318] SyS_epoll_ctl+0x778/0xc10 (unreliable)
[ 1327.891454] [c000003fdf663e30] [c00000000000b8e0] system_call+0x58/0x6c
[ 1327.891565] Instruction dump:
[ 1327.891623] e8410018 7fc9f378 7fbeeb78 4bffff9c 60000000 60420000 e9430000 2faa0000
[ 1327.891756] 419e0154 e92a0000 792807e1 4c820020 61270001 7d254b78 7faa4040
[ 1327.891897] ---[ end trace 6efc2b1ce3f4f85b ]---

Aafter this adapter is not recovered.. and we do not see the adapter in host 'lspci' until we reboot the host.

Steps to Reproduce:

Edit the guest in shutdown mode and add CX5 as hostdev device :

Start the guest, it fails to boot and Call Trace is seen on host where we hit EEH

Here, we are using in-box driver as MOFED is not present yet in HostOS.

Attaching lspci -vvv before starting guest:
lspci_vvv.txt

cde:info Mirrored with LTC bug #158978 </cde:info>

Memory hotplug/hotunplug: Memory hotunplug fails with Kernel alert messages while stress running inside the guest.

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=160904 </cde:info>

---Issue---
Seeing below Kernel messages while trying to hotunplug memory with stress running inside the guest:

[root@localhost ~]# dmesg -T --level=alert,crit,err,warn
[Thu Nov 2 03:28:34 2017] crashkernel: memory value expected
[Thu Nov 2 03:28:34 2017] Failed to allocate transformation for 'xts(aes)': -2
[Thu Nov 2 03:28:34 2017] alg: skcipher: Failed to load transform for p8_aes_xts: -2
[Thu Nov 2 03:28:34 2017] Warning: unable to open an initial console.
[Thu Nov 2 03:28:34 2017] This architecture does not have kernel memory protection.
[Thu Nov 2 03:28:34 2017] synth uevent: /devices/vio: failed to send uevent
[Thu Nov 2 03:28:34 2017] vio vio: uevent: failed to send synthetic uevent
[Thu Nov 2 03:28:36 2017] systemd: 25 output lines suppressed due to ratelimiting
[Thu Nov 2 03:28:37 2017] synth uevent: /devices/vio: failed to send uevent
[Thu Nov 2 03:28:37 2017] vio vio: uevent: failed to send synthetic uevent
[Thu Nov 2 03:31:38 2017] failed to isolate pfn 2400c
[Thu Nov 2 03:31:38 2017] raw: 01bffff000000000 0000000000000000 0000000000000000 00000011ffffffff
[Thu Nov 2 03:31:38 2017] raw: 5deadbeef0000100 5deadbeef0000200 0000000000000000 0000000000000000
[Thu Nov 2 03:31:38 2017] page dumped because: isolation failed
[Thu Nov 2 03:31:38 2017] failed to isolate pfn 2400c
[Thu Nov 2 03:31:38 2017] raw: 01bffff000000000 0000000000000000 0000000000000000 00000011ffffffff
[Thu Nov 2 03:31:38 2017] raw: 5deadbeef0000100 5deadbeef0000200 0000000000000000 0000000000000000
[Thu Nov 2 03:31:38 2017] page dumped because: isolation failed
[Thu Nov 2 03:31:38 2017] failed to isolate pfn 2400c
[Thu Nov 2 03:31:38 2017] raw: 01bffff000000000 0000000000000000 0000000000000000 00000011ffffffff
[Thu Nov 2 03:31:38 2017] raw: 5deadbeef0000100 5deadbeef0000200 0000000000000000 0000000000000000
[Thu Nov 2 03:31:38 2017] page dumped because: isolation failed
[Thu Nov 2 03:31:38 2017] failed to isolate pfn 2400c
[Thu Nov 2 03:31:38 2017] raw: 01bffff000000000 0000000000000000 0000000000000000 00000011ffffffff
[Thu Nov 2 03:31:38 2017] raw: 5deadbeef0000100 5deadbeef0000200 0000000000000000 0000000000000000
[Thu Nov 2 03:31:38 2017] page dumped because: isolation failed
[Thu Nov 2 03:31:38 2017] failed to isolate pfn 2400c
[Thu Nov 2 03:31:38 2017] raw: 01bffff000000000 0000000000000000 0000000000000000 00000011ffffffff
[Thu Nov 2 03:31:38 2017] raw: 5deadbeef0000100 5deadbeef0000200 0000000000000000 0000000000000000
[Thu Nov 2 03:31:38 2017] page dumped because: isolation failed
[Thu Nov 2 03:31:38 2017] pseries-hotplug-mem: Memory indexed-count-remove failed, adding any removed LMBs

---Steps to recreate---

Boot into the guest.
Start stress inside the guest as :
"stress --cpu 10 --io 10 --vm 10 --vm-bytes 256M --vm-stride 4096 --vm-hang 10 --timeout 500s"
Hotplug memory to the guest
Try Hotunplug memory from the guest - I see following logs in VM's ssh session :

Message from syslogd@localhost at Nov  2 03:31:39 ...
 kernel:page:c00a000000900300 count:17 mapcount:0 mapping:          (null) index:0x0

Message from syslogd@localhost at Nov  2 03:31:39 ...
 kernel:flags: 0x1bffff000000000()

Message from syslogd@localhost at Nov  2 03:31:39 ...
 kernel:page:c00a000000900300 count:17 mapcount:0 mapping:          (null) index:0x0

Message from syslogd@localhost at Nov  2 03:31:39 ...
 kernel:flags: 0x1bffff000000000()

Message from syslogd@localhost at Nov  2 03:31:39 ...
 kernel:page:c00a000000900300 count:17 mapcount:0 mapping:          (null) index:0x0

Message from syslogd@localhost at Nov  2 03:31:39 ...
 kernel:flags: 0x1bffff000000000()

Message from syslogd@localhost at Nov  2 03:31:39 ...
 kernel:page:c00a000000900300 count:17 mapcount:0 mapping:          (null) index:0x0

Message from syslogd@localhost at Nov  2 03:31:39 ...
 kernel:flags: 0x1bffff000000000()

Message from syslogd@localhost at Nov  2 03:31:39 ...
 kernel:page:c00a000000900300 count:17 mapcount:0 mapping:          (null) index:0x0

Message from syslogd@localhost at Nov  2 03:31:39 ...
 kernel:flags: 0x1bffff000000000()

[Regression] With 4.15.0-2.rc9.dev kernel memory hotplug is not working

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=164142 </cde:info>

Guest and Host are at the same level of kernel i.e. 4.15.0-2.rc9.dev.git03552b2.el7.centos.ppc64le
With 4.14.0-3.git68b4afb.el7.centos.ppc64le kernel able to do memory hotplug.

Steps to re-produce:

Start a guest with the following in the guest xml
<maxMemory slots='16' unit='KiB'>2621440</maxMemory>
Check the memory at guest before hotplug

# cat /proc/meminfo | grep -i memtotal
MemTotal:        1014464 kB

Hot plug memory from Host
hotplug xml:

# cat mem_hp_512m.xml
<memory model='dimm'>
<target>
<size unit='KiB'>524288</size>
</target>
</memory>

# virsh attach-device nrs mem_hp_512m.xml --live
Device attached successfully

Check the guest memory

# cat /proc/meminfo | grep -i memtotal
MemTotal:        1014464 kB

from dmesg:
[   65.938167] pseries-hotplug-mem: Attempting to hot-add 2 LMB(s) at index 80000004
[   65.975206] radix-mmu: Mapped 0xc000000040000000-0xc000000050000000 with 2.00 MiB pages
[   65.977148] radix-mmu: Mapped 0xc000000050000000-0xc000000060000000 with 2.00 MiB pages
[   65.978832] pseries-hotplug-mem: Memory at 40000000 (drc index 80000004) was hot-added
[   65.979941] pseries-hotplug-mem: Memory at 50000000 (drc index 80000005) was hot-added

From old kernel on guest: i.e.
4.14.0-3.git68b4afb.el7.centos.ppc64le
Before hotplug

# cat /proc/meminfo | grep -i memtotal
MemTotal:        1539456 kB

After hotplug

# cat /proc/meminfo | grep -i memtotal
MemTotal:        2063744 kB

Required services are running:

[root@localhost ~]# systemctl status rtas_errd.service
● rtas_errd.service - ppc64-diag rtas_errd (platform error handling) Service
   Loaded: loaded (/usr/lib/systemd/system/rtas_errd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2018-01-31 08:34:29 EST; 2min 21s ago
  Process: 808 ExecStart=/usr/sbin/rtas_errd (code=exited, status=0/SUCCESS)
 Main PID: 817 (rtas_errd)
   CGroup: /system.slice/rtas_errd.service
           └─817 /usr/sbin/rtas_errd

Jan 31 08:34:29 localhost.localdomain systemd[1]: Starting ppc64-diag rtas_er...
Jan 31 08:34:29 localhost.localdomain systemd[1]: Started ppc64-diag rtas_err...
Hint: Some lines were ellipsized, use -l to show in full.

Libvirt and QEMU rpm levels:

# rpm -qa | grep "qemu\|libvirt"
libvirt-daemon-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-config-network-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-storage-logical-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-secret-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-libs-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-storage-core-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-nwfilter-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-qemu-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-lxc-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-storage-disk-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-storage-iscsi-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-storage-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-nodedev-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-client-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-devel-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
qemu-img-2.11.0-1.dev.gite7153e0.el7.centos.ppc64le
qemu-common-2.11.0-1.dev.gite7153e0.el7.centos.ppc64le
qemu-system-ppc-2.11.0-1.dev.gite7153e0.el7.centos.ppc64le
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
qemu-2.11.0-1.dev.gite7153e0.el7.centos.ppc64le
libvirt-python-3.2.0-3.el7_4.1.ppc64le
libvirt-daemon-driver-network-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-config-nwfilter-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-storage-mpath-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-storage-scsi-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
libvirt-daemon-driver-interface-4.0.0-2.dev.git5e6f8a1.el7.centos.ppc64le
qemu-system-x86-2.11.0-1.dev.gite7153e0.el7.centos.ppc64le

Using the devel version of the packages.

Power8: Hit with "Oops: Kernel access of bad area, sig: 11" on latest nightly

Kernel Version: 4.13.0-3.rc3.dev.gitec0d270.el7.centos.ppc64le
Hit few mins after a fresh boot, tried to run avocado tests(just started).
Most of(sosreport, service restart, etc) command gets stuck after the crash.

[  909.585268] list_del corruption. prev->next should be c000000f23120760, but was c000000f23121760
[  909.585448] ------------[ cut here ]------------
[  909.585547] WARNING: CPU: 64 PID: 14123 at lib/list_debug.c:53 __list_del_entry_valid+0xd0/0x100
[  909.585705] Modules linked in: vhost_net vhost tap act_police cls_u32 sch_ingress cls_fw sch_sfq sch_htb xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas i2c_opal i2c_core powernv_op_panel ipmi_powernv ipmi_devintf ipmi_msghandler nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc kvm_hv kvm_pr kvm xfs libcrc32c tg3 ptp pps_core
[  909.586812] CPU: 64 PID: 14123 Comm: qemu-system-ppc Not tainted 4.13.0-3.rc3.dev.gitec0d270.el7.centos.ppc64le #1
[  909.586963] task: c000000f0c9cc600 task.stack: c000000f061a8000
[  909.587026] NIP: c0000000005a0770 LR: c0000000005a076c CTR: 00000000300304d0
[  909.587100] REGS: c000000f061ab6c0 TRAP: 0700   Not tainted  (4.13.0-3.rc3.dev.gitec0d270.el7.centos.ppc64le)
[  909.587197] MSR: 9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>
[  909.587205]   CR: 42024422  XER: 20000000
[  909.587291] CFAR: c00000000016e9c8 SOFTE: 1 
[  909.587291] GPR00: c0000000005a076c c000000f061ab940 c000000001397a00 0000000000000054 
[  909.587291] GPR04: 0000000000000000 c000000000098244 9000000000009033 0000000000000000 
[  909.587291] GPR08: 0000000000000001 0000000000000007 0000000000000006 9000000000001003 
[  909.587291] GPR12: 0000000000004400 c00000000fda8000 0000000000000000 0000000000000000 
[  909.587291] GPR16: 0000000000000000 0000000124cb8058 0000000124cb8038 00000001250ed8b8 
[  909.587291] GPR20: 00000001250ed8b0 00000001250ed8d0 c00000000138d820 c000000000d9c238 
[  909.587291] GPR24: 0000000000000001 5deadbeef0000100 c000000f061abb80 c000000000f24840 
[  909.587291] GPR28: c0000000013cbe50 0000000000000001 c000000f231215e0 c000000f23120750 
[  909.587927] NIP [c0000000005a0770] __list_del_entry_valid+0xd0/0x100
[  909.587990] LR [c0000000005a076c] __list_del_entry_valid+0xcc/0x100
[  909.588052] Call Trace:
[  909.588079] [c000000f061ab940] [c0000000005a076c] __list_del_entry_valid+0xcc/0x100 (unreliable)
[  909.588167] [c000000f061ab9a0] [c000000000988bbc] tcf_chain_destroy+0x2c/0xa0
[  909.588243] [c000000f061ab9d0] [c000000000988c84] tcf_block_put+0x54/0x90
[  909.588308] [c000000f061aba00] [d000000014d3178c] htb_destroy_class.isra.11+0x5c/0x80 [sch_htb]
[  909.588401] [c000000f061aba30] [d000000014d318a8] htb_destroy+0xf8/0x1b0 [sch_htb]
[  909.588476] [c000000f061abab0] [c0000000009818a4] qdisc_destroy+0xe4/0x170
[  909.588539] [c000000f061abae0] [c00000000098332c] dev_shutdown+0xbc/0x100
[  909.588604] [c000000f061abb20] [c00000000093f248] rollback_registered_many+0x2f8/0x560
[  909.588679] [c000000f061abbf0] [c00000000093f520] rollback_registered+0x70/0xb0
[  909.588755] [c000000f061abc40] [c000000000941908] unregister_netdevice_queue+0x128/0x180
[  909.588832] [c000000f061abcc0] [c00000000077a6cc] __tun_detach+0x22c/0x460
[  909.588895] [c000000f061abd20] [c00000000077a938] tun_chr_close+0x38/0x60
[  909.588959] [c000000f061abd50] [c00000000035abf8] __fput+0xd8/0x280
[  909.589024] [c000000f061abdb0] [c000000000120f20] task_work_run+0x140/0x1a0
[  909.589089] [c000000f061abe00] [c00000000001d810] do_notify_resume+0xf0/0x100
[  909.589164] [c000000f061abe30] [c00000000000bf44] ret_from_except_lite+0x70/0x74
[  909.589238] Instruction dump:
[  909.589295] 4bffffd4 3c62ff9b 3863f6d0 4bbce235 60000000 0fe00000 38600000 4bffffb8 
[  909.589435] 3c62ff9b 3863f690 4bbce219 60000000 <0fe00000> 38600000 4bffff9c 3c62ff9b 
[  909.589577] ---[ end trace c2b424e83e247e4b ]---
[  909.589685] Unable to handle kernel paging request for data at address 0x00000000
[  909.589823] Faulting instruction address: 0xc000000000988b48
[  909.589939] Oops: Kernel access of bad area, sig: 11 [#1]
[  909.590030] SMP NR_CPUS=1024 
[  909.590030] NUMA 
[  909.590101] PowerNV
[  909.590197] Modules linked in: vhost_net vhost tap act_police cls_u32 sch_ingress cls_fw sch_sfq sch_htb xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas i2c_opal i2c_core powernv_op_panel ipmi_powernv ipmi_devintf ipmi_msghandler nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc kvm_hv kvm_pr kvm xfs libcrc32c tg3 ptp pps_core
[  909.591279] CPU: 64 PID: 14123 Comm: qemu-system-ppc Tainted: G        W       4.13.0-3.rc3.dev.gitec0d270.el7.centos.ppc64le #1
[  909.591481] task: c000000f0c9cc600 task.stack: c000000f061a8000
[  909.591596] NIP: c000000000988b48 LR: c000000000988c04 CTR: 00000000300304d0
[  909.591733] REGS: c000000f061ab6f0 TRAP: 0300   Tainted: G        W        (4.13.0-3.rc3.dev.gitec0d270.el7.centos.ppc64le)
[  909.591913] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>
[  909.591919]   CR: 42024422  XER: 20000000
[  909.592080] CFAR: c0000000000087d8 DAR: 0000000000000000 DSISR: 40000000 SOFTE: 1 
[  909.592080] GPR00: c000000000988c04 c000000f061ab970 c000000001397a00 c000000f23120750 
[  909.592080] GPR04: 0000000000000000 c000000000098244 9000000000009033 0000000000000000 
[  909.592080] GPR08: 0000000000000001 0000000000000000 5deadbeef0000100 9000000000001003 
[  909.592080] GPR12: 0000000000004400 c00000000fda8000 0000000000000000 0000000000000000 
[  909.592080] GPR16: 0000000000000000 0000000124cb8058 0000000124cb8038 00000001250ed8b8 
[  909.592080] GPR20: 00000001250ed8b0 00000001250ed8d0 c00000000138d820 c000000000d9c238 
[  909.592080] GPR24: 0000000000000001 5deadbeef0000100 c000000f061abb80 c000000000f24840 
[  909.592080] GPR28: c0000000013cbe50 0000000000000001 c000000f231215e0 c000000f23120750 
[  909.593263] NIP [c000000000988b48] tcf_chain_flush+0x28/0x70
[  909.593377] LR [c000000000988c04] tcf_chain_destroy+0x74/0xa0
[  909.593491] Call Trace:
[  909.593540] [c000000f061ab970] [0000000000000001] 0x1 (unreliable)
[  909.593654] [c000000f061ab9a0] [c000000000988c04] tcf_chain_destroy+0x74/0xa0
[  909.593783] [c000000f061ab9d0] [c000000000988c84] tcf_block_put+0x54/0x90
[  909.593847] [c000000f061aba00] [d000000014d3178c] htb_destroy_class.isra.11+0x5c/0x80 [sch_htb]
[  909.593935] [c000000f061aba30] [d000000014d318a8] htb_destroy+0xf8/0x1b0 [sch_htb]
[  909.594013] [c000000f061abab0] [c0000000009818a4] qdisc_destroy+0xe4/0x170
[  909.594076] [c000000f061abae0] [c00000000098332c] dev_shutdown+0xbc/0x100
[  909.594140] [c000000f061abb20] [c00000000093f248] rollback_registered_many+0x2f8/0x560
[  909.594217] [c000000f061abbf0] [c00000000093f520] rollback_registered+0x70/0xb0
[  909.594292] [c000000f061abc40] [c000000000941908] unregister_netdevice_queue+0x128/0x180
[  909.594369] [c000000f061abcc0] [c00000000077a6cc] __tun_detach+0x22c/0x460
[  909.594433] [c000000f061abd20] [c00000000077a938] tun_chr_close+0x38/0x60
[  909.594496] [c000000f061abd50] [c00000000035abf8] __fput+0xd8/0x280
[  909.594563] [c000000f061abdb0] [c000000000120f20] task_work_run+0x140/0x1a0
[  909.594628] [c000000f061abe00] [c00000000001d810] do_notify_resume+0xf0/0x100
[  909.594704] [c000000f061abe30] [c00000000000bf44] ret_from_except_lite+0x70/0x74
[  909.594778] Instruction dump:
[  909.594816] 7c0803a6 4e800020 3c4c00a1 3842eee0 7c0802a6 60000000 7c0802a6 fbe1fff8 
[  909.594895] f8010010 f821ffd1 7c7f1b78 e9230008 <e9490000> 2faa0000 419e001c 39400000 
[  909.594975] ---[ end trace c2b424e83e247e4c ]---
[  909.601138]

cde:info Mirrored with LTC bug #158177 </cde:info>

kdump service not starting with crashkernel=auto

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=164225 </cde:info>

crashkernel with auto:

dmesg | grep crash
[ 0.000000] crashkernel: memory value expected
[ 0.000000] Kernel command line: root=/dev/mapper/host_os_ltc--wspoon5-root ro crashkernel=auto console=tty0 rd.lvm.lv=host_os_ltc-wspoon5/root rd.lvm.lv=host_os_ltc-wspoon5/swap

crashkernel with explicit value:
[root@ltc-wspoon5 ~]# dmesg | grep -i crash
[ 0.000000] Reserving 256MB of memory at 128MB for crashkernel (System RAM: 524288MB)
[ 0.000000] Kernel command line: root=/dev/mapper/host_os_ltc--wspoon5-root ro crashkernel=256M console=tty0 rd.lvm.lv=host_os_ltc-wspoon5/root rd.lvm.lv=host_os_ltc-wspoon5/swap

Call Trace noticed in Fedora24 guest after suspend/resume.

The below Call Trace is noticed in Fedora24 guest after suspend/resume operations.
[ 302.603321] INFO: rcu_sched self-detected stall on CPU
[ 302.603433] 0-...: (91 GPs behind) idle=60d/1/0 softirq=2818/2818 fqs=0
[ 302.603524](t=15081 jiffies g=4573 c=4572 q=18)
[ 302.603899] rcu_sched kthread starved for 15081 jiffies! g4573 c4572 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
[ 302.604030] rcu_sched S 0000000000000000 0 7 2 0x00000800
[ 302.604179] Call Trace:
[ 302.604215] [c00000009eb1b8c0] [c000000001338d08] sysctl_sched_migration_cost+0x0/0x4 (unreliable)
[ 302.604375] [c00000009eb1ba90] [c000000000016274] __switch_to+0x2e4/0x410
[ 302.604464] [c00000009eb1baf0] [c0000000009ae828] __schedule+0x328/0x9d0
[ 302.604553] [c00000009eb1bb80] [c0000000009aef18] schedule+0x48/0xc0
[ 302.604640] [c00000009eb1bbb0] [c0000000009b313c] schedule_timeout+0x16c/0x340
[ 302.604743] [c00000009eb1bca0] [c000000000140eac] rcu_gp_kthread+0x8ec/0xc00
[ 302.604846] [c00000009eb1bd80] [c0000000000e5f90] kthread+0x110/0x130
[ 302.604993] [c00000009eb1be30] [c0000000000095b0] ret_from_kernel_thread+0x5c/0xac
[ 302.605145] Task dump for CPU 0:
[ 302.605191] swapper/0 R running task 0 0 0 0x00000004
[ 302.605356] Call Trace:
[ 302.605429] [c0000000013075a0] [c0000000000fb720] sched_show_task+0xe0/0x180
[ 302.605472] INFO: rcu_sched self-detected stall on CPU
[ 302.605475] 2-...: (1 ticks this GP) idle=909/1/0 softirq=3974/3974 fqs=0
[ 302.605475]
[ 302.605477](t=15081 jiffies g=4573 c=4572 q=18)
_GP_WAIT_FQS(3) ->state=0x1
[ 302.605479] rcu_sched S
[ 302.605480] 0000000000000000
[ 302.605481] 0 7 2 0x00000800
[ 302.605481] Call Trace:
+0x0/0x4
302.605484
[ 302.605487] INFO: rcu_sched self-detected stall on CPU
[ 302.605490] [c00000009eb1ba90] [c000000000016274] __switch_to+0x2e4/0x410
[ 302.605492] [c00000009eb1baf0] [c0000000009ae828] __schedule+0x328/0x9d0
[ 302.605495] [c00000009eb1bb80] [c0000000009aef18] schedule+0x48/0xc0
0
[ 302.605499] [c00000009eb1bca0] [c000000000140eac] rcu_gp_kthread+0x8ec/0xc00
[ 302.605502] [c00000009eb1bd80] [c0000000000e5f90] kthread+0x110/0x130
[ 302.605505] 7-...: (49 GPs behind) idle=e19/1/0 softirq=1338/1338 fqs=0
[ 302.605506]
/0xac
[ 302.605510](t=15081 jiffies g=4573 c=4572 q=18)
_GP_WAIT_FQS(3) ->state=0x1
[ 302.605514] rcu_sched S
[ 302.605514] 0000000000000000
[ 302.605516] 0 7 2 0x00000800

The guest has an multifunction NIC adapter hot-plugged, and after suspend/resume, the Call Trace is noticed.
Without the NIC Hot-plugged, the issue isn't noticed in the guest after suspend/resume.
Apart from the Call Trace being noticed, the guest resumes fine without any further errors being noticed.

Guest Details

uname -a

Linux localhost.localdomain 4.5.5-300.fc24.ppc64le #1 SMP Tue May 24 12:23:26 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

Steps to Reproduce

Hot-plug an multifunction adapter to Fedora24 guest using virsh:

virsh attach-device fedora24-san hotadd-nic.xml --live

Device attached successfully

Inside the guest, the device is listed as:

lspci -nn

00:01.0 Ethernet controller [0200]: Red Hat, Inc Virtio network device [1af4:1000]
00:02.0 USB controller [0c03]: Apple Inc. KeyLargo/Intrepid USB [106b:003f]
00:03.0 SCSI storage controller [0100]: Red Hat, Inc Virtio block device [1af4:1001]
00:04.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon [1af4:1002]
00:05.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme II BCM57800 1/10 Gigabit Ethernet [14e4:168a](rev 10)
00:05.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme II BCM57800 1/10 Gigabit Ethernet [14e4:168a](rev 10)
00:05.2 Ethernet controller [0200]: Broadcom Corporation NetXtreme II BCM57800 1/10 Gigabit Ethernet [14e4:168a](rev 10)
00:05.3 Ethernet controller [0200]: Broadcom Corporation NetXtreme II BCM57800 1/10 Gigabit Ethernet [14e4:168a](rev 10)
00:0f.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194](rev 03)

[00:05.0 till 00:05.3] are the hot-plugged device.

Once the device is hot-plugged to the guest, suspend the guest:

virsh suspend fedora24-san

Domain fedora24-san suspended
After resuming the guest, the Call Trace is noticed.

virsh resume fedora24-san

Domain fedora24-san resumed
Without the NIC Hot-plugged, the CallTrace is unreproducible with suspend/resume.
dmesg.txt

Latest devel build update +reboot crashed host

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=160569 </cde:info>

Action: yum update + reboot
https://ltc-jenkins.aus.stglabs.ibm.com/job/HostOS_CI/842/consoleText

         Stopping Replay Read-Ahead Data...
[  OK  ] Reached target Shutdown.
[119099.239708] Unable to handle kernel paging request for data at address 0x00000010
[119099.239794] Faulting instruction address: 0xd00000000730064c
cpu 0x0: Vector: 300 (Data Access) at [c0000007f86077d0]
    pc: d00000000730064c: bm_evict_inode+0x2c/0x80 [binfmt_misc]
    lr: c00000000039003c: evict+0xfc/0x260
    sp: c0000007f8607a50
   msr: 900000010280b033
   dar: 10
 dsisr: 40000000
  current = 0xc0000007f8580080
  paca    = 0xc00000000fd60000   softe: 0        irq_happened: 0x01
    pid   = 1, comm = systemd
Linux version 4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-17) (GCC)) #1 SMP Fri Oct 20 22:55:44 -02 2017
enter ? for help
[c0000007f8607a80] c00000000039003c evict+0xfc/0x260
[c0000007f8607ac0] c000000000389258 dentry_unlink_inode+0x148/0x1c0
[c0000007f8607af0] c00000000038ad58 __dentry_kill+0xe8/0x2a0
[c0000007f8607b30] c00000000038b634 shrink_dentry_list+0x1e4/0x4e0
[c0000007f8607ba0] c00000000038bb84 shrink_dcache_parent+0x54/0xb0
[c0000007f8607c00] c00000000038bc08 do_one_tree+0x28/0x60
[c0000007f8607c30] c00000000038ce4c shrink_dcache_for_umount+0x4c/0xc0
[c0000007f8607ca0] c00000000036a92c generic_shutdown_super+0x3c/0x190
[c0000007f8607d10] c00000000036af08 kill_litter_super+0x48/0x70
[c0000007f8607d40] c00000000036b45c deactivate_locked_super+0xac/0xf0
[c0000007f8607d70] c000000000397f94 cleanup_mnt+0x64/0xb0
[c0000007f8607da0] c0000000001287c0 task_work_run+0x140/0x1a0
[c0000007f8607e00] c00000000001ca70 do_notify_resume+0xf0/0x100
[c0000007f8607e30] c00000000000bec4 ret_from_except_lite+0x70/0x74
--- Exception: c00 (System Call) at 00007fff8c6a50a8
SP (7fffee70e770) is in userspace

Memory hotplug/hotunplug continuously hit with Call Trace in the VM

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=160740 </cde:info>

---ISSUE---
Hotplugging and Hotunplugging continuously giving continuous Call Trace inside the guest.

[ 271.020588] WARNING: CPU: 3 PID: 6 at arch/powerpc/mm/pgtable.c:194 set_pte_at+0x38/0x1a0
[ 271.021669] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables virtio_balloon virtio_net virtio_scsi
[ 271.026247] CPU: 3 PID: 6 Comm: kworker/u64:0 Tainted: G B W 4.13.0-4.rel.git49564cb.el7.centos.ppc64le #1
[ 271.027489] Workqueue: pseries hotplug workque pseries_hp_work_fn
[ 271.028204] task: c000000e37444200 task.stack: c000000e3748c000
[ 271.028901] NIP: c0000000000675d8 LR: c00000000007354c CTR: 0000000000000000
[ 271.029720] REGS: c000000e3748f4f0 TRAP: 0700 Tainted: G B W (4.13.0-4.rel.git49564cb.el7.centos.ppc64le)
[ 271.030990] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>
[ 271.030995] CR: 48002044 XER: 00000000
[ 271.032204] CFAR: c000000000073548 SOFTE: 1
GPR00: c00000000007354c c000000e3748f770 c000000001397a00 c000000001341c50
GPR04: 0000000000000001 c000001cc00010f8 8e01e0ff1c000080 8e01e0ff1c000080
GPR08: 000000008000001c 06000000000000c0 8e01e0ff1c0000c0 0000000000000003
GPR12: 0000000000002000 c00000000fd81e00 c000000000124348 c000000e3b220e40
GPR16: 0000000000000000 0000000000000010 c000001d3fffec28 0000000000000000
GPR20: c000001d3ffcf800 c0000000015465f0 c00000000153ad58 c000001d3ffff000
GPR24: c000001cc0000100 c000001cffe00000 800000000000018e 0000000000000ff8
GPR28: c00000000153ad68 0000000000200000 c000000001341c50 c000001cffe00000
[ 271.041048] NIP [c0000000000675d8] set_pte_at+0x38/0x1a0
[ 271.041733] LR [c00000000007354c] radix__map_kernel_page+0x27c/0x670
[ 271.042559] Call Trace:
[ 271.042875] [c000000e3748f770] [c000000e3748f7b0] 0xc000000e3748f7b0 (unreliable)
[ 271.043842] [c000000e3748f790] [c00000000007354c] radix__map_kernel_page+0x27c/0x670
[ 271.044845] [c000000e3748f800] [c000000000ae9aa4] create_physical_mapping+0x188/0x20c
[ 271.045861] [c000000e3748f8a0] [c000000000072334] create_section_mapping+0x24/0x60
[ 271.046843] [c000000e3748f8c0] [c000000000067108] arch_add_memory+0x78/0xf0
[ 271.047757] [c000000e3748f950] [c0000000003223cc] add_memory_resource+0x15c/0x2c0
[ 271.048734] [c000000e3748f9e0] [c0000000003225fc] add_memory+0xcc/0x1d0
[ 271.049598] [c000000e3748fa60] [c0000000000be7b8] dlpar_add_lmb+0x248/0x420
[ 271.050501] [c000000e3748fb40] [c0000000000bfcc0] dlpar_memory+0xc80/0xd80
[ 271.051394] [c000000e3748fbf0] [c0000000000b7638] handle_dlpar_errorlog+0xf8/0x160
[ 271.052373] [c000000e3748fc60] [c0000000000b7734] pseries_hp_work_fn+0x94/0xa0
[ 271.053314] [c000000e3748fc90] [c00000000011bc00] process_one_work+0x1a0/0x490
[ 271.054251] [c000000e3748fd30] [c00000000011bf88] worker_thread+0x98/0x520
[ 271.055140] [c000000e3748fdc0] [c0000000001244a8] kthread+0x168/0x1b0
[ 271.055976] [c000000e3748fe30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 271.056955] Instruction dump:
[ 271.057343] 7c0802a6 f8010010 f821ffe1 e9450000 7944cfe3 41820024 3d200700 792907c6
[ 271.058350] 612900c0 7d494838 2ba900c0 419e000c <0fe00000> 60420000 78c70022 54ca403e
[ 271.059379] ---[ end trace 6da919e9ea1c5e99 ]---

---Steps to reproduce---

Boot in to guest with 2 numa as:

    <numa>
      <cell id='0' cpus='0-15' memory='4587520' unit='KiB'/>
      <cell id='1' cpus='16-31' memory='4587520' unit='KiB'/>
    </numa>

Continuously hotplug memory to numa 0 for 4 times

    <memory model='dimm'>
    <target>
    <size unit='KiB'>12582912</size>
    <node>0</node>
    </target>
    </memory>

Continuously hotplug memory to numa 1 for 5 times

    <memory model='dimm'>
    <target>
    <size unit='KiB'>12582912</size>
    <node>1</node>
    </target>
    </memory>

Try to hotunplug, hotunplug may fail
Reboot guest.
Now try to hotunplug memory continuously from numa 1 for 5 times - hotunplug works fine.
When you try hotunplug from numa 1 for 6th time - it hits with continuous Call Trace inside the vm.

Upstreamed: Suraj's patchset to implement halt polling

This records the fact that a set of patches by Suraj Jitindar Singh to implement halt polling in HV KVM are in hostos 1.0 and in upstream v4.9-rc1, but not in upstream 4.8.

PCI passthrough: mpt3sas firmware fault inside guest

This is the PCI device I'm passing through to a guest VM:

0001:03:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)

Inside the guest, the mpt3sas driver fails to initialize due to firmware faults. Sometimes when it does initialize it ends up getting IO errors.

Host is running HostOS with kernel 4.9.0 and qemu 2.7.0. Guest OS is stock CentOS 7.3.

Logs and config attached.
guest-xml.txt
guest-dmesg.txt
host-dmesg.txt

Upstreamed: Bug fix for copy_tofrom_user

This records the fact that hostos-1.0 includes the following commit, which fixes a bug which led to occasional data corruption in network stress tests:

da2b047 ("powerpc/64: Fix incorrect return value from __copy_tofrom_user", 2016-10-11)

It is now upstream in v4.9-rc1 with commit ID 1a34439 and is marked for inclusion in the stable trees. It is already in v4.8.5.

Upstreamed: Two bug fixes in idle/wakeup code

This records the fact that the two commits below in the hostos-1.0 release were submitted upstream and accepted after v4.8 was released. They are:

490b36e ("powerpc/64: Re-fix race condition between going idle and entering guest", 2016-10-21)
945f419 ("powerpc/64: Fix race condition in setting lock bit in idle/wakeup code", 2016-10-21)

These commits fix a sporadic soft-lockup in ktime_get_ts64() which has been observed on several systems.

These patches are in v4.9-rc3 with commit IDs 56c4622 and 09b7e37 and are marked for the stable trees.

Hardlockup during a VM boot on zz with D2.0 with latest devel branch

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=162352 </cde:info>

Hitting hardlockup with latest hostos devel branch while it was booting VM(hostos guest and ran yum update test inside the guest, I guess test irrespective of test, guest boot caused it.)

kernel: 4.14.0-3.dev.git68b4afb.el7.centos.ppc64le

Machine used:
zz - DD2.0

# lscpu
Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                128
On-line CPU(s) list:   0-127
Thread(s) per core:    4
Core(s) per socket:    16
Socket(s):             2
NUMA node(s):          2
Model:                 2.0 (pvr 004e 1200)
Model name:            POWER9 (raw), altivec supported
CPU max MHz:           3800.0000
CPU min MHz:           2283.0000
Hypervisor vendor:     (null)
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              512K
L3 cache:              10240K
NUMA node0 CPU(s):     0-63
NUMA node8 CPU(s):     64-127

FW: fips910/b1117d_1748.910

JOB LOG : /home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/results/job-2017-12-08T08.18-fe6f991/job.log
(1/5) guest_import.qemu.qcow2.virtio_scsi.smp2.virtio_net.HostOS.ppc64le.powerkvm-qemu.unattended_install.import.import.default_install.aio_native: PASS (47.01 s)
(2/5) guest_update.yum.qemu.qcow2.virtio_scsi.smp2.virtio_net.HostOS.ppc64le.powerkvm-qemu.guest_test.isa_serial_operations: PASS (58.74 s)
...

[ 7475.715366] virbr0: topology change detected, propagating
[ 7553.518889] virbr0: port 2(vnet0) entered disabled state
[ 7553.525649] device vnet0 left promiscuous mode
[ 7553.525718] virbr0: port 2(vnet0) entered disabled state
[ 7569.408657] Watchdog CPU:32 detected Hard LOCKUP other CPUS:117

Message from syslogd@ltczzj2 at Dec  8 08:21:19 ...
 kernel:Watchdog CPU:32 detected Hard LOCKUP other CPUS:117
[ 7584.519034] Watchdog CPU:94 detected Hard LOCKUP other CPUS:44
[ 7584.519142] Watchdog CPU:44 Hard LOCKUP
[ 7584.519206] Modules linked in: vhost_net vhost tap binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ebtable_nat ebtable_broute ib_ipoib rdma_ucm ib_ucm ib_uverbs bridge ib_umad stp llc rdma_cm ib_cm iw_cm ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat mlx5_ib ib_core nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas ipmi_powernv ipmi_devintf powernv_op_panel ipmi_msghandler opal_prd nfsd auth_rpcgss oid_registry
[ 7584.519872]  nfs_acl lockd grace sunrpc kvm_hv kvm lpfc mlx5_core bnx2x scsi_transport_fc mdio libcrc32c ptp pps_core
[ 7584.519983] CPU: 44 PID: 35342 Comm: CPU 30/KVM Not tainted 4.14.0-3.dev.git68b4afb.el7.centos.ppc64le #1
[ 7584.520072] task: c000003f7da1ce00 task.stack: c000003f7dad4000
[ 7584.520135] NIP:  c0000000001b9464 LR: c0000000001b9408 CTR: c0000000000420e0
[ 7584.520209] REGS: c000003f7dad74c0 TRAP: 0e81   Not tainted  (4.14.0-3.dev.git68b4afb.el7.centos.ppc64le)
[ 7584.520296] MSR:  9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 44224484  XER: 20040000
[ 7584.520378] CFAR: c0000000001b9470 SOFTE: 1
[ 7584.520378] GPR00: c0000000001b9408 c000003f7dad7740 c000000001403900 0000000000000075
[ 7584.520378] GPR04: 0000000000000400 0000000000000040 0000000000000000 c000000001439d78
[ 7584.520378] GPR08: 0000000000000001 0000000000000003 c000203994b48da0 0000000000000010
[ 7584.520378] GPR12: c0000000000420e0 c00000000fd7ce00
[ 7584.520689] NIP [c0000000001b9464] smp_call_function_many+0x384/0x420
[ 7584.520753] LR [c0000000001b9408] smp_call_function_many+0x328/0x420
[ 7584.520816] Call Trace:
[ 7584.520844] [c000003f7dad7740] [c0000000001b9408] smp_call_function_many+0x328/0x420 (unreliable)
[ 7584.520934] [c000003f7dad77b0] [c0000000000711c8] serialize_against_pte_lookup+0x38/0x50
[ 7584.521011] [c000003f7dad77d0] [c000000000073360] radix__pmdp_huge_get_and_clear+0x60/0x80
[ 7584.521088] [c000003f7dad7800] [c00000000033ae2c] zap_huge_pmd+0x8c/0x510
[ 7584.521153] [c000003f7dad78a0] [c0000000002d47f0] unmap_page_range+0xe80/0x10a0
[ 7584.521229] [c000003f7dad79e0] [c0000000002d4f04] unmap_vmas+0x84/0xf0
[ 7584.521293] [c000003f7dad7a30] [c0000000002e3508] exit_mmap+0xe8/0x1f0
[ 7584.521376] [c000003f7dad7af0] [c0000000000fa9f8] mmput+0xb8/0x1f0
[ 7584.521494] [c000003f7dad7b20] [c000000000104cf8] do_exit+0x358/0xcd0
[ 7584.521609] [c000003f7dad7be0] [c000000000105744] do_group_exit+0x64/0x100
[ 7584.521725] [c000003f7dad7c20] [c0000000001155d0] get_signal+0x210/0x700
[ 7584.521841] [c000003f7dad7d10] [c00000000001c65c] do_signal+0x6c/0x2d0
[ 7584.521957] [c000003f7dad7e00] [c00000000001ca74] do_notify_resume+0xd4/0x100
[ 7584.522098] [c000003f7dad7e30] [c00000000000bec4] ret_from_except_lite+0x70/0x74
[ 7584.522238] Instruction dump:
[ 7584.522310] 7d495214 812a0018 792807e1 41820034 4800001c 60000000 60000000 60000000
[ 7584.522451] 60000000 60000000 60420000 7c210b78 <7c421378> 812a0018 792807e1 4082fff0

Message from syslogd@ltczzj2 at Dec  8 08:21:33 ...
 kernel:Watchdog CPU:94 detected Hard LOCKUP other CPUS:44

Message from syslogd@ltczzj2 at Dec  8 08:21:33 ...
 kernel:Watchdog CPU:44 Hard LOCKUP

Upstreaming: Alexey's patchset "powerpc/spapr/vfio: Put pages on VFIO container shutdown"

This is to record the fact that hostos-1.0 includes the patch set "powerpc/spapr/vfio: Put pages on VFIO container shutdown", which is not upstream yet. This patch set fixes a bug where memory used by a guest can remain pinned for a long time after the guest is terminated.

The commits are:

5fa3141 ("powerpc/iommu: Pass mm_struct to init/cleanup helpers", 2016-10-24)
e96f99c ("powerpc/iommu: Stop using @current in mm_iommu_xxx", 2016-10-24)
f27109c ("vfio/spapr: Reference mm in tce_container", 2016-10-24)
200b22d ("powerpc/mm/iommu, vfio/spapr: Put pages on VFIO container shutdown", 2016-10-24)

systemtap: not to use spin_unlock_wait anymore

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=160774 </cde:info>

Systemtap (3.1-5.dev.git39b62b4) seems to be broken in current hostos devel (4.14 rc4 kernel) for a reason that it refers to calling spin_unlock_wait() which is no more in kernel as in its commit d3a024a

  locking: Remove spin_unlock_wait() generic definitions 

  There is no agreed-upon definition of spin_unlock_wait()'s semantics,
  and it appears that all callers could do just as well with a lock/unlock
  pair.  This commit therefore removes spin_unlock_wait() and related
  definitions from core code.

Hence, executing systemtap ends up with below error:

~]# stap -v -e 'probe vfs.read {exit()}' 
Pass 1: parsed user script and 490 library scripts using 174720virt/57280res/7744shr/47296data kb, in 640usr/20sys/653real ms.
Pass 2: analyzed script: 1 probe, 1 function, 7 embeds, 0 globals using 325248virt/209920res/9792shr/197824data kb, in 3170usr/700sys/3885real ms.
Pass 3: translated to C into "/tmp/staptlhTut/stap_83fd83b4a24b54380d462a6bd5886728_2819_src.c" using 325248virt/210112res/9984shr/197824data kb, in 30usr/150sys/187real ms.
In file included from /usr/share/systemtap/runtime/stp_utrace.c:30:0,
                 from /usr/share/systemtap/runtime/linux/task_finder2.c:4,
                 from /usr/share/systemtap/runtime/linux/task_finder.c:17,
                 from /usr/share/systemtap/runtime/linux/runtime.h:222,
                 from /usr/share/systemtap/runtime/runtime.h:26,
                 from /tmp/staptlhTut/stap_83fd83b4a24b54380d462a6bd5886728_2819_src.c:25:
/usr/share/systemtap/runtime/stp_helper_lock.h: In function ‘stp_spin_unlock_wait’:
/usr/share/systemtap/runtime/stp_helper_lock.h:60:1: error: implicit declaration of function ‘spin_unlock_wait’ [-Werror=implicit-function-declaration]
 static inline void stp_spin_unlock_wait(spinlock_t *lock) { spin_unlock_wait(lock); }
 ^
cc1: all warnings being treated as errors
make[1]: *** [/tmp/staptlhTut/stap_83fd83b4a24b54380d462a6bd5886728_2819_src.o] Error 1
make: *** [_module_/tmp/staptlhTut] Error 2
WARNING: kbuild exited with status: 2
Pass 4: compiled C into "stap_83fd83b4a24b54380d462a6bd5886728_2819.ko" in 14550usr/1330sys/15774real ms.
Pass 4: compilation failed.  [man error::pass4]

Upstream systemtap fix to getrid of spin_unlock_wait:

https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=commit;f=runtime/stp_utrace.c;h=0643ca2b7fd8cb6407aa84f41d26a71d2f2f8e90

Applied fix locally With that,

~]# stap -v -e 'probe oneshot { println("hello world") }'
Pass 1: parsed user script and 490 library scripts using 174720virt/57280res/7744shr/47296data kb, in 630usr/20sys/652real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 175488virt/57280res/7744shr/48064data kb, in 10usr/0sys/10real ms.
Pass 3: using cached /root/.systemtap/cache/56/stap_56041bf7d0574051b4814be457798d87_1190.c
Pass 4: using cached /root/.systemtap/cache/56/stap_56041bf7d0574051b4814be457798d87_1190.ko
Pass 5: starting run.
ERROR: module version mismatch (#1 SMP Thu Oct 26 22:52:08 -02 2017 vs #1 SMP Tue Oct 24 00:46:42 UTC 2017), release 4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le
WARNING: /usr/bin/staprun exited with status: 1
Pass 5: run completed in 10usr/50sys/332real ms.
Pass 5: run failed.  [man error::pass5]

Tried wiping out ~/.systemtap/cache/ but did not help. So, matched /usr/src/kernels/.../include/generated/compile.h, for stap and it worked (just a workaround to avoid recompiling stap).

~]# stap -v -e 'probe oneshot { println("hello world") }'
Pass 1: parsed user script and 490 library scripts using 174720virt/57216res/7744shr/47296data kb, in 630usr/20sys/653real ms.
Pass 2: analyzed script: 1 probe, 1 function, 0 embeds, 0 globals using 175488virt/57216res/7744shr/48064data kb, in 0usr/0sys/10real ms.
Pass 3: translated to C into "/tmp/stapG4SOWi/stap_56041bf7d0574051b4814be457798d87_1190_src.c" using 175616virt/59648res/9280shr/48192data kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into "stap_56041bf7d0574051b4814be457798d87_1190.ko" in 2850usr/510sys/3250real ms.
Pass 5: starting run.
hello world
Pass 5: run completed in 20usr/70sys/809real ms.

Please include mentioned patch for systemtap to work.

cpu hotplug-unplug in parallel with suspend resume.. guest becomes unresponsive..

I tried cpu hotplug-unplug in parallel with suspend resume.. guest becomes unresponsive..
i. On host run
terminal 1:
for i in {1..100};do sleep 2;virsh suspend srikanth_1710_Cdrom;sleep 2;virsh resume srikanth_1710_Cdrom;done
terminal 2:
for i in {1..100};do sleep 5;virsh setvcpus srikanth_1710_Cdrom 4 --live;sleep 5;virsh setvcpus srikanth_1710_Cdrom 2 --live;done

ii. After/while both above commands complete, will start seeing below messages on guest console:

[ 1209.368538] INFO: task jbd2/dm-0-8:1088 blocked for more than 120 seconds.
[ 1209.368633] Not tainted 4.12.0-11-generic #12-Ubuntu
[ 1209.368694] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

And the guest goes unresponsive..

I could'nt run sosreport on guest since it was unresponsive..

I tested same scenario as mentioned in previous comment with latest hostos guest.. I am hitting similar stall issues there as well...
I am attaching the traces of hostos here..
Guest is hung.. so could not get sosreports...

So these stall issues present on Ubuntu 17.10 [kernel: 4.13.0-11] and HostOS guest [kernel: 4.13.0-4.rel ]

In an internal comment I will post machine access details and guest details..

cde:info Mirrored with LTC bug #159338 </cde:info>

Set GitHub default branch other than 'powerkvm-v3.1.1'

Another user experience point.

Today when an user visits the hostos linux repo on github [1] they see the 'powerkvm-3.1.1' branch by default.
They'd probably be better routed to the hostos-stable/devel branches.

Can the default branch be changed to one of those?
See [2] for the few steps required.

Thanks

[1] https://github.com/open-power-host-os/linux/
[2] https://help.github.com/articles/setting-the-default-branch/

Upstreamed: Fix for spurious guest NMI watchdog lockup reports

This is to record the fact that hostos-1.0 includes two commits that fix a bug that cause guests to give spurious NMI watchdog reports. The commits are:

2f1956b ("KVM: PPC: Book3S: Treat VTB as a per-subcore register, not per-thread", 2016-09-15)
3d4f21c ("KVM: PPC: Book3S HV: Take out virtual core piggybacking code", 2016-09-15)

These commits are upstream in v4.9-rc1 with commit IDs 88b02cf and b009031.

Upstreamed: Fix for host crash when not using in-kernel XICS emulation

This records the fact that hostos-1.0 contains two commits which fix a bug which can cause a host crash if userspace (usually QEMU) elects not to use in-kernel XICS (interrupt controller) emulation. The commits are:

f995d4f ("KVM: PPC: Book3S: Don't crash if irqfd used with no in-kernel XICS emulation", 2016-08-10)
2886de6 ("KVM: PPC: Implement kvm_arch_intc_initialized() for PPC", 2016-08-10)

These commits are in v4.9-rc1 with commit IDs e48ba1c and 34a75b0.

Need to have CONFIG_HAVE_RELIABLE_STACKTRACE for klp

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=169888 </cde:info>

Currently when we try to live patch kernel on hostos-devel we get an error saying:

Jul 23 11:54:04 kernel: livepatch: This architecture doesn't have support for the livepatch consistency model.

Looks like we need below two configs to be enabled for klp:

static inline bool klp_have_reliable_stack(void)
{
return IS_ENABLED(CONFIG_STACKTRACE) &&
IS_ENABLED(CONFIG_HAVE_RELIABLE_STACKTRACE);
}

On hostos we currently have one of them:

# cat /boot/config-4.17.0-1.dev.git5ce3eac.el7.ppc64le | grep CONFIG_STACKTRACE 
CONFIG_STACKTRACE_SUPPORT=y 
CONFIG_STACKTRACE=y 
# cat /boot/config-4.17.0-1.dev.git5ce3eac.el7.ppc64le | grep CONFIG_HAVE_RELIABLE_STACKTRACE 
#

We need hostos kernel to be built with CONFIG_HAVE_RELIABLE_STACKTRACE option as well to support klp [for testing]

Out-of-tree kernel modules fail to build on hostos

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=169899 </cde:info>

kernel-devel package does not seem to contain arch/powerpc/kernel/module.lds.

Manually copying that file to said location resolve issue. But we need to include module.lds in kernel-devel..

Similar issue was as well reported in https://bugzilla.redhat.com/show_bug.cgi?id=1574604 and https://bugzilla.redhat.com/show_bug.cgi?id=1531182..

soft lockups during VM poweron and poweroff

Seeing this multiple times per day while doing manual power on and off.

Message from syslogd@NTNX-S240876X6C18782-A at May  3 13:19:46 ...
 kernel:NMI watchdog: BUG: soft lockup - CPU#80 stuck for 23s! [libvirtd:7559]

Host is 2 sockets x 8 cores, 512GB memory. The VM being powered on is configured with 4 cores x 8 threads, with 8GB of memory, backed by hugetlbfs.

Host is running hostos-2.0 release from 2017-03-31.
Some googling turned up this patch: https://lkml.org/lkml/2017/2/28/378.

Can not allocate hugepages after some iterations of the test

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=163537 </cde:info>

Can't get hugepages allocated when the test case does the following,

For each iteration of the test case:
echo n > /proc/sys/vm/nr_hugepages
Start the guest backed with 1G hugepages
Memory hotplug with 1G
echo 0 > /proc/sys/vm/nr_hugepages

After running 5 iterations of the above can't allocate any more hugepages.

# free -h
total used free shared buff/cache available
Mem: 31G 24G 3.2G 37M 3.8G 3.3G
Swap: 15G 836M 15G

Kernel version: 4.14.0-3.git68b4afb.el7.centos.ppc64le

From /etc/os-release:
NAME="CentOS Linux"
VERSION="7 (AltArch)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (AltArch)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
SIG_FAMILY="AltArch ppc64le"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

# cat /proc/buddyinfo
Node 0, zone DMA 52 156 342 190 98 26 16 18 888
Node 8, zone DMA 11 88 73 42 16 2 44 25 407
[root@zzfp365-lp1 ~]# free -h
total used free shared buff/cache available
Mem: 31G 7.5G 21G 27M 3.1G 21G
Swap: 15G 0B 15G
# numactl -H
available: 2 nodes (0,8)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
node 0 size: 16202 MB
node 0 free: 14767 MB
node 8 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
node 8 size: 16238 MB
node 8 free: 6819 MB
node distances:
node 0 8
0: 10 40
8: 40 10

# collectl -sb --verbose -oT
waiting for 1 second sample...
# MEMORY FRAGMENTATION SUMMARY (64K pages)
#Time 1Pg 2Pgs 4Pgs 8Pgs 16Pgs 32Pgs 64Pgs 128Pgs 256Pgs 512Pgs 1024Pgs
18:19:45 174 149 127 156 101 93 73 43 1677 0 0
18:19:46 170 149 128 156 101 93 73 43 1677 0 0
18:19:47 165 148 128 156 101 93 73 43 1677 0 0
18:19:48 166 149 127 156 101 93 73 43 1677 0 0
18:19:49 166 149 127 156 101 93 73 43 1677 0 0
18:19:50 162 149 127 156 101 93 73 43 1677 0 0
18:19:51 159 149 127 156 101 93 73 43 1677 0 0
18:19:52 156 149 127 156 101 93 73 43 1677 0 0

hit with host crash while running host stress tests

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=160748 </cde:info>

While trying to reproduce with host stress, I hit with the below host crash during xfs stress tests

enter ? for help
[link register ] c0000000002543f0 irq_work_run+0x30/0x50
[c000000ffff53cc0] c000000ffff53cf0 (unreliable)
[c000000ffff53cf0] c0000000001b7ca0 flush_smp_call_function_queue+0xf0/0x200
[c000000ffff53d70] c0000000000477ec smp_ipi_demux_relaxed+0x9c/0x110
[c000000ffff53db0] c0000000000903d4 icp_native_ipi_action+0x64/0x80
[c000000ffff53dd0] c000000000179420 __handle_irq_event_percpu+0x90/0x2d0
[c000000ffff53e90] c000000000179698 handle_irq_event_percpu+0x38/0x90
[c000000ffff53ed0] c00000000017fcf4 handle_percpu_irq+0x84/0xd0
[c000000ffff53f00] c000000000177b7c generic_handle_irq+0x4c/0x80
[c000000ffff53f20] c0000000000165d4 __do_irq+0x94/0x200
[c000000ffff53f90] c000000000029fa4 call_do_irq+0x14/0x24
[c0000007f87f3a50] c0000000000167dc do_IRQ+0x9c/0x110
[c0000007f87f3aa0] c000000000008c58 hardware_interrupt_common+0x158/0x160
--- Exception: 501 (Hardware Interrupt) at c0000000008eb664 snooze_loop+0xa4/0x190
[c0000007f87f3d90] c0000007f87f3dc0 (unreliable)
[c0000007f87f3dd0] c0000000008e83a4 cpuidle_enter_state+0xc4/0x3d0
[c0000007f87f3e30] c00000000015f73c call_cpuidle+0x4c/0x80
[c0000007f87f3e50] c00000000015fbe0 do_idle+0x2b0/0x350
[c0000007f87f3ec0] c00000000015fe8c cpu_startup_entry+0x3c/0x50
[c0000007f87f3ef0] c000000000048a74 start_secondary+0x4e4/0x530
[c0000007f87f3f90] c00000000000b16c start_secondary_prolog+0x10/0x14
b:mon>

Upstreamed: 3 miscellaneous KVM bug fixes

This records the fact that hostos-1.0 includes the following three commits, which fix minor bugs in KVM:

4f053d0 ("KVM: PPC: Book3S: Remove duplicate setting of the B field in tlbie", 2016-09-16)
2365f6b ("KVM: PPC: Book3S PR: Support 64kB page size on POWER8E and POWER8NVL", 2016-09-21)
fa73c3b ("KVM: PPC: Book3s PR: Allow access to unprivileged MMCR2 register", 2016-09-21)

These commits are upstream in v4.9-rc1 with commit IDs 4f053d0, 2365f6b and fa73c3b.

Upstreaming - Yong Ji's patchset "vfio-pci: Allow to mmap sub-page MMIO BARs and MSI-X table"

This records the fact that a set of PCI pass-through optimizations by Yong Ji Xie are in hostos 1.0 but not upstream yet.

hard lockup detected when vm starts with ftrace on

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=160778 </cde:info>

When I try to start a vm on host (4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le) with ftrace on (using function tracer), I see watchdog kickoff for cpu hard lockup as in below trace.

[ 3230.222246] Watchdog CPU:96 detected Hard LOCKUP other CPUS:24
[ 3276.051870] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 3276.051979] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=2985 
[ 3276.052082] 	(detected by 56, t=6002 jiffies, g=17311, c=17310, q=5714)
[ 3276.052209] Sending NMI from CPU 56 to CPUs 24:
[ 3286.061806] NETDEV WATCHDOG: net0 (tg3): transmit queue 0 timed out
[ 3286.061936] ------------[ cut here ]------------
[ 3286.062008] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x34c/0x360
[ 3286.062103] Modules linked in: vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas i2c_opal ipmi_powernv i2c_core ipmi_devintf ipmi_msghandler powernv_op_panel nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc kvm_hv kvm_pr kvm scsi_dh_alua dm_service_time dm_multipath tg3 ptp pps_core
[ 3286.062979] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le #1
[ 3286.063089] task: c00000000136e000 task.stack: c0000000013fc000
[ 3286.063166] NIP:  c0000000009a8fac LR: c0000000009a8fa8 CTR: c0000000001b1800
[ 3286.063258] REGS: c0000000013ff4b0 TRAP: 0700   Not tainted  (4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le)
[ 3286.063380] MSR:  9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 22002822  XER: 20000000
[ 3286.063502] CFAR: c000000000176f28 SOFTE: 1 
GPR00: c0000000009a8fa8 c0000000013ff730 c000000001403900 0000000000000037 
GPR04: c0000007ff50abd0 c0000007ff521410 9000000000009033 0000000000000020 
GPR08: 0000000000000000 c000000000f6126c 00000007fe5b0000 9000000000001003 
GPR12: 0000000000002200 c00000000fd60000 0000000000000000 0000000000000000 
GPR16: 0000000000200102 0000000100048e71 c0000000013fc000 0000000000000000 
GPR20: c000000000f74f00 c000000001433b00 c000000000f74f00 000000000000000a 
GPR24: 0000000000000000 ffffffffffffffff 0000000000000000 0000000000000000 
GPR28: 0000000000000004 c000000001433b00 c000000ff108e000 0000000000000000 
[ 3286.064360] NIP [c0000000009a8fac] dev_watchdog+0x34c/0x360
[ 3286.064427] LR [c0000000009a8fa8] dev_watchdog+0x348/0x360
[ 3286.064491] Call Trace:
[ 3286.064528] [c0000000013ff730] [c0000000009a8fa8] dev_watchdog+0x348/0x360 (unreliable)
[ 3286.064631] [c0000000013ff7d0] [c000000000198c30] call_timer_fn+0x60/0x1d0
[ 3286.064715] [c0000000013ff860] [c000000000198f20] expire_timers+0x140/0x1e0
[ 3286.064799] [c0000000013ff8d0] [c000000000199098] run_timer_softirq+0xd8/0x240
[ 3286.064897] [c0000000013ff960] [c000000000b0f6fc] __do_softirq+0x15c/0x3a4
[ 3286.064981] [c0000000013ffa50] [c0000000001064e8] irq_exit+0x118/0x130
[ 3286.065104] [c0000000013ffa70] [c00000000002469c] timer_interrupt+0xac/0xe0
[ 3286.065241] [c0000000013ffaa0] [c000000000009208] decrementer_common+0x158/0x160
[ 3286.065402] --- interrupt: 901 at replay_interrupt_return+0x0/0x4
    LR = arch_local_irq_restore+0x74/0x90
[ 3286.065613] [c0000000013ffd90] [c00000000143c0cc] cpu_idle_force_poll+0x0/0x4 (unreliable)
[ 3286.065779] [c0000000013ffdb0] [c0000000008e83f0] cpuidle_enter_state+0x110/0x3d0
[ 3286.065937] [c0000000013ffe10] [c00000000015f73c] call_cpuidle+0x4c/0x80
[ 3286.066074] [c0000000013ffe30] [c00000000015fbe0] do_idle+0x2b0/0x350
[ 3286.066207] [c0000000013ffea0] [c00000000015fe88] cpu_startup_entry+0x38/0x50
[ 3286.066367] [c0000000013ffed0] [c00000000000d918] rest_init+0xe8/0x100
[ 3286.066502] [c0000000013fff00] [c000000000e542d0] start_kernel+0x54c/0x568
[ 3286.066638] [c0000000013fff90] [c00000000000b27c] start_here_common+0x1c/0x520
[ 3286.066795] Instruction dump:
[ 3286.066880] 3d02fff3 7fc3f378 99282246 4bfc4ce1 60000000 7fc4f378 7fe6fb78 7c651b78 
[ 3286.067056] 3c62ff9e 3863d898 4b7cdf3d 60000000 <0fe00000> 4bffff84 60000000 60000000 
[ 3286.067233] ---[ end trace 7d5fcf569e0e59ec ]---
[ 3286.067344] tg3 0005:05:00.0 net0: transmit timed out, resetting
[ 3287.176015] rcu_sched kthread starved for 1113 jiffies! g17311 c17310 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=56
[ 3287.176185] rcu_sched       I    0     9      2 0x00000800
[ 3287.176254] Call Trace:
[ 3287.176294] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 3287.176396] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 3287.176480] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 3287.176564] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 3287.176647] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 3287.176746] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 3287.176844] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 3287.176929] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 3287.177052] Watchdog CPU:56 Hard LOCKUP
[ 3287.177054] Modules linked in: vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas i2c_opal ipmi_powernv i2c_core ipmi_devintf ipmi_msghandler powernv_op_panel nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc kvm_hv kvm_pr kvm scsi_dh_alua dm_service_time dm_multipath tg3 ptp pps_core
[ 3287.177133] CPU: 56 PID: 0 Comm: swapper/56 Tainted: G        W       4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le #1
[ 3287.177135] task: c000000ffb967e00 task.stack: c000000ffba40000
[ 3287.177137] NIP:  c0000000000165c8 LR: c0000000000165a4 CTR: c000000000182050
[ 3287.177139] REGS: c00000003fd5fd80 TRAP: 0900   Tainted: G        W        (4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le)
[ 3287.177141] MSR:  9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28002042  XER: 20000000
[ 3287.177153] CFAR: c000000000090558 SOFTE: 0 
GPR00: c0000000000165a4 c000001fffdcbf20 c000000001403900 0000000000000010 
GPR04: c000001fffdcbe60 0000000000000002 0000000000000000 00000000000001c0 
GPR08: 0000000000000000 b000000000009033 0000000000000000 0000000000000000 
GPR12: c0000000000903f0 c00000000fd84c00 c000000ffba43f90 0000000000000000 
GPR16: 0000000000200042 0000000100048ee0 c000000ffba40000 0000000000000000 
GPR20: c000000000f74f00 c000000001433b00 c000000000f74f00 000000000000000a 
GPR24: c000000ffba40000 c000000ffba40000 c000000001432200 c000000ffba43b30 
GPR28: 0000000000000000 c000000000f74ee0 c000000ffba436e0 c000001fffdc8000 
[ 3287.177200] NIP [c0000000000165c8] __do_irq+0x88/0x200
[ 3287.177203] LR [c0000000000165a4] __do_irq+0x64/0x200
[ 3287.177205] Call Trace:
[ 3287.177208] [c000001fffdcbf20] [c0000000000165a4] __do_irq+0x64/0x200 (unreliable)
[ 3287.177213] [c000001fffdcbf90] [c000000000029fa4] call_do_irq+0x14/0x24
[ 3287.177217] [c000000ffba43620] [c0000000000167dc] do_IRQ+0x9c/0x110
[ 3287.177222] [c000000ffba43670] [c000000000008c58] hardware_interrupt_common+0x158/0x160
[ 3287.177228] --- interrupt: 501 at arch_local_irq_restore+0x5c/0x90
    LR = arch_local_irq_restore+0x40/0x90
[ 3287.177232] [c000000ffba43960] [c000000000141904] vtime_account_irq_enter+0x64/0x80 (unreliable)
[ 3287.177238] [c000000ffba43980] [c000000000b0f690] __do_softirq+0xf0/0x3a4
[ 3287.177242] [c000000ffba43a70] [c0000000001064e8] irq_exit+0x118/0x130
[ 3287.177247] [c000000ffba43a90] [c00000000002469c] timer_interrupt+0xac/0xe0
[ 3287.177251] [c000000ffba43ac0] [c000000000009208] decrementer_common+0x158/0x160
[ 3287.177258] --- interrupt: 901 at replay_interrupt_return+0x0/0x4
    LR = arch_local_irq_restore+0x74/0x90
[ 3287.177261] [c000000ffba43db0] [c00000000143c0cc] cpu_idle_force_poll+0x0/0x4 (unreliable)
[ 3287.177268] [c000000ffba43dd0] [c0000000008e83f0] cpuidle_enter_state+0x110/0x3d0
[ 3287.177272] [c000000ffba43e30] [c00000000015f73c] call_cpuidle+0x4c/0x80
[ 3287.177276] [c000000ffba43e50] [c00000000015fbe0] do_idle+0x2b0/0x350
[ 3287.177281] [c000000ffba43ec0] [c00000000015fe88] cpu_startup_entry+0x38/0x50
[ 3287.177285] [c000000ffba43ef0] [c000000000048a74] start_secondary+0x4e4/0x530
[ 3287.177290] [c000000ffba43f90] [c00000000000b16c] start_secondary_prolog+0x10/0x14
[ 3287.177292] Instruction dump:
[ 3287.177295] 7d2903a6 7d2c4b78 4e800421 e8410018 894d028b 7948f7e3 554a003c 994d028b 
[ 3287.177306] 40820010 e92d0020 61298000 7d210164 <2fa30000> 41de012c 48161569 60000000 
[ 3287.181946] Watchdog CPU:56 became unstuck
[ 3287.327533] tg3 0005:05:00.0 net0: 0x00000000: 0x165714e4, 0x00100546, 0x02000001, 0x00800000
[ 3287.327647] tg3 0005:05:00.0 net0: 0x00000010: 0x0000000c, 0x00002501, 0x0001000c, 0x00002501
[ 3287.327756] tg3 0005:05:00.0 net0: 0x00000020: 0x0002000c, 0x00002501, 0x00000000, 0x04201014
[ 3287.327865] tg3 0005:05:00.0 net0: 0x00000030: 0x00000000, 0x00000048, 0x00000000, 0x00000100
[ 3287.327974] tg3 0005:05:00.0 net0: 0x00000040: 0x00000000, 0x03000000, 0xc8035001, 0x64002008
[ 3287.328083] tg3 0005:05:00.0 net0: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000
[ 3287.328191] tg3 0005:05:00.0 net0: 0x00000060: 0x00000000, 0x00000000, 0xf0010298, 0x00380080
[ 3287.328300] tg3 0005:05:00.0 net0: 0x00000070: 0x000710b0, 0xf4f6fffe, 0x00000000, 0x00000000
[ 3287.328409] tg3 0005:05:00.0 net0: 0x00000080: 0x165714e4, 0x0000001e, 0x00000000, 0x00000628
[ 3287.328517] tg3 0005:05:00.0 net0: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x00000361
[ 3287.328626] tg3 0005:05:00.0 net0: 0x000000a0: 0x8010ac11, 0x00000004, 0x00000124, 0x00020010
[ 3287.328734] tg3 0005:05:00.0 net0: 0x000000b0: 0x10648d81, 0x0010242e, 0x00055c41, 0x10410000
[ 3287.328843] tg3 0005:05:00.0 net0: 0x000000d0: 0x0000001f, 0x00000010, 0x00000000, 0x00010001
[ 3287.328952] tg3 0005:05:00.0 net0: 0x000000f0: 0x00000000, 0x05719001, 0x00000000, 0xffffffff
[ 3287.329060] tg3 0005:05:00.0 net0: 0x00000100: 0x13c10001, 0x00000000, 0x00000000, 0x00062030
[ 3287.329168] tg3 0005:05:00.0 net0: 0x00000110: 0x00000000, 0x00002000, 0x000001e0, 0x00000000
[ 3287.329277] tg3 0005:05:00.0 net0: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
[ 3287.329385] tg3 0005:05:00.0 net0: 0x00000140: 0x94028f64, 0x000098be, 0x00000000, 0x00000000
[ 3287.329494] tg3 0005:05:00.0 net0: 0x00000150: 0x16010004, 0x00000000, 0x0007812c, 0x00000000
[ 3287.329601] tg3 0005:05:00.0 net0: 0x00000160: 0x23010002, 0x00000000, 0x00000000, 0x00000000
[ 3287.329710] tg3 0005:05:00.0 net0: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
[ 3287.329818] tg3 0005:05:00.0 net0: 0x00000200: 0x00000000, 0x03000000, 0x00000000, 0xf8000000
[ 3287.329926] tg3 0005:05:00.0 net0: 0x00000210: 0x00000000, 0x7c000000, 0x00000000, 0x14000000
[ 3287.330035] tg3 0005:05:00.0 net0: 0x00000220: 0x00000000, 0xd3000000, 0x00000000, 0x00000000
[ 3287.330144] tg3 0005:05:00.0 net0: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x00000361
[ 3287.330252] tg3 0005:05:00.0 net0: 0x00000280: 0x00000000, 0x00000628, 0x00000000, 0x0000007c
[ 3287.330361] tg3 0005:05:00.0 net0: 0x00000290: 0x00000000, 0x00000321, 0x00000000, 0x000000d4
[ 3287.330469] tg3 0005:05:00.0 net0: 0x00000300: 0x00000000, 0x0000005c, 0x00000000, 0x00000000
[ 3287.330578] tg3 0005:05:00.0 net0: 0x00000400: 0x18e04804, 0x00400000, 0x00001000, 0x00000900
[ 3287.330687] tg3 0005:05:00.0 net0: 0x00000410: 0x000098be, 0x94028f64, 0x000098be, 0x94028f64
[ 3287.330795] tg3 0005:05:00.0 net0: 0x00000420: 0x000098be, 0x94028f64, 0x000098be, 0x94028f64
[ 3287.330904] tg3 0005:05:00.0 net0: 0x00000430: 0x00000400, 0x00000000, 0x000000e7, 0x000005f2
[ 3287.331012] tg3 0005:05:00.0 net0: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x04384400
[ 3287.331121] tg3 0005:05:00.0 net0: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000112
[ 3287.331229] tg3 0005:05:00.0 net0: 0x00000460: 0x00000008, 0x00002620, 0x01ff0006, 0x00000000
[ 3287.331337] tg3 0005:05:00.0 net0: 0x00000470: 0x80000000, 0x00000000, 0x00080000, 0x40000000
[ 3287.331445] tg3 0005:05:00.0 net0: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
[ 3287.331553] tg3 0005:05:00.0 net0: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000
[ 3287.331661] tg3 0005:05:00.0 net0: 0x00000590: 0x00e00000, 0x00000000, 0x00000000, 0x00000000
[ 3287.331770] tg3 0005:05:00.0 net0: 0x000005b0: 0x00000000, 0xc0000000, 0x00000000, 0x00000000
[ 3287.331890] tg3 0005:05:00.0 net0: 0x000005c0: 0xd384af98, 0x86411db1, 0x00000000, 0x00000000
[ 3287.331999] tg3 0005:05:00.0 net0: 0x00000600: 0xffffffff, 0x00f800d1, 0x00000000, 0x00001f04
[ 3287.332108] tg3 0005:05:00.0 net0: 0x00000610: 0xffffffff, 0x00000000, 0x07c00344, 0x00000000
[ 3287.332216] tg3 0005:05:00.0 net0: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
[ 3287.332325] tg3 0005:05:00.0 net0: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 3287.332432] tg3 0005:05:00.0 net0: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 3287.332541] tg3 0005:05:00.0 net0: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 3287.332650] tg3 0005:05:00.0 net0: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 3287.332758] tg3 0005:05:00.0 net0: 0x00000670: 0x88ec6860, 0xe62ee10c, 0x337b4659, 0x664e2a50
[ 3287.332867] tg3 0005:05:00.0 net0: 0x00000680: 0x83c13961, 0x3e5dd26e, 0x96e38c98, 0x9629f348
[ 3287.332975] tg3 0005:05:00.0 net0: 0x00000690: 0xab3402e3, 0xb7282180, 0x00000000, 0x00000000
[ 3287.333083] tg3 0005:05:00.0 net0: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000
[ 3287.333192] tg3 0005:05:00.0 net0: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[ 3287.333299] tg3 0005:05:00.0 net0: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[ 3287.333408] tg3 0005:05:00.0 net0: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[ 3287.333516] tg3 0005:05:00.0 net0: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.333624] tg3 0005:05:00.0 net0: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.333733] tg3 0005:05:00.0 net0: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.333842] tg3 0005:05:00.0 net0: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
[ 3287.333950] tg3 0005:05:00.0 net0: 0x00000880: 0x00000fca, 0x0000d0f7, 0x00000000, 0x00000001
[ 3287.334059] tg3 0005:05:00.0 net0: 0x00000890: 0x00000010, 0x00000014, 0x00000000, 0x00000000
[ 3287.334167] tg3 0005:05:00.0 net0: 0x000008f0: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[ 3287.334276] tg3 0005:05:00.0 net0: 0x00000900: 0x00021e0e, 0xffffffff, 0x00000000, 0x00000000
[ 3287.334384] tg3 0005:05:00.0 net0: 0x00000910: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[ 3287.334492] tg3 0005:05:00.0 net0: 0x00000920: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[ 3287.334601] tg3 0005:05:00.0 net0: 0x00000930: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.334710] tg3 0005:05:00.0 net0: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.334819] tg3 0005:05:00.0 net0: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.334928] tg3 0005:05:00.0 net0: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x0000033c
[ 3287.335036] tg3 0005:05:00.0 net0: 0x00000970: 0x00000008, 0x00000005, 0x00000000, 0x00000000
[ 3287.335145] tg3 0005:05:00.0 net0: 0x00000980: 0x007a9d77, 0x0000d0f7, 0x00000000, 0x00000411
[ 3287.335253] tg3 0005:05:00.0 net0: 0x00000990: 0x0000d2bd, 0x00007623, 0x00000000, 0x00000000
[ 3287.335361] tg3 0005:05:00.0 net0: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
[ 3287.335470] tg3 0005:05:00.0 net0: 0x00000c10: 0x00000000, 0x00000000, 0x00000000, 0x00280000
[ 3287.335578] tg3 0005:05:00.0 net0: 0x00000c80: 0x00000349, 0x00000000, 0x00000000, 0x00000000
[ 3287.335688] tg3 0005:05:00.0 net0: 0x00000ce0: 0xec720310, 0x08000007, 0x00000028, 0x00041028
[ 3287.335796] tg3 0005:05:00.0 net0: 0x00000cf0: 0x00000000, 0x5000005c, 0x00000000, 0x00000000
[ 3287.335904] tg3 0005:05:00.0 net0: 0x00001000: 0x00000002, 0x00000000, 0xa0004f50, 0x00000000
[ 3287.336013] tg3 0005:05:00.0 net0: 0x00001010: 0x005c05c1, 0x00004f50, 0x00000000, 0x00000000
[ 3287.336122] tg3 0005:05:00.0 net0: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[ 3287.336231] tg3 0005:05:00.0 net0: 0x00001440: 0x0000005c, 0x0000005c, 0x0000005c, 0x0000005c
[ 3287.336340] tg3 0005:05:00.0 net0: 0x00001450: 0x0000005c, 0x0000005c, 0x0000005c, 0x0000005c
[ 3287.336448] tg3 0005:05:00.0 net0: 0x00001460: 0x0000005c, 0x0000005c, 0x0000005c, 0x0000005c
[ 3287.336557] tg3 0005:05:00.0 net0: 0x00001470: 0x0000005c, 0x0000005c, 0x0000005c, 0x0000005c
[ 3287.336665] tg3 0005:05:00.0 net0: 0x00001480: 0x00001111, 0x00000000, 0x00000000, 0x00000000
[ 3287.336774] tg3 0005:05:00.0 net0: 0x00001800: 0x00000016, 0x00000000, 0x0000005c, 0x00000000
[ 3287.336883] tg3 0005:05:00.0 net0: 0x00001830: 0x00000000, 0x00000000, 0x00000000, 0xbbec0000
[ 3287.336991] tg3 0005:05:00.0 net0: 0x00001840: 0xbbec0000, 0x0800000f, 0x00000201, 0xc0000000
[ 3287.337099] tg3 0005:05:00.0 net0: 0x00001850: 0x0000001f, 0x00000000, 0x000041a0, 0x005c005c
[ 3287.337208] tg3 0005:05:00.0 net0: 0x00001860: 0x02000000, 0x00000000, 0xbbec05a0, 0x0800000f
[ 3287.337317] tg3 0005:05:00.0 net0: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[ 3287.337427] tg3 0005:05:00.0 net0: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[ 3287.337535] tg3 0005:05:00.0 net0: 0x00002010: 0x00000181, 0x00000001, 0x00780003, 0x00000000
[ 3287.337644] tg3 0005:05:00.0 net0: 0x00002100: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.337753] tg3 0005:05:00.0 net0: 0x00002110: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.337862] tg3 0005:05:00.0 net0: 0x00002120: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.337970] tg3 0005:05:00.0 net0: 0x00002130: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338078] tg3 0005:05:00.0 net0: 0x00002140: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338187] tg3 0005:05:00.0 net0: 0x00002150: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338297] tg3 0005:05:00.0 net0: 0x00002160: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338406] tg3 0005:05:00.0 net0: 0x00002170: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338515] tg3 0005:05:00.0 net0: 0x00002180: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338623] tg3 0005:05:00.0 net0: 0x00002190: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338732] tg3 0005:05:00.0 net0: 0x000021a0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338841] tg3 0005:05:00.0 net0: 0x000021b0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338949] tg3 0005:05:00.0 net0: 0x000021c0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.339058] tg3 0005:05:00.0 net0: 0x000021d0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.339168] tg3 0005:05:00.0 net0: 0x000021e0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.339276] tg3 0005:05:00.0 net0: 0x000021f0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.339385] tg3 0005:05:00.0 net0: 0x00002200: 0x00014e97, 0x00000000, 0x00000000, 0x00000000
[ 3287.339493] tg3 0005:05:00.0 net0: 0x00002400: 0x00010012, 0x00000000, 0x00400001, 0x00000000
[ 3287.339602] tg3 0005:05:00.0 net0: 0x00002410: 0x0000000f, 0x00005d00, 0x00000000, 0x00000000
[ 3287.339711] tg3 0005:05:00.0 net0: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00044400
[ 3287.339821] tg3 0005:05:00.0 net0: 0x00002450: 0x0800000f, 0xf33d0000, 0x08001800, 0x00040000
[ 3287.339929] tg3 0005:05:00.0 net0: 0x00002470: 0x00000000, 0x00000299, 0x00000000, 0x00000000
[ 3287.340038] tg3 0005:05:00.0 net0: 0x00002500: 0x00000000, 0x00000000, 0x00000002, 0x00044800
[ 3287.340146] tg3 0005:05:00.0 net0: 0x00002510: 0x00000000, 0x00000000, 0x00000002, 0x00040400
[ 3287.340255] tg3 0005:05:00.0 net0: 0x00002520: 0x00000000, 0x00000000, 0x00000002, 0x00044c00
[ 3287.340363] tg3 0005:05:00.0 net0: 0x00002530: 0x00000000, 0x00000000, 0x00000002, 0x00040800
[ 3287.340472] tg3 0005:05:00.0 net0: 0x00002540: 0x00000000, 0x00000000, 0x00000002, 0x00045000
[ 3287.340581] tg3 0005:05:00.0 net0: 0x00002550: 0x00000000, 0x00000000, 0x00000002, 0x00040c00
[ 3287.340688] tg3 0005:05:00.0 net0: 0x00002560: 0x00000000, 0x00000000, 0x00000002, 0x00045400
[ 3287.340797] tg3 0005:05:00.0 net0: 0x00002570: 0x00000000, 0x00000000, 0x00000002, 0x00041000
[ 3287.340905] tg3 0005:05:00.0 net0: 0x00002580: 0x00000000, 0x00000000, 0x00000002, 0x00045800
[ 3287.341013] tg3 0005:05:00.0 net0: 0x00002590: 0x00000000, 0x00000000, 0x00000002, 0x00041400
[ 3287.341122] tg3 0005:05:00.0 net0: 0x000025a0: 0x00000000, 0x00000000, 0x00000002, 0x00045c00
[ 3287.341229] tg3 0005:05:00.0 net0: 0x000025b0: 0x00000000, 0x00000000, 0x00000002, 0x00041800
[ 3287.341338] tg3 0005:05:00.0 net0: 0x000025c0: 0x00000000, 0x00000000, 0x00000002, 0x00046000
[ 3287.341446] tg3 0005:05:00.0 net0: 0x000025d0: 0x00000000, 0x00000000, 0x00000002, 0x00041c00
000000, 0x00000000
[ 3287.346116] tg3 0005:05:00.0 net0: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000
[ 3287.346225] tg3 0005:05:00.0 net0: 0x00003c30: 0x00000000, 0x00000000, 0x0800000f, 0xe3af0000
[ 3287.346333] tg3 0005:05:00.0 net0: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000
[ 3287.346441] tg3 0005:05:00.0 net0: 0x00003c50: 0x00000000, 0x0000029c, 0x00000000, 0x00000000
[ 3287.346549] tg3 0005:05:00.0 net0: 0x00003c80: 0x0000062b, 0x0000007c, 0x00000321, 0x000000d4
[ 3287.346657] tg3 0005:05:00.0 net0: 0x00003cc0: 0x0000005c, 0x00000000, 0x00000000, 0x00000000
[ 3287.346766] tg3 0005:05:00.0 net0: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000
[ 3287.346874] tg3 0005:05:00.0 net0: 0x00003d00: 0x0800000f, 0xc2120000, 0x0800000f, 0xf23e0000
[ 3287.346982] tg3 0005:05:00.0 net0: 0x00003d10: 0x0800000f, 0xe6a70000, 0x0800000f, 0xe54b0000
[ 3287.347090] tg3 0005:05:00.0 net0: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[ 3287.347199] tg3 0005:05:00.0 net0: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[ 3287.347307] tg3 0005:05:00.0 net0: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[ 3287.347415] tg3 0005:05:00.0 net0: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[ 3287.347523] tg3 0005:05:00.0 net0: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[ 3287.347632] tg3 0005:05:00.0 net0: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[ 3287.347740] tg3 0005:05:00.0 net0: 0x00003fc0: 0x0000786f, 0x00000000, 0x00000000, 0x00000000
[ 3287.347848] tg3 0005:05:00.0 net0: 0x00004000: 0x00000002, 0x00000000, 0x001692bd, 0x000d625e
[ 3287.347956] tg3 0005:05:00.0 net0: 0x00004010: 0x00000000, 0x002e7012, 0x00000480, 0x00847042
[ 3287.348064] tg3 0005:05:00.0 net0: 0x00004020: 0x00000000, 0x00000000, 0x00000010, 0x00000000
[ 3287.348173] tg3 0005:05:00.0 net0: 0x00004030: 0x00000010, 0x00000050, 0x00000000, 0x00000000
[ 3287.348281] tg3 0005:05:00.0 net0: 0x00004040: 0x00000000, 0x00000000, 0x01083620, 0x00000000
[ 3287.348388] tg3 0005:05:00.0 net0: 0x00004050: 0x00000000, 0x00000000, 0x002e7010, 0x0048f002
[ 3287.348497] tg3 0005:05:00.0 net0: 0x00004060: 0x00400000, 0x00000000, 0x00000000, 0x00000000
[ 3287.348605] tg3 0005:05:00.0 net0: 0x00004400: 0x80000006, 0x00000000, 0x00010000, 0x0000a000
[ 3287.348713] tg3 0005:05:00.0 net0: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000
[ 3287.348822] tg3 0005:05:00.0 net0: 0x00004420: 0x0000003d, 0x00000000, 0x00000000, 0x00000000
[ 3287.348929] tg3 0005:05:00.0 net0: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x04f14052
[ 3287.349039] tg3 0005:05:00.0 net0: 0x00004450: 0x0002033f, 0x00e800e9, 0x00000000, 0x00000000
[ 3287.349147] tg3 0005:05:00.0 net0: 0x00004800: 0x180303fe, 0x00000000, 0x00000000, 0x00000100
[ 3287.349254] tg3 0005:05:00.0 net0: 0x00004810: 0x00000000, 0x00000008, 0x05929c80, 0x00000000
[ 3287.349363] tg3 0005:05:00.0 net0: 0x00004820: 0x0000009a, 0x00000000, 0x00080000, 0x00000000
[ 3287.349472] tg3 0005:05:00.0 net0: 0x00004840: 0x00000000, 0x00000000, 0x000e2200, 0x0042f446
[ 3287.349580] tg3 0005:05:00.0 net0: 0x00004850: 0xfe1f915e, 0x804f054a, 0x8f8e8f8e, 0x00000000
[ 3287.349689] tg3 0005:05:00.0 net0: 0x00004860: 0x0000009a, 0x113e0007, 0x00000800, 0x6a000000
[ 3287.349797] tg3 0005:05:00.0 net0: 0x00004870: 0x00000080, 0x00000000, 0x00000000, 0x00000000
[ 3287.349907] tg3 0005:05:00.0 net0: 0x00004900: 0x00090404, 0x00305407, 0x00000000, 0x00000000
[ 3287.350015] tg3 0005:05:00.0 net0: 0x00004910: 0x000f001c, 0x00000000, 0x00000000, 0x00000000
[ 3287.350123] tg3 0005:05:00.0 net0: 0x00004a00: 0x180303fe, 0x00200000, 0x00200020, 0x7a9d0000
[ 3287.350232] tg3 0005:05:00.0 net0: 0x00004a10: 0xf33d6320, 0x008c0904, 0x00200010, 0x00000000
[ 3287.350340] tg3 0005:05:00.0 net0: 0x00004a20: 0x00000000, 0x00000000, 0xf02c0000, 0xf33d6380
[ 3287.350449] tg3 0005:05:00.0 net0: 0x00004a30: 0x00000000, 0x00000128, 0x00000128, 0x00000000
[ 3287.350557] tg3 0005:05:00.0 net0: 0x00004a40: 0xf33d62e0, 0xf33d6300, 0xf33d6320, 0xf33d62c0
[ 3287.350666] tg3 0005:05:00.0 net0: 0x00004a50: 0x00200020, 0x00200020, 0x00200020, 0x00200020
[ 3287.350776] tg3 0005:05:00.0 net0: 0x00004a70: 0x00090404, 0x00305407, 0x000f001c, 0x00000000
[ 3287.350884] tg3 0005:05:00.0 net0: 0x00004b00: 0x180303fe, 0x00420003, 0x30000000, 0x00280120
[ 3287.350992] tg3 0005:05:00.0 net0: 0x00004b10: 0x00420040, 0x00280002, 0x0042dc90, 0x00000000
[ 3287.351101] tg3 0005:05:00.0 net0: 0x00004b20: 0x00000051, 0x02278000, 0xac3904e9, 0x0f420048
[ 3287.351210] tg3 0005:05:00.0 net0: 0x00004b30: 0xec7202e9, 0x0f280028, 0xac3924e9, 0x0f420048
[ 3287.351319] tg3 0005:05:00.0 net0: 0x00004b40: 0xec720311, 0x0f280028, 0xcecebebe, 0xfafa9595
[ 3287.351427] tg3 0005:05:00.0 net0: 0x00004b50: 0xf03a0000, 0xac390530, 0xef050000, 0x514f0050
[ 3287.351535] tg3 0005:05:00.0 net0: 0x00004b60: 0xf0390000, 0xec720310, 0x8f030000, 0x4000005c
[ 3287.351644] tg3 0005:05:00.0 net0: 0x00004b70: 0xf03a0000, 0xac392530, 0xef050000, 0x000000ff
[ 3287.351753] tg3 0005:05:00.0 net0: 0x00004b80: 0x00000051, 0x113e0007, 0x00000800, 0x6a000000
[ 3287.351872] tg3 0005:05:00.0 net0: 0x00004b90: 0x00000080, 0x00090404, 0x00305407, 0x000f001c
[ 3287.351980] tg3 0005:05:00.0 net0: 0x00004ba0: 0x00f00024, 0x00000000, 0x00000000, 0x00000000
[ 3287.352089] tg3 0005:05:00.0 net0: 0x00004bb0: 0xf12cd0e9, 0xac3908e9, 0xec720169, 0xf12cd8e9
[ 3287.352199] tg3 0005:05:00.0 net0: 0x00004bc0: 0xac3904ee, 0xec7202e8, 0xac3924ee, 0xec720310
[ 3287.352307] tg3 0005:05:00.0 net0: 0x00004bd0: 0xf12cd0ee, 0xac3908ee, 0xec720168, 0xf12cd8ee
[ 3287.352416] tg3 0005:05:00.0 net0: 0x00004be0: 0x00280042, 0x00280042, 0x0042006a, 0x006a0180
[ 3287.352524] tg3 0005:05:00.0 net0: 0x00004bf0: 0xf0390000, 0xec720338, 0x8f030000, 0x00005bad
[ 3287.352633] tg3 0005:05:00.0 net0: 0x00004c00: 0x200003fe, 0x00000000, 0x00000000, 0x00000000
[ 3287.352742] tg3 0005:05:00.0 net0: 0x00004c10: 0x0000003f, 0x00000000, 0x00000006, 0x00000000
[ 3287.352850] tg3 0005:05:00.0 net0: 0x00004c20: 0x00000000, 0x00000000, 0x00000000, 0x00000006
[ 3287.352958] tg3 0005:05:00.0 net0: 0x00004c30: 0x00000000, 0x001f0000, 0x00000089, 0x00000089
[ 3287.353068] tg3 0005:05:00.0 net0: 0x00004c40: 0x00000000, 0xbb60c540, 0x001d0020, 0x00140020
[ 3287.353177] tg3 0005:05:00.0 net0: 0x00004c50: 0x2007b62a, 0x0062a0d3, 0xd432107c, 0x001c1111
[ 3287.353286] tg3 0005:05:00.0 net0: 0x00004c60: 0x00000020, 0x00000000, 0x00000000, 0x00000000
[ 3287.353394] tg3 0005:05:00.0 net0: 0x00005000: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 3287.353503] tg3 0005:05:00.0 net0: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x08001d54
[ 3287.353612] tg3 0005:05:00.0 net0: 0x00005020: 0x30632000, 0x00000000, 0x00000000, 0x40000020
[ 3287.353720] tg3 0005:05:00.0 net0: 0x00005030: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.353828] tg3 0005:05:00.0 net0: 0x00005040: 0x00000000, 0x00000000, 0x0800180a, 0x00000000
[ 3287.353938] tg3 0005:05:00.0 net0: 0x00005080: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 3287.354046] tg3 0005:05:00.0 net0: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x08001800
[ 3287.354155] tg3 0005:05:00.0 net0: 0x000050a0: 0x00641824, 0x00000000, 0x00000000, 0x40000020
[ 3287.354263] tg3 0005:05:00.0 net0: 0x000050b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.354372] tg3 0005:05:00.0 net0: 0x000050c0: 0x00000000, 0x00000000, 0x080024a0, 0x00000000
[ 3287.354481] tg3 0005:05:00.0 net0: 0x00005100: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[ 3287.354590] tg3 0005:05:00.0 net0: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08002988
[ 3287.354699] tg3 0005:05:00.0 net0: 0x00005120: 0x0a0005f9, 0x00000000, 0x00000000, 0x40000020
[ 3287.354808] tg3 0005:05:00.0 net0: 0x00005130: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.354916] tg3 0005:05:00.0 net0: 0x00005140: 0x00000000, 0x00000000, 0x08001d66, 0x00000000
[ 3287.355025] tg3 0005:05:00.0 net0: 0x00005180: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 3287.355134] tg3 0005:05:00.0 net0: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x08001850
[ 3287.355243] tg3 0005:05:00.0 net0: 0x000051a0: 0x30632000, 0x00000000, 0x00000000, 0x40000020
[ 3287.355352] tg3 0005:05:00.0 net0: 0x000051b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.355460] tg3 0005:05:00.0 net0: 0x000051c0: 0x00000000, 0x00000000, 0x0800180a, 0x00000000
[ 3287.355569] tg3 0005:05:00.0 net0: 0x00005200: 0x00000000, 0x14600011, 0x00000000, 0xb49a89ab
[ 3287.355678] tg3 0005:05:00.0 net0: 0x00005210: 0x1460019f, 0x08006f98, 0xc0000000, 0x00000005
[ 3287.355786] tg3 0005:05:00.0 net0: 0x00005220: 0x00000000, 0xffb7fbfb, 0x00000000, 0x08006f80
[ 3287.355895] tg3 0005:05:00.0 net0: 0x00005230: 0x00000005, 0x00000000, 0xffb7fbfb, 0x00000000
[ 3287.356004] tg3 0005:05:00.0 net0: 0x00005240: 0x08006f80, 0x00000005, 0x00000000, 0xffb7fbfb
[ 3287.356113] tg3 0005:05:00.0 net0: 0x00005250: 0x00000000, 0x08006f80, 0x00000005, 0x00000000
[ 3287.356222] tg3 0005:05:00.0 net0: 0x00005260: 0xffb7fbfb, 0x00000000, 0x08006f80, 0x00000005
[ 3287.356331] tg3 0005:05:00.0 net0: 0x00005270: 0x00000000, 0xffb7fbfb, 0x00000000, 0x08006f80
[ 3287.356440] tg3 0005:05:00.0 net0: 0x00005280: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 3287.356548] tg3 0005:05:00.0 net0: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x08001850
[ 3287.356657] tg3 0005:05:00.0 net0: 0x000052a0: 0x1464002b, 0x00000000, 0x00000000, 0x40000020
[ 3287.356766] tg3 0005:05:00.0 net0: 0x000052b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.356874] tg3 0005:05:00.0 net0: 0x000052c0: 0x00000000, 0x00000000, 0x0800180a, 0x00000000
[ 3287.356982] tg3 0005:05:00.0 net0: 0x00005300: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[ 3287.357090] tg3 0005:05:00.0 net0: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x080017e8
[ 3287.357199] tg3 0005:05:00.0 net0: 0x00005320: 0x1485000d, 0x00000000, 0x00000000, 0x40000020
[ 3287.357307] tg3 0005:05:00.0 net0: 0x00005330: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.357415] tg3 0005:05:00.0 net0: 0x00005340: 0x00000000, 0x00000000, 0x080029c0, 0x00000000
[ 3287.357524] tg3 0005:05:00.0 net0: 0x00005380: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 3287.357633] tg3 0005:05:00.0 net0: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x08001e10
[ 3287.357741] tg3 0005:05:00.0 net0: 0x000053a0: 0xafbf0014, 0x00000000, 0x00000000, 0x40000020
[ 3287.357850] tg3 0005:05:00.0 net0: 0x000053b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.357959] tg3 0005:05:00.0 net0: 0x000053c0: 0x00000000, 0x00000000, 0x08001d66, 0x00000000
[ 3287.358068] tg3 0005:05:00.0 net0: 0x00005800: 0x03000000, 0x03000000, 0xfb000000, 0x00000000
[ 3287.358177] tg3 0005:05:00.0 net0: 0x00005810: 0x7c000000, 0x00000000, 0x14000000, 0x00000000
[ 3287.358285] tg3 0005:05:00.0 net0: 0x00005820: 0xd3000000, 0x00000000, 0x00000000, 0x00000000
[ 3287.358394] tg3 0005:05:00.0 net0: 0x00005860: 0x00000000, 0x00000000, 0x00000364, 0x00000364
[ 3287.358502] tg3 0005:05:00.0 net0: 0x00005880: 0x0000062b, 0x0000062b, 0x0000007c, 0x0000007c
[ 3287.358611] tg3 0005:05:00.0 net0: 0x00005890: 0x00000321, 0x00000321, 0x000000d4, 0x000000d4
[ 3287.358720] tg3 0005:05:00.0 net0: 0x00005900: 0x0000005c, 0x0000005c, 0x00000000, 0x00000000
[ 3287.358829] tg3 0005:05:00.0 net0: 0x00005980: 0x0000005c, 0x00000000, 0x00000000, 0x00000000
[ 3287.358937] tg3 0005:05:00.0 net0: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000
[ 3287.359046] tg3 0005:05:00.0 net0: 0x00006000: 0x00010082, 0x00000000, 0x00000000, 0x00000000
[ 3287.359154] tg3 0005:05:00.0 net0: 0x00006400: 0x00000000, 0x00000000, 0x00010991, 0xc0000000
[ 3287.359264] tg3 0005:05:00.0 net0: 0x00006410: 0x0a000064, 0x0a000064, 0x00000000, 0x00000000
[ 3287.359372] tg3 0005:05:00.0 net0: 0x00006430: 0x00000000, 0x14e41657, 0x04201014, 0x01020000
[ 3287.359480] tg3 0005:05:00.0 net0: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000000
[ 3287.359589] tg3 0005:05:00.0 net0: 0x000064c0: 0x00000010, 0x00000004, 0x00000124, 0x00000000
[ 3287.359698] tg3 0005:05:00.0 net0: 0x000064d0: 0x00000000, 0x10008d81, 0x00000000, 0x00315e41
[ 3287.359806] tg3 0005:05:00.0 net0: 0x000064e0: 0x00000031, 0x0000001f, 0x00000000, 0x00000000
[ 3287.359914] tg3 0005:05:00.0 net0: 0x000064f0: 0x00000002, 0x00000031, 0x00000000, 0x00000000
[ 3287.360023] tg3 0005:05:00.0 net0: 0x00006500: 0x01e10003, 0x94028f64, 0x000098be, 0x00000003
[ 3287.360132] tg3 0005:05:00.0 net0: 0x00006510: 0x0007812c, 0x00058116, 0x0004610a, 0x00000000
[ 3287.360239] tg3 0005:05:00.0 net0: 0x00006530: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[ 3287.360347] tg3 0005:05:00.0 net0: 0x00006550: 0x00000000, 0x02800000, 0x00000000, 0x00000000
[ 3287.360456] tg3 0005:05:00.0 net0: 0x000065f0: 0x00000000, 0x00000109, 0x00000000, 0x00000000
[ 3287.360565] tg3 0005:05:00.0 net0: 0x00006800: 0x041b0034, 0x20089082, 0x01060408, 0xc0ac6cfe
[ 3287.360674] tg3 0005:05:00.0 net0: 0x00006810: 0x01020000, 0xffffffff, 0x00000000, 0x00000000
[ 3287.360782] tg3 0005:05:00.0 net0: 0x00006830: 0xffffffff, 0xffffffff, 0x00000000, 0x00000000
[ 3287.360890] tg3 0005:05:00.0 net0: 0x00006840: 0x00000000, 0x00000001, 0x00000000, 0x00000000
[ 3287.360999] tg3 0005:05:00.0 net0: 0x00006890: 0x00000000, 0x88003800, 0x00000000, 0x04102040
[ 3287.361108] tg3 0005:05:00.0 net0: 0x000068a0: 0x00000020, 0x00000001, 0x03ff03ff, 0x00000000
[ 3287.361216] tg3 0005:05:00.0 net0: 0x000068b0: 0xe0011514, 0x00000000, 0x00000000, 0x00000000
[ 3287.361325] tg3 0005:05:00.0 net0: 0x000068e0: 0x00000000, 0x00000000, 0x00000000, 0x0000e204
[ 3287.361433] tg3 0005:05:00.0 net0: 0x000068f0: 0x00ff000e, 0x00ff0000, 0x00000000, 0x04444444
[ 3287.361542] tg3 0005:05:00.0 net0: 0x00006900: 0xb3c3f9a0, 0x14f28bed, 0x00000000, 0x00000000
[ 3287.361650] tg3 0005:05:00.0 net0: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000
[ 3287.361759] tg3 0005:05:00.0 net0: 0x00007000: 0x00000188, 0x00000000, 0x00000000, 0x0000022c
[ 3287.361879] tg3 0005:05:00.0 net0: 0x00007010: 0x04200420, 0x010080f3, 0x00d70081, 0x03008200
[ 3287.361988] tg3 0005:05:00.0 net0: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000
[ 3287.362097] tg3 0005:05:00.0 net0: 0x00007030: 0x00020000, 0x00000230, 0x001f0000, 0x00000000
[ 3287.362208] tg3 0005:05:00.0 net0: 0: Host status block [00000001:00000011:(0000:0241:0000):(0000:005c)]
[ 3287.362319] tg3 0005:05:00.0 net0: 0: NAPI info [00000011:00000011:(005c:005c:01ff):0000:(036e:0000:0000:0000)]
[ 3287.362444] tg3 0005:05:00.0 net0: 1: Host status block [00000001:00000004:(0000:0000:0000):(0634:0000)]
[ 3287.362555] tg3 0005:05:00.0 net0: 1: NAPI info [00000004:00000004:(0000:0000:01ff):0634:(0634:0634:0000:0000)]
[ 3287.362679] tg3 0005:05:00.0 net0: 2: Host status block [00000001:0000007c:(007c:0000:0000):(0000:0000)]
[ 3287.362790] tg3 0005:05:00.0 net0: 2: NAPI info [0000007c:0000007c:(0000:0000:01ff):007c:(007c:007c:0000:0000)]
[ 3287.362914] tg3 0005:05:00.0 net0: 3: Host status block [00000001:00000014:(0000:0000:0000):(0000:0000)]
[ 3287.363025] tg3 0005:05:00.0 net0: 3: NAPI info [00000014:00000014:(0000:0000:01ff):0321:(0321:0321:0000:0000)]
[ 3287.363148] tg3 0005:05:00.0 net0: 4: Host status block [00000001:000000d4:(0000:0000:00d5):(0000:0000)]
[ 3287.363260] tg3 0005:05:00.0 net0: 4: NAPI info [000000d4:000000d4:(0000:0000:01ff):00d5:(00d5:00d5:0000:0000)]
[ 3287.392917] tg3 0005:05:00.0 net0: Link is down
[ 3289.256766] tg3 0005:05:00.0 net0: Link is up at 100 Mbps, full duplex
[ 3289.256859] tg3 0005:05:00.0 net0: Flow control is on for TX and on for RX
[ 3289.256941] tg3 0005:05:00.0 net0: EEE is disabled
[ 3314.141595] Watchdog CPU:64 detected Hard LOCKUP other CPUS:0
[ 3314.141692] Watchdog CPU:0 Hard LOCKUP
[ 3314.141744] Modules linked in: vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas i2c_opal ipmi_powernv i2c_core ipmi_devintf ipmi_msghandler powernv_op_panel nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc kvm_hv kvm_pr kvm scsi_dh_alua dm_service_time dm_multipath tg3 ptp pps_core
[ 3314.142565] CPU: 0 PID: 9200 Comm: kworker/0:0 Tainted: G        W       4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le #1
[ 3314.142689] Workqueue: events wait_rcu_exp_gp
[ 3314.142753] task: c0000007f008b000 task.stack: c0000007f0160000
[ 3314.142829] NIP:  c0000000001b8028 LR: c000000000189f14 CTR: c0000000001b7f60
[ 3314.142919] REGS: c0000007f01638d0 TRAP: 0501   Tainted: G        W        (4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le)
[ 3314.143039] MSR:  9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 44002022  XER: 20000000
[ 3314.143156] CFAR: c0000000001b8030 SOFTE: 1 
GPR00: c000000000189420 c0000007f0163b50 c000000001403900 0000000000000060 
GPR04: c0000007ff526700 c00000000138ac80 0000000000000000 0000000000000000 
GPR08: 0000000000000001 0000000000000001 0000000000000001 c00000000138ac80 
GPR12: c000000000090180 c00000000fd60000 
[ 3314.143563] NIP [c0000000001b8028] smp_call_function_single+0xd8/0x1a0
[ 3314.143642] LR [c000000000189f14] sync_rcu_exp_select_cpus+0x3c4/0x550
[ 3314.143718] Call Trace:
[ 3314.143753] [c0000007f0163b50] [c0000000001b7f60] smp_call_function_single+0x10/0x1a0 (unreliable)
[ 3314.143866] [c0000007f0163bc0] [c000000000189f14] sync_rcu_exp_select_cpus+0x3c4/0x550
[ 3314.143962] [c0000007f0163c50] [c00000000018a7b8] wait_rcu_exp_gp+0x38/0x60
[ 3314.144044] [c0000007f0163c80] [c000000000122d60] process_one_work+0x1a0/0x490
[ 3314.144138] [c0000007f0163d20] [c0000000001230e8] worker_thread+0x98/0x520
[ 3314.144220] [c0000007f0163dc0] [c00000000012b578] kthread+0x168/0x1b0
[ 3314.144301] [c0000007f0163e30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 3314.144394] Instruction dump:
[ 3314.144443] 7c895214 81240018 792a07e1 41820030 48000018 60000000 60000000 60000000 
[ 3314.144555] 60000000 60420000 7c210b78 7c421378 <81240018> 792807e1 4082fff0 7c2004ac 
[ 3379.041240] sd 0:2:0:0: [sda] tag#0 Resetting device
[ 3409.101139] ipr 0001:08:00.0: Timed out waiting for aborted commands
[ 3409.101252] sd 0:2:1:0: [sdb] tag#1 Resetting device
[ 3439.180914] ipr 0001:08:00.0: Timed out waiting for aborted commands
[ 3439.181028] ipr 0001:08:00.0: Adapter being reset as a result of error recovery.
[ 3456.100538] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 3456.100665] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=10988 
[ 3456.100763] 	(detected by 32, t=24007 jiffies, g=17311, c=17310, q=15846)
[ 3456.100871] Sending NMI from CPU 32 to CPUs 24:
[ 3467.225107] rcu_sched kthread starved for 1110 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=96
[ 3467.225212] rcu_sched       I    0     9      2 0x00000800
[ 3467.225281] Call Trace:
[ 3467.225321] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 3467.225422] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 3467.225507] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 3467.225590] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 3467.225673] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 3467.225771] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 3467.225867] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 3467.225951] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 3636.149208] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 3636.149318] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=19079 
[ 3636.149418] 	(detected by 64, t=42012 jiffies, g=17311, c=17310, q=24830)
[ 3636.149526] Sending NMI from CPU 64 to CPUs 24:
[ 3647.273942] rcu_sched kthread starved for 1109 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=104
[ 3647.274056] rcu_sched       I    0     9      2 0x00000800
[ 3647.274126] Call Trace:
[ 3647.274168] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 3647.274271] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 3647.274356] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 3647.274440] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 3647.274524] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 3647.274623] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 3647.274720] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 3647.274805] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 3647.274941] Watchdog CPU:64 detected Hard LOCKUP other CPUS:128,152
[ 3816.197876] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 3816.197983] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=27054 
[ 3816.198083] 	(detected by 88, t=60017 jiffies, g=17311, c=17310, q=33768)
[ 3816.198200] Sending NMI from CPU 88 to CPUs 24:
[ 3827.322627] rcu_sched kthread starved for 1110 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=112
[ 3827.322732] rcu_sched       I    0     9      2 0x00000800
[ 3827.322801] Call Trace:
[ 3827.322843] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 3827.322945] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 3827.323032] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 3827.323116] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 3827.323199] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 3827.323298] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 3827.323396] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 3827.323480] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 3996.246543] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 3996.246639] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=34980 
[ 3996.246740] 	(detected by 88, t=78022 jiffies, g=17311, c=17310, q=42662)
[ 3996.246832] Sending NMI from CPU 88 to CPUs 24:
[ 4007.256506] Watchdog CPU:128 detected Hard LOCKUP other CPUS:88
[ 4007.371456] rcu_sched kthread starved for 1113 jiffies! g17311 c17310 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=88
[ 4007.371560] rcu_sched       I    0     9      2 0x00000800
[ 4007.371629] Call Trace:
[ 4007.371667] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 4007.371768] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 4007.371852] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 4007.371937] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 4007.372020] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 4007.372119] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 4007.372216] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 4007.372301] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 4008.387138] Watchdog CPU:88 became unstuck
[ 4176.295213] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 4176.295317] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=42870 
[ 4176.295417] 	(detected by 64, t=96027 jiffies, g=17311, c=17310, q=51572)
[ 4176.295531] Sending NMI from CPU 64 to CPUs 24:
[ 4187.419949] rcu_sched kthread starved for 1110 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=104
[ 4187.420053] rcu_sched       I    0     9      2 0x00000800
[ 4187.420122] Call Trace:
[ 4187.420164] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 4187.420267] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 4187.420351] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 4187.420435] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 4187.420519] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 4187.420619] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 4187.420716] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 4187.420800] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 4356.343983] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 4356.344090] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=50769 
[ 4356.344191] 	(detected by 40, t=114032 jiffies, g=17311, c=17310, q=60472)
[ 4356.344299] Sending NMI from CPU 40 to CPUs 24:
[ 4367.468407] rcu_sched kthread starved for 1109 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=88
[ 4367.468521] rcu_sched       I    0     9      2 0x00000800
[ 4367.468590] Call Trace:
[ 4367.468630] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 4367.468733] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 4367.468817] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 4367.468900] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 4367.468984] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 4367.469083] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 4367.469179] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 4367.469263] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 4536.392782] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 4536.392881] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=58758 
[ 4536.392981] 	(detected by 48, t=132037 jiffies, g=17311, c=17310, q=69558)
[ 4536.393101] Sending NMI from CPU 48 to CPUs 24:
[ 4547.517491] rcu_sched kthread starved for 1110 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=112
[ 4547.517620] rcu_sched       I    0     9      2 0x00000800
[ 4547.517689] Call Trace:
[ 4547.517731] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 4547.517833] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 4547.517919] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 4547.518003] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 4547.518086] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 4547.518185] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 4547.518281] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 4547.518365] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 4547.518490] Watchdog CPU:48 Hard LOCKUP
[ 4547.518491] Modules linked in: vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas i2c_opal ipmi_powernv i2c_core ipmi_devintf ipmi_msghandler powernv_op_panel nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc kvm_hv kvm_pr kvm scsi_dh_alua dm_service_time dm_multipath tg3 ptp pps_core
[ 4547.518583] CPU: 48 PID: 0 Comm: swapper/48 Tainted: G        W       4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le #1
[ 4547.518586] task: c000000ffb972e00 task.stack: c000000ffba20000
[ 4547.518588] NIP:  c0000000000165c8 LR: c0000000000165a4 CTR: c000000000182050
[ 4547.518590] REGS: c00000003fdbfd80 TRAP: 0900   Tainted: G        W        (4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le)
[ 4547.518591] MSR:  9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28002044  XER: 20000000
[ 4547.518604] CFAR: c00000000009052c SOFTE: 0 
GPR00: c0000000000165a4 c000001fffe0bf20 c000000001403900 00000000000001a7 
GPR04: c000001fffe0be60 000000000000301a 0000000000000000 0000000000000180 
GPR08: 0000000000000000 b000000000009033 0000000000000000 0000000000000000 
GPR12: c0000000000903f0 c00000000fd7f800 c000000ffba23f90 0000000000000000 
GPR16: 0000000000200042 0000000100067b33 c000000ffba20000 0000000000000000 
GPR20: c000000000f74f00 c000000001433b00 c000000000f74f00 000000000000000a 
GPR24: c000000ffba20000 c000000ffba20000 c000000001432200 c000000ffba23b30 
GPR28: 0000000000000000 c000000000f74ee0 c000000ffba236e0 c000001fffe08000 
[ 4547.518654] NIP [c0000000000165c8] __do_irq+0x88/0x200
[ 4547.518657] LR [c0000000000165a4] __do_irq+0x64/0x200
[ 4547.518658] Call Trace:
[ 4547.518661] [c000001fffe0bf20] [c0000000000165a4] __do_irq+0x64/0x200 (unreliable)
[ 4547.518667] [c000001fffe0bf90] [c000000000029fa4] call_do_irq+0x14/0x24
[ 4547.518671] [c000000ffba23620] [c0000000000167dc] do_IRQ+0x9c/0x110
[ 4547.518676] [c000000ffba23670] [c000000000008c58] hardware_interrupt_common+0x158/0x160
[ 4547.518683] --- interrupt: 501 at arch_local_irq_restore+0x5c/0x90
    LR = arch_local_irq_restore+0x40/0x90
[ 4547.518688] [c000000ffba23960] [c000000000141904] vtime_account_irq_enter+0x64/0x80 (unreliable)
[ 4547.518694] [c000000ffba23980] [c000000000b0f690] __do_softirq+0xf0/0x3a4
[ 4547.518698] [c000000ffba23a70] [c0000000001064e8] irq_exit+0x118/0x130
[ 4547.518703] [c000000ffba23a90] [c00000000002469c] timer_interrupt+0xac/0xe0
[ 4547.518708] [c000000ffba23ac0] [c000000000009208] decrementer_common+0x158/0x160
[ 4547.518714] --- interrupt: 901 at replay_interrupt_return+0x0/0x4
    LR = arch_local_irq_restore+0x74/0x90
[ 4547.518717] [c000000ffba23db0] [c00000000143c0cc] cpu_idle_force_poll+0x0/0x4 (unreliable)
[ 4547.518726] [c000000ffba23dd0] [c0000000008e83f0] cpuidle_enter_state+0x110/0x3d0
[ 4547.518730] [c000000ffba23e30] [c00000000015f73c] call_cpuidle+0x4c/0x80
[ 4547.518735] [c000000ffba23e50] [c00000000015fbe0] do_idle+0x2b0/0x350
[ 4547.518739] [c000000ffba23ec0] [c00000000015fe8c] cpu_startup_entry+0x3c/0x50
[ 4547.518743] [c000000ffba23ef0] [c000000000048a74] start_secondary+0x4e4/0x530
[ 4547.518748] [c000000ffba23f90] [c00000000000b16c] start_secondary_prolog+0x10/0x14
[ 4547.518750] Instruction dump:
[ 4547.518754] 7d2903a6 7d2c4b78 4e800421 e8410018 894d028b 7948f7e3 554a003c 994d028b 
[ 4547.518765] 40820010 e92d0020 61298000 7d210164 <2fa30000> 41de012c 48161569 60000000 
[ 4547.522790] Watchdog CPU:48 became unstuck
[ 4716.441578] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 4716.441680] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=66701 
[ 4716.441781] 	(detected by 96, t=150042 jiffies, g=17311, c=17310, q=78447)
[ 4716.441894] Sending NMI from CPU 96 to CPUs 24:
[ 4727.566112] rcu_sched kthread starved for 1113 jiffies! g17311 c17310 f0x0 RCU_GP_WAIT4fc/0xa60
[ 5267.713414] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 5267.713498] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 5268.788484] Watchdog CPU:104 became unstuck
[ 5436.636769] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 5436.636852] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=98515 
[ 5436.636953] 	(detected by 112, t=222062 jiffies, g=17311, c=17310, q=115020)
[ 5436.637067] Sending NMI from CPU 112 to CPUs 24:
[ 5447.761232] rcu_sched kthread starved for 1111 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=104
[ 5447.761360] rcu_sched       I    0     9      2 0x00000800
[ 5447.761429] Call Trace:
[ 5447.761469] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 5447.761571] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 5447.761656] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 5447.761740] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 5447.761824] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 5447.761923] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 5447.762019] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 5447.762104] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 5616.685701] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=106527 
[ 5616.685812] 	(detected by 8, t=240067 jiffies, g=17311, c=17310, q=123911)
[ 5616.685921] Sending NMI from CPU 8 to CPUs 24:
[ 5627.810360] rcu_sched kthread starved for 1110 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=112
[ 5627.810475] rcu_sched       I    0     9      2 0x00000800
[ 5627.810544] Call Trace:
[ 5627.810584] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 5627.810686] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 5627.810771] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 5627.810854] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 5627.810937] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 5627.811036] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 5627.811132] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 5627.811216] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c

Earlier, I have enabled function tracing as in below script:

mkdir -p /debug
mount -t debugfs nodev /debug 2>&1
echo '*' >/debug/tracing/set_ftrace_filter
echo function >/debug/tracing/current_tracer
echo 1 >/debug/tracing/tracing_on
read -p "press ENTER key to cancel .." var
echo 0 >/debug/tracing/tracing_on
cat /debug/tracing/trace > /tmp/tracing.out$$
echo "/tmp/tracing.out$$ is created .."

Host Cpu Hardlockup and unstuck observed sometimes during guest boot.

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=169389 </cde:info>

Env:
HostOS CI P9:
Host: HW: P9-Boston
Kernel: 4.17.0-1.dev.git5ce3eac.el7.ppc64le
Qemu: qemu-system-ppc-2.12.0-2.dev.gitd36f3ee.el7.ppc64le
Libvirt: libvirt-4.3.0-1.dev.git3096ff1.el7.ppc64le
Guest: HostOS (4.17.0-1.dev.git5ce3eac.el7.ppc64le)

Test: Guest Boot through libvirt.
Only First guest boot test got this issue, not further boot/any tests..

Test log link: https://ltc-jenkins.aus.stglabs.ibm.com/job/HostOS_CI_P9/10/artifact/avocado-fvt-wrapper/results/job-2018-07-01T15.50-ca32c56/test-results/001-guest_sanity.cpu.import.qemu.qcow2.virtio_scsi.smp2.virtio_net.Guest.HostOS.ppc64le.powerkvm-qemu.unattended_install.import.import.default_install.aio_native

Testcase failure output:

15:51:30 DEBUG| make_create_command() setting up command for nic: {'netdst': 'virbr0', 'ip': None, 'nic_name': 'nic1', 'mac': '52:54:00:6d:6e:6f', 'nettype': 'bridge', 'nic_model': 'virtio', 'g_nic_name': None}
15:51:30 DEBUG| vm.make_create_command.add_nic returning:  --network=bridge=virbr0,model=virtio,mac=52:54:00:6d:6e:6f
15:51:30 INFO | Running libvirt command (reformatted):
15:51:30 INFO | /usr/bin/virt-install 
15:51:30 INFO |     --connect=qemu:///system 
15:51:30 INFO |     --hvm 
15:51:30 INFO |     --accelerate 
15:51:30 INFO |     --name 'virt-tests-vm1' 
15:51:30 INFO |     --machine pseries 
15:51:30 INFO |     --memory=32768 
15:51:30 INFO |     --vcpu=32,sockets=1,cores=32,threads=1 
15:51:30 INFO |     --import 
15:51:30 INFO |     --nographics 
15:51:30 INFO |     --serial pty 
15:51:30 INFO |     --memballoon model=virtio 
15:51:30 INFO |     --controller type=scsi,model=virtio-scsi 
15:51:30 INFO |     --disk path=/home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/data/avocado-vt/images/hostos-ppc64le.qcow2,bus=scsi,size=10,format=qcow2 
15:51:30 INFO |     --network=bridge=virbr0,model=virtio,mac=52:54:00:6d:6e:6f 
15:51:30 INFO |     --noautoconsole
15:51:30 INFO | Running '/usr/bin/virt-install --connect=qemu:///system --hvm --accelerate --name 'virt-tests-vm1' --machine pseries --memory=32768 --vcpu=32,sockets=1,cores=32,threads=1 --import --nographics --serial pty --memballoon model=virtio --controller type=scsi,model=virtio-scsi --disk path=/home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/data/avocado-vt/images/hostos-ppc64le.qcow2,bus=scsi,size=10,format=qcow2 --network=bridge=virbr0,model=virtio,mac=52:54:00:6d:6e:6f --noautoconsole'
15:51:31 DEBUG| [stderr] WARNING  No operating system detected, VM performance may suffer. Specify an OS with --os-variant for optimal results.
15:51:33 DEBUG| [stdout] 
15:51:33 DEBUG| [stdout] Starting install...
15:51:33 DEBUG| [stdout] Domain creation completed.
15:51:34 INFO | Command '/usr/bin/virt-install --connect=qemu:///system --hvm --accelerate --name 'virt-tests-vm1' --machine pseries --memory=32768 --vcpu=32,sockets=1,cores=32,threads=1 --import --nographics --serial pty --memballoon model=virtio --controller type=scsi,model=virtio-scsi --disk path=/home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/data/avocado-vt/images/hostos-ppc64le.qcow2,bus=scsi,size=10,format=qcow2 --network=bridge=virbr0,model=virtio,mac=52:54:00:6d:6e:6f --noautoconsole' finished with 0 after 3.39120984077s
15:51:34 DEBUG| waiting for domain virt-tests-vm1 to start (0.000012 secs)
15:51:34 INFO | Waiting for installation to finish. Timeout set to 180 s (3 min)
15:51:34 DEBUG| Monitoring serial console log for completion message: /home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/results/job-2018-07-01T15.50-ca32c56/test-results/001-guest_sanity.cpu.import.qemu.qcow2.virtio_scsi.smp2.virtio_net.Guest.HostOS.ppc64le.powerkvm-qemu.unattended_install.import.import.default_install.aio_native/serial-serial0-virt-tests-vm1-4en2.log
15:51:34 DEBUG| Attempting to log into 'virt-tests-vm1' via serial console (timeout 10s)
15:51:55 WARNI| Error occur when update VM address cache: Login timeout expired    (output: 'exceeded 10 s timeout')
15:51:56 DEBUG| Attempting to log into 'virt-tests-vm1' via serial console (timeout 10s)
15:52:15 DEBUG| Updated HWADDR (52:54:00:6d:6e:6f)<->(192.168.122.186) IP pair into address cache
15:52:17 WARNI| Error occur when update VM address cache: Login timeout expired    (output: 'exceeded 10 s timeout')
15:52:19 DEBUG| cleaning up threads and mounts that may be active
15:52:19 INFO | Guest reported successful installation after 44 s (0 min)
15:52:19 INFO | Wait for guest to shutdown cleanly
15:52:20 DEBUG| Waiting for guest to shutdown 59
15:52:21 DEBUG| Waiting for guest to shutdown 58
15:52:22 DEBUG| Waiting for guest to shutdown 57
15:52:23 DEBUG| Waiting for guest to shutdown 56
15:52:24 DEBUG| Waiting for guest to shutdown 55
15:52:25 DEBUG| Waiting for guest to shutdown 54
15:52:26 DEBUG| Waiting for guest to shutdown 53
15:52:27 DEBUG| Waiting for guest to shutdown 52
15:52:28 DEBUG| Waiting for guest to shutdown 51
15:52:29 DEBUG| Waiting for guest to shutdown 50
15:52:29 DEBUG| Shutdown took 10 seconds
15:52:29 DEBUG| VM virt-tests-vm1 shut down
15:52:30 INFO | Guest managed to shutdown cleanly
15:52:30 WARNI| Requested MAC address release from persistent vm virt-tests-vm1. Ignoring.
15:52:30 DEBUG| Checking image file /home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/data/avocado-vt/images/hostos-ppc64le.qcow2
15:52:30 DEBUG| Run qemu-img info comamnd on /home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/data/avocado-vt/images/hostos-ppc64le.qcow2
15:52:30 INFO | Running '/usr/bin/qemu-img info -U /home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/data/avocado-vt/images/hostos-ppc64le.qcow2'
15:52:31 DEBUG| [stdout] image: /home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/data/avocado-vt/images/hostos-ppc64le.qcow2
15:52:31 DEBUG| [stdout] file format: qcow2
15:52:31 INFO | Command '/usr/bin/qemu-img info -U /home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/data/avocado-vt/images/hostos-ppc64le.qcow2' finished with 0 after 0.0559370517731s
15:52:31 DEBUG| [stdout] virtual size: 30G (32212254720 bytes)
15:52:31 DEBUG| [stdout] disk size: 30G
15:52:31 DEBUG| [stdout] cluster_size: 65536
15:52:31 DEBUG| [stdout] Format specific information:
15:52:31 DEBUG| [stdout]     compat: 1.1
15:52:31 DEBUG| [stdout]     lazy refcounts: true
15:52:31 DEBUG| [stdout]     refcount bits: 16
15:52:31 DEBUG| [stdout]     corrupt: false
15:52:31 INFO | Running 'true'
15:52:31 INFO | Command 'true' finished with 0 after 0.00186491012573s
15:52:31 INFO | Running 'ps -o comm 1'
15:52:31 DEBUG| [stdout] COMMAND
15:52:31 INFO | Command 'ps -o comm 1' finished with 0 after 0.0576908588409s
15:52:31 DEBUG| [stdout] systemd
15:52:31 INFO | Running 'true'
15:52:31 INFO | Command 'true' finished with 0 after 0.00177407264709s
15:52:31 INFO | Running 'ps -o comm 1'
15:52:31 DEBUG| [stdout] COMMAND
15:52:31 INFO | Command 'ps -o comm 1' finished with 0 after 0.0565540790558s
15:52:31 DEBUG| [stdout] systemd
15:52:31 DEBUG| Setting ignore_status to True.
15:52:31 INFO | Running 'systemctl reset-failed libvirtd.service'
15:52:31 INFO | Command 'systemctl reset-failed libvirtd.service' finished with 0 after 0.00678014755249s
15:52:31 DEBUG| Setting ignore_status to True.
15:52:31 INFO | Running 'systemctl restart libvirtd.service'
15:52:32 INFO | Command 'systemctl restart libvirtd.service' finished with 0 after 0.0816829204559s
15:52:32 INFO | Running 'virsh list'
15:52:33 DEBUG| [stdout]  Id    Name                           State
15:52:33 INFO | Command 'virsh list' finished with 0 after 0.953235149384s
15:52:33 DEBUG| [stdout] ----------------------------------------------------
15:52:33 DEBUG| [stdout] 
15:52:33 INFO | Running 'dmesg -C'
15:52:33 INFO | Command 'dmesg -C' finished with 0 after 0.00171399116516s
15:52:33 ERROR| 
15:52:33 ERROR| Reproduced traceback from: /usr/lib/python2.7/site-packages/avocado_plugins_vt-62.0-py2.7.egg/avocado_vt/test.py:454
15:52:33 ERROR| Traceback (most recent call last):
15:52:33 ERROR|   File "/usr/lib/python2.7/site-packages/avocado_plugins_vt-62.0-py2.7.egg/virttest/error_context.py", line 135, in new_fn
15:52:33 ERROR|     return fn(*args, **kwargs)
15:52:33 ERROR|   File "/usr/lib/python2.7/site-packages/avocado_plugins_vt-62.0-py2.7.egg/virttest/env_process.py", line 1406, in postprocess
15:52:33 ERROR|     raise RuntimeError("Failures occurred while postprocess:\n%s" % err)
15:52:33 ERROR| RuntimeError: Failures occurred while postprocess:
15:52:33 ERROR| 
15:52:33 ERROR| Host dmesg verification failed: Found failures in host dmesg log Please check host dmesg log /home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/results/job-2018-07-01T15.50-ca32c56/test-results/001-guest_sanity.cpu.import.qemu.qcow2.virtio_scsi.smp2.virtio_net.Guest.HostOS.ppc64le.powerkvm-qemu.unattended_install.import.import.default_install.aio_native/host_dmesg.log.
15:52:33 ERROR| 
15:52:33 INFO | cleaning libvirtd logs...
15:52:33 ERROR| 
15:52:33 ERROR| Reproduced traceback from: /usr/lib/python2.7/site-packages/avocado_framework-62.0-py2.7.egg/avocado/core/test.py:832
15:52:33 ERROR| Traceback (most recent call last):
15:52:33 ERROR|   File "/usr/lib/python2.7/site-packages/avocado_plugins_vt-62.0-py2.7.egg/avocado_vt/test.py", line 297, in runTest
15:52:33 ERROR|     raise self.__status  # pylint: disable=E0702
15:52:33 ERROR| RuntimeError: Failures occurred while postprocess:
15:52:33 ERROR| 
15:52:33 ERROR| Host dmesg verification failed: Found failures in host dmesg log Please check host dmesg log /home/workspace/runAvocadoFVTTest/avocado-fvt-wrapper/results/job-2018-07-01T15.50-ca32c56/test-results/001-guest_sanity.cpu.import.qemu.qcow2.virtio_scsi.smp2.virtio_net.Guest.HostOS.ppc64le.powerkvm-qemu.unattended_install.import.import.default_install.aio_native/host_dmesg.log.
15:52:33 ERROR| ```

Host dmesg log:
[Sat Jun 30 02:08:16 2018] watchdog: CPU 144 detected hard LOCKUP on other CPUs 102,135-136
[Sat Jun 30 02:08:16 2018] watchdog: CPU 102 Hard LOCKUP
[Sat Jun 30 02:08:16 2018] watchdog: CPU 135 Hard LOCKUP
[Sat Jun 30 02:08:16 2018] watchdog: CPU 136 Hard LOCKUP
[Sat Jun 30 02:08:16 2018] watchdog: CPU 102 became unstuck
[Sat Jun 30 02:08:16 2018] watchdog: CPU 135 became unstuck
[Sat Jun 30 03:19:52 2018] watchdog: CPU 135 detected hard LOCKUP on other CPUs 69
[Sat Jun 30 03:19:52 2018] watchdog: CPU 69 Hard LOCKUP
[Sat Jun 30 03:19:52 2018] watchdog: CPU 69 became unstuck

vcpu hotplug triggers a guest kernel warning `WARNING: workqueue cpumask: online intersect > possible intersect`

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=168836 </cde:info>

Guest kernel prints a kernel warning after it receives vcpu hotplug event.

ENV:

Host: 4.17.0-1.dev.git5ce3eac.el7.ppc64le
Guest: 4.17.0-1.dev.git5ce3eac.el7.ppc64le

qemu: qemu-2.12.0-2.dev.gitd36f3ee.el7.ppc64le
libvirt: libvirt-4.3.0-1.dev.git3096ff1.el7.ppc64le

testcase: 685-guest.with_numa.with_hugepage.with_hugepage_pin.with_pin.import.qemu.qcow2.virtio_scsi.smp2.virtio_net.Guest.HostOS.ppc64le.powerkvm-libvirt.libvirt_vcpu_plug_unplug.positive_test.vcpu_set.live.vm_operate.reboot

Steps to reproduce:

Boot a guest with numa
Do vcpuhotplug
-------------> guest kernel shows warning message.

vcpu hotplug from host:
21:50:49 DEBUG| Running virsh command: setvcpus virt-tests-vm1 3 --live
21:50:49 INFO | Running '/bin/virsh setvcpus virt-tests-vm1 3 --live'
21:50:49 DEBUG| [stdout]
21:50:49 INFO | Command '/bin/virsh setvcpus virt-tests-vm1 3 --live' finished with 0 after 0.067715883255s
21:50:49 DEBUG| status: 0


Guest console:
2018-06-12 21:50:49: [   27.794386] WARNING: workqueue cpumask: online intersect > possible intersect

Power9: Host crash during SMT change with guest emulator thread pinned "Oops: Kernel access of bad area, sig: 11 [#1]"

Host Kernel: 4.13.0-4.rel.git49564cb.el7.centos.ppc64le

Steps to reproduce:

Boot a guest(vm1)
pin emulator thread to last host cpu
virsh emulatorpin vm1 79 --live --config
Change host SMT from 4 to 2
ppc64_cpu --smt=2
====> Host hit with crash and become unresposive

part of guest xml

<domain type='kvm'>
  <name>vm1</name>
  <uuid>8914b703-4133-4564-bb39-108159f0f2b8</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <emulatorpin cpuset='79'/>
  </cputune>
  <os>
    <type arch='ppc64le' machine='pseries-2.10'>hvm</type>
    <boot dev='hd'/>
  </os>
  <cpu>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>

Host hung and unresponsive, needs a external reboot to bring back.

# [175192.775110] IRQ 33: no longer affine to CPU2
[175193.513117] IRQ 51: no longer affine to CPU7
[175193.918060] IRQ 36: no longer affine to CPU10
[175194.898718] IRQ 32: no longer affine to CPU15
[175195.497593] IRQ 24: no longer affine to CPU23
[175195.847274] IRQ 59: no longer affine to CPU27
[175196.156829] IRQ 39: no longer affine to CPU31
[175196.514113] IRQ 38: no longer affine to CPU35
[175196.845370] IRQ 52: no longer affine to CPU38
[175197.016417] IRQ 50: no longer affine to CPU39
[175197.935579] irq_migrate_all_off_this_cpu: 1 callbacks suppressed
[175197.935582] IRQ 69: no longer affine to CPU51
[175198.195199] IRQ 56: no longer affine to CPU55
[175198.345390] IRQ 57: no longer affine to CPU62
[175198.506220] IRQ 28: no longer affine to CPU63
[175199.224386] IRQ 66: no longer affine to CPU71
[175199.554113] IRQ 35: no longer affine to CPU75
[175199.694068] IRQ 37: no longer affine to CPU78
[175199.852866] Unable to handle kernel paging request for data at address 0x000008c8
[175199.852938] Faulting instruction address: 0xc0000000001d0184
[175199.852953] Oops: Kernel access of bad area, sig: 11 [#1]
[175199.853004] SMP NR_CPUS=1024 
[175199.853005] NUMA 
[175199.853045] PowerNV
[175199.853098] Modules linked in: target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas ipmi_powernv ipmi_devintf ipmi_msghandler powernv_op_panel opal_prd nfsd auth_rpcgss oid_registry nfs_acl
[175199.853785]  lockd grace kvm_hv sunrpc kvm tg3 ptp pps_core
[175199.853856] CPU: 79 PID: 64710 Comm: kworker/79:2 Not tainted 4.13.0-4.rel.git49564cb.el7.centos.ppc64le #1
[175199.853961] Workqueue: events cpuset_hotplug_workfn
[175199.854014] task: c0000003a2a22600 task.stack: c0000003a2ac8000
[175199.854077] NIP: c0000000001d0184 LR: c0000000001d0170 CTR: c0000000001d0130
[175199.854153] REGS: c0000003a2acb710 TRAP: 0300   Not tainted  (4.13.0-4.rel.git49564cb.el7.centos.ppc64le)
[175199.854241] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>
[175199.854249]   CR: 448e2022  XER: 20040000
[175199.854349] CFAR: c0000000001c3db0 DAR: 00000000000008c8 DSISR: 40000000 SOFTE: 1 
[175199.854349] GPR00: c0000000001d0170 c0000003a2acb990 c000000001397a00 0000000000000000 
[175199.854349] GPR04: c0000003a2acb9b0 0000000000000000 c0000003a2acbab0 c000000245975678 
[175199.854349] GPR08: c000000245975678 c0000003a2acb948 c0000000015a7a00 0000000000000000 
[175199.854349] GPR12: c0000000001d0130 c00000000fdb1600 c000000000124348 c000000036de4e80 
[175199.854349] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000001 
[175199.854349] GPR20: c000000005be6940 c000000005be6960 0000000000000000 0000000000000000 
[175199.854349] GPR24: c000000001334de0 c0000000015a09e0 c000000001264488 c0000003a2acbab0 
[175199.854349] GPR28: c0000003aae63c00 c0000003a2acbaa0 c0000003a2acba10 0000000000000000 
[175199.855075] NIP [c0000000001d0184] cpuset_can_attach+0x54/0x1a0
[175199.855191] LR [c0000000001d0170] cpuset_can_attach+0x40/0x1a0
[175199.855304] Call Trace:
[175199.855355] [c0000003a2acb990] [c0000000001d0170] cpuset_can_attach+0x40/0x1a0 (unreliable)
[175199.855519] [c0000003a2acb9f0] [c0000000001c4dd4] cgroup_migrate_execute+0xc4/0x4c0
[175199.855657] [c0000003a2acba60] [c0000000001cc3d4] cgroup_transfer_tasks+0x1e4/0x380
[175199.855796] [c0000003a2acbb90] [c0000000001d2810] cpuset_hotplug_workfn+0x6e0/0x900
[175199.855934] [c0000003a2acbc90] [c00000000011bc00] process_one_work+0x1a0/0x490
[175199.856072] [c0000003a2acbd30] [c00000000011bf88] worker_thread+0x98/0x520
[175199.856188] [c0000003a2acbdc0] [c0000000001244a8] kthread+0x168/0x1b0
[175199.856304] [c0000003a2acbe30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[175199.856441] Instruction dump:
[175199.856513] fbc1fff0 fbe1fff8 f8010010 f821ffa1 38810020 7c7d1b78 4bff3c6d 60000000 
[175199.856655] 3f42ffed 3d420021 eb610020 3b5aca88 <e92308c8> 7f43d378 e9290000 f92a90c8 
[175199.856800] ---[ end trace 5aa84a7cf504a434 ]---
[175199.868456] 
[175201.708433] process 150492 (vhost-150463) no longer affine to cpu79

cde:info Mirrored with LTC bug #159341 </cde:info>

Host crashed while running memhotplug guest_sanity tests with latest devel branch

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=160986 </cde:info>

Host was running guest_sanity tests.
Kernel: 4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le

    lr: d00000000b30e498: kvmppc_book3s_hv_page_fault+0xbb8/0xc40 [kvm_hv]
    sp: c0000000ae89f850
   msr: 900000010280b033
   dar: d00000002b5bb20c
 dsisr: 40000000
  current = 0xc0000001c4003080
  paca    = 0xc00000000fd8f400   softe: 0        irq_happened: 0x01
    pid   = 46914, comm = CPU 3/KVM
Linux version 4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le ([email protected]) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-17) (GCC)) #1 SMP Fri Oct 20 22:55:44 -02 2017
[66906.130198] KVM: CPU 44 seems to be stuck
[66906.130257] KVM: CPU 46 seems to be stuck
enter ? for help
[c0000000aee2b8b0] d00000000b30e498 kvmppc_book3s_hv_page_fault+0xbb8/0xc40 [kvm_hv]
[c0000000aee2b9e0] d00000000b30a078 kvmppc_vcpu_run_hv+0xdf8/0x1300 [kvm_hv]
[c0000000aee2bb30] d00000000b1348c4 kvmppc_vcpu_run+0x34/0x50 [kvm]
[c0000000aee2bb50] d00000000b130d54 kvm_arch_vcpu_ioctl_run+0x114/0x2a0 [kvm]
[c0000000aee2bbd0] d00000000b1239d8 kvm_vcpu_ioctl+0x598/0x7a0 [kvm]
[c0000000aee2bd40] c0000000003832e0 do_vfs_ioctl+0xd0/0x8c0
[c0000000aee2bde0] c000000000383ba4 SyS_ioctl+0xd4/0x130
[c0000000aee2be30] c00000000000b8e0 system_call+0x58/0x6c
--- Exception: c00 (System Call) at 00007fff8d0b674c
SP (7fff597fde60) is in userspace
8:mon> ```

PCI passthrough: Frozen PE / EEH recovery happens in the host if driver is loaded after the guest is shutdown and device is reattached to the host

Scenario: PCI passthrough of the SAS3008-based PCIe adapter in the 8001-22C system.

# lspci -nnv -s 1:3:0.0 | head -n2
0001:03:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
Subsystem: Super Micro Computer Inc Device [15d9:0808]

Steps to reproduce:

Host: detach the adapter (virsh nodedev-detach pci_0001_03_00_0)
Host: start a guest with PCI passthrough (virsh start --console <guest>)
Guest: load the driver (initializes the adapter, scans for disks, etc) (modprobe mpt3sas)
Guest: shutdown (poweroff)
Host: reattach the adapter (virsh nodedev-reattach pci_0001_03_00_0)
Host: load the driver (starts to init the adapter and hits Frozen PE/EEH recovery) (modprobe mpt3sas)

During driver initialization the following Frozen PE / EEH recovery is consistently observed.
There is an Oops in the driver code afterward, but that's another problem which I'll be looking at.

Decoding the PEST bits tells this is a DMA write w/ invalid page access. The suspicion is there are pending operations/configuration from the guest, and since the PE was not reset in a way that could actually clear these in this adapter, the problem is hit.

In that scenario, this problem is expected to be resolved by the patch series which was applied downstream on PowerKVM [1], and now is being worked in a VFIO-based approach by @aik .

[1] https://lists.ozlabs.org/pipermail/linuxppc-dev/2015-February/124867.html

[  759.825059] mpt3sas 0001:03:00.0: enabling device (0400 -> 0402)
[  759.825165] mpt3sas 0001:03:00.0: Using 64-bit DMA iommu bypass
[  759.825223] mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (535679552 kB)
[  759.882919] mpt3sas_cm0: MSI-X vectors supported: 96, no of cores: 16, max_msix_vectors: -1
[  759.883772] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 706
[  759.883819] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 707
[  759.883863] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 708
[  759.883906] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 709
[  759.883949] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 710
[  759.883993] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 711
[  759.884035] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 712
[  759.884080] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 713
[  759.884123] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 714
[  759.884166] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 715
[  759.884210] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 716
[  759.884297] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 717
[  759.884339] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 718
[  759.884382] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 719
[  759.884427] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 720
[  759.884471] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 721
[  759.884516] mpt3sas_cm0: iomem(0x00003fe080140000), mapped(0xd0000800810a0000), size(65536)
[  759.884582] mpt3sas_cm0: ioport(0x0000000000000000), size(0)
[  759.975501] mpt3sas_cm0: Allocated physical memory: size(8887 kB)
[  759.975563] mpt3sas_cm0: Current Controller Queue Depth(2936),Max Controller Queue Depth(3072)
[  759.975636] mpt3sas_cm0: Scatter Gather Elements per IO(128)
[  760.021015] EEH: Frozen PE#fd on PHB#1 detected
[  760.021106] EEH: PE location: PLX Slot1, PHB location: N/A
[  760.021873] EEH: This PCI device has failed 1 times in the last hour
[  760.021927] EEH: Notify device drivers to shutdown
[  760.021970] mpt3sas_cm0: PCI error: detected callback, state(2)!!
[  760.022317] EEH: Collect temporary log
[  760.022378] EEH: of node=0001:03:00.0
[  760.022414] EEH: PCI device/vendor: 00971000
[  760.022461] EEH: PCI cmd/status register: 00180142
[  760.022503] EEH: PCI-E capabilities and status follow:
[  760.022558] EEH: PCI-E 00: 0002a810 10008025 0000281e 00415083 
[  760.022620] EEH: PCI-E 10: 10830000 00000000 00000000 00000000 
[  760.022675] EEH: PCI-E 20: 00000000 
[  760.022706] EEH: PCI-E AER capability register set follows:
[  760.022758] EEH: PCI-E AER 00: 1e020001 00000000 00000000 00462031 
[  760.022821] EEH: PCI-E AER 10: 00000000 00002000 000001e0 00000000 
[  760.022881] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000 
[  760.022935] EEH: PCI-E AER 30: 00000000 00000000 
[  760.022979] PHB3 PHB#1 Diag-data (Version: 1)
[  760.023022] brdgCtl:     00000002
[  760.023059] RootSts:     0000000f 00400000 b0830008 00100147 00002000
[  760.023112] PhbSts:      0000001c00000000 0000001c00000000
[  760.023156] Lem:         0000000004000000 42498e367f502eae 0000000000000000
[  760.023210] InAErr:      0000000000004000 0000000000004000 00000000612400fd 04000000000000fd
[  760.023284] PE[253] A/B: 8000302500000000 8000000061240000
[  760.023325] EEH: Reset without hotplug activity
[  762.174778] EEH: Notify device drivers the completion of reset
[  762.174860] mpt3sas_cm0: PCI error: slot reset callback!!
[  762.174985] mpt3sas 0001:03:00.0: Using 64-bit DMA iommu bypass
[  762.175044] mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (535679552 kB)
[  762.232259] mpt3sas_cm0: MSI-X vectors supported: 96, no of cores: 16, max_msix_vectors: -1
[  762.233046] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 706
[  762.233091] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 707
[  762.233135] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 708
[  762.233179] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 709
[  762.233223] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 710
[  762.233266] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 711
[  762.233309] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 712
[  762.233352] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 713
[  762.233395] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 714
[  762.233439] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 715
[  762.233482] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 716
[  762.233525] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 717
[  762.233569] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 718
[  762.233612] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 719
[  762.233656] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 720
[  762.233699] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 721
[  762.233743] mpt3sas_cm0: iomem(0x00003fe080140000), mapped(0xd0000800813b0000), size(65536)
[  762.233806] mpt3sas_cm0: ioport(0x0000000000000000), size(0)
[  762.234135] mpt3sas_cm0: _base_event_notification: timeout
[  762.234182] mf:
	[  762.234204] 07000000 
00000000 [  762.234238] 00000000 
00000000 [  762.234272] 00000000 
0f2f7fff [  762.234305] ffffff7c 
ffffffff [  762.234339] 
[  762.234339] 	
ffffffff [  762.234384] 00000000 
00000000 [  762.234418] 
[  762.236160] Unable to handle kernel paging request for data at address 0xd0000800813b0030
[  762.236230] Faulting instruction address: 0xd000000031fb072c
[  762.236286] Oops: Kernel access of bad area, sig: 11 [#1]
[  762.236329] SMP NR_CPUS=1024 [  762.236351] NUMA 
[  762.236374] PowerNV
[  762.236399] Modules linked in: mpt3sas raid_class scsi_transport_sas vhost_net vhost macvtap macvlan ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_nat nf_conntrack libcrc32c at24 nvmem_core ofpart ipmi_powernv powernv_flash ipmi_msghandler opal_prd mtd i2c_opal kvm_hv nfsd kvm_pr auth_rpcgss oid_registry nfs_acl lockd kvm grace sunrpc joydev ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops i40e ttm ixgbe mdio ptp drm pps_core i2c_core [last unloaded: raid_class][  762.237373] CPU: 8 PID: 779 Comm: eehd Tainted: G        W       4.9.0-4.el7.centos.ppc64le #1
[  762.237448] task: c000003fcf301500 task.stack: c000003fcf384000
[  762.237501] NIP: d000000031fb072c LR: d000000031fb070c CTR: c000000000115490
[  762.237564] REGS: c000003fcf3874b0 TRAP: 0300   Tainted: G        W        (4.9.0-4.el7.centos.ppc64le)
[  762.237638] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>[  762.237811]   CR: 24002084  XER: 20000000
[  762.237844] CFAR: c000000000a276a8 DAR: d0000800813b0030 DSISR: 40000000 SOFTE: 1 
GPR00: d000000031fb070c c000003fcf387730 d000000031fef390 d0000800813b0030 
GPR04: c000003fcf301500 0000000003fde404 00000060e3c47241 0000000000000000 
GPR08: c000003fed20ed00 d0000800813b0000 0000000000000000 00000000ffffffff 
GPR12: 0000000000002200 c00000000fdc4800 c0000000000fbd18 c000007949100040 
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
GPR24: c000003fcf387920 0000000000000003 0000000000000005 0000000040000000 
GPR28: 0000000000001388 00000000c0000000 c000001f04e84810 0000000000000001 
NIP [d000000031fb072c] _base_wait_for_doorbell_ack+0x8c/0x1f0 [mpt3sas]
[  762.238822] LR [d000000031fb070c] _base_wait_for_doorbell_ack+0x6c/0x1f0 [mpt3sas]
[  762.238886] Call Trace:
[  762.238914] [c000003fcf387730] [d000000031fb070c] _base_wait_for_doorbell_ack+0x6c/0x1f0 [mpt3sas] (unreliable)
[  762.239015] [c000003fcf3877c0] [d000000031fb1c6c] _base_handshake_req_reply_wait+0x15c/0x7e0 [mpt3sas]
[  762.243871] [c000003fcf387880] [d000000031fb689c] _base_get_ioc_facts+0x10c/0x460 [mpt3sas]
[  762.250568] mpt3sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:8830/_scsih_probe()!
[  762.260515] [c000003fcf387950] [d000000031fb96d8] mpt3sas_base_hard_reset_handler+0x2c8/0x600 [mpt3sas]
[  762.270219] [c000003fcf387a30] [d000000031fbeba4] scsih_pci_slot_reset+0xa4/0x100 [mpt3sas]
[  762.278537] [c000003fcf387ab0] [c000000000042d48] eeh_report_reset+0x128/0x170
[  762.285474] [c000003fcf387b00] [c000000000041128] eeh_pe_dev_traverse+0x98/0x170
[  762.292412] [c000003fcf387b90] [c00000000004347c] eeh_handle_normal_event+0x3ec/0x510
[  762.300722] [c000003fcf387c30] [c000000000043858] eeh_handle_event+0x178/0x360
[  762.307665] [c000003fcf387ce0] [c000000000043bf8] eeh_event_handler+0x1b8/0x1c0
[  762.314598] [c000003fcf387d80] [c0000000000fbe20] kthread+0x110/0x130
[  762.321520] [c000003fcf387e30] [c00000000000c360] ret_from_kernel_thread+0x5c/0x7c
[  762.328465] Instruction dump:
[  762.332603] 40820074 386003e8 388005dc 48028219 e8410018 393f0001 7f9c4840 793f0020 
[  762.339539] 41de010c e93e00a8 38690030 7c0004ac <81290030> 0c090000 4c00012c 2f89ffff 
[  762.430835] ---[ end trace ee34b74dd6657653 ]---
[  762.430881]

$ ./pest 8000302500000000 8000000061240000
Transaction type: DMA Write
TCE Page Fault
TCE Access Fault
LEM Bit Number 37
Requestor 0:0.0
MSI Data 0x0000
Fault Address = 0x0000000061240000

Power8: Host stuck during booting with latest devel branch(4.16.0-1.rc7.dev.git58079f0.el7)

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=166290 </cde:info>

Boot power8 with latest devel branch kernel 4.16.0-1.rc7.dev.git58079f0.el7

[   39.095205] systemd[1]: Detected architecture ppc64-le.
[   39.095264] systemd[1]: Running in initial RAM disk.
[   39.095454] systemd[1]: Set hostname to <ltc-test-ci2.aus.stglabs.ibm.com>.
[   39.148724] systemd[1]: Cannot add dependency job for unit blk-availability.service, ignoring: Unit not found.
[   39.149958] systemd[1]: Created slice Root Slice.
[   39.150027] systemd[1]: Starting Root Slice.
[   39.150187] systemd[1]: Listening on Journal Socket.
[   39.150249] systemd[1]: Starting Journal Socket.
[   39.150341] systemd[1]: Reached target Timers.
[   39.417008] tg3.c:v3.137 (May 11, 2014)
[   39.417068] pci 0005:02:09.0: enabling device (0141 -> 0143)
[   39.417144] tg3 0005:05:00.0: enabling device (0140 -> 0142)
[   39.419020] synth uevent: /devices/vio: failed to send uevent
[   39.419026] vio vio: uevent: failed to send synthetic uevent
[  OK  ] Started Device-Mapper Multipath Device Controller.
[  OK  ] Started Show Plymouth Boot Screen.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Basic System.
[   39.449997] tg3 0005:05:00.0: Using 64-bit DMA iommu bypass
[   39.450672] tg3 0005:05:00.0 eth0: Tigon3 [partno(00RX892) rev 5719001] (PCI Express) MAC address 98:be:94:02:8f:64
[   39.450780] tg3 0005:05:00.0 eth0: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[   39.450885] tg3 0005:05:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   39.450965] tg3 0005:05:00.0 eth0: dma_rwctrl[00000000] dma_mask[64-bit]
[   39.451214] tg3 0005:05:00.1: enabling device (0140 -> 0142)
[   39.455812] device-mapper: multipath service-time: version 0.3.0 loaded
[   39.491062] tg3 0005:05:00.1: Using 64-bit DMA iommu bypass
[   39.491705] tg3 0005:05:00.1 eth1: Tigon3 [partno(00RX892) rev 5719001] (PCI Express) MAC address 98:be:94:02:8f:65
[   39.491825] tg3 0005:05:00.1 eth1: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[   39.491930] tg3 0005:05:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   39.492010] tg3 0005:05:00.1 eth1: dma_rwctrl[00000000] dma_mask[64-bit]
[   39.492252] tg3 0005:05:00.2: enabling device (0140 -> 0142)
[   39.501713] WARNING: CPU: 37 PID: 1241 at kernel/workqueue.c:1513 __queue_delayed_work+0xc8/0xf0
[   39.501807] Modules linked in: dm_service_time dm_multipath tg3(+)
[   39.501882] CPU: 37 PID: 1241 Comm: systemd-udevd Not tainted 4.16.0-1.rc7.dev.git58079f0.el7.centos.ppc64le #1
[   39.501978] NIP:  c00000000012d278 LR: c00000000012d2fc CTR: c00000000012d2a0
[   39.502049] REGS: c0000007ee793870 TRAP: 0700   Not tainted  (4.16.0-1.rc7.dev.git58079f0.el7.centos.ppc64le)
[   39.502143] MSR:  9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 8a002884  XER: 00000000
[   39.502220] CFAR: c00000000012d1e4 SOFTE: 1 
[   39.502220] GPR00: c00000000012d2fc c0000007ee793af0 c00000000146a600 c0000007f5690710 
[   39.502220] GPR04: c000000002fd0400 c0000007f56906f0 0000000000000000 0000000000000001 
[   39.502220] GPR08: 0000000000000000 c00000000012d170 0000000000000400 d00000000b9750d8 
[   39.502220] GPR12: c00000000012d2a0 c00000000fd59700 0000000100093ec0 0000000000000000 
[   39.502220] GPR16: 0000000100091300 0000000100091380 00000001000d0900 0000000100092f10 
[   39.502220] GPR20: 00000001000d0030 000001000e824667 000001000e8247a7 0000000000000007 
[   39.502220] GPR24: 0000000000000000 0000000000000000 c0000007ee793ca8 c0000007ee793ca0 
[   39.502220] GPR28: 0000000000000000 c0000007f15a0170 0000000000000000 0000000000000001 
[   39.502826] NIP [c00000000012d278] __queue_delayed_work+0xc8/0xf0
[   39.502886] LR [c00000000012d2fc] queue_delayed_work_on+0x5c/0x90
[   39.502946] Call Trace:
[   39.502971] [c0000007ee793af0] [c00000000012d2fc] queue_delayed_work_on+0x5c/0x90 (unreliable)
[   39.503057] [c0000007ee793b20] [d00000000b970cb8] __pg_init_all_paths+0x108/0x190 [dm_multipath]
[   39.503141] [c0000007ee793b60] [d00000000b970d8c] pg_init_all_paths+0x4c/0x80 [dm_multipath]
[   39.503225] [c0000007ee793ba0] [d00000000b972ac8] multipath_prepare_ioctl+0x138/0x150 [dm_multipath]
[   39.503310] [c0000007ee793bf0] [c0000000008e6900] dm_get_bdev_for_ioctl+0x120/0x1b0
[   39.503382] [c0000007ee793c40] [c0000000008e6d80] dm_blk_ioctl+0x50/0x110
[   39.503443] [c0000007ee793cc0] [c000000000595794] blkdev_ioctl+0x5f4/0xb80
[   39.503505] [c0000007ee793d20] [c0000000003df5c4] block_ioctl+0x54/0xa0
[   39.503566] [c0000007ee793d40] [c0000000003a02a4] do_vfs_ioctl+0xd4/0x8c0
[   39.503626] [c0000007ee793de0] [c0000000003a0b64] SyS_ioctl+0xd4/0x130
[   39.503688] [c0000007ee793e30] [c00000000000b8e0] system_call+0x58/0x6c
[   39.503748] Instruction dump:
[   39.503785] e8010010 7c0803a6 4e800020 60000000 60000000 60420000 7d435378 4bfff8c4 
[   39.503858] 0fe00000 4bffff98 0fe00000 4bffff80 <0fe00000> 4bffff6c 0fe00000 4bffff50 
[   39.503933] ---[ end trace 47786a0f55475f74 ]---
[   39.503985] WARNING: CPU: 37 PID: 1241 at kernel/workqueue.c:1515 __queue_delayed_work+0xb8/0xf0
[   39.504067] Modules linked in: dm_service_time dm_multipath tg3(+)
[   39.504130] CPU: 37 PID: 1241 Comm: systemd-udevd Tainted: G        W        4.16.0-1.rc7.dev.git58079f0.el7.centos.ppc64le #1
[   39.504235] NIP:  c00000000012d268 LR: c00000000012d2fc CTR: c00000000012d2a0
[   39.504307] REGS: c0000007ee793870 TRAP: 0700   Tainted: G        W         (4.16.0-1.rc7.dev.git58079f0.el7.centos.ppc64le)
[   39.504412] MSR:  9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 8a002884  XER: 00000000
[   39.504487] CFAR: c00000000012d200 SOFTE: 1 
[   39.504487] GPR00: c00000000012d2fc c0000007ee793af0 c00000000146a600 c0000007f5690710 
[   39.504487] GPR04: c000000002fd0400 c0000007f56906f0 0000000000000000 0000000000000001 
[   39.504487] GPR08: 0000000000000000 c0000007f56906f8 0000000000000400 d00000000b9750d8 
[   39.504487] GPR12: c00000000012d2a0 c00000000fd59700 0000000100093ec0 0000000000000000 
[   39.504487] GPR16: 0000000100091300 0000000100091380 00000001000d0900 0000000100092f10 
[   39.504487] GPR20: 00000001000d0030 000001000e824667 000001000e8247a7 0000000000000007 
[   39.504487] GPR24: 0000000000000000 0000000000000000 c0000007ee793ca8 c0000007ee793ca0 
[   39.504487] GPR28: 0000000000000000 c0000007f15a0170 0000000000000000 0000000000000001 
[   39.505091] NIP [c00000000012d268] __queue_delayed_work+0xb8/0xf0
[   39.505150] LR [c00000000012d2fc] queue_delayed_work_on+0x5c/0x90
[   39.505209] Call Trace:
[   39.505235] [c0000007ee793af0] [c00000000012d2fc] queue_delayed_work_on+0x5c/0x90 (unreliable)
[   39.505320] [c0000007ee793b20] [d00000000b970cb8] __pg_init_all_paths+0x108/0x190 [dm_multipath]
[   39.505403] [c0000007ee793b60] [d00000000b970d8c] pg_init_all_paths+0x4c/0x80 [dm_multipath]
[   39.505488] [c0000007ee793ba0] [d00000000b972ac8] multipath_prepare_ioctl+0x138/0x150 [dm_multipath]
[   39.505571] [c0000007ee793bf0] [c0000000008e6900] dm_get_bdev_for_ioctl+0x120/0x1b0
[   39.505644] [c0000007ee793c40] [c0000000008e6d80] dm_blk_ioctl+0x50/0x110
[   39.505705] [c0000007ee793cc0] [c000000000595794] blkdev_ioctl+0x5f4/0xb80
[   39.505766] [c0000007ee793d20] [c0000000003df5c4] block_ioctl+0x54/0xa0
[   39.505826] [c0000007ee793d40] [c0000000003a02a4] do_vfs_ioctl+0xd4/0x8c0
[   39.505887] [c0000007ee793de0] [c0000000003a0b64] SyS_ioctl+0xd4/0x130
[   39.505947] [c0000007ee793e30] [c00000000000b8e0] system_call+0x58/0x6c
[   39.506007] Instruction dump:
[   39.506043] 40de0050 4807fbed 60000000 38210020 e8010010 7c0803a6 4e800020 60000000 
[   39.506117] 60000000 60420000 7d435378 4bfff8c4 <0fe00000> 4bffff98 0fe00000 4bffff80 
[   39.506191] ---[ end trace 47786a0f55475f75 ]---
[   39.506242] WARNING: CPU: 37 PID: 1241 at kernel/workqueue.c:1444 __queue_work+0x160/0x5c0
[   39.506313] Modules linked in: dm_service_time dm_multipath tg3(+)
[   39.506375] CPU: 37 PID: 1241 Comm: systemd-udevd Tainted: G        W        4.16.0-1.rc7.dev.git58079f0.el7.centos.ppc64le #1
[   39.506481] NIP:  c00000000012cc80 LR: c00000000012cc54 CTR: c00000000012d2a0
[   39.506552] REGS: c0000007ee793790 TRAP: 0700   Tainted: G        W         (4.16.0-1.rc7.dev.git58079f0.el7.centos.ppc64le)
[   39.506657] MSR:  9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 2a002844  XER: 00000000
[   39.506732] CFAR: c000000000b637b4 SOFTE: 1 
[   39.506732] GPR00: c00000000012cc54 c0000007ee793a10 c00000000146a600 c0000007fc210800 
[   39.506732] GPR04: c0000007f56906f0 0000000000000000 0000000000000000 c000000001499d70 
[   39.506732] GPR08: 0000000000000000 0000000000000001 0000000000000000 d00000000b9750d8 
[   39.506732] GPR12: c00000000012d2a0 c00000000fd59700 0000000100093ec0 0000000000000000 
[   39.506732] GPR16: 0000000100091300 0000000100091380 c000000000d97f30 0000000000000001 
[   39.506732] GPR20: 0000000000000000 c000000000fbc8a8 c000000001624bd8 0000000000000000 
[   39.506732] GPR24: c000000001624bd0 c0000007ff526e00 0000000000000025 c000000000fbc8a8 
[   39.506732] GPR28: 0000000000000400 c000000002fd0400 c0000007f56906f0 c0000007e75b0000 
[   39.507336] NIP [c00000000012cc80] __queue_work+0x160/0x5c0
[   39.507384] LR [c00000000012cc54] __queue_work+0x134/0x5c0
[   39.507432] Call Trace:
[   39.507457] [c0000007ee793a10] [c00000000012cc54] __queue_work+0x134/0x5c0 (unreliable)
[   39.507530] [c0000007ee793af0] [c00000000012d2fc] queue_delayed_work_on+0x5c/0x90
[   39.507603] [c0000007ee793b20] [d00000000b970cb8] __pg_init_all_paths+0x108/0x190 [dm_multipath]
[   39.507687] [c0000007ee793b60] [d00000000b970d8c] pg_init_all_paths+0x4c/0x80 [dm_multipath]
[   39.507771] [c0000007ee793ba0] [d00000000b972ac8] multipath_prepare_ioctl+0x138/0x150 [dm_multipath]
[   39.507855] [c0000007ee793bf0] [c0000000008e6900] dm_get_bdev_for_ioctl+0x120/0x1b0
[   39.507927] [c0000007ee793c40] [c0000000008e6d80] dm_blk_ioctl+0x50/0x110
[   39.507988] [c0000007ee793cc0] [c000000000595794] blkdev_ioctl+0x5f4/0xb80
[   39.508049] [c0000007ee793d20] [c0000000003df5c4] block_ioctl+0x54/0xa0
[   39.508109] [c0000007ee793d40] [c0000000003a02a4] do_vfs_ioctl+0xd4/0x8c0
[   39.508170] [c0000007ee793de0] [c0000000003a0b64] SyS_ioctl+0xd4/0x130
[   39.508231] [c0000007ee793e30] [c00000000000b8e0] system_call+0x58/0x6c
[   39.508290] Instruction dump:
[   39.508326] 48a36b09 60000000 813f0018 2f890000 41de0314 60000000 7fc9f378 e9490009 
[   39.508400] 7d295278 7d290074 7929d182 69290001 <0b090000> 2fa90000 40de0360 815f0010 
[   39.508474] ---[ end trace 47786a0f55475f76 ]---
[   39.530965] tg3 0005:05:00.2: Using 64-bit DMA iommu bypass
[   39.531375] tg3 0005:05:00.2 eth2: Tigon3 [partno(00RX892) rev 5719001] (PCI Express) MAC address 98:be:94:02:8f:66
[   39.531479] tg3 0005:05:00.2 eth2: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[   39.531581] tg3 0005:05:00.2 eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   39.531657] tg3 0005:05:00.2 eth2: dma_rwctrl[00000000] dma_mask[64-bit]
[   39.531900] tg3 0005:05:00.3: enabling device (0140 -> 0142)
[   39.570962] tg3 0005:05:00.3: Using 64-bit DMA iommu bypass
[   39.571363] tg3 0005:05:00.3 eth3: Tigon3 [partno(00RX892) rev 5719001] (PCI Express) MAC address 98:be:94:02:8f:67
[   39.571469] tg3 0005:05:00.3 eth3: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[   39.571572] tg3 0005:05:00.3 eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] TSOcap[1]
[   39.571649] tg3 0005:05:00.3 eth3: dma_rwctrl[00000000] dma_mask[64-bit]
[   39.573648] tg3 0005:05:00.0 enP5p5s0f0: renamed from eth0
[   39.762117] tg3 0005:05:00.3 enP5p5s0f3: renamed from eth3
[   39.822009] tg3 0005:05:00.1 enP5p5s0f1: renamed from eth1
[   39.911987] tg3 0005:05:00.2 enP5p5s0f2: renamed from eth2

open-power-host-os / linux Goto Github PK

linux's Introduction

OpenPOWER Host OS web page

linux's People

Contributors

Stargazers

Watchers

Forkers

linux's Issues

uname -a

virsh attach-device fedora24-san hotadd-nic.xml --live

lspci -nn

virsh suspend fedora24-san

virsh resume fedora24-san

Recommend Projects

Recommend Topics

Recommend Org