Console with modified backend about cosmos HOT 5 CLOSED

Hi,

Does it get stuck there on:

"Mapping TX rings
0 map errors"

It doesn't show anything else?

Could you tell me your hardware specifications? and the output of these commands:

xl vcpu-list
xl info
xl dmesg
dmesg

I indeed have heard of cases where it gets stuck on mapping phase being unable to map the RX rings. But I was unable to reproduce it in my hardware.

Thanks,
Cheers,
Joao

from cosmos.

Hi Joao,

Thanks for your prompt reply. It does get stuck at:

"Mapping TX rings
0 map errors"

and shows nothing else. The hardware specs are as follows:

CPU: Intel(R) Xeon(R) CPU E5-1607 v2 @ 3.00 GHz (4-core) (3 assigned to Dom-0)
RAM: 16 GB (with 8 GB assigned to Dom-0)
Network Card: Intel Corporation 82599 10 Gigabit TN Network Connection (dual-port)

Let me know if you require any other hardware specification.

The output to "xl vcpu-list" is:

Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 0 -b- 511.7 0
Domain-0 0 1 1 r-- 397.9 1
Domain-0 0 2 2 -b- 410.2 2
clickos 7 0 3 r-- 367.0 3

The output to "xl info" is:

Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 0 -b- 511.7 0
Domain-0 0 1 1 r-- 397.9 1
Domain-0 0 2 2 -b- 410.2 2
clickos 7 0 3 r-- 367.0 3
nfv1@nfv1-Precision-T3610:~/Documents/nfv-project$ sudo xl info
host : nfv1-Precision-T3610
release : 3.13.0-37-generic
version : #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014
machine : x86_64
nr_cpus : 4
max_cpu_id : 3
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 1
cpu_mhz : 2992
hw_caps : bfebfbff:2c100800:00000000:00006f00:77bee3ff:00000000:00000001:00000281
virt_caps : hvm hvm_directio
total_memory : 16328
free_memory : 7908
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 4
xen_extra : .0
xen_version : 4.4.0
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset :
xen_commandline : placeholder dom0_mem=8192M,max:8192M dom0_max_vcpus=3 dom0_vcpus_pin
cc_compiler : gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2
cc_compile_by : stefan.bader
cc_compile_domain : canonical.com
cc_compile_date : Tue Sep 30 19:19:31 UTC 2014
xend_config_format : 4

The output of "xl dmesg" is:

(XEN) Xen version 4.4.0 (Ubuntu 4.4.0-0ubuntu5.2) ([email protected]) (gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2) debug=n Tue Sep 30 19:19:31 UTC 2014
(XEN) Bootloader: GRUB 2.02~beta2-9ubuntu1
(XEN) Command line: placeholder dom0_mem=8192M,max:8192M dom0_max_vcpus=3 dom0_vcpus_pin
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: none; EDID transfer time: 0 seconds
(XEN) EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN) Found 1 MBR signatures
(XEN) Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 0000000000095400 (usable)
(XEN) 0000000000095400 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 0000000087ed6000 (usable)
(XEN) 0000000087ed6000 - 0000000088000000 (reserved)
(XEN) 0000000088000000 - 0000000088775000 (usable)
(XEN) 0000000088775000 - 0000000088800000 (reserved)
(XEN) 0000000088800000 - 0000000088ee9000 (usable)
(XEN) 0000000088ee9000 - 0000000089000000 (ACPI data)
(XEN) 0000000089000000 - 000000008a5a8000 (usable)
(XEN) 000000008a5a8000 - 000000008a800000 (ACPI NVS)
(XEN) 000000008a800000 - 000000008c594000 (usable)
(XEN) 000000008c594000 - 000000008c743000 (reserved)
(XEN) 000000008c743000 - 000000008c766000 (usable)
(XEN) 000000008c766000 - 000000008c7ec000 (ACPI NVS)
(XEN) 000000008c7ec000 - 000000008d000000 (usable)
(XEN) 0000000090000000 - 00000000a0000000 (reserved)
(XEN) 00000000fed1c000 - 00000000fed40000 (reserved)
(XEN) 00000000ff000000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000000470000000 (usable)
(XEN) ACPI: RSDP 000F0490, 0024 (r2 DELL )
(XEN) ACPI: XSDT 88F29088, 008C (r1 DELL CBX3 1072009 AMI 10013)
(XEN) ACPI: FACP 88F32130, 010C (r5 DELL CBX3 1072009 AMI 10013)
(XEN) ACPI: DSDT 88F291A0, 8F8C (r2 DELL CBX3 23 INTL 20091112)
(XEN) ACPI: FACS 8A7F7080, 0040
(XEN) ACPI: APIC 88F32240, 0090 (r3 DELL CBX3 1072009 AMI 10013)
(XEN) ACPI: FPDT 88F322D0, 0044 (r1 DELL CBX3 1072009 AMI 10013)
(XEN) ACPI: MCFG 88F32318, 003C (r1 DELL OEMMCFG. 1072009 MSFT 97)
(XEN) ACPI: HPET 88F32358, 0038 (r1 DELL CBX3 1072009 AMI. 5)
(XEN) ACPI: ASF! 88F32390, 00A0 (r32 INTEL HCG 1 TFSM F4240)
(XEN) ACPI: SSDT 88F32430, CD380 (r2 INTEL CpuPm 4000 INTL 20051117)
(XEN) ACPI: EINJ 88FFF7B0, 0130 (r1 AMI AMI EINJ 0 0)
(XEN) ACPI: ERST 88FFF8E0, 0230 (r1 AMIER AMI ERST 0 0)
(XEN) ACPI: HEST 88FFFB10, 00A8 (r1 AMI AMI HEST 0 0)
(XEN) ACPI: BERT 88FFFBB8, 0030 (r1 AMI AMI BERT 0 0)
(XEN) ACPI: DMAR 88FFFBE8, 00B4 (r1 A M I OEMDMAR 1 INTL 1)
(XEN) ACPI: SLIC 88FFFCA0, 0176 (r3 DELL CBX3 1072009 MSFT 10013)
(XEN) System RAM: 16328MB (16720112kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - 8a7f7080/0000000000000000, using 32
(XEN) Processor #0 7:14 APIC version 21
(XEN) Processor #2 7:14 APIC version 21
(XEN) Processor #4 7:14 APIC version 21
(XEN) Processor #6 7:14 APIC version 21
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 2, version 32, address 0xfec01000, GSI 24-47
(XEN) Enabling APIC mode: Flat. Using 2 I/O APICs
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2992.900 MHz processor.
(XEN) Initing memory sharing.
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using old ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) - APIC Register Virtualization
(XEN) - Virtual Interrupt Delivery
(XEN) - Posted Interrupt Processing
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 4 CPUs
(XEN) verify_tsc_reliability: TSC warp detected, disabling TSC_RELIABLE
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x2403000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 000000045c000000->0000000460000000 (2076086 pages to be allocated)
(XEN) Init. ramdisk: 000000046edb6000->000000046ffff6ce
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff81000000->ffffffff82403000
(XEN) Init. ramdisk: ffffffff82403000->ffffffff8364c6ce
(XEN) Phys-Mach map: ffffffff8364d000->ffffffff8464d000
(XEN) Start info: ffffffff8464d000->ffffffff8464d4b4
(XEN) Page tables: ffffffff8464e000->ffffffff84675000
(XEN) Boot stack: ffffffff84675000->ffffffff84676000
(XEN) TOTAL: ffffffff80000000->ffffffff84800000
(XEN) ENTRY ADDRESS: ffffffff81d361f0
(XEN) Dom0 has maximum 3 VCPUs
(XEN) Scrubbing Free RAM: ................................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 280kB init memory.

The output of "dmesg" is:

[65497.038538] xen_netback/xen_netback:xenvif_netmap_unmap: vif6.0: revoking 1034 pages
[65497.038542] xen:grant_table: WARNING: g.e. 0x207 still in use!
[65497.038544] deferring g.e. 0x207 (pfn 0xffffffffffffffff)
[65497.038545] xen:grant_table: WARNING: g.e. 0x40e still in use!
[65497.038546] deferring g.e. 0x40e (pfn 0xffffffffffffffff)
[65497.038547] xen:grant_table: WARNING: g.e. 0x40f still in use!
[65497.038548] deferring g.e. 0x40f (pfn 0xffffffffffffffff)
[65497.038548] xen:grant_table: WARNING: g.e. 0x410 still in use!
[65497.038549] deferring g.e. 0x410 (pfn 0xffffffffffffffff)
[65497.038550] xen:grant_table: WARNING: g.e. 0x411 still in use!
[65497.038551] deferring g.e. 0x411 (pfn 0xffffffffffffffff)
[65497.109658] 085.609299 [ 719] nm_bdg_detach deleting priv
[65497.120414] xen:balloon: Cannot add additional memory (-17)
[65498.035669] xen:grant_table: freeing g.e. 0x207
[65498.035675] xen:grant_table: freeing g.e. 0x40e
[65498.035677] xen:grant_table: freeing g.e. 0x40f
[65498.035680] xen:grant_table: freeing g.e. 0x410
[65498.035682] xen:grant_table: freeing g.e. 0x411
[65497.038633] xen_netback/xen_netback:xenvif_netmap_detach: netmap_detach vif6.0
[65522.831735] 111.331376 [2259] netmap_attach success for vif7.0
[65522.844806] IPv6: ADDRCONF(NETDEV_UP): vif7.0: link is not ready
[65522.845175] IPv6: ADDRCONF(NETDEV_UP): vif7.0: link is not ready
[65522.898519] 111.398160 [1456] netmap_set_ringid deprecated API, old ringid 0x0 -> ringid 0 reg 1
[65522.901717] xen_netback/xen_netback:xenvif_netmap_map: vif7.0: granted 10 1024 (ring/buffers)

Best wishes,
Hassan

from cosmos.

Hey Hassan,

Everything looks ok on the backend side and configurations.

Could you try booting the VM in one of the cores of Domain-0 ? For example in core 2 instead of core 3, then pinning it back to core 3.

Thanks,
Cheers

from cosmos.

HI Joao,

Great, it worked. I specified the cpu in the .cfg file and booted the VM. The console this time didn't get stuck. After that I pinned it back to core 3 by:

xl vcpu-pin clickos 0 3

Thanks a lot! Any reason why this worked?

from cosmos.

Not sure yet but will let you know when I fix the issue, in mini-os. Thanks a lot!

from cosmos.