verify_qp_creation() QP creation failed on ConnectX-6 IPoIB interface,about mellanox/libvma

igor-ivanov commented on August 17, 2024

Hello, @Honggang-LI ipoib is not supported in li ma now. So it looks as valid warning.

from libvma.

Honggang-LI commented on August 17, 2024

Hello, @Honggang-LI ipoib is not supported in li ma now. So it looks as valid warning.

Do you imply connectx-6 RoCE will work?

QP creation failed because ibv_create_qp_ex was called with source_qpn. How to call ibv_create_qp_ex with source_qpn?

Thanks

from libvma.

Honggang-LI commented on August 17, 2024

Hello, @Honggang-LI ipoib is not supported in li ma now.
Why ipoib was disabled?

from libvma.

igor-ivanov commented on August 17, 2024

QP creation failed because ibv_create_qp_ex was called with source_qpn. How to call ibv_create_qp_ex with source_qpn?

Thanks

libvma/src/vma/ib/base/verbs_extra.h

Lines 122 to 134 in 48ec52f

    
           #ifdef DEFINED_IBV_QP_INIT_SOURCE_QPN 
        
           #define vma_ibv_create_qp(pd, attr)                ibv_create_qp_ex((pd)->context, attr) 
        
           typedef struct ibv_qp_init_attr_ex                 vma_ibv_qp_init_attr; 
        
           #define vma_ibv_qp_create_flags(attr)              (attr).create_flags 
        
           #define vma_ibv_qp_source_qpn(attr)                (attr).source_qpn 
        
           #define VMA_IBV_QP_INIT_QPN_CREATE_FLAGS           IBV_QP_CREATE_SOURCE_QPN 
        
           #define VMA_IBV_QP_INIT_QPN_MASK                   IBV_QP_INIT_ATTR_CREATE_FLAGS 
        
           #define vma_ibv_qp_init_attr_comp_mask(_pd, _attr) { (_attr).pd = _pd; (_attr).comp_mask |= IBV_QP_INIT_ATTR_PD; } 
        
           #else 
        
           #define vma_ibv_create_qp(pd, attr)                 ibv_create_qp(pd, attr) 
        
           typedef struct ibv_qp_init_attr                     vma_ibv_qp_init_attr; 
        
           #define vma_ibv_qp_init_attr_comp_mask(_pd, _attr)  { NOT_IN_USE(_pd); NOT_IN_USE(_attr); } 
        
           #endif

https://docs.nvidia.com/networking/display/VMAv952/Changes+and+New+Features

IPoIB is temporarily unavailable when working with MLNX_OFED v5.1 and above.

from libvma.

Honggang-LI commented on August 17, 2024

https://docs.nvidia.com/networking/display/VMAv952/Changes+and+New+Features
IPoIB is temporarily unavailable when working with MLNX_OFED v5.1 and above.

It sounds connectx-6 kernel space driver does not support source_qpn. I tried rhel-8.2 inbox rdma stack and rhel-8.2 + MLNX_OFED_LINUX-5.0-2.1.8.0-rhel8.2-x86_64.iso . Tested the connectx-6 device with libibverbs-utils, it works as expected . But libvma over connectx-6 IPoIB never worked.

from libvma.

DanielLibenson commented on August 17, 2024

Hi Honggang,

IPoIB is not working since VMA v9.1.1 (OFED v5.1) and above.
Legacy implementation of IPoIB was based on mlnx libraries (experimental verbs) and current implementation is based on rdma-core.
Due to driver issue, IPoIB is not working with rdma-core (only for user space).

Regarding VMA legacy implementation with mlnx libraries:
MLNX_OFED-4.9 and below: supported by default.
MLNX_OFED-5.0 need to provide --mlnx-libs option to the installation script (rdma-core is the default)
MLNX_OFED-5.1 and above: is not supported.

Daniel

from libvma.

igor-ivanov commented on August 17, 2024

@Honggang-LI can it be closed?

from libvma.

Honggang-LI commented on August 17, 2024

@Honggang-LI can it be closed?

Thanks for help. Close it.

from libvma.

verify_qp_creation() QP creation failed on ConnectX-6 IPoIB interface about libvma HOT 8 CLOSED

Comments (8)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	#ifdef DEFINED_IBV_QP_INIT_SOURCE_QPN
	#define vma_ibv_create_qp(pd, attr) ibv_create_qp_ex((pd)->context, attr)
	typedef struct ibv_qp_init_attr_ex vma_ibv_qp_init_attr;
	#define vma_ibv_qp_create_flags(attr) (attr).create_flags
	#define vma_ibv_qp_source_qpn(attr) (attr).source_qpn
	#define VMA_IBV_QP_INIT_QPN_CREATE_FLAGS IBV_QP_CREATE_SOURCE_QPN
	#define VMA_IBV_QP_INIT_QPN_MASK IBV_QP_INIT_ATTR_CREATE_FLAGS
	#define vma_ibv_qp_init_attr_comp_mask(_pd, _attr) { (_attr).pd = _pd; (_attr).comp_mask \|= IBV_QP_INIT_ATTR_PD; }
	#else
	#define vma_ibv_create_qp(pd, attr) ibv_create_qp(pd, attr)
	typedef struct ibv_qp_init_attr vma_ibv_qp_init_attr;
	#define vma_ibv_qp_init_attr_comp_mask(_pd, _attr) { NOT_IN_USE(_pd); NOT_IN_USE(_attr); }
	#endif