Development repository for RXE user space code. Soft RDMA over Ethernet (RoCE) Driver

Source

Kernel Space Driver

This repository contains a full kernel source tree, with the RoCE driver code located in the directory drivers/infiniband/hw/rxe.

Github: https://github.com/SoftRoCE/rxe-dev.git Active Branch: master-next User Space Library

Github: https://github.com/SoftRoCE/librxe-dev.git (this repository) Current Version: librxe-1.0.0 Build Instructions

Compile and install kernel:

Clone kernel git: git clone https://github.com/SoftRoCE/rxe-dev.git Compile kernel: Enter the source directory cd rxe-dev cp /boot/config-$(uname –r) .config make menuconfig Need to enable “Software RDMA over Ethernet (RoCE) driver” in category "Device Drivers -> Infiniband" Need to enable CONFIG_INFINIBAND_ADDR_TRANS=y and CONFIG_INFINIBAND_ADDR_TRANS_CONFIGFS=y in new config file .config make –j 32 make modules_install make install Verify that the new kernel entry is added (e.g. to grub); if not, need to add it manually. Boot with new kernel. Install user space library (librxe):

Install the following package (example shown using RedHat): yum install perl-Switch (name might vary according to distribution) Make sure that the following upstream user space libraries are installed: libibverbs libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils Compile and install user space library librxe: git clone https://github.com/SoftRoCE/librxe-dev.git cd librxe-dev ./configure --libdir=/usr/lib64/ --prefix= make make install Configure Soft-RoCE (RXE):

Load rdma_rxe kernel module using the rxe_cfg script included in the librxe RPM: rxe_cfg start (this might require sudo or root privileges) Create RXE device over network interface (e.g. eth0): rxe_cfg add eth0 Use the status command to display the current configuration: rxe_cfg status If configured successfully, you should see output similar to the following: Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
eth0 yes mlx4_en rxe0 1024 (3) If you are using a Mellanox HCA: Need to make sure that the mlx4_ib kernel module is not loaded (modprobe –rv mlx4_ib) in the soft-RoCE machine. Now you have an Infiniband device called “rxe0” that can be used to run any RoCE app.