rdma-example's Introduction

RDMA programming example

Send 2 numbers to server then send back the sum

System required:

RDMA capable NIC, OFED driver with libverbs and librdmacm.

How to use:

Git clone this repository at both client and server

# git clone https://github.com/w180112/RDMA-example.git

Set iptables to make UDP port 4791 open

On client side,

# gcc -o client rdma_write_client.c -lrdmacm -libverbs
# ./client <servername or ip> <val1> <val2>

e.g.

# ./client 192.168.0.168 123 456

On server side,

# gcc -o server rdma_write_server.c -lrdmacm -libverbs
# ./server

Test environment

Mellanox Connectx-4 Lx with SRIOV enable
Rocky Linux 9.2, with kernel 5.14.0-284.11.1.el9_2.x86_64

rdma-example's People

Contributors

Stargazers

Watchers

rdma-example's Issues

No RDMA Write sync and faulty byte order conversion

There are two current existing issues to report here:

When client is posting the RDMA Write operation to write data to remote memory of server, the server gets no feedback about when the RDMA Write operation is finished. As a result, the server could likely process the line 188 in rdma_write_server.c
buf[0] = htonl(ntohl(buf[0]) + ntohl(buf[1]));
faster than the client can write the user-defined numbers to this memory. Potential solution: Add a RDMA send operation to sync server and client.
Line 166 in rdma_write_server.c:
rep_pdata.buf_va = htonl((uintptr_t)buf);
You need htonll() because .buf_va is 64 bytes defined.
Line 210 in rdma_write_client.c:
send_wr.wr.rdma.remote_addr = ntohl(server_pdata.buf_va);
You need ntohll() because .remote_addr is 64 bytes defined.

server blocked

the function prepare_recv_notify_before_using_rdma_write should be called before calling rdma_accept, otherwise the notification sent by the client may not be received in time, leading to a block in the server.

While loop won't stop running

The while loop in the rdma_write_server.c won't stop.

IBV_WR_RDMA_WRITE

The values val1 and val2 that are not being written on the server side when using the IBV_WR_RDMA_WRITE verb. However, when I use the IBV_WR_SEND verb on the client side, I am able to retrieve the values sent on the server side successfully.

I've also attempted to use the be64toh and htobe64 functions for the 64-bit address fields, but unfortunately, that didn't resolve the problem.

I would greatly appreciate any insights or suggestions you may have to help me troubleshoot this issue.
Thank you.

Recommend Projects