Unnecessary overflow check

async-rdma

A framework for writing RDMA applications with high-level abstraction and asynchronous APIs.

It provides a few major components:

Tools for establishing connections with rdma endpoints such as RdmaBuilder.
High-level APIs for data transmission between endpoints including read, write, send, receive.
High-level APIs for rdma memory region management including alloc_local_mr, request_remote_mr, send_mr, receive_local_mr, receive_remote_mr.
A framework including agent and event_listener working behind APIs for memory region management and executing rdma requests such as post_send and poll.

The ChangeLog file contains a brief summary of changes for each release.

Environment Setup

This section is for RDMA novices who want to try this library.

You can skip if your Machines have been configured with RDMA.

Next we will configure the RDMA environment in an Ubuntu20.04 VM. If you are using another operating system distribution, please search and replace the relevant commands.

1. Check whether the current kernel supports RXE

Run the following command and if the CONFIG_RDMA_RXE = y or m, the current operating system supports RXE. If not you need to search how to recompile the kernel to support RDMA.

cat /boot/config-$(uname -r) | grep RXE

2. Install Dependencies

sudo apt install -y libibverbs1 ibverbs-utils librdmacm1 libibumad3 ibverbs-providers rdma-core libibverbs-dev iproute2 perftest build-essential net-tools git librdmacm-dev rdmacm-utils cmake libprotobuf-dev protobuf-compiler clang curl

3. Configure RDMA netdev

(1) Load kernel driver

modprobe rdma_rxe

(2) User mode RDMA netdev configuration.

sudo rdma link add rxe_0 type rxe netdev ens33

rxe_0 is the RDMA device name, and you can name it whatever you want. ens33 is the name of the network device. The name of the network device may be different in each VM, and we can see it by running command "ifconfig".

(3) Check the RDMA device state

Run the following command and check if the state is ACTIVE.

rdma link

(4) Test it

Ib_send_bw is a program used to test the bandwidth of RDMA SEND operations.

Run the following command in a terminal.

ib_send_bw -d rxe_0

And run the following command in another terminal.

ib_send_bw -d rxe_0 localhost

4. Install Rust

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
source $HOME/.cargo/env

5. Try an example

git clone https://github.com/datenlord/async-rdma.git
cd async-rdma
cargo run --example rpc

if run rpc example failed, you can try run it with sudo permission.

cargo build --example rpc
sudo ./target/debug/examples/rpc

Example

A simple example: client request a remote memory region and put data into this remote memory region by rdma write. And finally client send_mr to make server aware of this memory region. Server receive_local_mr, and then get data from this mr.

use async_rdma::{LocalMrReadAccess, LocalMrWriteAccess, RdmaBuilder};
use portpicker::pick_unused_port;
use std::{
    alloc::Layout,
    io::{self, Write},
    net::{Ipv4Addr, SocketAddrV4},
    time::Duration,
};

async fn client(addr: SocketAddrV4) -> io::Result<()> {
    let layout = Layout::new::<[u8; 8]>();
    let rdma = RdmaBuilder::default().connect(addr).await?;
    // alloc 8 bytes remote memory
    let mut rmr = rdma.request_remote_mr(layout).await?;
    // alloc 8 bytes local memory
    let mut lmr = rdma.alloc_local_mr(layout)?;
    // write data into lmr
    let _num = lmr.as_mut_slice().write(&[1_u8; 8])?;
    // write the second half of the data in lmr to the rmr
    rdma.write(&lmr.get(4..8).unwrap(), &mut rmr.get_mut(4..8).unwrap())
        .await?;
    // send rmr's meta data to the remote end
    rdma.send_remote_mr(rmr).await?;
    Ok(())
}

#[tokio::main]
async fn server(addr: SocketAddrV4) -> io::Result<()> {
    let rdma = RdmaBuilder::default().listen(addr).await?;
    // receive mr's meta data from client
    let lmr = rdma.receive_local_mr().await?;
    let data = *lmr.as_slice();
    println!("Data written by the client using RDMA WRITE: {:?}", data);
    assert_eq!(data, [[0_u8; 4], [1_u8; 4]].concat());
    Ok(())
}

#[tokio::main]
async fn main() {
    let addr = SocketAddrV4::new(Ipv4Addr::new(127, 0, 0, 1), pick_unused_port().unwrap());
    std::thread::spawn(move || server(addr));
    tokio::time::sleep(Duration::new(1, 0)).await;
    client(addr)
        .await
        .map_err(|err| println!("{}", err))
        .unwrap();
}

Getting Help

First, see if the answer to your question can be found in the found API doc or Design doc. If the answer is not here, please open an issue and describe your problem in detail.

Related Projects

rdma-sys: Rust bindings for RDMA fundamental libraries: libibverbs-dev and librdmacm-dev.

	/// Creat a random u64 id.
	///
	/// Both `WorkRequetId` and `AgentRequestId` depend on this, so make this fn independent.
	/// To avoid id duplication, this fn concatenates `SystemTime` and random number into a U64.
	/// The syscall may have some overhead, which can be improved later by balancing the pros and cons.
	pub(crate) fn random_u64() -> u64 {
	let start = SystemTime::now();
	// No time can be earlier than Unix Epoch
	#[allow(clippy::unwrap_used)]
	let since_the_epoch = start.duration_since(UNIX_EPOCH).unwrap();
	let time = since_the_epoch.subsec_micros();
	let rand = rand::thread_rng().gen::<u32>();
	let left: u64 = time.into();
	let right: u64 = rand.into();
	left.overflow_shl(32) \| right
	}

	/// Get a local mr slice
	#[inline]
	pub fn get(&self, i: Range<usize>) -> io::Result<LocalMrSlice> {
	// SAFETY: `self` is checked to be valid and in bounds above.
	if i.start >= i.end \|\| i.end > self.length() {
	Err(io::Error::new(io::ErrorKind::Other, "wrong range of lmr"))
	} else {
	Ok(LocalMrSlice::new(
	self,
	self.addr().overflow_add(i.start),
	i.len(),
	))
	}
	}

	/// Get a mutable local mr slice
	#[inline]
	pub fn get_mut(&mut self, i: Range<usize>) -> io::Result<LocalMrSliceMut> {
	// SAFETY: `self` is checked to be valid and in bounds above.
	if i.start >= i.end \|\| i.end > self.length() {
	Err(io::Error::new(io::ErrorKind::Other, "wrong range of lmr"))
	} else {
	Ok(LocalMrSliceMut::new(
	self,
	self.addr().overflow_add(i.start),
	i.len(),
	))
	}
	}

	/// Get a remote mr slice
	#[inline]
	pub fn get(&self, i: Range<usize>) -> io::Result<RemoteMrSlice> {
	// SAFETY: `self` is checked to be valid and in bounds above.
	if i.start >= i.end \|\| i.end > self.length() {
	Err(io::Error::new(io::ErrorKind::Other, "wrong range of rmr"))
	} else {
	let slice_token = MrToken {
	addr: self.addr().overflow_add(i.start),
	len: i.end.overflow_sub(i.start),
	rkey: self.rkey(),
	};
	Ok(RemoteMrSlice::new_from_token(self, slice_token))
	}
	}

	/// Get a mutable remote mr slice
	#[inline]
	pub fn get_mut(&mut self, i: Range<usize>) -> io::Result<RemoteMrSliceMut> {
	// SAFETY: `self` is checked to be valid and in bounds above.
	if i.start >= i.end \|\| i.end > self.length() {
	Err(io::Error::new(io::ErrorKind::Other, "wrong range of rmr"))
	} else {
	let slice_token = MrToken {
	addr: self.addr().overflow_add(i.start),
	len: i.end.overflow_sub(i.start),
	rkey: self.rkey(),
	};
	Ok(RemoteMrSliceMut::new_from_token(self, slice_token))
	}
	}

	let end = (start.overflow_add(self.max_sr_data_len)).min(lm_len);
	let kind = RequestKind::SendData(SendDataRequest {
	len: end.overflow_sub(start),
	});

	/// Get the memory region as slice
	#[inline]
	fn as_slice(&self) -> &[u8] {
	unsafe { slice::from_raw_parts(self.as_ptr(), self.length()) }
	}

	/// Get the memory region as mut slice
	#[inline]
	fn as_mut_slice(&mut self) -> &mut [u8] {
	unsafe { slice::from_raw_parts_mut(self.as_mut_ptr(), self.length()) }
	}

	/// Server can't aware of rdma `read` or `write`, so we need sync with client
	/// before client `read` and after `write`.
	async fn sync_with_client(rdma: &Rdma) {
	let mut lmr_sync = rdma
	.alloc_local_mr(Layout::new::<Request>())
	.map_err(\|err\| println!("{}", &err))
	.unwrap();
	//write data to lmr
	unsafe { (lmr_sync.as_mut_ptr() as mut Request) = Request::Sync };
	rdma.send(&lmr_sync)
	.await
	.map_err(\|err\| println!("{}", &err))
	.unwrap();
	}

	/// Poll one work completion from CQ
	pub(crate) fn poll_single(&self) -> io::Result<WorkCompletion> {
	let polled = self.poll(1)?;
	polled
	.into_iter()
	.next()
	.ok_or_else(\|\| io::Error::new(io::ErrorKind::WouldBlock, ""))
	}

datenlord / async-rdma Goto Github PK

async-rdma's Introduction

async-rdma

Environment Setup

1. Check whether the current kernel supports RXE

2. Install Dependencies

3. Configure RDMA netdev

4. Install Rust

5. Try an example

Example

Getting Help

Related Projects

async-rdma's People

Contributors

Stargazers

Watchers

Forkers

async-rdma's Issues

Env

Description

Recommend Projects

Recommend Topics

Recommend Org