tvegas1 / xpmem Goto Github PK

# Introduction
This is an experimental version of XPMEM based on a version provided by Cray
and uploaded to https://code.google.com/p/xpmem. This version supports any
Linux kernel 3.12 and newer (tested up to 5.8.x). Keep in mind there may be bugs and
this version may cause kernel panics, code crashes, eat your cat, etc.

XPMEM is a Linux kernel module that enables a process to map the
memory of another process into its virtual address space. Source code
can be obtained by cloning the Git repository, original Mercurial
repository or by downloading a tarball from the link above.

The XPMEM API has three main functions:

  xpmem_make()    
  xpmem_get()
  xpmem_attach()

A process calls xpmem_make() to export a region of its virtual address
space. Other processes can then attach to the region by calling
xpmem_get() and xpmem_attach(). After a memory region is attached, it
is accessed via direct loads and stores. This enables upper-level
protocols such as MPI and SHMEM to perform single-copy address-space
to address-space transfers, completely at user-level.

Note, there is a limitation to the usage of an attached region. Any
system call that will call get_user_pages() on the region from the
non-owning process with get EFAULT. This include pthread mutexes
and condition variable, and SYS V semaphores. We intend to address
this limitation in a future release.

XPMEM regions are free to have "holes" in them, meaning virtual memory
regions that are not allocated. This makes XPMEM somewhat more
flexible than mmap(). A process could, for example, export a region
via XPMEM starting at address 0 and extending 4 GB. Accesses to
allocated (valid) virtual addresses in this region proceed normally,
and pages are mapped between address spaces on demand. A segfault will
occur if the source process or any other process mapping the region
tries to access an unallocated (invalid) virtual address in the
region.

# Known issues

* Memory regions mapped with XPMEM cannot be pinned with
  [ibv_dereg_mr](https://linux.die.net/man/3/ibv_reg_mr)