nminoru / pib Goto Github PK

Pseudo InfiniBand HCA driver (for Linux)

License: Other

Makefile 0.27% Shell 0.97% C 98.66% Awk 0.10%

pib's Introduction

pib - Pseudo InfiniBand HCA driver

pib is a software-based InfiniBand HCA driver. It provides InfiniBand functions without real IB HCA & fabric. pib aims to simulate InfiniBand behavior accurately but not to get speed.

pib contains the three components.

pib.ko - Linux kernel module
libpib - Userspace plug-in module for libibverbs
pibnetd - IB switch emulator for multi-host-mode

Features

In single-host-mode, pib creates up to 4 InfiniBand HCA (The default is 2). These IB devices are pib_0, pib_1, pib_2 and pib_3. Each HCA contains up to 32 ports (The default is 2).

In addition, pib creates one internal InfiniBand switch too. All ports of pib's HCA are connected to this switch.

The current version of pib enables to drive the following interface:

kernel-level Verbs (in-linux kernel)
kernel-level MAD (in-linux kernel)
uVerbs (libibverbs)
uMAD (libibmad & libibumad)
Subnet Manager (opensm)
IPoIB (in-linux kernel)
RDMA Connection Manager (librdmacm)
IB diagnostic utilities (infiniband-diags)

Debugging support features:

Inspect IB objects (ucontext, PD, MR, SRQ, CQ, AH, QP)
Trace API invocations, packet sending/receiving, async events/errors
Inject a specified error (QP/CQ/SRQ Error)
Select some implementation dependent behaviour and enforce error checking.
Show a warning of pitfalls that IB programs should avoid.

Other features:

The maximum size of inline data is 2048 bytes.

Limitation

The current version is EXPERIMENTAL.

The following features are not supported:

Unreliable Connected (UC)
Fast Memory Region (FMR)
Memory Windows (MW)
SEND Invalidate operation
Virtual Lane (VL)
Flow control

Supported OS

pib supports the following Linux:

Red Hat Enterprise Linux 6.x
CentOS 6.x

pib conflicts with Mellanox OFED. Mustn't install an environment to deploy Mellanox OFED.

Preparation

The following software packages are required for building pib:

rdma
libibverbs
kernel-devel
opensm
opensm-libs

The following packages are recommended:

libibverbs-devel (for developing Verbs API programs)
libibverbs-utils
librdmacm
librdmacm-utils
librdmacm-devel (for developing RDMA API programs)
infiniband-diags (IB diagnostic tools)

Building

First, acquire the source code by cloning the git repository.

$ git clone https://github.com/nminoru/pib.git

pib.ko

If you want to compile the pib.ko kernel module from source code, input the following commands.

$ cd pib/driver/
$ make
# make modules_install

If you want to create binary RPM file, input the following commands.

First, create libpib's source RPM from source code.

$ cp -r pib/driver pib-0.4.5
$ tar czvf $(HOME)/rpmbuild/SOURCES/pib-0.4.5.tar.gz pib-0.4.5/
$ cp pib/driver/pib.conf $(HOME)/rpmbuild/SOURCES/
$ cp pib/driver/pib.files $(HOME)/rpmbuild/SOURCES/
$ rpmbuild -bs pib/driver/pib.spec

Next, build the binary RPM from the source RPM.

$ rpmbuild --rebuild $(HOME)/rpmbuild/SRPMS/pib-0.4.5-1.el6.src.rpm

Finally, install the built binary RPM.

# rpm -ihv $(HOME)/rpmbuild/RPMS/x86_64/kmod-pib-0.4.5-1.el6.x86_64.rpm

libpib

The libpib userspace plug-in module will be installed from the binary RPM.

$ cp -r pib/libpib libpib-0.0.6
$ tar czvf $(HOME)/rpmbuild/SOURCES/libpib-0.0.6.tar.gz libpib-0.0.6/
$ rpmbuild -bs pib/libpib/libpib.spec

$ rpmbuild --rebuild $(HOME)/rpmbuild/SRPMS/libpib-0.0.6-1.el6.src.rpm

# rpm -ihv $(HOME)/rpmbuild/RPMS/x86_64/libpib-0.0.6-1.el6.x86_64.rpm

pibnetd

If you want to compile the pibnetd daemon from source code, input the following commands.

$ cd pib/pibnet/
$ make
# install -m 755 -D pibnetd                     /usr/sbin/pibnetd
# install -m 755 -D scripts/redhat-pibnetd.init /etc/rc.d/init.d/pibnetd

If you want to create binary RPM file, input the following commands.

$ cp -r pib/pibnetd pibnetd-0.4.1
$ tar czvf $(HOME)/rpmbuild/SOURCES/pibnetd-0.4.1.tar.gz pibnetd-0.4.1/
$ rpmbuild -bs pib/pibnetd/pibnetd.spec

$ rpmbuild --rebuild $(HOME)/rpmbuild/SRPMS/pibnetd-0.4.1-1.el6.src.rpm

# rpm -ihv $(HOME)/rpmbuild/RPMS/x86_64/pibnetd-0.4.1-1.el6.x86_64.rpm

Download

You can get source and binary RPMs for RHEL6 or CentOS6 on this link http://www.nminoru.jp/~nminoru/network/infiniband/src/

Loading (single-host-mode)

First, load some modules which pib.ko is dependent on.

# /etc/rc.d/init.d/rdma start

Next, load pib.ko.

# modprobe pib

Finally, run opensm

# /etc/rc.d/init.d/opensm start

pib.ko options

debug_level
num_hca
phys_port_cnt
behavior
manner_warn
manner_err
addr

Loading (multi-host-mode)

In multi-host-mode mode, pib enables to connect up to 32 hosts (To be precise, up to 32 ports).

   Host A           Host X           Host B
 (10.0.0.1)       (10.0.0.2)       (10.0.0.3)
+----------+     +-----------+     +----------+
| +------+ |     | +-------+ |     | +------+ |
| |pib.ko| |-----| |pibnetd| |-----| |pib.ko| |
| +------+ |     | +-------+ |     | +------+ |
|          |     |           |     | +------+ |
|          |     |           |     | |opensm| |
|          |     |           |     | +------+ | 
+----------+     +-----------+     +----------+

First, run pibnetd on a host.

# /etc/rc.d/init.d/pibnetd start

Next, load pib.ko by running modprobe command with the addr parameter specified by the pibnetd's IP address.

# /etc/rc.d/init.d/rdma start
# modprobe pib addr=10.0.0.2

On th default parameters, pib creates 2 IB devices of 2 ports. You had better limit 1 IB device of 1 port by specifying the num_hca and phys_port_cnt parameters in multi-host-mode.

# modprobe pib addr=10.0.0.2 num_hca=1 phys_port_cnt=1

Finally, run opensm on one of hosts that load pib.ko.

# /etc/rc.d/init.d/opensm start

Running

For instance, ibv_devinfo (includes libibverbs-utils package) show such an result.

$ ibv_devinfo
hca_id: pib_0
        transport:                      InfiniBand (0)
        fw_ver:                         0.2.000
        node_guid:                      000c:2925:551e:0400
        sys_image_guid:                 000c:2925:551e:0200
        vendor_id:                      0x0001
        vendor_part_id:                 1
        hw_ver:                         0x0
        phys_port_cnt:                  2

Performance counter

# perfquery

Debugging support

pib provides some debugging functions via debugfs to help developing IB programs.

First ensure that debugfs is mounted.

# mount -t debugfs none /sys/kernel/debug

A list of available debugging functions can be found in /sys/kernel/debug/pib/pib_X/.

See detailed information on DEBUGFS.md.

FAQ

Failed to call ibv_reg_mr() with more than 64 KB

pib permits an unprivileged program to use InfiniBand userspace verbs. However Linux operating system limits the maximum memory size that an unprivileged process may lock via mlock() and ibv_reg_mr() calls mlock() internally. This default max-locked-memory is only 64 K bytes.

To avoid this trouble, run your program under privileged mode or increase max-locked-memory limit for unprivileged user.

If you choose the latter, add the following two lines in the file /etc/security/limits.conf and then reboot.

* soft memlock unlimited
* hard memlock unlimited

Or you can also set it temporarily to do ulimit -l unlimited.

Future work

IB functions

Fast Memory Registration(FMR)
Peer-Direct
Alternate path
Unreliable Connection(UC)
Extended Reliable Connected (XRC)
Memory Window

Debugging support

Packet filtering

Software components

MPI
User Direct Access Programming Library (uDAPL)
iSCSI Extensions for RDMA (iSER)
SCSI RDMA Protocol (SRP)

Other

Systemd init script support
Other Linux distributions support
Kernel update package
IPv6 support
Translate Japanese into English in comments of source codes :-)

Contact

https://twitter.com/nminoru_jp

[email protected]

License

GPL version 2 or BSD license

pib's People

Contributors

Stargazers

Watchers

Forkers

neocat hkimura rajkamal2013 hiroyuki-sato daveti grafoo thmo vsharma13 nambrok ullbeking maquefel kyeah01

pib's Issues

no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1

問題

ibv_devinfoで次のようになってしまう。

# ibv_devinfo
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0

環境

OS: CentOS 6.5 x86_64
pib: c739087
kernel: 2.6.32-431.el6.x86_64

手順

インストール時の導入パッケージ

@Additional Development
@Base
@Core
@Development tools
@Ruby Support
@SNMP Support
@base
@core
compat-libstdc++-33
dhcp
libaio
openldap-servers
postfix
screen
-sendmail

必要なモジュールの導入

yum install -y libibverbs-devel libibverbs-utils librdmacm librdmacm-utils librdmacm-devel infiniband-diags opensm

pidの導入

git clone https://github.com/nminoru/pib.git
cd pib/driver/
make
cd ../libpib/
make
cd ../pibnetd/
make

実行

/sbin/service rdma start
Loading OpenIB kernel modules:                             [  OK  ]
cd pib/driver/
insmod pib.ko
# /sbin/service opensm start
Starting IB Subnet Manager:                                [  OK  ]
# ibv_devinfo
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
No IB devices found

マルチホストモード時にphys_port_cntを1にするとinsmodが失敗する

問題

insmod pib.ko addr=server_ip num_hca=1 phys_port_cnt=1

エラー

pib: In single-host-mode, the value of num_hca * phys_port_cn must be 2 or more.

修正案

diff --git a/driver/pib_main.c b/driver/pib_main.c
index 72f5c09..a68bab2 100644
--- a/driver/pib_main.c
+++ b/driver/pib_main.c
@@ -971,7 +971,7 @@ static int __init pib_init(void)
                return -EINVAL;
        }

-       if (pib_num_hca * pib_phys_port_cnt < 2) {
+       if (!server_addr && pib_num_hca * pib_phys_port_cnt < 2) {
                pr_err("pib: In single-host-mode, the value of num_hca * phys_port_cn must be 2 or more.\n");
                return -EINVAL;
        }

Mainline submission?

Hello,

I think your driver would help us with automated testing of NetworkManager. I'm wondering if you've considered submitting it for mainline kernel inclusion so that we could rely on it being maintained and shipped with Linux distributions?

Thank you,
Lubo

ibnetdiscoverがマルチモードで動かない

問題

クライアント側のでibnetdiscoverを実行した際にタイムアウトが発生する。

環境

CentOS 6.5
be95598

補足

サーバ、クライアントとは、pibnetd/opensmを起動したホストをサーバ、指定いないホストをクライアントと呼んでいる。

サーバ側のibnetdiscover(正常)

# ibnetdiscover 
#
# Topology file: generated on Sat Mar 15 10:09:39 2014
#
# Initiated from node 005056000aed0300 port 005056000aed0301

vendid=0x0
devid=0x1
sysimgguid=0x5056000aed
switchguid=0x5056000bed(5056000bed)
Switch  32 "S-0000005056000bed"     # "Pseudo InfiniBand HCA switch" base port 0 lid 7 lmc 0
[1] "H-005056000aed0300"[1](5056000aed0301)         # "Pseudo InfiniBand HCA driver" lid 1 12xQDR
[2] "H-005056000aed0300"[2](5056000aed0302)         # "Pseudo InfiniBand HCA driver" lid 4 12xQDR
[3] "H-005056000aee0300"[1](5056000aee0301)         # "Pseudo InfiniBand HCA driver" lid 6 12xQDR
[4] "H-005056000aee0300"[2](5056000aee0302)         # "Pseudo InfiniBand HCA driver" lid 8 12xQDR

vendid=0x0
devid=0x1
sysimgguid=0x5056000aee0200
caguid=0x5056000aee0300
Ca  2 "H-005056000aee0300"      # "Pseudo InfiniBand HCA driver"
[1](5056000aee0301)     "S-0000005056000bed"[3]     # lid 6 lmc 0 "Pseudo InfiniBand HCA switch" lid 7 12xQDR
[2](5056000aee0302)     "S-0000005056000bed"[4]     # lid 8 lmc 0 "Pseudo InfiniBand HCA switch" lid 7 12xQDR

クライアント側ibnetdiscover(タイムアウト)

# ibnetdiscover 
src/query_smp.c:196; umad (DR path slid 0; dlid 0; 0,1,1 Attr 0x11:0) bad status 110; Connection timed out
src/query_smp.c:196; umad (DR path slid 0; dlid 0; 0,1,2 Attr 0x11:0) bad status 110; Connection timed out
src/query_smp.c:196; umad (DR path slid 0; dlid 0; 0,1,3 Attr 0x11:0) bad status 110; Connection timed out
src/query_smp.c:196; umad (DR path slid 0; dlid 0; 0,1,4 Attr 0x11:0) bad status 110; Connection timed out
#
# Topology file: generated on Sat Mar 15 12:05:33 2014
#
# Initiated from node 005056000aee0300 port 005056000aee0301

vendid=0x0
devid=0x1
sysimgguid=0x5056000aed
switchguid=0x5056000bed(5056000bed)
Switch  32 "S-0000005056000bed"     # "Pseudo InfiniBand HCA switch" base port 0 lid 7 lmc 0
[3] "H-005056000aee0300"[1](5056000aee0301)         # "Pseudo InfiniBand HCA driver" lid 6 12xQDR

vendid=0x0
devid=0x1
sysimgguid=0x5056000aee0200
caguid=0x5056000aee0300
Ca  2 "H-005056000aee0300"      # "Pseudo InfiniBand HCA driver"
[1](5056000aee0301)     "S-0000005056000bed"[3]     # lid 6 lmc 0 "Pseudo InfiniBand HCA switch" lid 7 12xQDR

サーバ側

# ./pibnetd -v 
pibnetd: Pseudo InfiniBand HCA switch v0.3.2

rmdaの起動

/sbin/service rdma start

pibのロード

insmod pib.ko addr=server_ip num_hca=1

opensmの起動

/sbin/service opensm start

クライアント側

rmdaの起動

/sbin/service rdma start

pibのロード

insmod pib.ko addr=server_ip num_hca=1

ibv_reg_mr fails for larger buffers

ibv_reg_mr seems to fail (returns NULL) when trying to register a 400KB region. It works for smaller buffers, e.g. 32k. Is there an option to allow larger memory regions?

ib_atomic_bw, ib_atomic_lat performance report

ib_atomic_bw、ib_atomic_lat の測定情報

環境

VMWware vSphere 5.5 上の仮想マシン(仮想 CPU 2)
OS: CentOS 6.5
Kernel: 2.6.32-431.5.1.el6.x86_64
pib be95598
perftest-2.0-2.el6.x86_64

ib_atomic_bw (FETCH_AND_ADD)

Server

$ ib_atomic_bw -d pib_0
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic FETCH_AND_ADD BW Test
 Dual-port       : OFF          Device         : pib_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x02 QPN 0xe90f75 PSN 0x9ab1fa
 remote address: LID 0x04 QPN 0xba1c8f PSN 0xa653e2

Client

$ ib_atomic_bw -d pib_1 localhost
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic FETCH_AND_ADD BW Test
 Dual-port       : OFF          Device         : pib_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x04 QPN 0xba1c8f PSN 0xa653e2
 remote address: LID 0x02 QPN 0xe90f75 PSN 0x9ab1fa
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 8          1000           1.66               1.48                 0.193769
---------------------------------------------------------------------------------------

ib_atomic_lat (FETCH_AND_ADD)

Server

$ ib_atomic_lat -d pib_0
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic FETCH_AND_ADD Latency Test
 Dual-port       : OFF          Device         : pib_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x02 QPN 0xe90f76 PSN 0x6c2a4f
 remote address: LID 0x04 QPN 0xba1c90 PSN 0x930942
---------------------------------------------------------------------------------------

Client

$ ib_atomic_lat -d pib_1 localhost
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic FETCH_AND_ADD Latency Test
 Dual-port       : OFF          Device         : pib_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 TX depth        : 1
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x04 QPN 0xba1c90 PSN 0x930942
 remote address: LID 0x02 QPN 0xe90f76 PSN 0x6c2a4f
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
 8       1000          15.01          98.22        18.86
---------------------------------------------------------------------------------------

ib_atomic_bw (CMP_AND_SWAP)

Server

$ ib_atomic_bw -d pib_0 -A CMP_AND_SWAP
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic CMP_AND_SWAP BW Test
 Dual-port       : OFF          Device         : pib_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x02 QPN 0xe90f77 PSN 0x985402
 remote address: LID 0x04 QPN 0xba1c91 PSN 0xd03f5e
---------------------------------------------------------------------------------------

Client

$ ib_atomic_bw -d pib_1 -A CMP_AND_SWAP localhost
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic CMP_AND_SWAP BW Test
 Dual-port       : OFF          Device         : pib_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x04 QPN 0xba1c93 PSN 0xbb1ff7
 remote address: LID 0x02 QPN 0xe90f79 PSN 0x316dee
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 8          1000           1.70               1.67                 0.219031
---------------------------------------------------------------------------------------

ib_atomic_lat (CMP_AND_SWAP)

Server

$ ib_atomic_lat -d pib_0 -A CMP_AND_SWAP
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic CMP_AND_SWAP Latency Test
 Dual-port       : OFF          Device         : pib_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x02 QPN 0xe90f78 PSN 0x55d67a
 remote address: LID 0x04 QPN 0xba1c92 PSN 0x316cc3
---------------------------------------------------------------------------------------

Client

$ ib_atomic_lat -d pib_1 -A CMP_AND_SWAP localhost
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic CMP_AND_SWAP Latency Test
 Dual-port       : OFF          Device         : pib_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 TX depth        : 1
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x04 QPN 0xba1c92 PSN 0x316cc3
 remote address: LID 0x02 QPN 0xe90f78 PSN 0x55d67a
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
 8       1000          8.55           164.53       9.27
---------------------------------------------------------------------------------------

ib_read_bw failed

問題

ib_read_bwが停止してしまう

環境

OS: CentOS 6.5
kernel : 2.6.32-431.el6.x86_64
pib : 0.3.4 (6a9436e)

その他

ハードウェアが貧弱なのでそのせいかもしれません。ESXi5.5上のCentOSです。

server

# ib_read_bw -d pib_0 -a 
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF      Device         : pib_0
 Number of qps   : 1        Transport type : IB
 Connection type : RC
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x01 QPN 0xfcdfdb PSN 0x5d2d4b OUT 0x10 RKey 0x31b06000 VAddr 0x007f75894a7000
 remote address: LID 0x03 QPN 0x27d939 PSN 0x8899fc OUT 0x10 RKey 0x84309000 VAddr 0x007fe9b02b0000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
pp_read_keys: Success
Couldn't read remote address
 Unable to read to socket/rdam_cm
Failed to close connection between server and client

client

# ib_read_bw -d pib_1 -a server_ip
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF      Device         : pib_1
 Number of qps   : 1        Transport type : IB
 Connection type : RC
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x03 QPN 0x27d939 PSN 0x8899fc OUT 0x10 RKey 0x84309000 VAddr 0x007fe9b02b0000
 remote address: LID 0x01 QPN 0xfcdfdb PSN 0x5d2d4b OUT 0x10 RKey 0x31b06000 VAddr 0x007f75894a7000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          1000           0.09               0.06             0.032045
 4          1000           0.17               0.12             0.031849
 8          1000           0.35               0.25             0.032629
 16         1000           0.70               0.49             0.032218
 32         1000           1.40               0.99             0.032367
 64         1000           2.39               1.90             0.031055
 128        1000           4.74               3.79             0.031012
 256        1000           9.64               7.41             0.030341
 512        1000           22.21              15.54            0.031817
 1024       1000           43.95              31.30            0.032048
 2048       1000           79.26              61.98            0.031735
 4096       1000           161.80             116.89           0.029924
 8192       1000           184.29             134.07           0.017161
 16384      1000           149.20             141.54           0.009058
 32768      1000           144.68             144.27           0.004617
 65536      1000           149.00             147.91           0.002367
 131072     1000           1.65               1.64             0.000013
 262144     1000           1.65               1.64             0.000007
 524288     1000           1.64               1.64             0.000003
 1048576    1000           141.82             141.82           0.000142
 2097152    1000           134.89             134.66           0.000067
 4194304    1000           121.97             121.76           0.000030
Problems with warm up

nminoru / pib Goto Github PK

pib's Introduction

pib - Pseudo InfiniBand HCA driver

Features

Limitation

Supported OS

Preparation

Building

pib.ko

libpib

pibnetd

Download

Loading (single-host-mode)

pib.ko options

Loading (multi-host-mode)

Running

Performance counter

Debugging support

FAQ

Failed to call ibv_reg_mr() with more than 64 KB

Future work

IB functions

Debugging support

Software components

Other

Contact

License

pib's People

Contributors

Stargazers

Watchers

Forkers

pib's Issues

問題

環境

手順

インストール時の導入パッケージ

必要なモジュールの導入

pidの導入

実行

問題

エラー

修正案

問題

環境

補足

サーバ側のibnetdiscover(正常)

クライアント側ibnetdiscover(タイムアウト)

サーバ側

rmdaの起動

pibのロード

opensmの起動

クライアント側

rmdaの起動

pibのロード

環境

ib_atomic_bw (FETCH_AND_ADD)

ib_atomic_lat (FETCH_AND_ADD)

ib_atomic_bw (CMP_AND_SWAP)

ib_atomic_lat (CMP_AND_SWAP)

問題

環境

その他

server

client

Recommend Projects

Recommend Topics

Recommend Org