nminoru / pib Goto Github PK

View Code? Open in Web Editor NEW

30.0 6.0 13.0 443 KB

Pseudo InfiniBand HCA driver (for Linux)

License: Other

Makefile 0.27% Shell 0.97% C 98.66% Awk 0.10%

pib's Issues

ib_atomic_bw, ib_atomic_lat performance report

ib_atomic_bw、ib_atomic_lat の測定情報

環境

VMWware vSphere 5.5 上の仮想マシン(仮想 CPU 2)
OS: CentOS 6.5
Kernel: 2.6.32-431.5.1.el6.x86_64
pib be95598
perftest-2.0-2.el6.x86_64

ib_atomic_bw (FETCH_AND_ADD)

Server

$ ib_atomic_bw -d pib_0
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic FETCH_AND_ADD BW Test
 Dual-port       : OFF          Device         : pib_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x02 QPN 0xe90f75 PSN 0x9ab1fa
 remote address: LID 0x04 QPN 0xba1c8f PSN 0xa653e2

Client

$ ib_atomic_bw -d pib_1 localhost
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic FETCH_AND_ADD BW Test
 Dual-port       : OFF          Device         : pib_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x04 QPN 0xba1c8f PSN 0xa653e2
 remote address: LID 0x02 QPN 0xe90f75 PSN 0x9ab1fa
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 8          1000           1.66               1.48                 0.193769
---------------------------------------------------------------------------------------

ib_atomic_lat (FETCH_AND_ADD)

Server

$ ib_atomic_lat -d pib_0
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic FETCH_AND_ADD Latency Test
 Dual-port       : OFF          Device         : pib_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x02 QPN 0xe90f76 PSN 0x6c2a4f
 remote address: LID 0x04 QPN 0xba1c90 PSN 0x930942
---------------------------------------------------------------------------------------

Client

$ ib_atomic_lat -d pib_1 localhost
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic FETCH_AND_ADD Latency Test
 Dual-port       : OFF          Device         : pib_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 TX depth        : 1
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x04 QPN 0xba1c90 PSN 0x930942
 remote address: LID 0x02 QPN 0xe90f76 PSN 0x6c2a4f
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
 8       1000          15.01          98.22        18.86
---------------------------------------------------------------------------------------

ib_atomic_bw (CMP_AND_SWAP)

Server

$ ib_atomic_bw -d pib_0 -A CMP_AND_SWAP
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic CMP_AND_SWAP BW Test
 Dual-port       : OFF          Device         : pib_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x02 QPN 0xe90f77 PSN 0x985402
 remote address: LID 0x04 QPN 0xba1c91 PSN 0xd03f5e
---------------------------------------------------------------------------------------

Client

$ ib_atomic_bw -d pib_1 -A CMP_AND_SWAP localhost
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic CMP_AND_SWAP BW Test
 Dual-port       : OFF          Device         : pib_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x04 QPN 0xba1c93 PSN 0xbb1ff7
 remote address: LID 0x02 QPN 0xe90f79 PSN 0x316dee
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 8          1000           1.70               1.67                 0.219031
---------------------------------------------------------------------------------------

ib_atomic_lat (CMP_AND_SWAP)

Server

$ ib_atomic_lat -d pib_0 -A CMP_AND_SWAP
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic CMP_AND_SWAP Latency Test
 Dual-port       : OFF          Device         : pib_0
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x02 QPN 0xe90f78 PSN 0x55d67a
 remote address: LID 0x04 QPN 0xba1c92 PSN 0x316cc3
---------------------------------------------------------------------------------------

Client

$ ib_atomic_lat -d pib_1 -A CMP_AND_SWAP localhost
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    Atomic CMP_AND_SWAP Latency Test
 Dual-port       : OFF          Device         : pib_1
 Number of qps   : 1            Transport type : IB
 Connection type : RC
 TX depth        : 1
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x04 QPN 0xba1c92 PSN 0x316cc3
 remote address: LID 0x02 QPN 0xe90f78 PSN 0x55d67a
---------------------------------------------------------------------------------------
 #bytes #iterations    t_min[usec]    t_max[usec]  t_typical[usec]
 8       1000          8.55           164.53       9.27
---------------------------------------------------------------------------------------

Mainline submission?

Hello,

I think your driver would help us with automated testing of NetworkManager. I'm wondering if you've considered submitting it for mainline kernel inclusion so that we could rely on it being maintained and shipped with Linux distributions?

Thank you,
Lubo

マルチホストモード時にphys_port_cntを1にするとinsmodが失敗する

問題

insmod pib.ko addr=server_ip num_hca=1 phys_port_cnt=1

エラー

pib: In single-host-mode, the value of num_hca * phys_port_cn must be 2 or more.

修正案

diff --git a/driver/pib_main.c b/driver/pib_main.c
index 72f5c09..a68bab2 100644
--- a/driver/pib_main.c
+++ b/driver/pib_main.c
@@ -971,7 +971,7 @@ static int __init pib_init(void)
                return -EINVAL;
        }

-       if (pib_num_hca * pib_phys_port_cnt < 2) {
+       if (!server_addr && pib_num_hca * pib_phys_port_cnt < 2) {
                pr_err("pib: In single-host-mode, the value of num_hca * phys_port_cn must be 2 or more.\n");
                return -EINVAL;
        }

ib_read_bw failed

問題

ib_read_bwが停止してしまう

環境

OS: CentOS 6.5
kernel : 2.6.32-431.el6.x86_64
pib : 0.3.4 (6a9436e)

その他

ハードウェアが貧弱なのでそのせいかもしれません。ESXi5.5上のCentOSです。

server

# ib_read_bw -d pib_0 -a 
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF      Device         : pib_0
 Number of qps   : 1        Transport type : IB
 Connection type : RC
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x01 QPN 0xfcdfdb PSN 0x5d2d4b OUT 0x10 RKey 0x31b06000 VAddr 0x007f75894a7000
 remote address: LID 0x03 QPN 0x27d939 PSN 0x8899fc OUT 0x10 RKey 0x84309000 VAddr 0x007fe9b02b0000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
pp_read_keys: Success
Couldn't read remote address
 Unable to read to socket/rdam_cm
Failed to close connection between server and client

client

# ib_read_bw -d pib_1 -a server_ip
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF      Device         : pib_1
 Number of qps   : 1        Transport type : IB
 Connection type : RC
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 4096[B]
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs     : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x03 QPN 0x27d939 PSN 0x8899fc OUT 0x10 RKey 0x84309000 VAddr 0x007fe9b02b0000
 remote address: LID 0x01 QPN 0xfcdfdb PSN 0x5d2d4b OUT 0x10 RKey 0x31b06000 VAddr 0x007f75894a7000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 2          1000           0.09               0.06             0.032045
 4          1000           0.17               0.12             0.031849
 8          1000           0.35               0.25             0.032629
 16         1000           0.70               0.49             0.032218
 32         1000           1.40               0.99             0.032367
 64         1000           2.39               1.90             0.031055
 128        1000           4.74               3.79             0.031012
 256        1000           9.64               7.41             0.030341
 512        1000           22.21              15.54            0.031817
 1024       1000           43.95              31.30            0.032048
 2048       1000           79.26              61.98            0.031735
 4096       1000           161.80             116.89           0.029924
 8192       1000           184.29             134.07           0.017161
 16384      1000           149.20             141.54           0.009058
 32768      1000           144.68             144.27           0.004617
 65536      1000           149.00             147.91           0.002367
 131072     1000           1.65               1.64             0.000013
 262144     1000           1.65               1.64             0.000007
 524288     1000           1.64               1.64             0.000003
 1048576    1000           141.82             141.82           0.000142
 2097152    1000           134.89             134.66           0.000067
 4194304    1000           121.97             121.76           0.000030
Problems with warm up

ibv_reg_mr fails for larger buffers

ibv_reg_mr seems to fail (returns NULL) when trying to register a 400KB region. It works for smaller buffers, e.g. 32k. Is there an option to allow larger memory regions?

ibnetdiscoverがマルチモードで動かない

問題

クライアント側のでibnetdiscoverを実行した際にタイムアウトが発生する。

環境

CentOS 6.5
be95598

補足

サーバ、クライアントとは、pibnetd/opensmを起動したホストをサーバ、指定いないホストをクライアントと呼んでいる。

サーバ側のibnetdiscover(正常)

# ibnetdiscover 
#
# Topology file: generated on Sat Mar 15 10:09:39 2014
#
# Initiated from node 005056000aed0300 port 005056000aed0301

vendid=0x0
devid=0x1
sysimgguid=0x5056000aed
switchguid=0x5056000bed(5056000bed)
Switch  32 "S-0000005056000bed"     # "Pseudo InfiniBand HCA switch" base port 0 lid 7 lmc 0
[1] "H-005056000aed0300"[1](5056000aed0301)         # "Pseudo InfiniBand HCA driver" lid 1 12xQDR
[2] "H-005056000aed0300"[2](5056000aed0302)         # "Pseudo InfiniBand HCA driver" lid 4 12xQDR
[3] "H-005056000aee0300"[1](5056000aee0301)         # "Pseudo InfiniBand HCA driver" lid 6 12xQDR
[4] "H-005056000aee0300"[2](5056000aee0302)         # "Pseudo InfiniBand HCA driver" lid 8 12xQDR

vendid=0x0
devid=0x1
sysimgguid=0x5056000aee0200
caguid=0x5056000aee0300
Ca  2 "H-005056000aee0300"      # "Pseudo InfiniBand HCA driver"
[1](5056000aee0301)     "S-0000005056000bed"[3]     # lid 6 lmc 0 "Pseudo InfiniBand HCA switch" lid 7 12xQDR
[2](5056000aee0302)     "S-0000005056000bed"[4]     # lid 8 lmc 0 "Pseudo InfiniBand HCA switch" lid 7 12xQDR

クライアント側ibnetdiscover(タイムアウト)

# ibnetdiscover 
src/query_smp.c:196; umad (DR path slid 0; dlid 0; 0,1,1 Attr 0x11:0) bad status 110; Connection timed out
src/query_smp.c:196; umad (DR path slid 0; dlid 0; 0,1,2 Attr 0x11:0) bad status 110; Connection timed out
src/query_smp.c:196; umad (DR path slid 0; dlid 0; 0,1,3 Attr 0x11:0) bad status 110; Connection timed out
src/query_smp.c:196; umad (DR path slid 0; dlid 0; 0,1,4 Attr 0x11:0) bad status 110; Connection timed out
#
# Topology file: generated on Sat Mar 15 12:05:33 2014
#
# Initiated from node 005056000aee0300 port 005056000aee0301

vendid=0x0
devid=0x1
sysimgguid=0x5056000aed
switchguid=0x5056000bed(5056000bed)
Switch  32 "S-0000005056000bed"     # "Pseudo InfiniBand HCA switch" base port 0 lid 7 lmc 0
[3] "H-005056000aee0300"[1](5056000aee0301)         # "Pseudo InfiniBand HCA driver" lid 6 12xQDR

vendid=0x0
devid=0x1
sysimgguid=0x5056000aee0200
caguid=0x5056000aee0300
Ca  2 "H-005056000aee0300"      # "Pseudo InfiniBand HCA driver"
[1](5056000aee0301)     "S-0000005056000bed"[3]     # lid 6 lmc 0 "Pseudo InfiniBand HCA switch" lid 7 12xQDR

サーバ側

# ./pibnetd -v 
pibnetd: Pseudo InfiniBand HCA switch v0.3.2

rmdaの起動

/sbin/service rdma start

pibのロード

insmod pib.ko addr=server_ip num_hca=1

opensmの起動

/sbin/service opensm start

クライアント側

rmdaの起動

/sbin/service rdma start

pibのロード

insmod pib.ko addr=server_ip num_hca=1

no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1

問題

ibv_devinfoで次のようになってしまう。

# ibv_devinfo
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0

環境

OS: CentOS 6.5 x86_64
pib: c739087
kernel: 2.6.32-431.el6.x86_64

手順

インストール時の導入パッケージ

@Additional Development
@Base
@Core
@Development tools
@Ruby Support
@SNMP Support
@base
@core
compat-libstdc++-33
dhcp
libaio
openldap-servers
postfix
screen
-sendmail

必要なモジュールの導入

yum install -y libibverbs-devel libibverbs-utils librdmacm librdmacm-utils librdmacm-devel infiniband-diags opensm

pidの導入

git clone https://github.com/nminoru/pib.git
cd pib/driver/
make
cd ../libpib/
make
cd ../pibnetd/
make

実行

/sbin/service rdma start
Loading OpenIB kernel modules:                             [  OK  ]
cd pib/driver/
insmod pib.ko
# /sbin/service opensm start
Starting IB Subnet Manager:                                [  OK  ]
# ibv_devinfo
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs1
libibverbs: Warning: no userspace device-specific driver found for /sys/class/infiniband_verbs/uverbs0
No IB devices found

nminoru / pib Goto Github PK

pib's Issues

環境

ib_atomic_bw (FETCH_AND_ADD)

ib_atomic_lat (FETCH_AND_ADD)

ib_atomic_bw (CMP_AND_SWAP)

ib_atomic_lat (CMP_AND_SWAP)

問題

エラー

修正案

問題

環境

その他

server

client

問題

環境

補足

サーバ側のibnetdiscover(正常)

クライアント側ibnetdiscover(タイムアウト)

サーバ側

rmdaの起動

pibのロード

opensmの起動

クライアント側

rmdaの起動

pibのロード

問題

環境

手順

インストール時の導入パッケージ

必要なモジュールの導入

pidの導入

実行

Recommend Projects

Recommend Topics

Recommend Org