Running vma inside docker container about libvma HOT 9 OPEN

Hi @Cha465 the behavior is not container specific.
It is expected that scenario using tcp connection on the same server when both (server and client) processes under libvma whould work.

from libvma.

Hi, I tried again with sockperf, specifying tcp connection:
LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf sr --tcp -i 0.0.0.0 // server
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/rmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/rmem_max, we'll use default 229376
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/wmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/wmem_max, we'll use default 229376
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA_VERSION: 9.7.1-0 Development Snapshot built on Dec 19 2023 10:37:13
VMA INFO: Cmd Line: sockperf sr --tcp -i 0.0.0.0
VMA INFO: Current Time: Mon Dec 25 09:31:13 2023
VMA INFO: Pid: 77119
VMA INFO: OFED Version: MLNX_OFED_LINUX-5.4-3.7.5.0:
VMA INFO: Architecture: x86_64
VMA INFO: Node: e55bdebd6779
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA Spec Latency [VMA_SPEC]
VMA INFO: Log Level INFO [VMA_TRACELEVEL]
VMA INFO: Ring On Device Memory TX 16384 [VMA_RING_DEV_MEM_TX]
VMA INFO: Tx QP WRE 256 [VMA_TX_WRE]
VMA INFO: Tx QP WRE Batching 4 [VMA_TX_WRE_BATCHING]
VMA INFO: Rx QP WRE 256 [VMA_RX_WRE]
VMA INFO: Rx QP WRE Batching 4 [VMA_RX_WRE_BATCHING]
VMA INFO: Rx Poll Loops -1 [VMA_RX_POLL]
VMA INFO: Rx Prefetch Bytes Before Poll 256 [VMA_RX_PREFETCH_BYTES_BEFORE_POLL]
VMA INFO: GRO max streams 0 [VMA_GRO_STREAMS_MAX]
VMA INFO: Select Poll (usec) -1 [VMA_SELECT_POLL]
VMA INFO: Select Poll OS Force Enabled [VMA_SELECT_POLL_OS_FORCE]
VMA INFO: Select Poll OS Ratio 1 [VMA_SELECT_POLL_OS_RATIO]
VMA INFO: Select Skip OS 1 [VMA_SELECT_SKIP_OS]
VMA INFO: CQ Drain Interval (msec) 100 [VMA_PROGRESS_ENGINE_INTERVAL]
VMA INFO: CQ Interrupts Moderation Disabled [VMA_CQ_MODERATION_ENABLE]
VMA INFO: CQ AIM Max Count 128 [VMA_CQ_AIM_MAX_COUNT]
VMA INFO: CQ Adaptive Moderation Disabled [VMA_CQ_AIM_INTERVAL_MSEC]
VMA INFO: CQ Keeps QP Full Disabled [VMA_CQ_KEEP_QP_FULL]
VMA INFO: TCP nodelay 1 [VMA_TCP_NODELAY]
VMA INFO: Avoid sys-calls on tcp fd Enabled [VMA_AVOID_SYS_CALLS_ON_TCP_FD]
VMA INFO: Internal Thread Affinity 0 [VMA_INTERNAL_THREAD_AFFINITY]
VMA INFO: Thread mode Single [VMA_THREAD_MODE]
VMA INFO: ---------------------------------------------------------------------------
sockperf: == version #3.7-no.git ==
sockperf: [SERVER] listen on:
[ 0] IP = 0.0.0.0 PORT = 11111 # TCP
sockperf: Warmup stage (sending a few dummy messages)...
sockperf: [tid 77119] using recvfrom() to block on socket(s)

LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf pp -i 0.0.0.0 --tcp -t60 // client
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/rmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/rmem_max, we'll use default 229376
VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/wmem_max (errno 2 No such file or directory)
VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/wmem_max, we'll use default 229376
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA_VERSION: 9.7.1-0 Development Snapshot built on Dec 19 2023 10:37:13
VMA INFO: Cmd Line: sockperf pp -i 0.0.0.0 --tcp -t60
VMA INFO: Current Time: Mon Dec 25 09:31:31 2023
VMA INFO: Pid: 77125
VMA INFO: OFED Version: MLNX_OFED_LINUX-5.4-3.7.5.0:
VMA INFO: Architecture: x86_64
VMA INFO: Node: e55bdebd6779
VMA INFO: ---------------------------------------------------------------------------
VMA INFO: VMA Spec Latency [VMA_SPEC]
VMA INFO: Log Level INFO [VMA_TRACELEVEL]
VMA INFO: Ring On Device Memory TX 16384 [VMA_RING_DEV_MEM_TX]
VMA INFO: Tx QP WRE 256 [VMA_TX_WRE]
VMA INFO: Tx QP WRE Batching 4 [VMA_TX_WRE_BATCHING]
VMA INFO: Rx QP WRE 256 [VMA_RX_WRE]
VMA INFO: Rx QP WRE Batching 4 [VMA_RX_WRE_BATCHING]
VMA INFO: Rx Poll Loops -1 [VMA_RX_POLL]
VMA INFO: Rx Prefetch Bytes Before Poll 256 [VMA_RX_PREFETCH_BYTES_BEFORE_POLL]
VMA INFO: GRO max streams 0 [VMA_GRO_STREAMS_MAX]
VMA INFO: Select Poll (usec) -1 [VMA_SELECT_POLL]
VMA INFO: Select Poll OS Force Enabled [VMA_SELECT_POLL_OS_FORCE]
VMA INFO: Select Poll OS Ratio 1 [VMA_SELECT_POLL_OS_RATIO]
VMA INFO: Select Skip OS 1 [VMA_SELECT_SKIP_OS]
VMA INFO: CQ Drain Interval (msec) 100 [VMA_PROGRESS_ENGINE_INTERVAL]
VMA INFO: CQ Interrupts Moderation Disabled [VMA_CQ_MODERATION_ENABLE]
VMA INFO: CQ AIM Max Count 128 [VMA_CQ_AIM_MAX_COUNT]
VMA INFO: CQ Adaptive Moderation Disabled [VMA_CQ_AIM_INTERVAL_MSEC]
VMA INFO: CQ Keeps QP Full Disabled [VMA_CQ_KEEP_QP_FULL]
VMA INFO: TCP nodelay 1 [VMA_TCP_NODELAY]
VMA INFO: Avoid sys-calls on tcp fd Enabled [VMA_AVOID_SYS_CALLS_ON_TCP_FD]
VMA INFO: Internal Thread Affinity 0 [VMA_INTERNAL_THREAD_AFFINITY]
VMA INFO: Thread mode Single [VMA_THREAD_MODE]
VMA INFO: ---------------------------------------------------------------------------
sockperf: == version #3.7-no.git ==
sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

[ 0] IP = 0.0.0.0 PORT = 11111 # TCP
sockperf : Warmup stage ( sending a few du mmy mess ERSION: 9.7.1-0 Developm ent Snap shot bui sockperf: Test end (interrupted by timer)
sockperf: Test ended
sockperf: [Total Run] RunTime=60.000 sec; Warm up time=400 msec; SentMessages=4570324; ReceivedMessages=4570323
sockperf: ========= Printing statistics for Server No: 0
sockperf: [Valid Duration] RunTime=59.550 sec; SentMessages=4540388; ReceivedMessages=4540388
sockperf: ====> avg-latency=6.536 (std-dev=2.148)
sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0
sockperf: Summary: Latency is 6.536 usec
sockperf: Total 4540388 observations; each percentile contains 45403.88 observations
sockperf: ---> observation = 2635.175
sockperf: ---> percentile 99.999 = 69.872
sockperf: ---> percentile 99.990 = 17.672
sockperf: ---> percentile 99.900 = 12.896
sockperf: ---> percentile 99.000 = 10.863
sockperf: ---> percentile 90.000 = 6.807
sockperf: ---> percentile 75.000 = 6.566
sockperf: ---> percentile 50.000 = 6.485
sockperf: ---> percentile 25.000 = 6.368
sockperf: ---> observation = 4.769

The performance is not ideal, and vma_stats -p <client_pid> is not showing a single line. What is the cause of this? Maybe the VMA ERROR must be fixed?

from libvma.

could you try to set specific ip instead of 0.0.0.0.

from libvma.

127.0.0.1 or ip inside the docker container give the same result.

from libvma.

Of course, error should not be appeared.
Could you try on server first (w/o container) to separate potential issues.

Example on the same node:
Launch server

sudo env VMA_STATS_FILE=/tmp/libvma-sr.log LD_PRELOAD=libvma.so sockperf sr -i 1.1.1.10 --tcp

Statistic for server:

$ cat /tmp/libvma-sr.log
======================================================
        Fd=[24]
- TCP, Blocked
- Local Address   = [1.1.1.10:11111]
- Foreign Address = [1.1.1.10:50613]
Tx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx Offload: 4281 / 313152 / 0 / 1 [kilobytes/packets/eagains/errors]
Rx byte: cur 0 / max 14 / dropped 0 / limit 0
Rx pkt : cur 0 / max 1 / dropped 0
Rx poll: 2 / 313151 (100.00%) [miss/hit]
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [1.1.1.10:11111]
Rx poll: 3 / 0 (0.00%) [miss/hit]
======================================================

Launch client:

sudo env VMA_STATS_FILE=/tmp/libvma-cl.log LD_PRELOAD=libvma.so sockperf pp -i 1.1.1.10 --tcp

Statistic for client:

cat /tmp/libvma-cl.log
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [1.1.1.10:50613]
- Foreign Address = [1.1.1.10:11111]
Tx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx byte: cur 0 / max 14 / dropped 0 / limit 0
Rx pkt : cur 0 / max 1 / dropped 0
Rx poll: 0 / 313152 (100.00%) [miss/hit]
======================================================

from libvma.

I got these results:
server

cat /tmp/libvma-sr.log
======================================================
        Fd=[20]
- TCP, Blocked
Rx and Tx where not active
======================================================

client

cat /tmp/libvma-cl.log
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [0.0.0.0:48379]
Rx and Tx where not active
======================================================

from libvma.

Did you launch on Mellanox device? Did you see any error messages from VMA? Use direct ip as mentioned before.

from libvma.

• Yes, launched on Mellanox cx6, but I couldn't find a way to manually configure which NIC to use.
• No error messages except for a warning (I tried ulimit -l unlimited but did not make a difference):

 VMA WARNING: ************************************************************************
 VMA WARNING: Your current max locked memory is: 33762250752. Please change it to unlimited.
 VMA WARNING: Set this user's default to `ulimit -l unlimited`.
 VMA WARNING: Read more about this topic in the VMA's User Manual.
 VMA WARNING: ************************************************************************

• Yes, I used direct ip.

from libvma.

• Yes, launched on Mellanox cx6, but I couldn't find a way to manually configure which NIC to use.

Just use ip that is set on Mellanox NIC

• No error messages except for a warning (I tried ulimit -l unlimited but did not make a difference):

 VMA WARNING: ************************************************************************
 VMA WARNING: Your current max locked memory is: 33762250752. Please change it to unlimited.
 VMA WARNING: Set this user's default to `ulimit -l unlimited`.
 VMA WARNING: Read more about this topic in the VMA's User Manual.
 VMA WARNING: ************************************************************************

It is strange. Did you use different users to set ulimit and launch application?

• Yes, I used direct ip.

from libvma.