With a 25Gb Mellanox ConnectX-6 NIC on both sides, I can transfer at ~3.13+ GB/s which

My bad. Updated the title to 3.1.3 . <p dir="auto"

Ah, great point! Linux has 3.12 : <div class="high

Fixes speed issues with iperf3 3.1.3 about iperf3-win-builds HOT 5 CLOSED

ar51an commented on June 14, 2024

Fixes speed issues with iperf3 3.1.3

from iperf3-win-builds.

Comments (5)

ar51an commented on June 14, 2024

You are referring to iperf3 version 3.13 and 3.1.3 randomly in above issue. These are 2 different versions. Last official windows binary release was 3.1.3 and that was 7 years ago. Since then there are lot of improvements and bug fixes in iperf3. I assume you meant the 3.1.3 rather than 3.13

You should try running the client with parallel streams using the -P option. And check the row with [SUM]. Something like:
iperf3.exe -c <IP> -P 10

The biggest change from version 3.15 to 3.16 is utilization of multiple threads. Quoted below from official source code release:

Multiple test streams started with -P/--parallel will now be
serviced by different threads. This allows iperf3 to take
advantage of multiple CPU cores on modern processors, and will
generally result in significant throughput increases

I got consistent results at around 18-20 Gb/s for both upload and download (Reverse mode) using parallel streams. I do not know what kind of network setup you have. If it is direct Ethernet transfer or you are using some RAID controller. (May be you need to fine tune your NIC configuration)

I just compile and release the windows binaries from official source code, as it is without any change in the code. They do not release binaries anymore for any platform. Unix based compilation is straight forward. You can check it here in the official source code repo:
https://github.com/esnet/iperf/blob/master/docs/obtaining.rst

If you think there is an issue. You should open it in the official iperf3 source code repo at:
https://github.com/esnet/iperf

from iperf3-win-builds.

Sawtaytoes commented on June 14, 2024

My bad. Updated the title to 3.1.3.

My topology for these to machines:
Windows 25Gb Fiber <-> 25Gb port on Switch <-> 25Gb port on Switch <-> TrueNAS SCALE 25Gb Fiber on my own Epyc server hardware that's roughly the same as my PC Ryzen setup.

And there are 2 x SFP28 connectors per machine, but no LAGG or combined adapters right now. All 4 have IPs on the network and are connected to the same switch.

Windows -> Linux

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  2.89 GBytes  2.48 Gbits/sec                  receiver
[  8]   0.00-10.01  sec  2.89 GBytes  2.48 Gbits/sec                  receiver
[ 10]   0.00-10.01  sec  2.90 GBytes  2.49 Gbits/sec                  receiver
[ 12]   0.00-10.01  sec  2.89 GBytes  2.48 Gbits/sec                  receiver
[ 14]   0.00-10.01  sec  2.88 GBytes  2.47 Gbits/sec                  receiver
[ 16]   0.00-10.01  sec  2.88 GBytes  2.47 Gbits/sec                  receiver
[ 18]   0.00-10.01  sec  2.89 GBytes  2.48 Gbits/sec                  receiver
[ 20]   0.00-10.01  sec  2.90 GBytes  2.49 Gbits/sec                  receiver
[ 22]   0.00-10.01  sec  2.86 GBytes  2.46 Gbits/sec                  receiver
[ 24]   0.00-10.01  sec  2.89 GBytes  2.48 Gbits/sec                  receiver
[SUM]   0.00-10.01  sec  28.9 GBytes  24.8 Gbits/sec                  receiver

Windows -> Linux (Reverse mode)

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  2.31 GBytes  1.99 Gbits/sec    0             sender
[  8]   0.00-10.01  sec  2.31 GBytes  1.99 Gbits/sec    0             sender
[ 10]   0.00-10.01  sec  2.31 GBytes  1.99 Gbits/sec    0             sender
[ 12]   0.00-10.01  sec  2.31 GBytes  1.99 Gbits/sec    0             sender
[ 14]   0.00-10.01  sec  2.31 GBytes  1.99 Gbits/sec    0             sender
[ 16]   0.00-10.01  sec  2.31 GBytes  1.99 Gbits/sec    0             sender
[ 18]   0.00-10.01  sec  2.31 GBytes  1.99 Gbits/sec    0             sender
[ 20]   0.00-10.01  sec  2.31 GBytes  1.99 Gbits/sec    0             sender
[ 22]   0.00-10.01  sec  2.31 GBytes  1.99 Gbits/sec    0             sender
[ 24]   0.00-10.01  sec  2.31 GBytes  1.98 Gbits/sec    0             sender
[SUM]   0.00-10.01  sec  23.1 GBytes  19.9 Gbits/sec    0             sender

Linux -> Windows

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.93 GBytes  1.66 Gbits/sec                  receiver
[  8]   0.00-10.00  sec  11.5 GBytes  9.87 Gbits/sec                  receiver
[ 10]   0.00-10.00  sec  1.90 GBytes  1.63 Gbits/sec                  receiver
[ 12]   0.00-10.00  sec  1.95 GBytes  1.67 Gbits/sec                  receiver
[ 14]   0.00-10.00  sec  1.90 GBytes  1.63 Gbits/sec                  receiver
[ 16]   0.00-10.00  sec  1.92 GBytes  1.65 Gbits/sec                  receiver
[ 18]   0.00-10.00  sec  1.94 GBytes  1.66 Gbits/sec                  receiver
[ 20]   0.00-10.00  sec  1.93 GBytes  1.66 Gbits/sec                  receiver
[ 22]   0.00-10.00  sec  1.91 GBytes  1.64 Gbits/sec                  receiver
[ 24]   0.00-10.00  sec  1.95 GBytes  1.68 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  28.8 GBytes  24.8 Gbits/sec                  receiver

Linux -> Windows (Reverse mode)

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  3.21 GBytes  2.76 Gbits/sec                  sender
[  8]   0.00-10.00  sec  1.55 GBytes  1.33 Gbits/sec                  sender
[ 10]   0.00-10.00  sec  3.21 GBytes  2.76 Gbits/sec                  sender
[ 12]   0.00-10.00  sec  3.21 GBytes  2.76 Gbits/sec                  sender
[ 14]   0.00-10.00  sec  3.22 GBytes  2.76 Gbits/sec                  sender
[ 16]   0.00-10.00  sec  3.22 GBytes  2.76 Gbits/sec                  sender
[ 18]   0.00-10.00  sec  1.66 GBytes  1.43 Gbits/sec                  sender
[ 20]   0.00-10.00  sec  3.20 GBytes  2.74 Gbits/sec                  sender
[ 22]   0.00-10.00  sec  3.21 GBytes  2.76 Gbits/sec                  sender
[ 24]   0.00-10.00  sec  3.19 GBytes  2.74 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  28.9 GBytes  24.8 Gbits/sec                  sender

"Windows -> Linux (Reverse mode)" is the only one where I don't get the full bandwidth.

Figured you might know something, but if you're just compiling those builds, I guess there's nothing you could do.

from iperf3-win-builds.

ar51an commented on June 14, 2024

Are you using iperf3 version 3.16 on both Windows and Linux?

You got now around 20Gb/s in reverse mode with parallel streams from Windows > Linux. It is way better than what you were getting earlier.

My two cents. I do not see any issue with iperf3. Iperf3 gives you the throughput such that you can fine tune your network if it is not working at its optimal. May be you need to fine tune Windows network configuration and/or your windows NIC config (if it is doing onboard offloading)

from iperf3-win-builds.

Sawtaytoes commented on June 14, 2024

Ah, great point! Linux has 3.12:

# iperf3 -v
iperf 3.12 (cJSON 1.7.15)
Linux storeman 6.1.55-production+truenas #2 SMP PREEMPT_DYNAMIC Tue Oct 31 16:07:08 UTC 2023 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, SCTP, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication, bind to device, support IPv4 don't fragment

I can't upgrade it in TrueNAS SCALE, so that might be part of the issue.

20Gb is fine, but why can I do 25Gb from Linux to Windows if Windows is hosting the iperf3 server rather than if Linux is hosting the server with Reverse mode? That's the thing that doesn't make sense. Both are sending data to Windows with iperf3.

Is there a difference of making Windows the server rather than the client?

from iperf3-win-builds.

ar51an commented on June 14, 2024

Possible reason could be different buffer sizes are used. Use -V (--verbose) option in iperf3 command on the client to view buffer size in both cases. Is it same?

from iperf3-win-builds.

Fixes speed issues with iperf3 3.1.3 about iperf3-win-builds HOT 5 CLOSED

Comments (5)

Windows -> Linux

Windows -> Linux (Reverse mode)

Linux -> Windows

Linux -> Windows (Reverse mode)

Related Issues (3)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent