Multi-threaded iperf3 discussion thread

[bmah888]

Please feel free to post experiences with the multi-threaded iperf3 beta snapshot(s) or codeline here. Thanks!

Bruce.

[davidBar-On]

Great enhancement to iperf3!
I don't have the environment to test multi processing, so just tested that the change dose not break functionality under Windows Cygwin. The multi-threading works o.k., but since UDP multi-streams doesn't work under Windows Cygwin (see PR #1163) I found few minor issues related to error handling - same issues in both iperf_server_api.c and iperf_client_api.c:

1. "cleanup" tries to join the threads even if they were not created. Suggested changes:

• If pthread_create() fails, set sp->thr = NULL before calling "cleanup".
• In "cleanup" call pthread_join() only if sp->thr != NULL.

2. In "cleanup", the if (test->debug >= DEBUG_LEVEL_INFO) after calling pthread_join() should check test->debug_level.

[bmah888]

Thanks for testing, and I'm glad it mostly works on Cygwin (minus the UDP stuff, anyway). We'll see about implementing the changes for error handling.

[fgggid]

I build one on centos 7, use option "-b 100m -t 600" to test for 10 minutes, but it will keep stopping at around 120s, something like
server side:

[  5] 118.00-119.00 sec  11.9 MBytes  99.6 Mbits/sec                  
iperf3: error - idle timeout for receiving data

client side:

[  5] 119.00-120.00 sec  12.0 MBytes   101 Mbits/sec    0    140 KBytes       
[  5] 119.00-120.00 sec  12.0 MBytes   101 Mbits/sec    0    140 KBytes       
iperf3: error - control socket has closed unexpectedly

single thread version does not have this issue.

[bmah888]

Hmmm...interesting. What was the version number for the single-threaded iperf3 you tried?

[fgggid]

single thread version is 3.13 too
Note: They are both patched with PR 1278, I think this might not be a issue.

[davidBar-On]

The idle timeout issue also happens in Ubuntu WSL - after --rcv-timeout time (default is 120 sec.). In reverse mode it happens in the client. It seems that the issue is because the multi-thread solution does not go through the select(). A fix/work-around may be to set settings->rcv_timeout on the receiver side (both server and client for bidir) to a value greater than the test time duration, and when duration is 0 pass NULL to select() instead of timeout.

[rjmcmahon]

Some initial local host testing

`[root@mlmach1 iperf-3.13-mt1]# uname -r
5.14.10-300.fc35.x86_64

[root@mlmach1 iperf-3.13-mt1]# src/iperf3 -v
iperf 3.13-mt1 (cJSON 1.7.15)
Linux mlmach1 5.14.10-300.fc35.x86_64 #1 SMP Thu Oct 7 20:48:44 UTC 2021 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, bind to device, support IPv4 don't fragment, POSIX threads

[root@mlmach1 iperf-3.13-mt1]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
BIOS Vendor ID: Intel(R) Corporation
Model name: Intel(R) Xeon(R) Gold 6250 CPU @ 3.90GHz
BIOS Model name: Intel(R) Xeon(R) Gold 6250 CPU @ 3.90GHz
CPU family: 6
Model: 85
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 7
CPU max MHz: 4500.0000
CPU min MHz: 1200.0000
BogoMIPS: 7800.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc
art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm p
cid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single inte
l_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx5
12f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local
dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 256 KiB (8 instances)
L1i: 256 KiB (8 instances)
L2: 8 MiB (8 instances)
L3: 35.8 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerabilities:
Itlb multihit: KVM: Mitigation: VMX disabled
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Srbds: Not affected
Tsx async abort: Mitigation; TSX disabled

[root@mlmach1 iperf-3.13-mt1]# src/iperf3 -c localhost -t 3
Connecting to host localhost, port 5201
[ 5] local ::1 port 38282 connected to ::1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 5.86 GBytes 50.3 Gbits/sec 0 1.19 MBytes
[ 5] 1.00-2.00 sec 5.70 GBytes 49.0 Gbits/sec 0 1.19 MBytes
[ 5] 2.00-3.00 sec 5.70 GBytes 49.0 Gbits/sec 0 1.19 MBytes

---

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-3.00 sec 17.3 GBytes 49.4 Gbits/sec 0 sender
[ 5] 0.00-3.00 sec 17.3 GBytes 49.4 Gbits/sec receiver

iperf Done.
[root@mlmach1 iperf-3.13-mt1]# src/iperf3 -c localhost -t 3 -P 2
Connecting to host localhost, port 5201
[ 5] local ::1 port 38286 connected to ::1 port 5201
[ 7] local ::1 port 38288 connected to ::1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 5.13 GBytes 44.0 Gbits/sec 0 1.25 MBytes
[ 7] 0.00-1.00 sec 5.12 GBytes 43.9 Gbits/sec 0 1.25 MBytes
[SUM] 0.00-1.00 sec 10.2 GBytes 88.0 Gbits/sec 0

---

[ 5] 1.00-2.00 sec 5.15 GBytes 44.2 Gbits/sec 0 1.25 MBytes
[ 7] 1.00-2.00 sec 4.78 GBytes 41.0 Gbits/sec 0 1.25 MBytes
[SUM] 1.00-2.00 sec 9.93 GBytes 85.3 Gbits/sec 0

---

[ 5] 2.00-3.00 sec 5.08 GBytes 43.6 Gbits/sec 0 1.25 MBytes
[ 7] 2.00-3.00 sec 4.86 GBytes 41.8 Gbits/sec 0 1.25 MBytes
[SUM] 2.00-3.00 sec 9.94 GBytes 85.4 Gbits/sec 0

---

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-3.00 sec 15.4 GBytes 44.0 Gbits/sec 0 sender
[ 5] 0.00-3.00 sec 15.4 GBytes 43.9 Gbits/sec receiver
[ 7] 0.00-3.00 sec 14.8 GBytes 42.2 Gbits/sec 0 sender
[ 7] 0.00-3.00 sec 14.8 GBytes 42.2 Gbits/sec receiver
[SUM] 0.00-3.00 sec 30.1 GBytes 86.2 Gbits/sec 0 sender
[SUM] 0.00-3.00 sec 30.1 GBytes 86.2 Gbits/sec receiver

iperf Done.
[root@mlmach1 iperf-3.13-mt1]# src/iperf3 -c localhost -t 3 -P 4
Connecting to host localhost, port 5201
[ 5] local ::1 port 38292 connected to ::1 port 5201
[ 7] local ::1 port 38294 connected to ::1 port 5201
[ 9] local ::1 port 38296 connected to ::1 port 5201
[ 11] local ::1 port 38298 connected to ::1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 4.50 GBytes 38.6 Gbits/sec 0 1.06 MBytes
[ 7] 0.00-1.00 sec 4.47 GBytes 38.3 Gbits/sec 0 1.25 MBytes
[ 9] 0.00-1.00 sec 4.48 GBytes 38.5 Gbits/sec 0 1.06 MBytes
[ 11] 0.00-1.00 sec 4.46 GBytes 38.3 Gbits/sec 0 1.37 MBytes
[SUM] 0.00-1.00 sec 17.9 GBytes 154 Gbits/sec 0

---

[ 5] 1.00-2.00 sec 4.07 GBytes 34.9 Gbits/sec 0 1.06 MBytes
[ 7] 1.00-2.00 sec 4.53 GBytes 38.9 Gbits/sec 0 1.25 MBytes
[ 9] 1.00-2.00 sec 4.65 GBytes 39.9 Gbits/sec 0 1.06 MBytes
[ 11] 1.00-2.00 sec 4.54 GBytes 39.0 Gbits/sec 0 1.37 MBytes
[SUM] 1.00-2.00 sec 17.8 GBytes 153 Gbits/sec 0

---

[ 5] 2.00-3.00 sec 4.37 GBytes 37.5 Gbits/sec 0 1.06 MBytes
[ 7] 2.00-3.00 sec 4.20 GBytes 36.1 Gbits/sec 0 1.25 MBytes
[ 9] 2.00-3.00 sec 4.62 GBytes 39.7 Gbits/sec 0 1.06 MBytes
[ 11] 2.00-3.00 sec 4.57 GBytes 39.2 Gbits/sec 0 1.37 MBytes
[SUM] 2.00-3.00 sec 17.8 GBytes 153 Gbits/sec 0

---

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-3.00 sec 12.9 GBytes 37.0 Gbits/sec 0 sender
[ 5] 0.00-3.00 sec 12.9 GBytes 37.0 Gbits/sec receiver
[ 7] 0.00-3.00 sec 13.2 GBytes 37.8 Gbits/sec 0 sender
[ 7] 0.00-3.00 sec 13.2 GBytes 37.8 Gbits/sec receiver
[ 9] 0.00-3.00 sec 13.8 GBytes 39.4 Gbits/sec 0 sender
[ 9] 0.00-3.00 sec 13.8 GBytes 39.4 Gbits/sec receiver
[ 11] 0.00-3.00 sec 13.6 GBytes 38.8 Gbits/sec 0 sender
[ 11] 0.00-3.00 sec 13.6 GBytes 38.8 Gbits/sec receiver
[SUM] 0.00-3.00 sec 53.5 GBytes 153 Gbits/sec 0 sender
[SUM] 0.00-3.00 sec 53.5 GBytes 153 Gbits/sec receiver

iperf Done.
[root@mlmach1 iperf-3.13-mt1]# src/iperf3 -c localhost -t 3 -P 8
Connecting to host localhost, port 5201
[ 5] local ::1 port 38302 connected to ::1 port 5201
[ 7] local ::1 port 38304 connected to ::1 port 5201
[ 9] local ::1 port 38306 connected to ::1 port 5201
[ 11] local ::1 port 38310 connected to ::1 port 5201
[ 13] local ::1 port 38312 connected to ::1 port 5201
[ 15] local ::1 port 38314 connected to ::1 port 5201
[ 17] local ::1 port 38316 connected to ::1 port 5201
[ 19] local ::1 port 38318 connected to ::1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 3.41 GBytes 29.3 Gbits/sec 0 1.19 MBytes
[ 7] 0.00-1.00 sec 3.09 GBytes 26.5 Gbits/sec 0 1.56 MBytes
[ 9] 0.00-1.00 sec 3.42 GBytes 29.4 Gbits/sec 0 1.25 MBytes
[ 11] 0.00-1.00 sec 3.67 GBytes 31.5 Gbits/sec 0 1.44 MBytes
[ 13] 0.00-1.00 sec 3.12 GBytes 26.8 Gbits/sec 0 1.31 MBytes
[ 15] 0.00-1.00 sec 3.51 GBytes 30.1 Gbits/sec 0 1.06 MBytes
[ 17] 0.00-1.00 sec 3.37 GBytes 28.9 Gbits/sec 0 1.19 MBytes
[ 19] 0.00-1.00 sec 3.26 GBytes 28.0 Gbits/sec 0 1.25 MBytes
[SUM] 0.00-1.00 sec 26.8 GBytes 230 Gbits/sec 0

---

[ 5] 1.00-2.00 sec 3.46 GBytes 29.7 Gbits/sec 0 1.19 MBytes
[ 7] 1.00-2.00 sec 2.88 GBytes 24.7 Gbits/sec 0 1.56 MBytes
[ 9] 1.00-2.00 sec 3.50 GBytes 30.1 Gbits/sec 0 1.25 MBytes
[ 11] 1.00-2.00 sec 3.58 GBytes 30.7 Gbits/sec 0 1.44 MBytes
[ 13] 1.00-2.00 sec 3.06 GBytes 26.3 Gbits/sec 0 1.31 MBytes
[ 15] 1.00-2.00 sec 3.52 GBytes 30.2 Gbits/sec 0 1.06 MBytes
[ 17] 1.00-2.00 sec 3.39 GBytes 29.1 Gbits/sec 0 1.25 MBytes
[ 19] 1.00-2.00 sec 3.43 GBytes 29.5 Gbits/sec 0 1.25 MBytes
[SUM] 1.00-2.00 sec 26.8 GBytes 230 Gbits/sec 0

---

[ 5] 2.00-3.00 sec 3.48 GBytes 29.9 Gbits/sec 0 1.19 MBytes
[ 7] 2.00-3.00 sec 2.86 GBytes 24.6 Gbits/sec 0 1.56 MBytes
[ 9] 2.00-3.00 sec 3.49 GBytes 30.0 Gbits/sec 0 1.25 MBytes
[ 11] 2.00-3.00 sec 3.79 GBytes 32.5 Gbits/sec 0 1.44 MBytes
[ 13] 2.00-3.00 sec 3.17 GBytes 27.2 Gbits/sec 0 1.37 MBytes
[ 15] 2.00-3.00 sec 3.54 GBytes 30.4 Gbits/sec 0 1.06 MBytes
[ 17] 2.00-3.00 sec 3.42 GBytes 29.3 Gbits/sec 0 1.31 MBytes
[ 19] 2.00-3.00 sec 3.27 GBytes 28.1 Gbits/sec 0 1.31 MBytes
[SUM] 2.00-3.00 sec 27.0 GBytes 232 Gbits/sec 0

---

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-3.00 sec 10.3 GBytes 29.6 Gbits/sec 0 sender
[ 5] 0.00-3.00 sec 10.3 GBytes 29.6 Gbits/sec receiver
[ 7] 0.00-3.00 sec 8.83 GBytes 25.3 Gbits/sec 0 sender
[ 7] 0.00-3.00 sec 8.83 GBytes 25.3 Gbits/sec receiver
[ 9] 0.00-3.00 sec 10.4 GBytes 29.8 Gbits/sec 0 sender
[ 9] 0.00-3.00 sec 10.4 GBytes 29.8 Gbits/sec receiver
[ 11] 0.00-3.00 sec 11.0 GBytes 31.6 Gbits/sec 0 sender
[ 11] 0.00-3.00 sec 11.0 GBytes 31.5 Gbits/sec receiver
[ 13] 0.00-3.00 sec 9.35 GBytes 26.8 Gbits/sec 0 sender
[ 13] 0.00-3.00 sec 9.35 GBytes 26.8 Gbits/sec receiver
[ 15] 0.00-3.00 sec 10.6 GBytes 30.2 Gbits/sec 0 sender
[ 15] 0.00-3.00 sec 10.6 GBytes 30.2 Gbits/sec receiver
[ 17] 0.00-3.00 sec 10.2 GBytes 29.1 Gbits/sec 0 sender
[ 17] 0.00-3.00 sec 10.2 GBytes 29.1 Gbits/sec receiver
[ 19] 0.00-3.00 sec 9.97 GBytes 28.5 Gbits/sec 0 sender
[ 19] 0.00-3.00 sec 9.97 GBytes 28.5 Gbits/sec receiver
[SUM] 0.00-3.00 sec 80.7 GBytes 231 Gbits/sec 0 sender
[SUM] 0.00-3.00 sec 80.7 GBytes 231 Gbits/sec receiver

iperf Done.
[root@mlmach1 iperf-3.13-mt1]# src/iperf3 -c localhost -t 3 -P 16
Connecting to host localhost, port 5201
[ 5] local ::1 port 38322 connected to ::1 port 5201
[ 7] local ::1 port 38324 connected to ::1 port 5201
[ 9] local ::1 port 38326 connected to ::1 port 5201
[ 11] local ::1 port 38328 connected to ::1 port 5201
[ 13] local ::1 port 38330 connected to ::1 port 5201
[ 15] local ::1 port 38332 connected to ::1 port 5201
[ 17] local ::1 port 38334 connected to ::1 port 5201
[ 19] local ::1 port 38336 connected to ::1 port 5201
[ 21] local ::1 port 38340 connected to ::1 port 5201
[ 23] local ::1 port 38342 connected to ::1 port 5201
[ 25] local ::1 port 38344 connected to ::1 port 5201
[ 27] local ::1 port 38348 connected to ::1 port 5201
[ 29] local ::1 port 38350 connected to ::1 port 5201
[ 31] local ::1 port 38352 connected to ::1 port 5201
[ 33] local ::1 port 38354 connected to ::1 port 5201
[ 35] local ::1 port 38356 connected to ::1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.34 GBytes 11.5 Gbits/sec 25 3.18 MBytes
[ 7] 0.00-1.00 sec 1.50 GBytes 12.8 Gbits/sec 28 3.12 MBytes
[ 9] 0.00-1.01 sec 1.45 GBytes 12.3 Gbits/sec 29 3.12 MBytes
[ 11] 0.00-1.01 sec 1.46 GBytes 12.4 Gbits/sec 34 3.12 MBytes
[ 13] 0.00-1.01 sec 1.12 GBytes 9.54 Gbits/sec 28 3.18 MBytes
[ 15] 0.00-1.01 sec 1.21 GBytes 10.3 Gbits/sec 37 3.06 MBytes
[ 17] 0.00-1.01 sec 1.10 GBytes 9.37 Gbits/sec 21 2.43 MBytes
[ 19] 0.00-1.01 sec 1.18 GBytes 10.0 Gbits/sec 40 2.75 MBytes
[ 21] 0.00-1.01 sec 1.40 GBytes 11.9 Gbits/sec 37 3.12 MBytes
[ 23] 0.00-1.01 sec 1.40 GBytes 11.9 Gbits/sec 41 3.12 MBytes
[ 25] 0.00-1.01 sec 1.05 GBytes 8.94 Gbits/sec 20 3.18 MBytes
[ 27] 0.00-1.01 sec 1.07 GBytes 9.05 Gbits/sec 25 3.12 MBytes
[ 29] 0.00-1.01 sec 929 MBytes 7.71 Gbits/sec 15 3.12 MBytes
[ 31] 0.00-1.01 sec 1.32 GBytes 11.2 Gbits/sec 27 2.25 MBytes
[ 33] 0.00-1.01 sec 1.67 GBytes 14.2 Gbits/sec 58 3.12 MBytes
[ 35] 0.00-1.01 sec 1.09 GBytes 9.27 Gbits/sec 5 2.50 MBytes
[SUM] 0.00-1.00 sec 20.3 GBytes 174 Gbits/sec 470

---

[ 5] 1.00-2.00 sec 1.10 GBytes 9.48 Gbits/sec 41 3.18 MBytes
[ 7] 1.00-2.00 sec 1.60 GBytes 13.8 Gbits/sec 52 3.12 MBytes
[ 9] 1.01-2.01 sec 1.47 GBytes 12.7 Gbits/sec 65 3.12 MBytes
[ 11] 1.01-2.01 sec 1.42 GBytes 12.3 Gbits/sec 52 3.12 MBytes
[ 13] 1.01-2.01 sec 1.22 GBytes 10.5 Gbits/sec 37 3.18 MBytes
[ 15] 1.01-2.01 sec 1.24 GBytes 10.7 Gbits/sec 41 3.06 MBytes
[ 17] 1.01-2.01 sec 1.12 GBytes 9.62 Gbits/sec 23 3.18 MBytes
[ 19] 1.01-2.01 sec 1.21 GBytes 10.4 Gbits/sec 58 3.12 MBytes
[ 21] 1.01-2.01 sec 1.18 GBytes 10.1 Gbits/sec 66 3.12 MBytes
[ 23] 1.01-2.01 sec 1006 MBytes 8.47 Gbits/sec 47 3.12 MBytes
[ 25] 1.01-2.01 sec 1.13 GBytes 9.70 Gbits/sec 38 3.18 MBytes
[ 27] 1.01-2.01 sec 1.43 GBytes 12.3 Gbits/sec 35 3.12 MBytes
[ 29] 1.01-2.01 sec 1.16 GBytes 10.0 Gbits/sec 34 3.12 MBytes
[ 31] 1.01-2.01 sec 1.15 GBytes 9.89 Gbits/sec 37 3.18 MBytes
[ 33] 1.01-2.01 sec 1.51 GBytes 13.0 Gbits/sec 50 3.12 MBytes
[ 35] 1.01-2.01 sec 1.12 GBytes 9.61 Gbits/sec 29 3.18 MBytes
[SUM] 1.00-2.00 sec 20.0 GBytes 172 Gbits/sec 705

---

[ 5] 2.00-3.01 sec 1.08 GBytes 9.20 Gbits/sec 25 2.68 MBytes
[ 7] 2.00-3.01 sec 1.70 GBytes 14.5 Gbits/sec 41 3.12 MBytes
[ 9] 2.01-3.01 sec 1.68 GBytes 14.4 Gbits/sec 53 3.12 MBytes
[ 11] 2.01-3.01 sec 1.54 GBytes 13.2 Gbits/sec 37 3.12 MBytes
[ 13] 2.01-3.01 sec 1.20 GBytes 10.3 Gbits/sec 37 3.18 MBytes
[ 15] 2.01-3.01 sec 1.18 GBytes 10.1 Gbits/sec 43 3.18 MBytes
[ 17] 2.01-3.01 sec 1.05 GBytes 9.02 Gbits/sec 15 3.12 MBytes
[ 19] 2.01-3.01 sec 1.21 GBytes 10.3 Gbits/sec 41 2.56 MBytes
[ 21] 2.01-3.01 sec 1.16 GBytes 9.93 Gbits/sec 51 2.06 MBytes
[ 23] 2.01-3.01 sec 1.43 GBytes 12.2 Gbits/sec 89 3.12 MBytes
[ 25] 2.01-3.01 sec 1.11 GBytes 9.50 Gbits/sec 15 3.12 MBytes
[ 27] 2.01-3.01 sec 1.28 GBytes 11.0 Gbits/sec 23 3.12 MBytes
[ 29] 2.01-3.01 sec 1.10 GBytes 9.44 Gbits/sec 31 3.12 MBytes
[ 31] 2.01-3.01 sec 1.21 GBytes 10.4 Gbits/sec 47 3.18 MBytes
[ 33] 2.01-3.01 sec 1.43 GBytes 12.2 Gbits/sec 41 3.12 MBytes
[ 35] 2.01-3.01 sec 1.13 GBytes 9.67 Gbits/sec 43 3.18 MBytes
[SUM] 2.00-3.01 sec 20.5 GBytes 174 Gbits/sec 632

---

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-3.01 sec 3.52 GBytes 10.0 Gbits/sec 91 sender
[ 5] 0.00-3.01 sec 3.52 GBytes 10.0 Gbits/sec receiver
[ 7] 0.00-3.01 sec 4.81 GBytes 13.7 Gbits/sec 121 sender
[ 7] 0.00-3.01 sec 4.81 GBytes 13.7 Gbits/sec receiver
[ 9] 0.00-3.01 sec 4.60 GBytes 13.1 Gbits/sec 147 sender
[ 9] 0.00-3.01 sec 4.60 GBytes 13.1 Gbits/sec receiver
[ 11] 0.00-3.01 sec 4.42 GBytes 12.6 Gbits/sec 123 sender
[ 11] 0.00-3.01 sec 4.42 GBytes 12.6 Gbits/sec receiver
[ 13] 0.00-3.01 sec 3.54 GBytes 10.1 Gbits/sec 102 sender
[ 13] 0.00-3.01 sec 3.54 GBytes 10.1 Gbits/sec receiver
[ 15] 0.00-3.01 sec 3.63 GBytes 10.4 Gbits/sec 121 sender
[ 15] 0.00-3.01 sec 3.63 GBytes 10.4 Gbits/sec receiver
[ 17] 0.00-3.01 sec 3.27 GBytes 9.34 Gbits/sec 59 sender
[ 17] 0.00-3.01 sec 3.27 GBytes 9.34 Gbits/sec receiver
[ 19] 0.00-3.01 sec 3.60 GBytes 10.3 Gbits/sec 139 sender
[ 19] 0.00-3.01 sec 3.60 GBytes 10.3 Gbits/sec receiver
[ 21] 0.00-3.01 sec 3.74 GBytes 10.7 Gbits/sec 154 sender
[ 21] 0.00-3.01 sec 3.74 GBytes 10.7 Gbits/sec receiver
[ 23] 0.00-3.01 sec 3.81 GBytes 10.9 Gbits/sec 177 sender
[ 23] 0.00-3.01 sec 3.81 GBytes 10.9 Gbits/sec receiver
[ 25] 0.00-3.01 sec 3.29 GBytes 9.38 Gbits/sec 73 sender
[ 25] 0.00-3.01 sec 3.29 GBytes 9.38 Gbits/sec receiver
[ 27] 0.00-3.01 sec 3.78 GBytes 10.8 Gbits/sec 83 sender
[ 27] 0.00-3.01 sec 3.78 GBytes 10.8 Gbits/sec receiver
[ 29] 0.00-3.01 sec 3.17 GBytes 9.04 Gbits/sec 80 sender
[ 29] 0.00-3.01 sec 3.17 GBytes 9.04 Gbits/sec receiver
[ 31] 0.00-3.01 sec 3.68 GBytes 10.5 Gbits/sec 111 sender
[ 31] 0.00-3.01 sec 3.68 GBytes 10.5 Gbits/sec receiver
[ 33] 0.00-3.01 sec 4.60 GBytes 13.1 Gbits/sec 149 sender
[ 33] 0.00-3.01 sec 4.60 GBytes 13.1 Gbits/sec receiver
[ 35] 0.00-3.01 sec 3.33 GBytes 9.52 Gbits/sec 77 sender
[ 35] 0.00-3.01 sec 3.33 GBytes 9.52 Gbits/sec receiver
[SUM] 0.00-3.01 sec 60.8 GBytes 173 Gbits/sec 1807 sender
[SUM] 0.00-3.01 sec 60.8 GBytes 173 Gbits/sec receiver

iperf Done.

[rjmcmahon]

Iperf2 numbers - same system

`[root@mlmach1 iperf2-code]# src/iperf -c localhost -t 3 -e

Client connecting to localhost, TCP port 5001 with pid 229773 (1 flows)
Write buffer size: 131072 Byte
TOS set to 0x0 (Nagle on)
TCP window size: 2.50 MByte (default)

[ 1] local 127.0.0.1%lo port 49940 connected with 127.0.0.1 port 5001 (sock=3) (icwnd/mss/irtt=319/32741/21) (ct=0.06 ms) on 2023-05-17 10:25:46 (PDT)
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT(var) NetPwr
[ 1] 0.00-3.00 sec 8.02 GBytes 23.0 Gbits/sec 65740/0 0 1278K/370(659) us 7755631
[root@mlmach1 iperf2-code]# src/iperf -c localhost -t 3 -e -P 2

Client connecting to localhost, TCP port 5001 with pid 229777 (2 flows)
Write buffer size: 131072 Byte
TOS set to 0x0 (Nagle on)
TCP window size: 2.50 MByte (default)

[ 1] local 127.0.0.1%lo port 49944 connected with 127.0.0.1 port 5001 (sock=3) (icwnd/mss/irtt=319/32741/22) (ct=0.06 ms) on 2023-05-17 10:25:54 (PDT)
[ 2] local 127.0.0.1%lo port 49942 connected with 127.0.0.1 port 5001 (sock=4) (icwnd/mss/irtt=319/32741/22) (ct=0.06 ms) on 2023-05-17 10:25:54 (PDT)
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT(var) NetPwr
[ 2] 0.00-3.01 sec 8.63 GBytes 24.6 Gbits/sec 70720/0 0 1982K/1440(2802) us 2137594
[ 1] 0.00-3.01 sec 8.56 GBytes 24.4 Gbits/sec 70127/0 0 1982K/1450(2823) us 2105000
[SUM] 0.00-3.00 sec 17.2 GBytes 49.2 Gbits/sec 140847/0 0
[ CT] final connect times (min/avg/max/stdev) = 0.058/0.061/0.064/4.243 ms (tot/err) = 2/0
[root@mlmach1 iperf2-code]# src/iperf -c localhost -t 3 -e -P 4

Client connecting to localhost, TCP port 5001 with pid 229783 (4 flows)
Write buffer size: 131072 Byte
TOS set to 0x0 (Nagle on)
TCP window size: 2.50 MByte (default)

[ 2] local 127.0.0.1%lo port 49946 connected with 127.0.0.1 port 5001 (sock=3) (icwnd/mss/irtt=319/32741/22) (ct=0.06 ms) on 2023-05-17 10:26:02 (PDT)
[ 4] local 127.0.0.1%lo port 49952 connected with 127.0.0.1 port 5001 (sock=6) (icwnd/mss/irtt=319/32741/19) (ct=0.05 ms) on 2023-05-17 10:26:02 (PDT)
[ 1] local 127.0.0.1%lo port 49954 connected with 127.0.0.1 port 5001 (sock=8) (icwnd/mss/irtt=319/32741/17) (ct=0.05 ms) on 2023-05-17 10:26:02 (PDT)
[ 3] local 127.0.0.1%lo port 49948 connected with 127.0.0.1 port 5001 (sock=4) (icwnd/mss/irtt=319/32741/18) (ct=0.05 ms) on 2023-05-17 10:26:02 (PDT)
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT(var) NetPwr
[ 4] 0.00-3.01 sec 8.99 GBytes 25.6 Gbits/sec 73657/0 0 2046K/1558(3044) us 2057141
[ 2] 0.00-3.01 sec 8.98 GBytes 25.6 Gbits/sec 73604/0 0 1918K/1557(3053) us 2056954
[ 1] 0.00-3.01 sec 11.9 GBytes 34.0 Gbits/sec 97805/0 0 1982K/1471(2847) us 2893090
[ 3] 0.00-3.01 sec 9.39 GBytes 26.8 Gbits/sec 76941/0 0 1982K/1551(3028) us 2158596
[SUM] 0.00-3.00 sec 39.3 GBytes 113 Gbits/sec 322007/0 0
[ CT] final connect times (min/avg/max/stdev) = 0.047/0.053/0.061/5.795 ms (tot/err) = 4/0
[root@mlmach1 iperf2-code]# src/iperf -c localhost -t 3 -e -P 8

Client connecting to localhost, TCP port 5001 with pid 229793 (8 flows)
Write buffer size: 131072 Byte
TOS set to 0x0 (Nagle on)
TCP window size: 2.50 MByte (default)

[ 2] local 127.0.0.1%lo port 49960 connected with 127.0.0.1 port 5001 (sock=4) (icwnd/mss/irtt=319/32741/22) (ct=0.06 ms) on 2023-05-17 10:26:08 (PDT)
[ 4] local 127.0.0.1%lo port 49956 connected with 127.0.0.1 port 5001 (sock=3) (icwnd/mss/irtt=319/32741/24) (ct=0.06 ms) on 2023-05-17 10:26:08 (PDT)
[ 8] local 127.0.0.1%lo port 49974 connected with 127.0.0.1 port 5001 (sock=15) (icwnd/mss/irtt=319/32741/23) (ct=0.05 ms) on 2023-05-17 10:26:08 (PDT)
[ 5] local 127.0.0.1%lo port 49966 connected with 127.0.0.1 port 5001 (sock=7) (icwnd/mss/irtt=319/32741/23) (ct=0.05 ms) on 2023-05-17 10:26:08 (PDT)
[ 3] local 127.0.0.1%lo port 49958 connected with 127.0.0.1 port 5001 (sock=5) (icwnd/mss/irtt=319/32741/22) (ct=0.06 ms) on 2023-05-17 10:26:08 (PDT)
[ 6] local 127.0.0.1%lo port 49970 connected with 127.0.0.1 port 5001 (sock=12) (icwnd/mss/irtt=319/32741/26) (ct=0.06 ms) on 2023-05-17 10:26:08 (PDT)
[ 1] local 127.0.0.1%lo port 49964 connected with 127.0.0.1 port 5001 (sock=6) (icwnd/mss/irtt=319/32741/22) (ct=0.06 ms) on 2023-05-17 10:26:08 (PDT)
[ 7] local 127.0.0.1%lo port 49972 connected with 127.0.0.1 port 5001 (sock=13) (icwnd/mss/irtt=319/32741/23) (ct=0.06 ms) on 2023-05-17 10:26:08 (PDT)
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT(var) NetPwr
[ 6] 0.00-3.01 sec 8.29 GBytes 23.7 Gbits/sec 67880/0 0 3197K/64(7) us 46199414
[ 5] 0.00-3.01 sec 10.3 GBytes 29.5 Gbits/sec 84513/0 0 2046K/61(13) us 60351840
[ 4] 0.00-3.01 sec 8.72 GBytes 24.9 Gbits/sec 71453/0 0 3261K/1120(2192) us 2779167
[ 1] 0.00-3.01 sec 8.47 GBytes 24.2 Gbits/sec 69356/0 1 3197K/1128(2167) us 2678506
[ 8] 0.00-3.01 sec 8.80 GBytes 25.1 Gbits/sec 72074/0 0 3197K/1156(2229) us 2715841
[ 3] 0.00-3.01 sec 8.00 GBytes 22.9 Gbits/sec 65573/0 0 3261K/1161(2213) us 2460309
[ 2] 0.00-3.01 sec 9.57 GBytes 27.3 Gbits/sec 78367/0 0 2749K/57(6) us 59889625
[ 7] 0.00-3.01 sec 8.26 GBytes 23.6 Gbits/sec 67646/0 1 3197K/544(1000) us 5416980
[SUM] 0.00-3.00 sec 70.4 GBytes 202 Gbits/sec 576862/0 2
[ CT] final connect times (min/avg/max/stdev) = 0.050/0.057/0.064/4.472 ms (tot/err) = 8/0
[root@mlmach1 iperf2-code]# src/iperf -c localhost -t 3 -e -P 16

Client connecting to localhost, TCP port 5001 with pid 229812 (16 flows)
Write buffer size: 131072 Byte
TOS set to 0x0 (Nagle on)
TCP window size: 2.50 MByte (default)

[ 8] local 127.0.0.1%lo port 50000 connected with 127.0.0.1 port 5001 (sock=13) (icwnd/mss/irtt=319/32741/16) (ct=0.03 ms) on 2023-05-17 10:26:17 (PDT)
[ 1] local 127.0.0.1%lo port 49998 connected with 127.0.0.1 port 5001 (sock=6) (icwnd/mss/irtt=319/32741/10) (ct=0.03 ms) on 2023-05-17 10:26:17 (PDT)
[ 3] local 127.0.0.1%lo port 49984 connected with 127.0.0.1 port 5001 (sock=7) (icwnd/mss/irtt=319/32741/12) (ct=0.04 ms) on 2023-05-17 10:26:17 (PDT)
[ 11] local 127.0.0.1%lo port 49980 connected with 127.0.0.1 port 5001 (sock=4) (icwnd/mss/irtt=319/32741/20) (ct=0.05 ms) on 2023-05-17 10:26:17 (PDT)
[ 5] local 127.0.0.1%lo port 49996 connected with 127.0.0.1 port 5001 (sock=11) (icwnd/mss/irtt=319/32741/18) (ct=0.05 ms) on 2023-05-17 10:26:17 (PDT)
[ 14] local 127.0.0.1%lo port 50004 connected with 127.0.0.1 port 5001 (sock=19) (icwnd/mss/irtt=319/32741/14) (ct=0.04 ms) on 2023-05-17 10:26:17 (PDT)
[ 12] local 127.0.0.1%lo port 49990 connected with 127.0.0.1 port 5001 (sock=8) (icwnd/mss/irtt=319/32741/18) (ct=0.04 ms) on 2023-05-17 10:26:17 (PDT)
[ 13] local 127.0.0.1%lo port 50006 connected with 127.0.0.1 port 5001 (sock=24) (icwnd/mss/irtt=319/32741/20) (ct=0.05 ms) on 2023-05-17 10:26:17 (PDT)
[ 10] local 127.0.0.1%lo port 50012 connected with 127.0.0.1 port 5001 (sock=23) (icwnd/mss/irtt=319/32741/13) (ct=0.04 ms) on 2023-05-17 10:26:17 (PDT)
[ 9] local 127.0.0.1%lo port 49976 connected with 127.0.0.1 port 5001 (sock=3) (icwnd/mss/irtt=319/32741/22) (ct=0.06 ms) on 2023-05-17 10:26:17 (PDT)
[ 16] local 127.0.0.1%lo port 50010 connected with 127.0.0.1 port 5001 (sock=31) (icwnd/mss/irtt=319/32741/11) (ct=0.04 ms) on 2023-05-17 10:26:17 (PDT)
[ 15] local 127.0.0.1%lo port 50008 connected with 127.0.0.1 port 5001 (sock=27) (icwnd/mss/irtt=319/32741/12) (ct=0.04 ms) on 2023-05-17 10:26:17 (PDT)
[ 4] local 127.0.0.1%lo port 49988 connected with 127.0.0.1 port 5001 (sock=9) (icwnd/mss/irtt=319/32741/15) (ct=0.04 ms) on 2023-05-17 10:26:17 (PDT)
[ 6] local 127.0.0.1%lo port 50002 connected with 127.0.0.1 port 5001 (sock=12) (icwnd/mss/irtt=319/32741/16) (ct=0.05 ms) on 2023-05-17 10:26:17 (PDT)
[ 7] local 127.0.0.1%lo port 49986 connected with 127.0.0.1 port 5001 (sock=10) (icwnd/mss/irtt=319/32741/18) (ct=0.04 ms) on 2023-05-17 10:26:17 (PDT)
[ 2] local 127.0.0.1%lo port 49982 connected with 127.0.0.1 port 5001 (sock=5) (icwnd/mss/irtt=319/32741/21) (ct=0.05 ms) on 2023-05-17 10:26:17 (PDT)
[ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT(var) NetPwr
[ 2] 0.00-3.01 sec 5.67 GBytes 16.2 Gbits/sec 46423/0 4 3197K/862(1233) us 2344667
[ 15] 0.00-3.01 sec 5.53 GBytes 15.8 Gbits/sec 45276/0 1 3197K/1578(2775) us 1249159
[ 12] 0.00-3.01 sec 5.51 GBytes 15.7 Gbits/sec 45106/0 7 3197K/1616(2852) us 1215134
[ 14] 0.00-3.01 sec 5.41 GBytes 15.4 Gbits/sec 44295/0 3 3197K/1597(2805) us 1207493
[ 4] 0.00-3.01 sec 5.59 GBytes 16.0 Gbits/sec 45802/0 3 3197K/1599(2813) us 1247093
[ 3] 0.00-3.01 sec 5.56 GBytes 15.9 Gbits/sec 45518/0 0 3197K/1577(2743) us 1256694
[ 5] 0.00-3.01 sec 5.57 GBytes 15.9 Gbits/sec 45664/0 4 3197K/1572(2753) us 1264710
[ 9] 0.00-3.01 sec 5.53 GBytes 15.8 Gbits/sec 45307/0 1 3261K/128(140) us 15413166
[ 10] 0.00-3.01 sec 5.52 GBytes 15.7 Gbits/sec 45204/0 0 3197K/1576(2760) us 1248737
[ 6] 0.00-3.01 sec 5.23 GBytes 14.9 Gbits/sec 42808/0 7 3197K/358(420) us 5205771
[ 7] 0.00-3.01 sec 5.41 GBytes 15.4 Gbits/sec 44324/0 1 3261K/1558(2722) us 1238630
[ 16] 0.00-3.01 sec 5.56 GBytes 15.9 Gbits/sec 45586/0 1 3197K/1570(2773) us 1264201
[ 13] 0.00-3.01 sec 5.56 GBytes 15.9 Gbits/sec 45572/0 1 3197K/863(1231) us 2298917
[ 1] 0.00-3.01 sec 5.56 GBytes 15.9 Gbits/sec 45580/0 0 3197K/1561(2732) us 1271279
[ 8] 0.00-3.01 sec 5.22 GBytes 14.9 Gbits/sec 42785/0 6 3197K/1565(2733) us 1190220
[ 11] 0.00-3.01 sec 5.42 GBytes 15.5 Gbits/sec 44368/0 7 3197K/1609(2840) us 1200498
[SUM] 0.00-3.00 sec 87.8 GBytes 251 Gbits/sec 719618/0 46
[ CT] final connect times (min/avg/max/stdev) = 0.035/0.043/0.061/7.882 ms (tot/err) = 16/0

[davidBar-On]

Based on PR #1527 and testings done by @ffolkes1911 it may be useful to implement the related changes to multi-threading while it is being tested and before it becomes the master branch.

Following are the main related changes. If it will help, I can submit a PR with the related proposed changes.

1. iperf_api.c

• In iperf_parse_arguments remove this condition to allow setting rcv-timeout also for the client in sending mode. Note that with this change IERVRSONLYRCVTIMEOUT is redundant.
• In iperf_set_send_state() after writing the state byte - if this is the end of the test, write additional 3 bytes, since it may be that the server is waiting for JSON and that ensures that the server will not get stuck forever waiting for the 4 bytes JSON header. Something like this:

        if (state == IPERF_DONE || state == CLIENT_TERMINATE) {
            // Send additional bytes to complete the sent size to JSON length prefix,
            // in case the other size is waiting for JSON.
            l = sizeof(uint32_t) - sizeof(state);
            while (l-- > 0)
                Nwrite(test->ctrl_sck, &c, 1, Ptcp);
        }

• Add a function iperf_cancel_test_timers(test) that cancels the 4 timers like in this case.
  Also, replace canceling the 4 timers with call that function here, here and here (not including the congestion_used cancel).

2. iperf_client_api.c in iperf_crun_clienty()

• In iperf_run_client() define int64_t rcv_timeout_value_in_us and replace this if code (keep the else part) with:

    rcv_timeout_value_in_us = (test->settings->rcv_timeout.secs * SEC_TO_US) + test->settings->rcv_timeout.usecs;
    if (test->mode != SENDER)
        rcv_timeout_us = rcv_timeout_value_in_us;

• Replace this if statement with:

        // In reverse/bidir active mode client ensures data is received.
        // Same for receiving control messages at the end of the test. 
        if ( (rcv_timeout_us > 0 && test->state == TEST_RUNNING)
             || (rcv_timeout_value_in_us > 0
                 && (test->state == TEST_END
                     || test->state == EXCHANGE_RESULTS
                     || test->state == DISPLAY_RESULTS)) )
        {

• The write_set in the select() statement seem to be redundant and it can be replaced with NULL. If this is not done, then at least replace it with NULL at the end of the test, in states TEST_END, EXCHANGE_RESULTS and DISPLAY_RESULTS.
• Add call to the new iperf_cancel_test_timers() after call to stats_callbac(), as timers are not needed at test end and may interrupt with select() receive timeout.
• Do not cancel receive streams at TEST_END, to allow to continue receiving, in case network buffers are full and closing the streams will not allow receiving control messages from the server.

3. iperf_server_api.c

• Do not close the test streams on TEST_END, because client's select() returns immediately when monitoring closed sockets,
  and the client's select() timeout is not effective in this case. (Or close only receive streams.)
  If some streams are still closed, add sp->socket = -1; after closing.
• Replace this else if with with:

        } else if (test->mode != SENDER        // In non-reverse active mode server ensures data is received.
                   || test->state == TEST_END  // Same for receiving control messages at the end of the test.
                   || test->state == EXCHANGE_RESULTS
                   || test->state == DISPLAY_RESULTS)
        {

• Same comment as for the client about replacing write_set with NULL in the select() statement.