worker数量与多核心问题

[louishot]

服务器上一共有80个CPU，为什么在配置worker数量为80时测试发并发反而没有配置为8个worker QPS高呢（已启用event扩展）？

配置80个worker时测试结果如下:

root@sbox01:~# wrk -t1000 -c1000 -d30s http://192.168.1.2:80/ --latency
Running 30s test @ http://192.168.1.2:80/
  1000 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.54ms    4.63ms 144.98ms   91.58%
    Req/Sec    70.52     31.24   414.00     74.71%
  Latency Distribution
     50%    3.37ms
     75%    5.70ms
     90%    8.37ms
     99%   24.25ms
  1586330 requests in 30.10s, 11.98GB read
  Socket errors: connect 0, read 334175, write 0, timeout 0
Requests/sec:  52700.09
Transfer/sec:    407.55MB

配置8个worker时测试结果如下:

wrk -t1000 -c1000 -d30s http://192.168.1.2:80/ --latency
Running 30s test @ http://192.168.1.2:80/
  1000 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.73ms    1.59ms  96.66ms   91.69%
    Req/Sec   144.45     24.98     1.28k    96.07%
  Latency Distribution
     50%    6.73ms
     75%    7.09ms
     90%    7.61ms
     99%    9.34ms
  4352469 requests in 30.11s, 27.93GB read
  Socket errors: connect 0, read 11, write 0, timeout 0
Requests/sec: 144573.17
Transfer/sec:      0.93GB

是CPU核心太多相互竞争内耗严重导致吗？

[louishot]

perf top看 native_queued_spin_lock_slowpath 这个占用CPU很高

[louishot]

反复测试了，worker数量超过16时性能无明显提升，并且越大QPS还越低并更不稳定。
目前测试就只是返回一个HTML页面，非计算密集型，网卡为50Gbps，当前测试网卡最高才跑到9Gbps；
iperf测试过网络吞吐量能达到48Gbps，所以这块瓶颈应该在软件上。

[walkor]

试下nginx性能表现

[louishot]

@walkor
反复测试wrk线程数与链接数同worker数量各种组合的情况。
不开keepalive的情况测试最好的结果在27万QPS。wrk -t1000 -c1000 -d30s http://192.168.1.2:80/
打开keepalive最好的情况能到145万 wrk -t40 -c4000 -d30s http://192.168.1.2:80/

是不是每秒新建连接数在哪个阶段遇到瓶颈？

nginx之前keepalive测试好像能到180万

[walkor]

短链接会有建立连接 断开链接开销，QPS低是正常的

[walkor]

按照你之前的说法，14万QPS的时候带宽9G，网卡为50Gbps
现在145万QPS的时候带宽应该满了。

[joanhey]

Hi, I'll try to comment, because it is difficult to me understand Chinese (with translator).

In the Techempower benchmark, they changed to more powerful new servers (28 cores and 56 threads) some months ago, and Workerman scaled perfectly ( ~3x faster).
With the new servers, Workerman use 224 workers before was using 112 workers with the old servers.

Without Keep-alive, you are testing the TCP performance, and this job is done by the kernel not Workerman.
Please, check that your PHP is using the event extension, because the default select from linux is limited to 1024 connections.

The bottleneck can be anything, not necessarily the software (and I doubt that is Workerman).
We need more information, which OS, CPU, PHP version, the code used in Workerman, ....

For the CPU, send the output of lscpu.
If it's affected by Meltdown, Spectre, Retbleed, ... will be using the security mitigations, that have a very big performance drop. With more contexts switch, slower will be, so with more workers and high number of connections it will be slower.

And please send the code (you said that is simple).

The problem can be anything, but for sure that is not from Workerman.

Please send more information, to try to help you.

Thank you.

Update: with the benchmark numbers (req/s)

[joanhey]

Another question is:
Are you running wrk and Workerman in the same computer?
If it's so, both are fighting for the CPU and the TCP limit connections.

[louishot]

按照你之前的说法，14万QPS的时候带宽9G，网卡为50Gbps 现在145万QPS的时候带宽应该满了。

知道这个问题，所以我把body内容调整成2字节了，所以宽带没有瓶颈; 在48个worker时达到145万QPS极限，再增加worker数量QPS不增反降

[louishot]

Another question is: Are you running wrk and Workerman in the same computer? If it's so, both are fighting for the CPU and the TCP limit connections.
@joanhey
No, wrk on another server

[joanhey]

@louishot please in English

I am still waiting for your information.
In the Techempower benchmark with less cores and 40Gb network card , we have more req/s than with your server than have more cores.

And you think that the problem is still Workerman. I don't think so.

[louishot]

@joelhy @walkor
Looks like the bottleneck on Linux kernel I'm not sure, I also had a test with nginx, it can get 1.5M QPS with 24 cores (15% of CPU usage) and no more increase even tried adjust all knobs.
Use workerman with 32 cores can get the same QPS as nginx.
I checked and the number of packets during the test seemed to reach 3Mpps(2.63 Gbps/)
tx 1500381 pkts/s - rx 1500545 pkts/ on interface bond0
https://lwn.net/Articles/629155/

[joanhey]

I think that is your CPU, with the security mitigations.
Please send the output of lscpu in your terminal.

From the Techempower benchmark timeline.

Nginx with "Hello world" test and 28 cores

Nginx full timeline

Workerman full timeline

The benchmark still don't use Ubuntu 24.04, that come with the kernel 6.8.

Network-related: Linux 6.8 includes networking buffs that provide better cache efficiency. This is said to improve “TCP performances with many concurrent connections up to 40%” – a sizeable uplift, though to what degree most users will benefit is unclear.

[joanhey]

@walkor I see a drop in the last changes from v5

Perhaps will be good to check with the timeline, what changes we did in that time.

I send you the url, that show the exact time of each run.
https://tfb-status.techempower.com/timeline/workerman/plaintext

Possibly is not related to Workerman code, but an small inspection is recommended.
I will check the time of the event extension updates, that it can also be the problem.

UPDATE:
I seee the similar drop using Adapterman with other frameworks, and Adapterman use Workerman v4.
So I think that is not the code from Workerman v5. Checking event extension updates.

[walkor]

@joanhey
dc6d96b
a39c619
I think these changes have affected performance.
These commits added more checks, resulting in a decrease in performance.

[louishot]

@joanhey
both running Debian 11 & 5.10.197-1 kernel
Server

Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Byte Order:                           Little Endian
Address sizes:                        46 bits physical, 48 bits virtual
CPU(s):                               80
On-line CPU(s) list:                  0-79
Thread(s) per core:                   2
Core(s) per socket:                   20
Socket(s):                            2
NUMA node(s):                         2
Vendor ID:                            GenuineIntel
CPU family:                           6
Model:                                85
Model name:                           Intel(R) Xeon(R) Gold 6133 CPU @ 2.50GHz
Stepping:                             4
CPU MHz:                              1000.281
CPU max MHz:                          3000.0000
CPU min MHz:                          1000.0000
BogoMIPS:                             5000.00
Virtualization:                       VT-x
L1d cache:                            1.3 MiB
L1i cache:                            1.3 MiB
L2 cache:                             40 MiB
L3 cache:                             55 MiB
NUMA node0 CPU(s):                    0-19,40-59
NUMA node1 CPU(s):                    20-39,60-79
Vulnerability Gather data sampling:   Vulnerable: No microcode
Vulnerability Itlb multihit:          KVM: Mitigation: VMX disabled
Vulnerability L1tf:                   Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:                    Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:               Mitigation; PTI
Vulnerability Mmio stale data:        Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; IBRS, IBPB conditional, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Mitigation; Clear CPU buffers; SMT vulnerable
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pd
                                      pe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_c
                                      pl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdr
                                      and lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi fl
                                      expriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx sma
                                      p clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_loca
                                      l dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

Client

Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      46 bits physical, 48 bits virtual
CPU(s):                             40
On-line CPU(s) list:                0-39
Thread(s) per core:                 2
Core(s) per socket:                 10
Socket(s):                          2
NUMA node(s):                       2
Vendor ID:                          GenuineIntel
CPU family:                         6
Model:                              79
Model name:                         Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
Stepping:                           1
CPU MHz:                            1202.187
CPU max MHz:                        3400.0000
CPU min MHz:                        1200.0000
BogoMIPS:                           4800.10
Virtualization:                     VT-x
L1d cache:                          640 KiB
L1i cache:                          640 KiB
L2 cache:                           5 MiB
L3 cache:                           50 MiB
NUMA node0 CPU(s):                  0-9,20-29
NUMA node1 CPU(s):                  10-19,30-39
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        KVM: Mitigation: VMX disabled
Vulnerability L1tf:                 Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:                  Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:             Mitigation; PTI
Vulnerability Mmio stale data:      Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not 
                                    affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; Clear CPU buffers; SMT vulnerable
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr s
                                    se sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl 
                                    xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sd
                                    bg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f
                                    16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ss
                                    bd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 s
                                    mep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap intel_pt xsaveopt cqm_llc cqm_occup_llc cqm_mbm
                                    _total cqm_mbm_local dtherm ida arat pln pts md_clear flush_l1d

[joanhey]

Your server CPU:

The Xeon Gold CPUs, were the most affected with the security mitigations.
Only with the spectre/meldown https://www.phoronix.com/review/linux-419-mitigations/4
And if you are running VMs, the full mitigation is worst as halving the number of threads available.

The earlier graph from Techempower timeline:

Is from a Xeon Gold 5120 (now we have new servers)

Three homogeneous Dell R440 servers each equipped with an Intel Xeon Gold 5120 CPU, 32 GB of memory, and an enterprise SSD. Dedicated Cisco 10-gigabit Ethernet switch. Provided by Microsoft.

And show clearly how Nginx (and all frameworks) after all the recent security mitigations, the big drop in performance, from 4 millions req/s to the last runs with only ~1.5 millions req/s.

Now they use Xeon Gold 6330 CPUs, that are not affected by most vulnerabilities, and now with the same Nginx configuration, give ~6 millions req/s.

PD: the problem is in the fault CPUs, not in the software security mitigations needed to add to the OS kernels (linux, Windows, ...) to fix the Hardware problem.

[louishot]

@joanhey let me try phoronix test, now I use wrk there is no graph I cannot see the QPS over the time

[joanhey]

You don't need a bench over the time.

Only disable the mitigations and bench again. But remember that your server will be vulnerable.

https://jcvassort.open-web.fr/how-to-disable-cpu-mitigations/