gzip filter #914
Closed
gzip filter #914
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
v2 changes:
|
|
v3 changes:
|
|
v4 changes:
|
|
v5 changes:
|
|
v6 changes:
|
|
v7 changes:
|
|
v8 changes:
|
|
v9 changes:
|
|
Testing: $ cat gzip-v9.json
{
"settings": {
"http": { "log_route": true }
},
"listeners": {
"*:8080": { "pass": "routes/di" },
"*:8081": { "pass": "routes/n" },
"*:81": { "pass": "routes/o" },
"*:82": { "pass": "routes/s" },
"*:83": { "pass": "routes/a" },
"*:84": { "pass": "routes/u" },
"*:85": { "pass": "routes/r" }
},
"routes": {
"di": [{
"action": {
"share": "/srv/www/readme"
}
}],
"n": [{
"action": {
"share": "/srv/www/empty"
}
}],
"o": [{
"action": {
"proxy": "http://127.0.0.1:8081",
"compress": { "encoding": "gzip" }
}
}],
"s": [{
"action": {
"share": "/srv/www/unit/index.html",
"compress": { "encoding": "gzip" }
}
}],
"a": [{
"action": {
"pass": "applications/perl",
"compress": { "encoding": "gzip" }
}
}],
"u": [{
"action": {
"proxy": "http://127.0.0.1:8080",
"compress": { "encoding": "gzip" }
}
}],
"r": [{
"action": {
"return": 321,
"compress": { "encoding": "gzip" }
}
}]
},
"applications": {
"perl": {
"type": "perl",
"working_directory": "/home/alx/srv/www/perl/",
"script": "./sock.psgi"
}
}
}$ cat /home/alx/srv/www/perl/sock.psgi
my $app = sub {
return [
"200",
[ "Content-Type" => "text/plain" ],
[ "Hello, Perl on Unit!\n" ],
];
};$ curl localhost:8080
rdm$ curl localhost:8081$ curl localhost:81$ curl localhost:82
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.$ curl localhost:82 --compressed
idx$ curl localhost:83
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.$ curl localhost:83 --compressed
Hello, Perl on Unit!$ curl localhost:84
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.$ curl localhost:84 --compressed
rdmThis round of testing found some bug, triggered only by requests received on port 83, that is, the application. I'll need to fix that: |
|
v10 changes: Rebase to master. |
|
v11 changes: Drop patch that's unnecessary after the rebase. |
|
v12 changes:
New round of testing: Which results in a clean log: |
|
v13 changes:
All tests pass as in v12 |
|
v14 changes:
|
|
v14b changes:
|
|
v15 changes:
|
|
v16 changes:
|
|
v17 changes:
|
|
v18 changes:
Tested with different compression levels (-1, 0, 3, 6, 7, 9). Example of compression config: "compress": {
"encoding": "gzip",
"level": 0,
"content_length_threshold": 3
} |
|
v19 changes:
Tested with: "compress": {
"encoding": "gzip"
,"mime_types": [
"!text/javascript",
"!text/css",
"!text/*",
"~video/3gpp2?"
]
}and "compress": {
"encoding": "gzip"
,"mime_types": [
"!text/javascript",
"!text/css",
"text/*",
"~video/3gpp2?"
]
}Changes: |
|
Here goes some trivial sanity check that nothing blows up times (far from being a performance test, though): |
|
Performance test: $ sudo unit ctl http GET /config </dev/null
{
"settings": {
"http": {
"log_route": false
}
},
"listeners": {
"*:8080": {
"pass": "routes/d"
},
"*:8081": {
"pass": "routes/n"
},
"*:8082": {
"pass": "routes/i"
},
"*:81": {
"pass": "routes/o"
},
"*:82": {
"pass": "routes/s"
},
"*:83": {
"pass": "routes/a"
},
"*:84": {
"pass": "routes/u"
},
"*:85": {
"pass": "routes/r"
}
},
"routes": {
"d": [
{
"action": {
"share": "/srv/www/readme"
}
}
],
"n": [
{
"action": {
"share": "/srv/www/empty"
}
}
],
"i": [
{
"action": {
"pass": "applications/perl"
}
}
],
"o": [
{
"action": {
"proxy": "http://127.0.0.1:8081",
"compress": {
"encoding": "gzip"
}
}
}
],
"s": [
{
"action": {
"share": "/srv/www/unit/index.html",
"compress": {
"encoding": "gzip"
}
}
}
],
"a": [
{
"action": {
"pass": "applications/perl",
"compress": {
"encoding": "gzip"
}
}
}
],
"u": [
{
"action": {
"proxy": "http://127.0.0.1:8080",
"compress": {
"encoding": "gzip"
}
}
}
],
"r": [
{
"action": {
"return": 321,
"compress": {
"encoding": "gzip"
}
}
}
]
},
"applications": {
"perl": {
"type": "perl",
"working_directory": "/home/alx/srv/www/perl/",
"script": "./sock.psgi"
}
}
}alx@asus5775:~/etc/unitd$ wrk -t2 -c100 -d5s http://localhost:8080
Running 5s test @ http://localhost:8080
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 406.23us 264.82us 5.12ms 74.57%
Req/Sec 87.62k 15.33k 128.37k 64.00%
871886 requests in 5.00s, 138.03MB read
Requests/sec: 174359.62
Transfer/sec: 27.60MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c1000 -d10s http://localhost:8080
Running 10s test @ http://localhost:8080
2 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 10.39ms 49.59ms 791.17ms 97.76%
Req/Sec 108.33k 16.07k 136.13k 70.35%
2156437 requests in 10.10s, 395.04MB read
Socket errors: connect 0, read 392, write 0, timeout 0
Non-2xx or 3xx responses: 1939968
Requests/sec: 213596.65
Transfer/sec: 39.13MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c10000 -d20s http://localhost:8080
Running 20s test @ http://localhost:8080
2 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 5.84ms 40.49ms 1.30s 99.58%
Req/Sec 106.24k 27.19k 142.85k 47.99%
4219373 requests in 20.06s, 772.94MB read
Socket errors: connect 8981, read 799, write 0, timeout 0
Non-2xx or 3xx responses: 3795696
Requests/sec: 210312.61
Transfer/sec: 38.53MBalx@asus5775:~/etc/unitd$ wrk -t2 -c100 -d5s http://localhost:8081
Running 5s test @ http://localhost:8081
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 413.17us 348.52us 8.76ms 94.84%
Req/Sec 84.23k 21.41k 146.61k 78.00%
837638 requests in 5.00s, 129.41MB read
Requests/sec: 167474.18
Transfer/sec: 25.87MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c1000 -d10s http://localhost:8081
Running 10s test @ http://localhost:8081
2 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 9.56ms 46.44ms 609.56ms 98.11%
Req/Sec 107.27k 21.72k 141.48k 55.50%
2142163 requests in 10.09s, 345.32MB read
Socket errors: connect 0, read 403, write 0, timeout 0
Non-2xx or 3xx responses: 456438
Requests/sec: 212344.06
Transfer/sec: 34.23MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c10000 -d20s http://localhost:8081
Running 20s test @ http://localhost:8081
2 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 5.66ms 39.26ms 1.10s 99.61%
Req/Sec 103.62k 32.66k 143.12k 32.16%
4118041 requests in 20.06s, 657.44MB read
Socket errors: connect 8981, read 800, write 0, timeout 0
Non-2xx or 3xx responses: 674340
Requests/sec: 205314.20
Transfer/sec: 32.78MBalx@asus5775:~/etc/unitd$ wrk -t2 -c100 -d5s http://localhost:8082
Running 5s test @ http://localhost:8082
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 700.32us 285.79us 4.63ms 73.81%
Req/Sec 63.03k 6.17k 71.40k 70.00%
627179 requests in 5.00s, 97.49MB read
Requests/sec: 125417.41
Transfer/sec: 19.50MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c1000 -d10s http://localhost:8082
Running 10s test @ http://localhost:8082
2 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 13.44ms 47.79ms 703.84ms 98.31%
Req/Sec 57.92k 7.30k 74.08k 65.00%
1157093 requests in 10.09s, 179.87MB read
Socket errors: connect 0, read 399, write 0, timeout 0
Requests/sec: 114639.78
Transfer/sec: 17.82MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c10000 -d20s http://localhost:8082
Running 20s test @ http://localhost:8082
2 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 9.42ms 37.51ms 1.11s 99.67%
Req/Sec 60.96k 6.99k 74.54k 64.57%
2421044 requests in 20.03s, 376.35MB read
Socket errors: connect 8981, read 800, write 0, timeout 0
Requests/sec: 120859.31
Transfer/sec: 18.79MBalx@asus5775:~/etc/unitd$ wrk -t2 -c100 -d5s http://localhost:81
Running 5s test @ http://localhost:81
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.75ms 1.31ms 13.38ms 68.99%
Req/Sec 18.38k 1.40k 20.72k 65.00%
182849 requests in 5.00s, 28.25MB read
Requests/sec: 36564.68
Transfer/sec: 5.65MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c1000 -d10s http://localhost:81
Running 10s test @ http://localhost:81
2 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 8.70ms 8.45ms 613.41ms 84.71%
Req/Sec 15.75k 8.00k 32.03k 77.00%
313409 requests in 10.04s, 66.65MB read
Socket errors: connect 0, read 386, write 0, timeout 1
Non-2xx or 3xx responses: 313398
Requests/sec: 31217.89
Transfer/sec: 6.64MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c10000 -d20s http://localhost:81
Running 20s test @ http://localhost:81
2 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 8.12ms 5.92ms 782.23ms 55.91%
Req/Sec 9.81k 12.40k 31.53k 70.45%
370256 requests in 20.07s, 78.74MB read
Socket errors: connect 8981, read 966, write 0, timeout 490
Non-2xx or 3xx responses: 370222
Requests/sec: 18446.00
Transfer/sec: 3.92MBalx@asus5775:~/etc/unitd$ wrk -t2 -c100 -d5s http://localhost:82
Running 5s test @ http://localhost:82
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.99ms 1.79ms 14.49ms 72.06%
Req/Sec 28.95k 16.50k 46.36k 51.00%
288113 requests in 5.00s, 66.49MB read
Requests/sec: 57598.73
Transfer/sec: 13.29MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c1000 -d10s http://localhost:82
Running 10s test @ http://localhost:82
2 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 19.25ms 64.28ms 1.03s 97.92%
Req/Sec 47.14k 4.51k 61.54k 74.00%
938916 requests in 10.06s, 209.40MB read
Socket errors: connect 0, read 400, write 0, timeout 0
Non-2xx or 3xx responses: 849107
Requests/sec: 93373.84
Transfer/sec: 20.82MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c10000 -d20s http://localhost:82
Running 20s test @ http://localhost:82
2 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 11.24ms 50.91ms 1.72s 99.59%
Req/Sec 57.26k 6.28k 107.89k 78.70%
2276873 requests in 20.10s, 507.63MB read
Socket errors: connect 8981, read 699, write 0, timeout 0
Non-2xx or 3xx responses: 2079380
Requests/sec: 113296.21
Transfer/sec: 25.26MBalx@asus5775:~/etc/unitd$ wrk -t2 -c100 -d5s http://localhost:83
Running 5s test @ http://localhost:83
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 11.00ms 5.11ms 32.96ms 71.84%
Req/Sec 4.58k 501.63 5.69k 76.00%
45553 requests in 5.00s, 8.47MB read
Requests/sec: 9107.15
Transfer/sec: 1.69MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c1000 -d10s http://localhost:83
Running 10s test @ http://localhost:83
2 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 109.09ms 57.42ms 951.18ms 95.01%
Req/Sec 4.72k 1.30k 9.68k 76.00%
93939 requests in 10.06s, 17.47MB read
Socket errors: connect 0, read 361, write 0, timeout 0
Requests/sec: 9334.29
Transfer/sec: 1.74MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c10000 -d20s http://localhost:83
Running 20s test @ http://localhost:83
2 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 124.36ms 85.91ms 1.70s 84.62%
Req/Sec 4.06k 2.28k 7.05k 58.16%
161099 requests in 20.08s, 29.96MB read
Socket errors: connect 8981, read 670, write 0, timeout 0
Requests/sec: 8021.46
Transfer/sec: 1.49MBalx@asus5775:~/etc/unitd$ wrk -t2 -c100 -d5s http://localhost:84
Running 5s test @ http://localhost:84
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.00us 0.00us 0.00us -nan%
Req/Sec 0.00 0.00 0.00 -nan%
0 requests in 5.01s, 0.00B read
Socket errors: connect 0, read 31983, write 0, timeout 0
Requests/sec: 0.00
Transfer/sec: 0.00B
alx@asus5775:~/etc/unitd$ wrk -t2 -c1000 -d10s http://localhost:84
Running 10s test @ http://localhost:84
2 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.51ms 7.59ms 790.51ms 96.36%
Req/Sec 10.96k 9.94k 31.89k 53.40%
208512 requests in 10.05s, 44.34MB read
Socket errors: connect 0, read 374, write 0, timeout 110
Non-2xx or 3xx responses: 208512
Requests/sec: 20745.97
Transfer/sec: 4.41MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c10000 -d20s http://localhost:84
Running 20s test @ http://localhost:84
2 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 7.43ms 6.56ms 47.70ms 62.76%
Req/Sec 11.63k 11.51k 32.04k 41.91%
437320 requests in 20.08s, 93.00MB read
Socket errors: connect 8981, read 920, write 0, timeout 409
Non-2xx or 3xx responses: 437320
Requests/sec: 21775.63
Transfer/sec: 4.63MBalx@asus5775:~/etc/unitd$ wrk -t2 -c100 -d5s http://localhost:85
Running 5s test @ http://localhost:85
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 265.33us 470.58us 8.84ms 97.52%
Req/Sec 142.29k 13.51k 149.88k 91.00%
1414962 requests in 5.00s, 126.84MB read
Requests/sec: 282937.45
Transfer/sec: 25.36MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c1000 -d10s http://localhost:85
Running 10s test @ http://localhost:85
2 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 8.58ms 47.70ms 704.78ms 97.73%
Req/Sec 132.41k 11.25k 141.19k 93.00%
2647160 requests in 10.09s, 237.31MB read
Socket errors: connect 0, read 396, write 0, timeout 0
Requests/sec: 262259.88
Transfer/sec: 23.51MB
alx@asus5775:~/etc/unitd$ wrk -t2 -c10000 -d20s http://localhost:85
Running 20s test @ http://localhost:85
2 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.31ms 38.40ms 1.10s 99.61%
Req/Sec 128.32k 14.97k 139.48k 90.45%
5103994 requests in 20.07s, 457.55MB read
Socket errors: connect 8981, read 800, write 0, timeout 0
Requests/sec: 254306.35
Transfer/sec: 22.80MBalx@asus5775:~/etc/unitd$ curl localhost:8080
rdm
alx@asus5775:~/etc/unitd$ curl localhost:8081
alx@asus5775:~/etc/unitd$ curl localhost:8082
Hello, Perl on Unit!
alx@asus5775:~/etc/unitd$ curl localhost:81
alx@asus5775:~/etc/unitd$ curl localhost:82
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
alx@asus5775:~/etc/unitd$ curl localhost:83
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
alx@asus5775:~/etc/unitd$ curl localhost:84
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
alx@asus5775:~/etc/unitd$ curl localhost:85
alx@asus5775:~/etc/unitd$ curl --compressed localhost:82
idx
alx@asus5775:~/etc/unitd$ curl --compressed localhost:83
Hello, Perl on Unit!
alx@asus5775:~/etc/unitd$ curl --compressed localhost:84
rdmCompressing a share seems to be quite fast. There's also the weirdness that port 84 (the compressed proxy) has weird (0) numbers for a lowish number of connections. Here's a peak at the errors thrown by this test: To clarify, I think it's more appropriate to perform this tests with small files, because we don't try to measure the performance of zlib, but of the wrapping that we do in unit. Having the compression be virtually instantaneous helps not hide our numbers with those of zlib. |
|
Memory test: alx@asus5775:~$ # idle
alx@asus5775:~$ ps aux | head -n1
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 0.0 0.0 308956 3460 ? Sl 00:32 0:00 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.0 0.0 15692 5392 ? S 00:32 0:00 unit: "perl" application
alx 344923 0.0 0.0 6868 2176 pts/3 S+ 00:33 0:00 grep unit$ wrk -t2 -c10000 -d30m http://localhost:82
[...the results were accidentally lost...]alx@asus5775:~$ ps aux | head -n1
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
alx@asus5775:~$ # 5 s
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 20.7 0.1 308956 38476 ? Sl 00:32 0:14 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.0 0.0 15692 5392 ? S 00:32 0:00 unit: "perl" application
alx 344929 0.0 0.0 6868 2048 pts/3 S+ 00:34 0:00 grep unit
alx@asus5775:~$ # 1 min
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 138 0.1 308956 39812 ? Sl 00:32 2:52 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.0 0.0 15692 5392 ? S 00:32 0:00 unit: "perl" application
alx 344932 0.0 0.0 6868 2048 pts/3 S+ 00:35 0:00 grep unit
alx@asus5775:~$ # 5 min
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 210 0.1 308956 39916 ? Sl 00:32 13:11 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.0 0.0 15692 5392 ? S 00:32 0:00 unit: "perl" application
alx 344972 0.0 0.0 6868 2048 pts/3 S+ 00:39 0:00 grep unit
alx@asus5775:~$ # 15 min
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 213 0.1 308956 40624 ? Sl 00:32 34:15 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.0 0.0 15692 5392 ? S 00:32 0:00 unit: "perl" application
alx 345097 0.0 0.0 6868 2048 pts/3 S+ 00:49 0:00 grep unit
alx@asus5775:~$ # 29 min
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 211 0.1 308956 41120 ? Sl 00:32 63:35 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.0 0.0 15692 5392 ? S 00:32 0:00 unit: "perl" application
alx 345242 0.0 0.0 6868 1920 pts/3 S+ 01:03 0:00 grep unitalx@asus5775:~$ # 1 min idle
alx@asus5775:~$ ps aux | head -n1
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 204 0.1 308956 39388 ? Sl 00:32 65:38 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.0 0.0 15692 5392 ? S 00:32 0:00 unit: "perl" application
alx 345248 0.0 0.0 6868 2048 pts/3 S+ 01:05 0:00 grep unitalx@asus5775:~$ wrk -t2 -c10000 -d30m http://localhost:83
Running 30m test @ http://localhost:83
2 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 93.43ms 82.73ms 1.70s 98.44%
Req/Sec 5.37k 5.55k 27.42k 89.33%
19228780 requests in 30.00m, 3.49GB read
Socket errors: connect 8981, read 59535, write 0, timeout 0
Requests/sec: 10682.57
Transfer/sec: 1.99MBalx@asus5775:~$ # 5 s
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 192 0.1 8149264 59728 ? Sl 00:32 65:46 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.0 0.0 38900 17296 ? S 00:32 0:01 unit: "perl" application
alx 345313 0.0 0.0 6868 2048 pts/3 S+ 01:07 0:00 grep unit
alx@asus5775:~$ # 1 min
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 191 0.1 329444 64484 ? Sl 00:32 67:20 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.8 0.0 38900 17296 ? S 00:32 0:17 unit: "perl" application
alx 345315 0.0 0.0 6868 2048 pts/3 S+ 01:08 0:00 grep unit
alx@asus5775:~$ # 5 min
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 187 0.2 3677564 74244 ? Sl 00:32 73:14 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 2.7 0.0 38900 17296 ? S 00:32 1:03 unit: "perl" application
alx 345615 0.0 0.0 6868 2048 pts/3 S+ 01:12 0:00 grep unit
alx@asus5775:~$ # 15 min
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 173 0.2 329444 73580 ? Sl 00:32 85:20 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 3.2 0.0 38900 17296 ? S 00:32 1:34 unit: "perl" application
alx 345824 0.0 0.0 6868 2048 pts/3 S+ 01:22 0:00 grep unit
alx@asus5775:~$ # 29 min
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 161 0.2 4190172 75944 ? Sl 00:32 102:09 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 3.6 0.0 38900 17296 ? S 00:32 2:17 unit: "perl" application
alx 345859 0.0 0.0 6868 2176 pts/3 S+ 01:36 0:00 grep unitalx@asus5775:~$ # 1 min idle
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 158 0.2 329444 75960 ? Sl 00:32 103:22 unit: router
nobody 344916 0.0 0.0 9764 2060 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 3.5 0.0 38900 17296 ? S 00:32 2:20 unit: "perl" application
alx 345864 0.0 0.0 6868 2048 pts/3 S+ 01:38 0:00 grep unit
alx@asus5775:~$ # 10 min idle
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 139 0.2 329444 89656 ? Sl 00:32 103:22 unit: router
nobody 344916 0.0 0.0 9764 1932 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 3.1 0.0 38900 16656 ? S 00:32 2:20 unit: "perl" application
alx 346030 0.0 0.0 6868 2048 pts/3 S+ 01:47 0:00 grep unit
alx@asus5775:~$ # all night idle
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 16.9 0.2 329444 91832 ? Sl 00:32 103:22 unit: router
nobody 344916 0.0 0.0 9764 1932 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.3 0.0 38900 16656 ? S 00:32 2:20 unit: "perl" application
alx 350183 0.0 0.0 6868 2176 pts/3 S+ 10:43 0:00 grep unitalx@asus5775:~$ wrk -t2 -c10000 -d5m http://localhost:82
Running 5m test @ http://localhost:82
2 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 10.93ms 53.94ms 1.99s 99.63%
Req/Sec 58.83k 7.51k 88.93k 81.46%
35122975 requests in 5.00m, 7.64GB read
Socket errors: connect 8981, read 9045, write 0, timeout 8
Non-2xx or 3xx responses: 33199513
Requests/sec: 117043.18
Transfer/sec: 26.06MBalx@asus5775:~$ # 5 s
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 16.9 0.2 329444 91832 ? Sl 00:32 103:34 unit: router
nobody 344916 0.0 0.0 9764 1932 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.3 0.0 38900 16656 ? S 00:32 2:20 unit: "perl" application
alx 350192 0.0 0.0 6868 2048 pts/3 S+ 10:44 0:00 grep unit
alx@asus5775:~$ # 1 min
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 17.4 0.2 329444 94004 ? Sl 00:32 106:38 unit: router
nobody 344916 0.0 0.0 9764 1932 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.3 0.0 38900 16656 ? S 00:32 2:20 unit: "perl" application
alx 350196 0.0 0.0 6868 2048 pts/3 S+ 10:45 0:00 grep unitalx@asus5775:~$ # 1 min idle
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 18.5 0.2 329444 94004 ? Sl 00:32 114:20 unit: router
nobody 344916 0.0 0.0 9764 1932 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.3 0.0 38900 16656 ? S 00:32 2:20 unit: "perl" application
alx 350274 0.0 0.0 6868 1920 pts/3 S+ 10:49 0:00 grep unitalx@asus5775:~$ wrk -t2 -c10000 -d5m http://localhost:83
Running 5m test @ http://localhost:83
2 threads and 10000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 123.18ms 78.02ms 1.69s 87.48%
Req/Sec 4.08k 1.53k 8.03k 56.10%
2435842 requests in 5.00m, 452.98MB read
Socket errors: connect 8981, read 10014, write 0, timeout 0
Requests/sec: 8117.14
Transfer/sec: 1.51MBalx@asus5775:~$ # 5 s
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 18.5 0.3 2003504 102572 ? Sl 00:32 114:23 unit: router
nobody 344916 0.0 0.0 9764 1932 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.3 0.0 38900 16656 ? S 00:32 2:20 unit: "perl" application
alx 350279 0.0 0.0 6868 2048 pts/3 S+ 10:50 0:00 grep unit
alx@asus5775:~$ # 1 min
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 18.8 0.3 2003504 104916 ? Sl 00:32 116:22 unit: router
nobody 344916 0.0 0.0 9764 1932 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.3 0.0 38900 16656 ? R 00:32 2:26 unit: "perl" application
alx 350282 0.0 0.0 6868 2048 pts/3 S+ 10:51 0:00 grep unitalx@asus5775:~$ # 1 min idle
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 19.3 0.3 329444 105572 ? Sl 00:32 120:28 unit: router
nobody 344916 0.0 0.0 9764 1932 ? S 00:32 0:00 unit: "perl" prototype
nobody 344917 0.4 0.0 38900 16656 ? S 00:32 2:38 unit: "perl" application
alx 350346 0.0 0.0 6868 2048 pts/3 S+ 10:56 0:00 grep unitalx@asus5775:~$ echo '{}' | sudo unit ctl http PUT /config
[sudo] password for alx:
{
"success": "Reconfiguration done."
}
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 4904 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 4904 1664 ? S 00:32 0:00 unit: controller
nobody 344915 19.3 0.2 302676 96036 ? Sl 00:32 120:28 unit: router
alx 350382 0.0 0.0 6868 2048 pts/3 S+ 10:56 0:00 grep unit
alx@asus5775:~$ sudo unit ctl http PUT /config </home/alx/etc/unitd/gzip-v19-2.json
{
"success": "Reconfiguration done."
}
alx@asus5775:~$ ps aux | grep unit
root 344912 0.0 0.0 5548 2184 ? Ss 00:32 0:00 unit: main v1.31.0 [/opt/local/unit/gzip/sbin/unitd]
nobody 344914 0.0 0.0 5036 1664 ? S 00:32 0:00 unit: controller
nobody 344915 18.8 0.2 308956 97060 ? Sl 00:32 120:28 unit: router
nobody 350599 0.0 0.0 9764 2192 ? S 11:10 0:00 unit: "perl" prototype
nobody 350600 0.0 0.0 15692 5140 ? S 11:10 0:00 unit: "perl" application
alx 350602 0.0 0.0 6868 2048 pts/3 S+ 11:10 0:00 grep unitThe VSZ field has some peaks, but in the long term stays stable, so there seem to be no memory leaks. Here are the errors of this test session: alx@asus5775:~$ sudo wc -l /opt/local/unit/gzip/var/log/unit/unit.log
212237370 /opt/local/unit/gzip/var/log/unit/unit.log
alx@asus5775:~$ sudo cat /opt/local/unit/gzip/var/log/unit/unit.log | grep -o 'failed.*' | uniq -c
178787443 failed (24: Too many open files)
1001 failed (104: Connection reset by peer)
59694 failed (24: Too many open files)
131 failed (104: Connection reset by peer)
33212401 failed (24: Too many open files)
973 failed (104: Connection reset by peer)
10035 failed (24: Too many open files) |
|
About the QPS and memory test on the feature, maybe it's helpful to do the wrk test on NGINX with the same file size. |
Makes sense. Of course, I'll only do the share (so skip the perl app), since it's the only thing comparable to nginx. |
|
v20 changes:
|
|
v20b changes:
|
|
Hi, |
Hi, Most of those refactors are necessary for the feature. But I still need to do a review to see if some can be removed. I'll open for review when it's ready for review. |
|
Re-run tests after v33: Unit, compression 1: alx@asus5775:~/src/nginx/unit/actions$ sudo /opt/local/unit/gzip/sbin/unitd
2023/09/04 03:43:24 [info] 226248#226248 unit 1.31.0 started
alx@asus5775:~/src/nginx/unit/actions$ sudo unit ctl http GET /config </dev/null
{
"listeners": {
"*:80": {
"pass": "routes"
}
},
"routes": [
{
"action": {
"share": "/srv/www/unit/",
"index": "index.html",
"compress": {
"encoding": "gzip",
"level": 1,
"min_length": 1
}
}
}
]
}
alx@asus5775:~/src/nginx/unit/actions$ makepasswd --char 10 | sudo tee /srv/www/unit/index.html >/dev/null
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost
Running 1m test @ http://localhost
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 201.11us 324.79us 26.06ms 97.29%
Req/Sec 55.79k 4.43k 66.87k 82.53%
3336413 requests in 1.00m, 833.65MB read
Requests/sec: 55514.36
Transfer/sec: 13.87MB
alx@asus5775:~/src/nginx/unit/actions$ makepasswd --char 1000 | sudo tee /srv/www/unit/index.html >/dev/null
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost
Running 1m test @ http://localhost
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 177.71us 149.59us 12.97ms 96.93%
Req/Sec 55.88k 3.55k 58.78k 79.67%
3336698 requests in 1.00m, 3.17GB read
Requests/sec: 55611.37
Transfer/sec: 54.10MB
alx@asus5775:~/src/nginx/unit/actions$ makepasswd --char 1000111 | sudo tee /srv/www/unit/index.html >/dev/null
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost
Running 1m test @ http://localhost
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 84.19ms 55.01ms 262.56ms 74.98%
Req/Sec 119.09 21.43 151.00 59.50%
7124 requests in 1.00m, 4.93GB read
Requests/sec: 118.70
Transfer/sec: 84.18MB
alx@asus5775:~/src/nginx/unit/actions$ makepasswd --char 122000111 | sudo tee /srv/www/unit/index.html >/dev/null
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost
Running 1m test @ http://localhost
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.00us 0.00us 0.00us -nan%
Req/Sec 4.88 8.44 40.00 90.91%
56 requests in 1.00m, 4.93GB read
Socket errors: connect 0, read 0, write 0, timeout 56
Requests/sec: 0.93
Transfer/sec: 84.19MBUnit, compression level 9: alx@asus5775:~/src/nginx/unit/actions$ sudo unit ctl edit /config
{
"success": "Reconfiguration done."
}
alx@asus5775:~/src/nginx/unit/actions$ sudo unit ctl http GET /config </dev/null
{
"listeners": {
"*:80": {
"pass": "routes"
}
},
"routes": [
{
"action": {
"share": "/srv/www/unit/",
"index": "index.html",
"compress": {
"encoding": "gzip",
"level": 9,
"min_length": 1
}
}
}
]
}
alx@asus5775:~/src/nginx/unit/actions$ makepasswd --char 10 | sudo tee /srv/www/unit/index.html >/dev/null
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost
Running 1m test @ http://localhost
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 218.91us 323.27us 19.22ms 98.02%
Req/Sec 49.26k 1.79k 51.24k 90.85%
2946431 requests in 1.00m, 736.20MB read
Requests/sec: 49025.47
Transfer/sec: 12.25MB
alx@asus5775:~/src/nginx/unit/actions$ makepasswd --char 1000 | sudo tee /srv/www/unit/index.html >/dev/null
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost
Running 1m test @ http://localhost
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 349.57us 115.71us 4.06ms 86.04%
Req/Sec 28.66k 707.92 32.08k 87.67%
1710740 requests in 1.00m, 1.63GB read
Requests/sec: 28512.04
Transfer/sec: 27.74MB
alx@asus5775:~/src/nginx/unit/actions$ makepasswd --char 1000111 | sudo tee /srv/www/unit/index.html >/dev/null
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost
Running 1m test @ http://localhost
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 109.20ms 57.07ms 267.13ms 64.87%
Req/Sec 92.07 24.73 131.00 61.50%
5513 requests in 1.00m, 3.82GB read
Requests/sec: 91.80
Transfer/sec: 65.08MB
alx@asus5775:~/src/nginx/unit/actions$ makepasswd --char 122000111 | sudo tee /srv/www/unit/index.html >/dev/null
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost
Running 1m test @ http://localhost
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.00us 0.00us 0.00us -nan%
Req/Sec 5.19 8.32 20.00 81.25%
37 requests in 1.00m, 3.79GB read
Socket errors: connect 0, read 0, write 0, timeout 37
Requests/sec: 0.62
Transfer/sec: 64.65MBThe performance dropped too much for small files, so we should tune the parameters a little bit. A curiosity is that very large files (100M) seem to benefit of larger compression levels, but small files get worse performance. |
With this, short responses, that is, responses with a body of up to content_length_threshold bytes, won't be compressed. The default value is 20, as in NGINX. Signed-off-by: Alejandro Colomar <[email protected]>
Signed-off-by: Alejandro Colomar <[email protected]>
Signed-off-by: Alejandro Colomar <[email protected]>
|
v34 changes:
|
These are based on C23's <stdbit.h>. Signed-off-by: Alejandro Colomar <[email protected]>
|
v35 changes:
|
ac000
reviewed
Sep 5, 2023
|
On 2023-09-05 03:55, Andrew Clayton wrote:
@ac000 commented on this pull request.
> + if (0) {
+
+ } else {
+ return NXT_ERROR;
+ }
Dare I ask?
Heh, it looks funny :)
It's meant to be a long chain of ifs with all supported encodings.
Since at that point there are no supported encodings, it's a weirdo.
The reason to keep it weird is that patch 0080128
("HTTP: compress: added "encoding": "gzip".) looks better. Does it
sound good to you?
Cheers,
Alex
…--
<http://www.alejandro-colomar.es/>
GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
|
|
Test: Unit: alx@asus5775:~/src/nginx/unit/actions$ sudo unit ctl http GET /config </dev/null
{
"listeners": {
"*:80": {
"pass": "routes"
}
},
"routes": [
{
"action": {
"share": "/srv/www/unit$uri",
"index": "index.html",
"compress": {
"encoding": "gzip",
"level": 9,
"min_length": 1
}
}
}
]
}
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/10.html
Running 1m test @ http://localhost/10.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 70.29us 126.30us 11.14ms 99.78%
Req/Sec 130.97k 3.93k 133.38k 97.34%
7831737 requests in 1.00m, 1.91GB read
Requests/sec: 130312.14
Transfer/sec: 32.56MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/1k.html
Running 1m test @ http://localhost/1k.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 226.23us 84.83us 3.70ms 63.90%
Req/Sec 43.86k 2.43k 47.10k 49.25%
2623826 requests in 1.00m, 2.70GB read
Requests/sec: 43657.76
Transfer/sec: 46.01MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/1M.html
Running 1m test @ http://localhost/1M.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 82.36ms 28.96ms 165.86ms 74.64%
Req/Sec 121.82 4.56 141.00 83.00%
7283 requests in 1.00m, 5.04GB read
Requests/sec: 121.32
Transfer/sec: 86.03MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/100M.html
Running 1m test @ http://localhost/100M.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.00us 0.00us 0.00us -nan%
Req/Sec 5.76 11.07 40.00 82.35%
54 requests in 1.00m, 5.03GB read
Socket errors: connect 0, read 0, write 0, timeout 54
Requests/sec: 0.90
Transfer/sec: 85.77MBalx@asus5775:~/src/nginx/unit/actions$ sudo unit ctl edit /config/routes/0/action/compress/level
{
"success": "Reconfiguration done."
}
alx@asus5775:~/src/nginx/unit/actions$ sudo unit ctl http GET /config/routes/0/action/compress </dev/null
{
"encoding": "gzip",
"level": 1,
"min_length": 1
}
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/10.html
Running 1m test @ http://localhost/10.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 68.76us 70.97us 10.82ms 99.39%
Req/Sec 130.04k 1.93k 132.65k 85.33%
7762900 requests in 1.00m, 1.89GB read
Requests/sec: 129381.29
Transfer/sec: 32.33MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/1k.html
Running 1m test @ http://localhost/1k.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 232.91us 99.65us 3.23ms 67.55%
Req/Sec 43.33k 4.24k 50.27k 70.00%
2587472 requests in 1.00m, 2.66GB read
Requests/sec: 43124.35
Transfer/sec: 45.44MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/1M.html
Running 1m test @ http://localhost/1M.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 82.60ms 37.21ms 211.76ms 50.17%
Req/Sec 121.40 17.09 161.00 55.67%
7259 requests in 1.00m, 5.03GB read
Requests/sec: 120.94
Transfer/sec: 85.78MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/100M.html
Running 1m test @ http://localhost/100M.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.00us 0.00us 0.00us -nan%
Req/Sec 6.43 10.92 30.00 80.95%
54 requests in 1.00m, 5.02GB read
Socket errors: connect 0, read 0, write 0, timeout 54
Requests/sec: 0.90
Transfer/sec: 85.69MBalx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m http://localhost/10.html
Running 1m test @ http://localhost/10.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 88.85us 196.19us 10.05ms 98.07%
Req/Sec 122.65k 3.44k 126.12k 95.01%
7333995 requests in 1.00m, 1.36GB read
Requests/sec: 122031.02
Transfer/sec: 23.16MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m http://localhost/1k.html
Running 1m test @ http://localhost/1k.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 91.55us 202.33us 12.67ms 97.53%
Req/Sec 128.35k 2.07k 130.91k 82.20%
7675379 requests in 1.00m, 8.53GB read
Requests/sec: 127710.17
Transfer/sec: 145.30MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m http://localhost/1M.html
Running 1m test @ http://localhost/1M.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.51ms 573.02us 16.66ms 76.10%
Req/Sec 4.16k 60.99 4.23k 95.17%
248757 requests in 1.00m, 231.74GB read
Requests/sec: 4141.87
Transfer/sec: 3.86GB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m http://localhost/100M.html
Running 1m test @ http://localhost/100M.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 250.30ms 118.12ms 1.15s 76.93%
Req/Sec 30.92 12.95 78.00 68.62%
1847 requests in 1.00m, 210.22GB read
Requests/sec: 30.76
Transfer/sec: 3.50GBNGINX: alx@asus5775:~/src/nginx/unit/actions$ cat /etc/nginx/sites-enabled/default
server {
listen 80;
gzip on;
gzip_http_version 1.0;
gzip_comp_level 1;
gzip_min_length 1;
gzip_types *;
root /srv/www/unit/;
index index.html;
server_name _;
location / {
try_files $uri $uri/ =404;
}
access_log off;
sendfile off;
}
alx@asus5775:~/src/nginx/unit/actions$ sudo systemctl restart nginx
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/10.html
Running 1m test @ http://localhost/10.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 503.27us 1.59ms 22.61ms 92.91%
Req/Sec 109.23k 4.95k 114.12k 91.51%
6531889 requests in 1.00m, 1.76GB read
Requests/sec: 108683.50
Transfer/sec: 29.95MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/1k.html
Running 1m test @ http://localhost/1k.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 157.91us 389.54us 16.12ms 98.81%
Req/Sec 73.74k 4.43k 76.01k 97.33%
4402820 requests in 1.00m, 4.28GB read
Requests/sec: 73379.97
Transfer/sec: 73.06MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/1M.html
Running 1m test @ http://localhost/1M.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 264.39ms 8.90ms 273.76ms 99.56%
Req/Sec 37.73 4.19 40.00 77.33%
2264 requests in 1.00m, 1.60GB read
Requests/sec: 37.73
Transfer/sec: 27.30MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m -H 'Accept-Encoding: gzip' http://localhost/100M.html
Running 1m test @ http://localhost/100M.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.00us 0.00us 0.00us -nan%
Req/Sec 3.67 4.54 10.00 70.00%
34 requests in 1.00m, 3.05GB read
Socket errors: connect 0, read 0, write 0, timeout 34
Requests/sec: 0.57
Transfer/sec: 51.99MBalx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m http://localhost/10.html
Running 1m test @ http://localhost/10.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 119.14us 469.62us 16.34ms 98.55%
Req/Sec 122.59k 6.96k 130.23k 94.01%
7330615 requests in 1.00m, 1.68GB read
Requests/sec: 121973.24
Transfer/sec: 28.61MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m http://localhost/1k.html
Running 1m test @ http://localhost/1k.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 101.18us 339.17us 18.28ms 98.96%
Req/Sec 122.04k 5.66k 130.26k 88.69%
7296767 requests in 1.00m, 8.43GB read
Requests/sec: 121411.11
Transfer/sec: 143.57MB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m http://localhost/1M.html
Running 1m test @ http://localhost/1M.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.52ms 620.71us 12.45ms 75.09%
Req/Sec 4.19k 51.09 4.39k 85.17%
250373 requests in 1.00m, 233.26GB read
Requests/sec: 4169.37
Transfer/sec: 3.88GB
alx@asus5775:~/src/nginx/unit/actions$ wrk -t 1 -d 1m http://localhost/100M.html
Running 1m test @ http://localhost/100M.html
1 threads and 10 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 298.80ms 115.08ms 820.32ms 81.78%
Req/Sec 31.05 15.82 80.00 67.74%
1859 requests in 1.00m, 211.82GB read
Requests/sec: 30.94
Transfer/sec: 3.53GB
|
|
On Sun, 03 Sep 2023 18:13:16 -0700 Alejandro Colomar ***@***.***> wrote:
But I'm still puzzled by this:
```sh
$ curl -s -H 'Accept-Encoding: gzip' localhost | gunzip
gzip: stdin: not in gzip format
```
What does saving the content and running file(1) on it say?
|
|
On 2023-09-05 04:20, Andrew Clayton wrote:
On Sun, 03 Sep 2023 18:13:16 -0700
Alejandro Colomar ***@***.***> wrote:
> But I'm still puzzled by this:
>
> ```sh
> $ curl -s -H 'Accept-Encoding: gzip' localhost | gunzip
>
> gzip: stdin: not in gzip format
> ```
What does saving the content and running file(1) on it say?
It said something zlib. But I fixed it already. Details logged here:
<#914 (comment)>
<#914 (comment)>
Cheers,
Alex
…--
<http://www.alejandro-colomar.es/>
GPG key fingerprint: A9348594CE31283A826FBDD8D57633D441E25BB5
|
|
On Mon, 04 Sep 2023 19:02:04 -0700
Alejandro Colomar ***@***.***> wrote:
On 2023-09-05 03:55, Andrew Clayton wrote:
> @ac000 commented on this pull request.
>
>
>
>> + if (0) {
> +
> + } else {
> + return NXT_ERROR;
> + }
>
> Dare I ask?
Heh, it looks funny :)
It's meant to be a long chain of ifs with all supported encodings.
Since at that point there are no supported encodings, it's a weirdo.
The reason to keep it weird is that patch 0080128
("HTTP: compress: added "encoding": "gzip".) looks better. Does it
sound good to you?
Heh, sure.
|
When the content length is small, optimize zlib for low memory usage. Conversely, when the content length is large, use a similar amount of memory within zlib, as it will improve compression, and won't hurt significantly. Signed-off-by: Alejandro Colomar <[email protected]>
|
v36 changes:
I'm satisfied with the optimizations, and can't improve it further from other things I've tried. Unless/until I get more ideas, I'm done. |
|
BTW, we could add support for zlib compression, if that's something clients understand. curl(1) seemed to understand it. The thing is that the numbers were much faster when I was accidentally using zlib, and the implementation is just a few lines of code away, so it may be worth it. Although it doesn't have any checksum, so maybe not worth for most. |
|
On Mon, 04 Sep 2023 20:07:13 -0700 Alejandro Colomar ***@***.***> wrote:
BTW, we could add support for zlib compression, if that's something clients understand. curl(1) seemed to understand it. The thing is that the numbers were much faster when I was accidentally using zlib, and the implementation is just a few lines of code away, so it may be worth it.
IIRC that's called deflate (I used zlib in a project years ago). It'll
be faster because it's simpler and won't compress as well as gzip.
In fact gzip isn't a compression algorithm, it's a file format that
uses deflate internally but has certain properties that make it compress
better than just using deflate.
|
|
Here's a stable way to pull this patch set. |
|
Why is this PR closed? |
|
Hi @razvanphp, I don't work anymore at NGINX/F5. Since I don't agree with GitHub's abusive policies, I closed (removed) most of my repositories on GitHub, including the Unit one. Nevertheless, AFAIK, this code will be merged to Unit eventually (after code review), and I support it in my personal git server, which you can access here https://www.alejandro-colomar.es/src/alx/nginx/unit.git/tag/?h=gzip-v36 if you want to get the code. I know that @ac000 will be fine pulling from my personal git server. Maybe someone from the NGINX Unit team can tell about the status of the review. About the status of this code, I'm very confident of its quality, so if you need it, you can build Unit with the patches. I'll rebase in a moment on top of the 1.31.1 release, so you can build it with gzip. P.S.: If in the link above you're greeted by your browser with a warning, read here: https://www.alejandro-colomar.es/ssl and https://www.alejandro-colomar.es/random/trust. |
|
v37: Changes:
Pull from here: Also available via HTTPS: |
|
v38 changes:
Pull from here: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements a compression filter which can be applied to any actions to compress their body.
The feature is not yet finished, and has some known limitations which haven't been decided, or that have been accepted for the initial implementation but haven't been implemented yet:
And some limitations that we decided to defer for a future improvement release:
Varyresponse header field.Cache-Control,Expires,ETag)Here's an example configuration that can be used for trying this feature:
{ "settings": { "http": { "log_route": true } }, "listeners": { "*:8080": { "pass": "routes/di" }, "*:8081": { "pass": "routes/n" }, "*:81": { "pass": "routes/o" }, "*:82": { "pass": "routes/s" }, "*:83": { "pass": "routes/a" }, "*:84": { "pass": "routes/u" }, "*:85": { "pass": "routes/r" } }, "routes": { "di": [{ "action": { "share": "/srv/www/readme" } }], "n": [{ "action": { "share": "/srv/www/empty" } }], "o": [{ "action": { "proxy": "http://127.0.0.1:8081", "compress": { "encoding": "gzip" } } }], "s": [{ "action": { "share": "/srv/www/unit/index.html", "compress": { "encoding": "gzip" } } }], "a": [{ "action": { "pass": "applications/perl", "compress": { "encoding": "gzip" } } }], "u": [{ "action": { "proxy": "http://127.0.0.1:8080", "compress": { "encoding": "gzip" } } }], "r": [{ "action": { "return": 321, "compress": { "encoding": "gzip" } } }] }, "applications": { "perl": { "type": "perl", "working_directory": "/home/alx/srv/www/perl/", "script": "./sock.psgi" } } }