Some Benchmarks #411

josejachuf · 2022-01-19T19:20:13Z

josejachuf
Jan 19, 2022

Based on the work published on the YouTube channel [1] with different comparisons between different frameworks and languages to serialize JSON I wrote to Emmett [2] the same app that the author used for Flask, Fasti, Falcon, etc. and I sent it to Vagelis Prokopiou.

I also did some tests on my computer and I'm going to publish them here.

I understand that if Orjson (Rust) is installed, Emmett will use this instead of Rapid-Json. If this is so, I did not notice the difference.

I use apache bench

% ab -n 100000 -c 1000 http://127.0.0.1:8081/users

in all case I start sever with:

% uvicorn main:app --host 127.0.0.1 --port 8081 --workers 8

[1] https://www.youtube.com/channel/UCPnvXp_A6O7xREKK2cYwhIw
[2] https://github.com/josejachuf/web-api-python-emmett

josejachuf · 2022-01-19T19:26:23Z

josejachuf
Jan 19, 2022
Author

Falcon

https://github.com/Vagelis-Prokopiou/yt-web-api-python-falcon

1° Run

Document Path:          /users
Document Length:        114679 bytes

Concurrency Level:      1000
Time taken for tests:   155.527 seconds
Complete requests:      100000
Failed requests:        0
Total transferred:      11480800000 bytes
HTML transferred:       11467900000 bytes
Requests per second:    642.98 [#/sec] (mean)
Time per request:       1555.265 [ms] (mean)
Time per request:       1.555 [ms] (mean, across all concurrent requests)
Transfer rate:          72088.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   6.9      1     117
Processing:    59 1548 728.6   1479    5746
Waiting:       14 1152 656.5   1046    5707
Total:         59 1550 729.5   1483    5746

Percentage of the requests served within a certain time (ms)
  50%   1483
  66%   1799
  75%   1988
  80%   2103
  90%   2466
  95%   2888
  98%   3252
  99%   3474
 100%   5746 (longest request)

2° Run

Document Path:          /users
Document Length:        114679 bytes

Concurrency Level:      1000
Time taken for tests:   137.212 seconds
Complete requests:      100000
Failed requests:        0
Total transferred:      11480800000 bytes
HTML transferred:       11467900000 bytes
Requests per second:    728.80 [#/sec] (mean)
Time per request:       1372.123 [ms] (mean)
Time per request:       1.372 [ms] (mean, across all concurrent requests)
Transfer rate:          81710.76 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3  12.3      1     159
Processing:    34 1364 638.1   1285    4178
Waiting:        7 1022 576.0    929    4146
Total:         34 1367 638.5   1287    4181

Percentage of the requests served within a certain time (ms)
  50%   1287
  66%   1535
  75%   1719
  80%   1866
  90%   2230
  95%   2599
  98%   2852
  99%   3156
 100%   4181 (longest request)

0 replies

josejachuf · 2022-01-19T19:27:43Z

josejachuf
Jan 19, 2022
Author

Emmett (rapid-json)

https://github.com/josejachuf/web-api-python-emmett

1° Run

Document Path:          /users
Document Length:        32 bytes

Concurrency Level:      1000
Time taken for tests:   30.926 seconds
Complete requests:      100000
Failed requests:        0
Non-2xx responses:      100000
Total transferred:      21200000 bytes
HTML transferred:       3200000 bytes
Requests per second:    3233.53 [#/sec] (mean)
Time per request:       309.260 [ms] (mean)
Time per request:       0.309 [ms] (mean, across all concurrent requests)
Transfer rate:          669.44 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    5   6.7      3     106
Processing:     2  303 155.4    277     946
Waiting:        1  249 135.5    225     878
Total:          2  308 155.8    282     959

Percentage of the requests served within a certain time (ms)
  50%    282
  66%    349
  75%    396
  80%    427
  90%    524
  95%    606
  98%    689
  99%    766
 100%    959 (longest request)

2° Run

Document Path:          /users
Document Length:        32 bytes

Concurrency Level:      1000
Time taken for tests:   28.512 seconds
Complete requests:      100000
Failed requests:        0
Non-2xx responses:      100000
Total transferred:      21200000 bytes
HTML transferred:       3200000 bytes
Requests per second:    3507.35 [#/sec] (mean)
Time per request:       285.116 [ms] (mean)
Time per request:       0.285 [ms] (mean, across all concurrent requests)
Transfer rate:          726.13 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    5   5.9      3      62
Processing:     1  279 176.9    244    1382
Waiting:        1  230 150.9    199    1326
Total:          1  284 178.4    248    1389

Percentage of the requests served within a certain time (ms)
  50%    248
  66%    320
  75%    368
  80%    400
  90%    491
  95%    556
  98%    681
  99%   1112
 100%   1389 (longest request)

0 replies

josejachuf · 2022-01-19T19:29:09Z

josejachuf
Jan 19, 2022
Author

Emmett (orjson)

https://github.com/josejachuf/web-api-python-emmett

1° Run

Document Path:          /users
Document Length:        32 bytes

Concurrency Level:      1000
Time taken for tests:   28.688 seconds
Complete requests:      100000
Failed requests:        0
Non-2xx responses:      100000
Total transferred:      21200000 bytes
HTML transferred:       3200000 bytes
Requests per second:    3485.72 [#/sec] (mean)
Time per request:       286.884 [ms] (mean)
Time per request:       0.287 [ms] (mean, across all concurrent requests)
Transfer rate:          721.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    5   8.6      3     125
Processing:     1  280 151.0    256     996
Waiting:        1  231 136.0    203     962
Total:          2  286 153.2    261    1056

Percentage of the requests served within a certain time (ms)
  50%    261
  66%    323
  75%    369
  80%    392
  90%    490
  95%    586
  98%    676
  99%    757
 100%   1056 (longest request)

2° Run

Document Path:          /users
Document Length:        32 bytes

Concurrency Level:      1000
Time taken for tests:   28.596 seconds
Complete requests:      100000
Failed requests:        0
Non-2xx responses:      100000
Total transferred:      21200000 bytes
HTML transferred:       3200000 bytes
Requests per second:    3497.00 [#/sec] (mean)
Time per request:       285.959 [ms] (mean)
Time per request:       0.286 [ms] (mean, across all concurrent requests)
Transfer rate:          723.99 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    5   6.5      3      69
Processing:     1  280 140.1    263     799
Waiting:        0  230 122.0    213     675
Total:          1  285 140.4    268     804

Percentage of the requests served within a certain time (ms)
  50%    268
  66%    331
  75%    374
  80%    396
  90%    486
  95%    536
  98%    621
  99%    673
 100%    804 (longest request)

0 replies

josejachuf · 2022-01-19T19:30:18Z

josejachuf
Jan 19, 2022
Author

Fastapi

https://github.com/Vagelis-Prokopiou/yt-web-api-python-fastapi

1° Run

Document Path:          /users
Document Length:        105680 bytes

Concurrency Level:      1000
Time taken for tests:   1315.209 seconds
Complete requests:      100000
Failed requests:        0
Total transferred:      10580900000 bytes
HTML transferred:       10568000000 bytes
Requests per second:    76.03 [#/sec] (mean)
Time per request:       13152.086 [ms] (mean)
Time per request:       13.152 [ms] (mean, across all concurrent requests)
Transfer rate:          7856.48 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   4.9      0      75
Processing:   564 13034 26712.2   9840  519222
Waiting:      275 11782 26724.2   8454  518943
Total:        564 13034 26712.4   9840  519270

Percentage of the requests served within a certain time (ms)
  50%   9840
  66%  11057
  75%  12053
  80%  12963
  90%  15236
  95%  18898
  98%  44025
  99%  110412
 100%  519270 (longest request)

2° Run

Document Path:          /users
Document Length:        105680 bytes

Concurrency Level:      1000
Time taken for tests:   1076.026 seconds
Complete requests:      100000
Failed requests:        0
Total transferred:      10580900000 bytes
HTML transferred:       10568000000 bytes
Requests per second:    92.93 [#/sec] (mean)
Time per request:       10760.264 [ms] (mean)
Time per request:       10.760 [ms] (mean, across all concurrent requests)
Transfer rate:          9602.84 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   3.3      0      56
Processing:   135 10699 17446.4   8776  540862
Waiting:      103 9606 17440.9   7547  539985
Total:        191 10700 17446.4   8776  540862

Percentage of the requests served within a certain time (ms)
  50%   8776
  66%   9822
  75%  10264
  80%  10619
  90%  12172
  95%  16580
  98%  40775
  99%  78105
 100%  540862 (longest request)

0 replies

josejachuf · 2022-01-19T19:31:36Z

josejachuf
Jan 19, 2022
Author

Warp (Rust)

https://github.com/Vagelis-Prokopiou/yt-web-api-rust-warp

1° Run

Document Path:          /users
Document Length:        101680 bytes

Concurrency Level:      1000
Time taken for tests:   26.882 seconds
Complete requests:      100000
Failed requests:        0
Total transferred:      10179200000 bytes
HTML transferred:       10168000000 bytes
Requests per second:    3719.90 [#/sec] (mean)
Time per request:       268.825 [ms] (mean)
Time per request:       0.269 [ms] (mean, across all concurrent requests)
Transfer rate:          369781.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   11 104.6      0    1042
Processing:     4  256 114.3    262    3681
Waiting:        3  256 114.3    262    3681
Total:         25  267 184.1    262    4715

Percentage of the requests served within a certain time (ms)
  50%    262
  66%    264
  75%    267
  80%    268
  90%    277
  95%    302
  98%    351
  99%   1135
 100%   4715 (longest request)

2° Run

Document Path:          /users
Document Length:        101680 bytes

Concurrency Level:      1000
Time taken for tests:   27.478 seconds
Complete requests:      100000
Failed requests:        0
Total transferred:      10179200000 bytes
HTML transferred:       10168000000 bytes
Requests per second:    3639.23 [#/sec] (mean)
Time per request:       274.783 [ms] (mean)
Time per request:       0.275 [ms] (mean, across all concurrent requests)
Transfer rate:          361762.66 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   3.0      0      59
Processing:     5  273  35.6    261     405
Waiting:        3  273  35.6    261     405
Total:         22  273  35.1    261     422

Percentage of the requests served within a certain time (ms)
  50%    261
  66%    265
  75%    269
  80%    272
  90%    303
  95%    370
  98%    401
  99%    402
 100%    422 (longest request)

0 replies

josejachuf · 2022-01-21T12:46:50Z

josejachuf
Jan 21, 2022
Author

Other benchmarks, this time using wrk

% wrk -t4 -c400 -d30s http://127.0.0.1:8081/users

Emmett

4 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   450.46ms  262.73ms   1.98s    77.17%
    Req/Sec   202.83     76.20   514.00     71.21%
  23972 requests in 30.06s, 2.34GB read
  Socket errors: connect 7, read 2958, write 0, timeout 109
Requests/sec:    797.55
Transfer/sec:     79.74MB

Falcon

4 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   737.23ms  334.65ms   2.00s    67.42%
    Req/Sec   128.57     53.18   303.00     69.46%
  15336 requests in 30.08s, 1.64GB read
  Socket errors: connect 0, read 6, write 0, timeout 304
Requests/sec:    509.85
Transfer/sec:     55.84MB

Fastapi

4 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.21s   506.56ms   2.00s    60.19%
    Req/Sec    18.44     10.57    80.00     74.70%
  2004 requests in 30.10s, 202.22MB read
  Socket errors: connect 0, read 0, write 0, timeout 1896
Requests/sec:     66.58
Transfer/sec:      6.72MB

Warp (Rust)

4 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    70.05ms   42.14ms 346.92ms   68.29%
    Req/Sec     1.47k   138.06     2.58k    83.89%
  175964 requests in 30.05s, 16.68GB read
Requests/sec:   5854.98
Transfer/sec:    568.38MB

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some Benchmarks #411

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Some Benchmarks #411

josejachuf Jan 19, 2022

Replies: 6 comments

josejachuf Jan 19, 2022 Author

Falcon

1° Run

2° Run

josejachuf Jan 19, 2022 Author

Emmett (rapid-json)

1° Run

2° Run

josejachuf Jan 19, 2022 Author

Emmett (orjson)

1° Run

2° Run

josejachuf Jan 19, 2022 Author

Fastapi

1° Run

2° Run

josejachuf Jan 19, 2022 Author

Warp (Rust)

1° Run

2° Run

josejachuf Jan 21, 2022 Author

Emmett

Falcon

Fastapi

Warp (Rust)

josejachuf
Jan 19, 2022

josejachuf
Jan 19, 2022
Author

josejachuf
Jan 19, 2022
Author

josejachuf
Jan 19, 2022
Author

josejachuf
Jan 19, 2022
Author

josejachuf
Jan 19, 2022
Author

josejachuf
Jan 21, 2022
Author