(de)compression: reduce memory allocation to improve performance

[magurotuna]

Motivation

Currently, every time WrapBody::poll_frame is called, new instance of BytesMut is created with the default capacity, which is effectively 64 bytes. This ends up with a lot of memory allocation in certain situations, making the throughput significantly worse.

#520 describes that a manual implementation of decompression logic using async-compression gets the performance issue with tower_http::decompression to be resolved. To identify what is making this difference, I captured a flamegraph. Here's the result:

As annotated, tower_http::decompression has more memory allocation which is the reason why the performance is degraded.

Solution

To optimize memory allocation, WrapBody now gets BytesMut as its field, with initial capacity of 4096 bytes. This buffer will be reused as much as possible across multiple poll_frame calls, and only when its capacity becomes 0, new allocation of another 4096 bytes is performed.

The capacity of 4096 bytes is taken from the default capacity of the internal buffer used in tokio_util::io::ReaderStream: https://docs.rs/tokio-util/0.7.12/src/tokio_util/io/reader_stream.rs.html#8

Performance Improvement

With this optimization, the flamegraph is changed as follows, where memory allocation is significantly reduced and optimized.

Also, the throughput gets 10x better when we measure it using Deno's fetch implementation that internally uses hyper and tower_http::decompresson.
This graph shows how long each Deno version takes to handle 2k requests. The rightmost one named v2.0.0-rc.4-tower-http-patched is the one with this patch applied. The leftmost one, v1.45.2, does not use tower_http::decompression.

For more information on how this result was measured, please refer to https://github.com/magurotuna/deno_fetch_decompression_throughput

Fixes: #520

[seanmonstar]

Phenomenal write-up, thanks!

[Dav1dde]

I just wanna say amazing work!

I work on Relay, which get's quite a chunk of compressed traffic. I updated tower and axum the other day and noticed about a 20% decrease in tokio runtime busy threads, a 4x decrease in time spent reading (+decompressing) request bodies, a 50% reduction in the average latency and about 30% reduced latency in the max and p95's.

Now looking through several changelogs and git commits trying to find what caused this, I am relatively sure it was this PR!