High memory usage with encoder

[brenol]

Hi! Thanks for this pure go implementation.

I'm experiencing a very high memory usage (600MB encoding a few 200KB ~ 2mb files, ~5 every few seconds) and the profile is pretty specific. the high memory usage is because of zstd encoder.
Here is the offending line:

Some info about my logic:

• a few goroutines receiving a few []byte arrays, reusing the same encoder
• encoder was created globally within package: var encoder, _ = zstd.NewWriter(nil)
  I'm encoding with:
  dst := encoder.EncodeAll(data, make([]byte, 0, len(data))
• then I upload that file with something like: ... Body: bytes.NewReader(dst)

I tried to create a simple repro using net/http but unfortunately it didn't work.
I'm using github.com/klauspost/compress v1.15.15

[klauspost]

There are three ways to control memory:

First WithLowerEncoderMem will trade a few more allocations for less static memory.

Reducing WithWindowSize (default 8MB) will trade a bit of compression for less memory or WithEncoderConcurrency will allocate less encoders and therefore keep less buffers.

[brenol]

Thanks @klauspost. I saw those functions but I was still unsure what which one should I use, so I did not change them.

I have a great memory usage now (with the low concurrency I need right now) with a very good CPU usage too.
It seems the major offender was WithEncoderConcurrency, as only changing WithLowerEncoderMem and WithWindowSize even though they did result in a great size reduction, they went from those 750MB to ~250MB.

Changing WithEncoderConcurrency to 2 did change significantly and now I'm seeing ~50MB usages.

// package level variable
var enc, _ = zstd.NewWriter(
    nil,
    zstd.WithLowerEncoderMem(true),
    zstd.WithEncoderConcurrency(2),
    zstd.WithWindowSize(1<<20),
)

Thanks a lot, @klauspost.

[klauspost]

Great. Just note that it will never be able to run more than 2 encodes concurrently (they will block).