Lazily allocate context for chunk compression

[anacrolix]

Changelog

Reduce memory overhead of many concurrent writers.

Docs

The CustomCompressor interface now returns a new compressor instead of assuming to return the same one every time.

Description

The compression context created for each Writer can have a significant memory usage. Only allocate this as required when compressing chunks. When operating a large number of concurrent writers this significantly reduces memory overhead.

https://linear.app/foxglove/issue/FG-8734/inbox-listener-ooms-when-file-has-too-many-input-channels

[jtbandes]

Thanks for the PR. It still looks like there are some unrelated go module changes, would you be able to remove those so it can pass CI?

[anacrolix]

Feel free to build on top of this.

In particular to minimise any extra changes required I believe:

You can retain the original custom compressor interface, for now, as the compressor is just reused.

I think the go module changes are just residual go workspace stuff, it can all be reverted.

[anacrolix]

I can no longer see the related data platform PR but that can be modified to not bother with chunking topic split uploads: It's still a significant performance fix, and is more reliable than the original code. The chunking can be addressed in some other more complete PR with tests, or not at all.