use net.Buffers to reduce memory inuse bytes.. #421
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
use net.Buffers to reduce memory inuse bytes
Detail
In a typical clickhouse insert scenario, in order to improve the insert efficiency, the client usually inserts in batches, inserting every 10w:
The third step usually takes a long time, so in order to reduce memory usage, you can set []struct to nil after the first step to release []struct.
Since proto.Block is []byte, the proto.Input data will actually be copied to proto.Block, which results in double memory overhead.
Now set proto.Block to net.Buffers, which allows a series of memory areas to be written to the network connection at once, reducing the memory overhead by half.
Only Support *Col.Str for now.
Checklist
Delete items not relevant to your PR: