Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use net.Buffers to reduce memory inuse bytes.. #421

Closed

Conversation

zdyj3170101136
Copy link
Contributor

Summary

use net.Buffers to reduce memory inuse bytes

Detail

In a typical clickhouse insert scenario, in order to improve the insert efficiency, the client usually inserts in batches, inserting every 10w:

1, convert []struct to proto.Input
2, use ch.Client.Do to convert proto.Input to proto.Block
3, send proto.Block to clickhouse server.

The third step usually takes a long time, so in order to reduce memory usage, you can set []struct to nil after the first step to release []struct.

Since proto.Block is []byte, the proto.Input data will actually be copied to proto.Block, which results in double memory overhead.

Now set proto.Block to net.Buffers, which allows a series of memory areas to be written to the network connection at once, reducing the memory overhead by half.

Only Support *Col.Str for now.

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG
  • For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials

@ernado
Copy link
Collaborator

ernado commented Sep 18, 2024

BTW there is also WIP PR #413 that aims to reduce memory footprint

query.go Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants