feat: support backpressure #2236
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* refactor(quota): refactor `maybeRecordAndGetThrottleTimeMs` Signed-off-by: Ning Yu <[email protected]> * fix(quota): throttle the produce request whatever the acks is Signed-off-by: Ning Yu <[email protected]> * refactor(quota): separate `Request` in `ClientQuotaManager` and `RequestRate` in `BrokerQuotaManager` Signed-off-by: Ning Yu <[email protected]> * sytle: fix lint Signed-off-by: Ning Yu <[email protected]> * feat(quota): support to update broker request rate quota Signed-off-by: Ning Yu <[email protected]> * test(quota): test update quota Signed-off-by: Ning Yu <[email protected]> --------- Signed-off-by: Ning Yu <[email protected]>
* feat(quota): introduce `SLOW_FETCH` broker quota Signed-off-by: Ning Yu <[email protected]> * feat(quota): add slow fetch quota Signed-off-by: Ning Yu <[email protected]> * test(quota): test broker slow fetch quota Signed-off-by: Ning Yu <[email protected]> * test(quota): test zero quota value Signed-off-by: Ning Yu <[email protected]> --------- Signed-off-by: Ning Yu <[email protected]>
* feat(backpressure): init backpressure module Signed-off-by: Ning Yu <[email protected]> * feat(backpressure): implement `DefaultBackPressureManager` Signed-off-by: Ning Yu <[email protected]> * test(backpressure): test `DefaultBackPressureManager` Signed-off-by: Ning Yu <[email protected]> --------- Signed-off-by: Ning Yu <[email protected]>
Signed-off-by: Ning Yu <[email protected]>
Signed-off-by: Ning Yu <[email protected]>
Signed-off-by: Ning Yu <[email protected]>
* fix: fix logs Signed-off-by: Ning Yu <[email protected]> * feat(quota): support to get current quota metric value Signed-off-by: Ning Yu <[email protected]> * refactor(backpressure): remove `Regulator#minimize` Signed-off-by: Ning Yu <[email protected]> * perf(quota): increase the max of broker quota throttle time Signed-off-by: Ning Yu <[email protected]> * perf(backpressure): decrease cooldown time Signed-off-by: Ning Yu <[email protected]> * perf(quota): increase the max of broker quota throttle time Signed-off-by: Ning Yu <[email protected]> * docs: update comments Signed-off-by: Ning Yu <[email protected]> --------- Signed-off-by: Ning Yu <[email protected]>
* feat(quota): exclude internal client IDs from broker quota Signed-off-by: Ning Yu <[email protected]> * feat(autobalancer): mark producers and consumers internal clients Signed-off-by: Ning Yu <[email protected]> --------- Signed-off-by: Ning Yu <[email protected]>
Signed-off-by: Ning Yu <[email protected]>
Signed-off-by: Ning Yu <[email protected]>
* feat(backpressure): log it on recovery from backpressure Signed-off-by: Ning Yu <[email protected]> * feat: add metric fetch_limiter_waiting_task_num Signed-off-by: Ning Yu <[email protected]> * feat: add metric fetch_limiter_timeout_count Signed-off-by: Ning Yu <[email protected]> * feat: add metric fetch_limiter_time Signed-off-by: Ning Yu <[email protected]> * feat: add metric back_pressure_state Signed-off-by: Ning Yu <[email protected]> * feat: add metric broker_quota_limit Signed-off-by: Ning Yu <[email protected]> * fix(backpressure): run checkers with fixed delay Signed-off-by: Ning Yu <[email protected]> * style: fix lint Signed-off-by: Ning Yu <[email protected]> * perf: drop too large values Signed-off-by: Ning Yu <[email protected]> * refactor: record -1 for other states Signed-off-by: Ning Yu <[email protected]> * test: fix tests Signed-off-by: Ning Yu <[email protected]> --------- Signed-off-by: Ning Yu <[email protected]>
* feat(backpressure): make back pressure manager configurable Signed-off-by: Ning Yu <[email protected]> * test: test diabled Signed-off-by: Ning Yu <[email protected]> * refactor: move backpressure from s3stream to kafka.core Signed-off-by: Ning Yu <[email protected]> * refactor: init `BackPressureManager` in `BrokerServer` Signed-off-by: Ning Yu <[email protected]> * refactor: introduce `BackPressureConfig` Signed-off-by: Ning Yu <[email protected]> * feat: make `BackPressureManager` reconfigurable Signed-off-by: Ning Yu <[email protected]> * test: test reconfigurable Signed-off-by: Ning Yu <[email protected]> * refactor: rename config key Signed-off-by: Ning Yu <[email protected]> * refactor: move metric "back_pressure_state" from s3stream to core Signed-off-by: Ning Yu <[email protected]> --------- Signed-off-by: Ning Yu <[email protected]>
…2207) Signed-off-by: Ning Yu <[email protected]>
Signed-off-by: Ning Yu <[email protected]>
fix: fix metric value of back pressure state Signed-off-by: Ning Yu <[email protected]>
Signed-off-by: Ning Yu <[email protected]>
Chillax-0v0
requested review from
superhx,
SCNieh and
Gezi-lzq
as code owners
superhx
approved these changes
Dec 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick following PRs:
Checker#2162