Skip to content
This repository has been archived by the owner on Oct 7, 2024. It is now read-only.

Optimize rebalance #36

Closed
antban opened this issue Aug 10, 2016 · 4 comments
Closed

Optimize rebalance #36

antban opened this issue Aug 10, 2016 · 4 comments

Comments

@antban
Copy link
Contributor

antban commented Aug 10, 2016

Right now rebalance is made step by step with step size of 1 partition.
Maybe it will be better to transfer more partitions at a time (5, 10, or by leader count), and it will be faster.

@CyberDem0n
Copy link

Doesn't really make sense, kafka does rebalance step-by-step anyway. It process one partition and updates znode. When there is nothing left - removes it.

@antban
Copy link
Contributor Author

antban commented Aug 10, 2016

@CyberDem0n In some situations it may help. For example - bubuku have minimum step time - 0.5 seconds. That means, that if partition migration takes less time, than it's better to migrate several partitions at a time.

@antban antban changed the title Optimize rebalance speed Optimize rebalance Sep 9, 2016
@antban
Copy link
Contributor Author

antban commented Sep 9, 2016

Well, renamed task. There are problems with rebalance, and they should be fixed:

  1. Rebalance should do as less steps as it's possible (do not move data that should not be moved)
  2. Leaders should be balanced having in mind that there must be different leaders for partitions in single topic.
  3. During fatboyslim only replicas should be moved. Leaders should remain in the same position

@antban
Copy link
Contributor Author

antban commented Sep 13, 2016

Closed in favor of #63

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants