请问有没有一份最新的query node balancer的架构解释？

[xiaobingxia-at]

请教一下各位大佬。Cluster 2.5.2, 600k partitions. 我发现我的query node频繁load / release segments，我怀疑是balancer频繁在不同query node上切segments。我的balancer的设置如下：

queryCoord:
            autoBalance: true
            autoBalanceChannel: true
            memoryUsageMaxDifferencePercentage: 10
            checkBalanceInterval: 120000 # check balance every 120 seconds
            overloadedMemoryThresholdPercentage: 90

请问有没有什么问题？

我在源码中看到有两个balancer:
https://github.com/milvus-io/milvus/blob/master/internal/querycoordv2/balance/balance.go
https://github.com/milvus-io/milvus/blob/master/internal/querycoordv2/balance/score_based_balancer.go

config里默认的balancer type是score based balancer. 在运行中，这两个都在跑吗？还是只有score based balancer再跑？
目前有没有详细的材料解释一下所有的balancer的运行规则和使用手册？目前我只找到了这个： https://milvus.io/blog/2022-02-28-how-milvus-balances-query-load-across-nodes.md 但是感觉里面有些信息已经过时了。而且我发现score based的balancer，规则极为复杂。如果网站给篇doc详细解释一下就最好了。感谢！

[yhmo]

Wait @weiliu1031 comment

[weiliu1031]

During the iterative development of Milvus, several versions of the balancer have been implemented, with the score_based_balancer currently being the primary one in use. It is recommended to focus more on the implementation of this balancer. However, the implementation of balancers involves a wide range of considerations, such as multi-table mixing and memory heterogeneity, which introduces significant complexity.

Based on some internal documentation and code implementations of Milvus, I have compiled a simplified guide to help you understand the design and implementation of the balancer. I hope this will assist you in exploring its mechanisms effectively.

[weiliu1031]

Milvus Balance Rules and Mechanisms

1. Objectives of Balance

• Data Balancing: Ensure that each table's data is evenly distributed across all QueryNodes.
• Load Balancing: Achieve similar data loads across all QueryNodes, preventing resource imbalance.

---

2. Triggers for Balance

1. Periodic Checks: balance_checker is triggered every 3 seconds.
2. Event-Driven Triggers:
   • When a new node joins the cluster.
   • When a node goes offline.
   • When a collection is released.

---

3. Types of Balance

• Stopping Balance:
  • Handles nodes marked for stopping.
  • Migrates segments and channels from the stopping node to other nodes.
• Normal Balance:
  • Addresses uneven distribution of loaded data.
  • Prioritizes balancing replicas of a single collection, progressively equalizing data loads across nodes.

---

4. Execution Mechanism of Balance

a. Task Generation

1. Node Score Calculation (Compute Node Score):
   • current_score: Calculated based on the current data load on a node.
   • assign_score: Calculated based on the memory proportion allocated to the node.
   • Priority Calculation: priority = assign_score - current_score, determining a node's priority for receiving or migrating segments.
2. Selecting Segments for Migration:
   • Select segments from nodes where current_score > assign_score to achieve balance after migration.
3. Assigning Segments to Nodes:
   • Allocate segments based on target nodes' priorities.
   • Avoid migrations that could negatively impact existing balance.

b. Task Submission and Execution

• Balance tasks are submitted to the scheduler after checking for conflicts with existing tasks.
• Priority Rules:
  • Load and release tasks take precedence over balance tasks.
  • Tasks of the same type have equal priority.
• Rate Limiting: Limit the number of balance tasks submitted per round to prevent resource contention.
• Task Completion: Resources on the original node are released after the new node successfully loads the data.

---

5. Insights from Implementation

5.1 Key Components and Functions

• Score Calculation:

  • current_score: Represents the total data load currently on the QueryNode.

  • assign_score: Represents the target load for the node based on its available memory.

  • Priority: Calculated as priority = assign_score - current_score to determine the node's necessity for balancing.

  • Score Calculation:

    • The formula for calculating the weight of a table c1 on node n1 is as follows:
      • P(c1, n1) = RowCount(c1, n1) + ∑Rowcount(co, n1) × factor
    • Here:
      • RowCount(c1, n1) represents the data volume of table c1 on node n1.
      • Rowcount(co, n1) represents the data volume of other tables on node n1.
      • factor is a coefficient that determines the influence of other tables on the weight calculation.
    • Considerations:
      • This approach accounts for both the current table's data volume and the data volume of other tables on the same node, aiming to prevent node load imbalance caused by uneven distribution of a single table.

• Balancing Strategy:

  • Nodes with higher scores (heavier loads) are candidates for offloading data.
  • Nodes with lower scores (lighter loads) are prioritized to receive data.

• Segment Assignment:

  • Segments are selected from overloaded nodes to align their current_score with the target assign_score.
  • These segments are then assigned to underloaded nodes based on priority.
  • Avoids migrations that do not reduce the load imbalance.

5.2 Efficiency and Resource Awareness

• Limits the number of simultaneous migrations to avoid overloading the system.
• Considers heterogeneous QueryNode resources, including memory and computational capacity.
• Delegator nodes are given special handling to avoid excessive computational overhead.