ui.management.doc.bulkTopicReBalanceDoc.ftl Maven / Gradle / Ivy
- Goal is to have each topic being "balanced" on it's own
- "Balanced" topic means it has:
- equal number of partition replicas per each broker (equally use broker(s) disk and equally contribute to broker's I/O load)
- equal number of partition leaders per each broker (consumers and producers equally contribute to all brokers I/O load)
- Balance is achieved by taking topic's current assignment and finding minimal re-assignment to apply
- find minimal set of replicas to "move"
- find minimal set of partitions to change its leader replica
-
Performance impact:
- Moving replicas between brokers may use significant amount of I/O and affect (increased latency) producers and consumers
- You may want to use appropriate I/O throttling to limit speed of replication when broker(s)
start to pull whole partition's data from other brokers for all partitions that are newly being assigned to that broker
<#if (bulkReBalanceDoc??)>
-
NOTE when doing bulk re-balance on multiple topics
- You may want to limit number of re-assignments doing at once to reduce stress on cluster
- Big number of re-assigned topic-partitions by itself can also have impact on cluster (regardless of how much underlying data all those partition have)
-
Strategy when having large number of "dis-balanced" topics:
- Select a "batch" of topics to re-assign at once
- Wait for re-assignments to complete and do verify re-assignments completion (which will remove generated throttles)
- Repeat with new batch of topics until all topics are "balanced"
-
Downscaling cluster:
- One might want to reduce number of nodes in cluster
- Use bulk re-assignment of topic replicas with specifying which broker-ids not to include in assignments
- Once all topics have been re-assigned that particular broker(s) have no topic partition replicas, then such broke(s) can be decommissioned