ui.clusters.doc.incrementalBalanceDoc.ftl Maven / Gradle / Ivy

Go to download
Show more of this group Show more artifacts with this name
Show all versions of kafkistry-web Show documentation
The newest version!

    
    
        
            NOTE: consider doing incremental balancing if all topics are "balanced" on their own
            (having equal number of partition replicas and partition leaders per each broker)
        

        
            Why incremental balancing?
        
        
            End goal: achieving "near-equal" load on all brokers in cluster
                
                    Similar disk usage on all brokers
                    Similar network inbound on all brokers
                    Similar network outbound on all brokers
                
            
            Even if all topics are "balanced" on their own, there is still possibility that
                all partitions of some topic doesn't have equal amount of traffic going through which may cause
                uneven impact on brokers in cluster
            
            All topics may not have partition-count which is multiple of number of brokers in cluster,
                which will have uneven load impact on different brokers
            
        

        
            Why incremental instead computing optimal assignments for all topics and applying it in one step?
        
        
            Computing optimal assignments is nearly impossible when having large number of topic-partitions
            Even if there was an algorithm which could do that optimization problem fast,
                there would still be some problems with such approach:
                
                    Amount of traffic per topic-partition is likely to change in the future, so an optimal
                        assignment now may not be optimal "tomorrow"
                    
                    Optimal assignments are likely "very different" than current assignments which
                        applying would introduce a lot of stress on cluster when needing to move a large number of
                        partition replicas across cluster
                    
                
            
            Due to issues mentioned above, there is no point chasing an optimal load distribution across brokers
            
            Near similar load per different brokers should be good enough in most circumstances
        
        
            When to do incremental balancing?
        
        
            When all topics are balanced on their own but there is still significant dis-balance of load across
                brokers
            
        
        
            What incremental balancing does?
        
        
            
                Algorithm receives current state of assignments and loads as input
            
            
                Output of one iteration is:
                
                    migration of one partition replica
                    swap migrations of two partition replicas
                    change on preferred leader replica of particular topic-partition
                
            
            
                Resulting proposed change of current assignments is best found change (found within the time limit)
                such that it contributes to overall cluster balance the most (optimizing fo given balance objective)
            
            
                It is possible to compute multiple such iterations and apply them all at once
            
            
                After change is applied, you need to wait for migrations to complete and then verify new assignments
                to remove generated throttles
            
            
                If cluster disbalance is still too high, this process can be repeated again