All Downloads are FREE. Search and download functionalities are using the official Maven repository.

org.apache.kafka.clients.consumer.RangeAssignor Maven / Gradle / Ivy

There is a newer version: 3.3.8
Show newest version
/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.kafka.clients.consumer;

import org.apache.kafka.clients.consumer.internals.AbstractPartitionAssignor;
import org.apache.kafka.common.TopicPartition;

import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * 

The range assignor works on a per-topic basis. For each topic, we lay out the available partitions in numeric order * and the consumers in lexicographic order. We then divide the number of partitions by the total number of * consumers to determine the number of partitions to assign to each consumer. If it does not evenly * divide, then the first few consumers will have one extra partition. * *

For example, suppose there are two consumers C0 and C1, two topics t0 and * t1, and each topic has 3 partitions, resulting in partitions t0p0, t0p1, * t0p2, t1p0, t1p1, and t1p2. * *

The assignment will be: *

    *
  • C0: [t0p0, t0p1, t1p0, t1p1]
  • *
  • C1: [t0p2, t1p2]
  • *
* * Since the introduction of static membership, we could leverage group.instance.id to make the assignment behavior more sticky. * For the above example, after one rolling bounce, group coordinator will attempt to assign new member.id towards consumers, * for example C0 -> C3 C1 -> C2. * *

The assignment could be completely shuffled to: *

    *
  • C3 (was C0): [t0p2, t1p2] (before was [t0p0, t0p1, t1p0, t1p1]) *
  • C2 (was C1): [t0p0, t0p1, t1p0, t1p1] (before was [t0p2, t1p2]) *
* * The assignment change was caused by the change of member.id relative order, and * can be avoided by setting the group.instance.id. * Consumers will have individual instance ids I1, I2. As long as * 1. Number of members remain the same across generation * 2. Static members' identities persist across generation * 3. Subscription pattern doesn't change for any member * *

The assignment will always be: *

    *
  • I0: [t0p0, t0p1, t1p0, t1p1] *
  • I1: [t0p2, t1p2] *
*/ public class RangeAssignor extends AbstractPartitionAssignor { @Override public String name() { return "range"; } private Map> consumersPerTopic(Map consumerMetadata) { Map> topicToConsumers = new HashMap<>(); for (Map.Entry subscriptionEntry : consumerMetadata.entrySet()) { String consumerId = subscriptionEntry.getKey(); MemberInfo memberInfo = new MemberInfo(consumerId, subscriptionEntry.getValue().groupInstanceId()); for (String topic : subscriptionEntry.getValue().topics()) { put(topicToConsumers, topic, memberInfo); } } return topicToConsumers; } @Override public Map> assign(Map partitionsPerTopic, Map subscriptions) { Map> consumersPerTopic = consumersPerTopic(subscriptions); Map> assignment = new HashMap<>(); for (String memberId : subscriptions.keySet()) assignment.put(memberId, new ArrayList<>()); for (Map.Entry> topicEntry : consumersPerTopic.entrySet()) { String topic = topicEntry.getKey(); List consumersForTopic = topicEntry.getValue(); Integer numPartitionsForTopic = partitionsPerTopic.get(topic); if (numPartitionsForTopic == null) continue; Collections.sort(consumersForTopic); int numPartitionsPerConsumer = numPartitionsForTopic / consumersForTopic.size(); int consumersWithExtraPartition = numPartitionsForTopic % consumersForTopic.size(); List partitions = AbstractPartitionAssignor.partitions(topic, numPartitionsForTopic); for (int i = 0, n = consumersForTopic.size(); i < n; i++) { int start = numPartitionsPerConsumer * i + Math.min(i, consumersWithExtraPartition); int length = numPartitionsPerConsumer + (i + 1 > consumersWithExtraPartition ? 0 : 1); assignment.get(consumersForTopic.get(i).memberId).addAll(partitions.subList(start, start + length)); } } return assignment; } }




© 2015 - 2024 Weber Informatics LLC | Privacy Policy