Package com.linkedin.davinci.kafka.consumer

Class KafkaConsumerService

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    Direct Known Subclasses:
    PartitionWiseKafkaConsumerService, TopicWiseKafkaConsumerService
    public abstract class KafkaConsumerService
extends AbstractKafkaConsumerService
    KafkaConsumerService is used to manage a pool of consumption-related resources connected to a specific Kafka cluster. The reasons to have this pool are: 1. To reduce the unnecessary overhead of having one consumer per store-version, each of which includes the internal IO threads/connections to brokers and internal buffers; 2. To reduce the GC overhead when there are a lot of store versions bootstrapping/ingesting at the same time; 3. To have a predictable and configurable upper bound on the total amount of resources occupied by consumers become, no matter how many store-versions are being hosted in the same instance; The responsibilities of this class include: 1. Setting up a fixed size pool of consumption unit, where each unit contains exactly one: a) SharedKafkaConsumer b) ConsumptionTask c) ConsumerSubscriptionCleaner 2. Receive various calls to interrogate or mutate consumer state, and delegate them to the correct unit, by maintaining a mapping of which unit belongs to which version-topic and subscribed topic-partition. Notably, the startConsumptionIntoDataReceiver(PartitionReplicaIngestionContext, long, ConsumedDataReceiver) function allows the caller to start funneling consumed data into a receiver (i.e. into another task). 3. Provide a single abstract function that must be overridden by subclasses in order to implement a consumption load balancing strategy: pickConsumerForPartition(PubSubTopic, PubSubTopicPartition)
    See Also:
    which wraps one instance of this class per Kafka cluster.