Kafka Share Groups (Queues for Kafka, KIP-932) in Java

Kafka has always had one awkward limitation for work-queue use cases: in a consumer group, each partition is assigned to exactly one consumer. Six partitions means at most six busy consumers — add a seventh and it sits idle. If one slow message blocks a partition, everything behind it waits.

Share groups — delivered by KIP-932, "Queues for Kafka" — remove that coupling. They arrived as early access in Kafka 4.0, previewed in 4.1, and became production-ready in Apache Kafka 4.2. Here's how they work and how to use them from Java.

Think of it like this

A consumer group is a supermarket where each checkout lane has exactly one cashier, and customers are assigned to lanes by their surname. A share group is a bank with a single queue and many counters: whoever is free takes the next customer, and if a customer's paperwork isn't ready, they step aside and someone else is served.

How share groups work

Many consumers in a share group can read from the same partition at the same time. When a consumer fetches records, the broker acquires them for it with a time-limited lock — 30 seconds by default. The consumer then acknowledges each record individually:

ACCEPT — processed successfully; done.

RELEASE — give it back for another attempt, possibly by another consumer.

REJECT — unprocessable; never redeliver it.

RENEW — extend the lock because processing is taking longer (added in 4.2).

Kafka counts delivery attempts per record. Once a record hits the limit — five by default, set by group.share.delivery.count.limit — it is archived and not delivered again. If a consumer crashes, its locks simply expire and the records become available to others.

A share consumer in Java

The Java client provides KafkaShareConsumer. With share.acknowledgement.mode=explicit, every record returned by poll() must be acknowledged before the next poll — which gives you precise control over retries:

KycDocumentWorker.java
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "kyc-document-workers");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put("share.acknowledgement.mode", "explicit");

try (KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(props)) {
    consumer.subscribe(List.of("kyc-documents"));

    while (running) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> record : records) {
            try {
                verifier.verify(record.value());
                consumer.acknowledge(record, AcknowledgeType.ACCEPT);
            } catch (VendorTimeoutException e) {
                consumer.acknowledge(record, AcknowledgeType.RELEASE);   // retry, maybe elsewhere
            } catch (InvalidDocumentException e) {
                deadLetters.send(record);                                // keep it for humans
                consumer.acknowledge(record, AcknowledgeType.REJECT);    // never redeliver
            }
        }
        consumer.commitSync();   // send the acknowledgements to the broker
    }
}

VendorTimeoutException, InvalidDocumentException and deadLetters stand in for your own types. Note that KafkaShareConsumer is not thread-safe: run one per thread and scale by adding consumers, which is exactly what share groups are designed for. Recent Spring for Apache Kafka releases have been adding share-consumer support; the plain client above works regardless of framework version.

When to use share groups — and when not to

Use them when records are independent units of work: sending notifications, processing uploaded documents, calling slow third-party APIs, running image or AI inference jobs. You can scale consumers beyond the partition count for peak load, and one slow record no longer blocks its neighbours.

Keep using consumer groups when order matters. Share groups only guarantee order within a single delivered batch — not across batches, consumers or redeliveries. Event sourcing, change-data-capture and per-account ledgers still belong in classic consumer groups.

What's still missing (plan around it)

No built-in dead-letter queue yet — a DLQ feature (KIP-1191) is in development for a later Kafka release, so publish rejected records to your own DLQ topic before rejecting, as in the example. The group's start position is controlled by the group-level share.auto.offset.reset config (default latest), set with the configs tool rather than in consumer code. Exactly-once semantics are not part of share groups; design processing to be idempotent. And make sure your cluster is on Kafka 4.2 or later — early-access share groups from 4.0 were explicitly not upgradeable.

Wherever you run workers like these, give each processed record a traceable ID in your logs — the pattern in correlation IDs and structured logging carries over directly to Kafka headers.

Frequently asked questions

What are Kafka share groups?

Share groups, introduced by KIP-932 (Queues for Kafka), let multiple consumers cooperatively process records from the same partitions, with per-record acknowledgement and delivery counting. They bring queue-like consumption to Kafka without changing the underlying log.

Are Kafka share groups production-ready?

Yes. Share groups were early access in Kafka 4.0, preview in 4.1, and are production-ready in Apache Kafka 4.2.

Do Kafka share groups preserve message order?

Only within a single delivered batch for a share-partition. There is no ordering guarantee across batches, consumers or redeliveries, so ordered workloads should keep using consumer groups.

Share groups decouple consumer count from partition count.

Records are locked, acknowledged individually and redelivered up to a limit.

Production-ready since Kafka 4.2; Java API is KafkaShareConsumer.

Great for independent work items; keep consumer groups for ordered streams.

Build your own DLQ and make processing idempotent.