Kafka Exactly-Once Semantics: The Ultimate 2026 Deep Dive

Kafka exactly-once semantics is the most misunderstood feature in the Apache Kafka toolbox. Most teams flip processing.guarantee to exactly_once_v2, watch duplicates disappear from a test harness, and declare victory. Then a producer stalls at 03:00, a single partition stops delivering records to every read_committed consumer in the estate, and the on-call engineer discovers a transaction that has been sitting in the Ongoing state for forty minutes while every replication dashboard stayed reassuringly green.

This Kafka exactly-once semantics deep dive is written for engineers who already operate Kafka at scale and want the internals rather than the marketing. We will walk the full path: the idempotent producer, the two-phase commit run by the transaction coordinator, the durable transaction state machine, the last stable offset that governs the read path, what KIP-447 changed inside Kafka Streams, the configuration surface that actually matters, and a field-tested runbook for hanging transactions. Every non-obvious mechanism is illustrated with a diagram.

What this guide covers

• What Kafka exactly-once semantics actually guarantees
• The three building blocks of Kafka exactly-once semantics
• Inside the transaction coordinator: the commit protocol, step by step
• The transaction state machine and zombie fencing
• How the read path changes: LSO, control records and the aborted index
• Kafka exactly-once semantics in Kafka Streams: v1 versus exactly_once_v2
• The Kafka exactly-once semantics configuration surface
• Runbook: diagnosing a hanging transaction
• The real performance cost of Kafka exactly-once semantics
• Anti-patterns that quietly break correctness
• When not to use Kafka exactly-once semantics at all
• Frequently asked questions

What Kafka exactly-once semantics actually guarantees

The first correction to make about Kafka exactly-once semantics is semantic. Kafka does not promise that a record is physically delivered to a consumer precisely one time. Networks retry, consumers restart, and the same bytes can be fetched repeatedly. What Kafka guarantees is that a read-process-write cycle is atomic and idempotent from the perspective of the log: either every output record and the corresponding input offsets become visible together, or none of them do. Practitioners often call this effectively-once processing, and the distinction matters when you design sinks.

The second correction is scope. Kafka exactly-once semantics applies to work whose inputs and outputs are both Kafka partitions inside a single cluster. It does not extend to an external database, a REST call, an S3 bucket, or a second Kafka cluster. A transaction cannot span clusters, which is why cross-cluster replication with MirrorMaker 2 is at-least-once and why the transactional outbox pattern still exists.

The third correction is failure scope. Kafka exactly-once semantics protects you from producer retries, broker failovers, consumer rebalances and zombie instances. They do not protect you from non-deterministic application logic. If your processor calls Instant.now() or a random number generator and you replay an aborted batch, the output values will differ even though the commit was atomic.

The three building blocks of Kafka exactly-once semantics

Everything in Kafka exactly-once semantics rests on three independent mechanisms that must all be enabled together. Understanding which one is failing is the fastest way to debug a correctness incident.

1. The idempotent producer

When enable.idempotence=true, which has been the default since Kafka 3.0, the producer requests a producer ID (PID) from a broker and stamps every record batch with a triple: the PID, a producer epoch and a monotonically increasing sequence number per partition. The partition leader caches the last five sequence numbers for each PID and rejects anything it has already seen. That is how a retried batch after a network timeout does not become a duplicate, and it costs almost nothing.

Idempotence alone is weaker than most people assume. It is scoped to a single producer session and a single partition, it disappears when the producer restarts and acquires a new PID, and it says nothing about atomicity across partitions. It also requires acks=all and max.in.flight.requests.per.connection of five or fewer to preserve ordering.

2. Transactions and the coordinator

Setting transactional.id promotes the producer from idempotent to transactional. That identifier is stable across restarts, and it is the anchor for the entire protocol: the broker uses it to locate a transaction coordinator, to persist state, and to fence obsolete instances. The coordinator is not a special service. It is whichever broker leads the partition of the internal __transaction_state topic selected by hashing the transactional ID modulo the partition count, which defaults to fifty.

3. read_committed isolation

The third pillar of Kafka exactly-once semantics is isolation, and producer-side atomicity is pointless if readers can see uncommitted data. Consumers must set isolation.level=read_committed, which is not the default. This is the single most common misconfiguration we find during audits: a perfectly correct transactional pipeline feeding downstream consumers that happily read aborted records.

Inside the transaction coordinator: the commit protocol, step by step

The coordinator runs a two-phase commit whose write-ahead log is an ordinary compacted Kafka topic, and it is the beating heart of Kafka exactly-once semantics. Nothing is held in memory only, which is exactly why the protocol survives broker failure. The sequence below is what really happens between beginTransaction() and commitTransaction().

Three details in that flow deserve emphasis. First, beginTransaction() is purely a client-side state change and issues no RPC, so the coordinator only learns about a transaction when the first AddPartitionsToTxn arrives. Second, offsets for the input topic are committed by the producer through sendOffsetsToTransaction(), which is why auto-commit must be switched off in any transactional consumer. Third, once PrepareCommit is durable in __transaction_state, the outcome is decided; the coordinator will keep retrying transaction markers forever, and a newly elected coordinator replays the log to finish the job.

A minimal but correct transactional loop looks like this:

Properties p = new Properties();
p.put("bootstrap.servers", "broker1:9092");
p.put("transactional.id", "orders-tx-1");   // stable across restarts
p.put("enable.idempotence", "true");         // implied, but be explicit
p.put("acks", "all");
p.put("transaction.timeout.ms", "60000");    // must be <= broker max

KafkaProducer<String, String> producer = new KafkaProducer<>(p);
producer.initTransactions();                 // fences every older epoch

while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(200));
    if (records.isEmpty()) continue;
    producer.beginTransaction();
    try {
        for (ConsumerRecord<String, String> r : records) {
            producer.send(new ProducerRecord<>("payments", r.key(), transform(r.value())));
        }
        producer.sendOffsetsToTransaction(offsetsOf(records), consumer.groupMetadata());
        producer.commitTransaction();
    } catch (ProducerFencedException | OutOfOrderSequenceException e) {
        producer.close();                    // fatal: another instance owns this id
        break;
    } catch (KafkaException e) {
        producer.abortTransaction();         // retryable: replay the same batch
    }
}

Note the two distinct catch blocks. Treating every exception as retryable is the classic bug that turns Kafka exactly-once semantics into a silent duplicate factory, because a fenced producer that keeps calling abortTransaction() in a loop will never surrender its partitions.

The transaction state machine and zombie fencing

Every transition the coordinator makes is a record in __transaction_state, and the set of legal transitions is small enough to memorise. It is worth memorising, because most production incidents are simply a transaction that is stuck in one of these states.

Fencing is the part that makes the design safe under partition. Kafka has exactly one fencing primitive: the producer epoch. When a new instance calls initTransactions() with the same transactional ID, the coordinator increments the epoch, persists it, and moves any in-flight transaction to PrepareEpochFence so that it can be aborted on the zombie's behalf. The old instance, which may still be alive behind a long garbage-collection pause or a network partition, is rejected the moment it touches a coordinator or a leader that has already seen the higher epoch.

Two exceptions signal this and they are not interchangeable. ProducerFencedException is fatal and means another instance legitimately took over the transactional ID. InvalidProducerEpochException can be recoverable in modern clients, typically after a transaction timed out and the epoch was bumped underneath you. Modern releases also verify, before appending, that the partition leader genuinely belongs to an open transaction known to the coordinator, which closes the race that historically produced orphaned transactions after a producer timeout.

How the read path changes: LSO, control records and the aborted index

The write path is only half the story. The most surprising operational consequences of Kafka exactly-once semantics live on the consumer side, because a read_committed consumer no longer fetches up to the high watermark. It fetches up to the last stable offset, or LSO, which is the first offset of the oldest still-open transaction on that partition.

Three consequences follow directly from that picture. Records below the LSO are durable, fully replicated and completely invisible, so a single slow or stuck producer stalls every committed reader on the partitions it touched even though replication is healthy. Commit and abort markers are real control records that consume real offsets, so consumer lag arithmetic must tolerate gaps and offset counting is no longer a reliable message count. And filtering is genuinely cheap: each log segment carries an aborted transaction index, the .txnindex file, so the broker can tell the client which offset ranges to discard without scanning the data.

This is also where your Kafka exactly-once semantics alerting should live. Watch the per-partition LastStableOffsetLag gauge, which is the distance between the high watermark and the LSO. A value that grows and never returns to zero is the earliest reliable signal of a transactional problem, and it fires long before consumer lag alerts do.

Kafka exactly-once semantics in Kafka Streams: v1 versus exactly_once_v2

Kafka Streams was the first mainstream framework to make Kafka exactly-once semantics a one-line configuration change, and the evolution from the original implementation to exactly_once_v2 is the clearest example of why the internals matter.

The original design created one transactional producer per task, which means per input partition. On a topology with three hundred partitions, every rebalance re-initialised three hundred transactional IDs, each with its own coordinator round trip and its own set of markers. Recovery time scaled with partition count, and the transaction log grew accordingly.

KIP-447 removed that coupling by moving the fencing responsibility from the transactional ID to the consumer group. Because TxnOffsetCommit now carries the consumer group generation, the group coordinator itself rejects a zombie whose generation is stale, and a single producer per stream thread becomes safe. Fewer producers means fewer markers, a smaller __transaction_state topic and materially faster rebalances. The v1 mode was deprecated and then removed, so exactly_once_v2 is the only sensible choice on any modern cluster.

Two Streams-specific details are easy to miss. State stores participate in the transaction only through their changelog topic, so a RocksDB store that survives an aborted batch is repaired by replaying the changelog, not by rolling back the store in place. And commit.interval.ms stops being a throughput knob and becomes your end-to-end latency floor: with exactly-once processing enabled, the framework lowers the default to one hundred milliseconds precisely because nothing downstream is visible until the transaction commits.

The Kafka exactly-once semantics configuration surface that matters

Most Kafka exactly-once semantics outages trace back to a handful of settings. The table below is the shortlist we review during MinervaDB Kafka health checks.

On the observability side, four signals cover almost every failure mode: LastStableOffsetLag per partition, the producer metrics txn-commit-time-ns-total and txn-abort-time-ns-total, and the coordinator gauges UnknownDestinationQueueSize and LogAppendRetryQueueSize exposed by TransactionMarkerChannelManager. If the marker queues are non-empty and not draining, markers are not reaching their destinations and your LSOs are about to freeze.

Runbook: diagnosing a hanging transaction

A hanging transaction is the signature failure of Kafka exactly-once semantics: the coordinator and the partition leader disagree about whether a transaction is still open, so the marker that would advance the LSO never arrives. Replication is perfect, produce latency is normal, and committed consumers simply stop. The following sequence turns a two-hour investigation into a ten-minute one.

In practice you are comparing two sources of truth. The partition leader reports, through describe-producers, a producer whose currentTransactionStartOffset is set. The coordinator reports, through describe --transactional-id, what it believes that producer is doing. When the leader says a transaction is open, the coordinator has no matching ongoing state, and the start timestamp is older than transaction.timeout.ms, you have an orphan.

# 1. Which partition is stuck? Look for a non-zero, non-decreasing LSO lag
#    kafka.log:type=Log,name=LastStableOffsetLag,topic=orders,partition=0

# 2. Who owns the open transaction on that partition?
kafka-transactions.sh --bootstrap-server broker1:9092 \
  describe-producers --topic orders --partition 0

# 3. What does the coordinator think?
kafka-transactions.sh --bootstrap-server broker1:9092 \
  describe --transactional-id orders-tx-1

# 4. Only when the two disagree and the timeout has elapsed, abort the orphan
kafka-transactions.sh --bootstrap-server broker1:9092 \
  abort --topic orders --partition 0 --start-offset 109

Aborting by topic, partition and start offset is the safest Kafka exactly-once semantics repair you can make, and it is deliberate: it targets exactly one orphaned transaction instead of blowing away legitimate in-flight work. Run it too eagerly, on a transaction that is merely slow, and you will discard records that the application believed were committed. Confirm the timeout has genuinely elapsed first.

The real performance cost of Kafka exactly-once semantics

The overhead of Kafka exactly-once semantics is frequently misquoted in both directions. Throughput cost is modest when transactions are large: you pay for a handful of extra coordinator round trips plus one marker per partition per transaction, amortised over the whole batch. The cost that actually hurts is latency, and it is structural rather than incidental.

End-to-end latency for a committed reader is at minimum the commit interval, plus the coordinator round trip, plus the time to write markers to every participating partition, plus the fetch that follows the LSO advance. Halving the commit interval doubles the marker rate, so the tuning exercise is a genuine trade-off rather than a free win. Three rules keep it under control: keep transactions short in time but wide in records, keep the number of partitions touched per transaction small because every one of them needs a marker, and never let a transaction wait on an external system while it is open.

Anti-patterns that quietly break Kafka exactly-once semantics

• Sharing one transactional ID across live instances. Each initTransactions() call fences the other, and the two processes will take turns killing each other forever.
• Deriving the transactional ID from a hostname or pod name. On a rolling restart the new pod gets a new ID, so the old one is never fenced and its in-flight transaction blocks the LSO until it times out.
• Enclosing an external write inside the transaction. A database insert cannot be rolled back by a Kafka abort. Use the outbox pattern and let the sink be idempotent.
• Leaving consumers on read_uncommitted. The pipeline is atomic and the readers see aborted data anyway.
• A generous transaction.timeout.ms. That number is a promise about how long a crashed producer may block your readers.
• Assuming replication metrics prove health. LSO lag is invisible to every standard under-replicated-partition dashboard.

When not to use Kafka exactly-once semantics at all

Kafka exactly-once semantics is a correctness tool, not a default. For high-volume telemetry, clickstreams or metrics, where a duplicate is statistically irrelevant, the latency floor and the operational surface are not worth it. For a single-topic fan-out with no state, idempotence alone is sufficient. And when the destination is a database or object store with natural upsert keys, an idempotent sink with at-least-once delivery is simpler, cheaper and easier to reason about at 03:00 than a distributed transaction. Reserve transactions for pipelines where a duplicate is a financial or regulatory event.

Frequently asked questions

Do Kafka transactions work across two clusters?

No. A transaction is coordinated by one broker in one cluster, and replication tools copy records without preserving transactional atomicity. Treat cross-cluster delivery as at-least-once and make the consuming side idempotent.

What happens if the producer crashes mid-transaction?

The transaction stays Ongoing and continues to block the LSO on every partition it touched until transaction.timeout.ms expires or a replacement instance calls initTransactions() with the same transactional ID and fences it.

Does exactly-once processing slow down consumers that do not need it?

Only if they read a partition that carries transactional data. A read_committed consumer is bounded by the LSO regardless of who wrote the open transaction, which is a strong argument for not mixing transactional and non-transactional producers on the same topic.

Is the idempotent producer enough for Kafka exactly-once semantics on its own?

It is enough to remove duplicates caused by retries within one producer session on one partition. It is not enough for atomic multi-partition writes or for coupling output records to input offsets, which is the part that requires transactions.

Key takeaways on Kafka exactly-once semantics

• Kafka exactly-once semantics is atomic read-process-write inside one cluster, not magical de-duplication across systems.
• The coordinator runs a two-phase commit whose write-ahead log is the __transaction_state topic; PrepareCommit is the point of no return.
• The producer epoch is the only fencing primitive, so a stable transactional ID is a correctness requirement rather than a convention.
• On the read path the LSO, not the high watermark, decides visibility, which makes LastStableOffsetLag your most valuable alert.
• Use exactly_once_v2 in Kafka Streams; the per-task producer model of v1 scales badly and is gone.
• Keep transactions short, narrow and free of external calls, and treat transaction.timeout.ms as a service-level promise to your readers.

Further reading: the Apache Kafka delivery semantics documentation and the design notes for KIP-447 on producer scalability for exactly-once semantics are the canonical references for the behaviour described above.

If you are designing or repairing a transactional pipeline, these companion articles go deeper on the surrounding subsystems: Kafka internals, Kafka architecture for DBAs, producer configuration and cluster tuning, tiered storage and multi-region replication and scaling consumption with the parallel consumer.

Need a second pair of eyes on Kafka exactly-once semantics in your own cluster? MinervaDB engineers run transactional-correctness and performance audits for data infrastructure teams worldwide.