Kafka performance optimization after a 4.x upgrade usually starts with the same question: why did producer latency go up when nobody touched the config? Often the answer is linger.ms. Its default moved from 0 to 5 ms in Kafka 4.0, the release that also removed ZooKeeper, and nothing gets logged when it starts adding wait time to your batches. Kafka 4.4 will not make a cluster dramatically faster. What it does is make slowness easier to explain. Brokers now report disk I/O through JMX, the producer perf test can replay the same key distribution run after run, tiered storage uploads can be held back, and Kafka Streams no longer has to rebuild an entire state store after an exactly-once failure.
This post covers the changes we treat as performance work when we take a production cluster from 4.0 through 4.4. Each one comes with its configuration, the metric that proves it worked, and the conditions where it does not help. We have not published throughput numbers here. Measure on your own hardware with the test plan below, because a figure from someone else's cluster is not a planning number.
Kafka performance optimization across 4.0 to 4.4: what actually changed
Release notes list everything. Kafka performance optimization in production cares about the handful of changes that move latency, recovery time or cost. The table keeps only those, pinned to the release that introduced them.
| Release | Change | Performance effect | Action |
|---|---|---|---|
| 4.0 | ZooKeeper removed; KRaft only | Metadata propagation through the controller quorum | Clusters on 3.x must migrate on 3.9 first |
| 4.0 | linger.ms default 0 → 5 | Up to 5 ms added per producer batch wait | Set it explicitly per producer |
| 4.0 | KIP-848 consumer rebalance protocol GA | Incremental rebalances, no stop-the-world | group.protocol=consumer |
| 4.0 | remote.log.manager.thread.pool.size 10 → 2 | Lower default tiered storage upload concurrency | Re-check upload lag after upgrade |
| 4.2 | Share groups production-ready (KIP-932) | Consumer parallelism beyond partition count | New lag metrics (KIP-1226) |
| 4.3 | follower.fetch.last.tiered.offset.enable (KIP-1023) | New followers start from the last tiered offset | Off by default; test on large-retention topics |
| 4.4 | Transactional state stores (KIP-892) | Streams no longer wipe stores on error under EOS | Off by default |
| 4.4 | Delayed upload (KIP-1241) | Shorter duplication window in tiered storage | remote.copy.lag.ms per topic |
| 4.4 | Linux I/O metrics over JMX (KIP-1291) | Page-cache and write-amplification visibility | Scrape and baseline |
If you are still on 4.3, take 4.3.2 regardless. It fixes a size-based retention bug that could delete in-retention data from both tiers after a leader election (KAFKA-20732) and a time-index truncation across restarts (KAFKA-20979). Neither is a speed problem, but both are the kind of incident that ends a performance project.
Measure first: the Kafka performance optimization signal map
A slow pipeline has four places to lose time: the producer accumulator, the broker request path, the log and page cache, and the fetch side. Tiered storage, the KRaft quorum and Kafka Streams state restoration sit beside that path and show up in tail latency rather than averages. Before changing any setting, we capture one baseline per hop.
Most broker latency sits in the request handler and network thread pools. Watch RequestHandlerAvgIdlePercent and the produce TotalTimeMs breakdown before blaming disks. On 4.2 and later the controller exposes AvgIdleRatio, which makes a saturated quorum visible instead of guessed.
Linux I/O metrics in Kafka 4.4
KIP-1291 exposes all seven fields of /proc/self/io through JMX, including linux-disk-rchar, linux-disk-wchar and linux-disk-cancelled-write-bytes. The useful part is the ratio between bytes the broker asked to read and bytes that reached the block device. It is the closest thing Kafka has had to a page-cache hit ratio without attaching a kernel tracer.
# Kafka performance optimization: expose the Linux I/O metrics added in Kafka 4.4 (KIP-1291)
# Prometheus JMX exporter rule. Confirm the MBean path with jconsole on one broker before rollout.
rules:
- pattern: 'kafka.server<type=(.+), name=(linux-disk-.+)><>Value'
name: kafka_server_$1_$2
type: GAUGE
With the gauges scraped, two PromQL expressions give the numbers we baseline on every engagement. Metric names depend on your exporter rules, so adjust the series names to match.
# Approximate page-cache read hit ratio per broker: bytes the broker asked for (rchar) # versus bytes that actually came from the block device (read-bytes). 1 - ( rate(kafka_server_KafkaServer_linux_disk_read_bytes[5m]) / rate(kafka_server_KafkaServer_linux_disk_rchar[5m]) ) # Write amplification: device writes per byte of produce traffic accepted by the broker. rate(kafka_server_KafkaServer_linux_disk_write_bytes[5m]) / sum by (instance) (rate(kafka_server_BrokerTopicMetrics_BytesInPerSec_total[5m]))
A falling read hit ratio during consumer catch-up is expected, and it is not a Kafka performance optimization target. A falling ratio during steady tail reads usually means the heap or another process is competing for page cache, and no producer setting will fix that.
Kafka producer tuning after the linger.ms default change
The 4.0 change to linger.ms is the most common source of a latency regression that nobody can explain. There is no error and no warning. Producers that sent immediately at linger.ms=0 now wait up to 5 ms for a batch to fill. For a high-volume pipeline that is a gain, because fuller batches compress better and cost fewer requests. For a low-volume, latency-sensitive service it is 5 ms of added p99.
Kafka producer tuning is the first stop in any Kafka performance optimization review, and it starts from this baseline. Each value is set explicitly so that a future default change cannot alter behaviour silently:
# Kafka performance optimization: producer baseline for Apache Kafka 4.x clients
# Defaults shown in comments are the 4.x client defaults; change one value per test run.
bootstrap.servers=${KAFKA_BOOTSTRAP}
acks=all # default since 3.0; keep it for durability-sensitive topics
enable.idempotence=true # default since 3.0; requires acks=all
linger.ms=5 # default changed from 0 to 5 in 4.0
batch.size=65536 # default 16384; larger batches only pay off if linger.ms lets them fill
compression.type=zstd # default none; lz4 if producer CPU is the constraint
max.in.flight.requests.per.connection=5
buffer.memory=67108864 # default 33554432; raise only if record-queue-time-avg climbs
Batching only pays when linger.ms and batch.size move together. Raising batch.size with linger.ms=0 rarely changes anything, because batches are sent before they fill. On the producer, watch batch-size-avg, record-queue-time-avg and compression-rate-avg. If queue time rises while batch size stays flat, the producer is waiting on the broker, not on the accumulator.
A reproducible Kafka performance optimization test plan
Single benchmark runs are noise. Kafka 4.2 added --warmup-records to separate warmup from steady state (KIP-1052). Kafka 4.4 adds --key-distribution, --record-key-range and --random-seed (KIP-1299), so two runs can produce the same partition spread. Together they make a sweep repeatable enough to compare across versions.
#!/usr/bin/env bash
# Kafka performance optimization test plan: linger.ms sweep with a fixed key distribution.
# Requires Apache Kafka 4.4 tools (--key-distribution, --record-key-range, --random-seed: KIP-1299)
# and 4.2+ for --warmup-records (KIP-1052). Run against a staging cluster, never production.
set -euo pipefail
TOPIC="perf_linger_sweep"
RECORDS=5000000
for LINGER in 0 5 20 50; do
for RUN in 1 2 3 4 5; do
bin/kafka-producer-perf-test.sh \
--topic "${TOPIC}" \
--num-records "${RECORDS}" \
--record-size 1024 \
--throughput -1 \
--warmup-records 500000 \
--record-key-range 100000 \
--key-distribution random \
--random-seed 42 \
--producer-props bootstrap.servers="${KAFKA_BOOTSTRAP}" \
acks=all linger.ms="${LINGER}" batch.size=65536 compression.type=zstd \
| tail -n 1 | sed "s/^/linger=${LINGER} run=${RUN} /" >> linger_sweep.log
done
done
# Report the median and the spread per linger value, not the best run.
# Check bin/kafka-producer-perf-test.sh --help on your build for the exact key-range semantics.
Our rules for any Kafka performance optimization claim:
- Keep record size, key distribution, partition count and replication factor fixed across a sweep. Change one setting per run.
- Run each configuration at least five times. Report the median and the spread (min to max, or p25 to p75), never the best run.
- Record broker version, client version, instance type, disk type and whether the page cache was warm. A result without them is not reproducible.
- Run against staging. The perf tool can saturate a cluster on purpose.
Consumers and Kafka share groups
The KIP-848 rebalance protocol
On the fetch side, Kafka performance optimization starts with the rebalance protocol. The consumer rebalance protocol from KIP-848 has been GA since 4.0, and it is enabled on the server once the 4.0 upgrade is finalized. Clients still choose it with group.protocol=consumer. From 4.3 the client logs a warning when it starts on the classic protocol, which is a useful way to find the services nobody has touched. The gain is operational: partitions move incrementally, so a deploy no longer pauses every member of the group.
# Kafka 4.x consumer: the KIP-848 rebalance protocol plus fetch sizing
bootstrap.servers=${KAFKA_BOOTSTRAP}
group.id=${CONSUMER_GROUP}
group.protocol=consumer # KIP-848, GA in 4.0; 4.3 logs a warning when classic is used
fetch.min.bytes=65536 # default 1; trade a little latency for fewer, fuller fetches
fetch.max.wait.ms=100 # default 500; caps the latency that fetch.min.bytes can add
max.partition.fetch.bytes=1048576
max.poll.records=500
Fetch sizing is where consumer throughput is usually lost. fetch.min.bytes at its default of 1 returns as soon as any data exists, which is right for latency and wasteful for bulk consumers. Raise it together with a bounded fetch.max.wait.ms so the latency it can add has a ceiling.
Kafka share groups
Kafka share groups (KIP-932) decouple consumer parallelism from partition count. Several consumers can process records from the same partition, with per-record acknowledgement. The Apache Kafka 4.2 announcement calls them production-ready. It also added the RENEW acknowledgement type for long-running work and share partition lag metrics. Release 4.4 adds a dead-letter queue for records that exceed the delivery count or are rejected.
# Kafka share groups: enable the share group dead-letter queue (KIP-1191, Kafka 4.4)
# Step 1: upgrade the feature level once every broker runs 4.4.
bin/kafka-features.sh --bootstrap-server "${KAFKA_BOOTSTRAP}" \
upgrade --feature share.version=2
# Step 2: set the DLQ topic for one share group (dynamic group config; verify the
# entity type and key names against the 4.4 documentation for your build).
bin/kafka-configs.sh --bootstrap-server "${KAFKA_BOOTSTRAP}" \
--alter --entity-type groups --entity-name "${SHARE_GROUP}" \
--add-config errors.deadletterqueue.topic.name="${SHARE_GROUP}_dlq"
The performance trap with share groups is monitoring, not throughput. Lag dashboards built on consumer group offsets return nothing for a share group. Wire up the share partition lag metrics before the first production workload moves, or the first backlog will be found by a customer.
Kafka tiered storage: delayed upload and faster follower bootstrap
Kafka tiered storage moves closed segments to object storage, so local disks hold only recent data. The cost is a window in which each segment exists in both tiers, which makes the upload schedule part of Kafka performance optimization. KIP-1241 in Kafka 4.4 adds remote.copy.lag.ms and remote.copy.lag.bytes, with broker-level equivalents, to delay the upload and shorten that window.
# Kafka tiered storage: delay remote upload on one topic (KIP-1241, Kafka 4.4)
# Values are illustrative. remote.copy.lag.ms must sit well inside local.retention.ms.
bin/kafka-configs.sh --bootstrap-server "${KAFKA_BOOTSTRAP}" \
--alter --entity-type topics --entity-name "${TOPIC}" \
--add-config remote.storage.enable=true,local.retention.ms=86400000,retention.ms=604800000,remote.copy.lag.ms=21600000
# Faster broker replacement on large-retention topics (KIP-1023, Kafka 4.3, default false):
# follower.fetch.last.tiered.offset.enable=true # broker config, set in server.properties
Two cautions. First, 4.0 lowered the default remote.log.manager.thread.pool.size from 10 to 2, so clusters upgraded from 3.x with heavy tiered topics should check upload lag before adding any copy delay. Second, KIP-1023 in 4.3 lets a new follower start from the last tiered offset instead of copying the whole local log. On large-retention topics that is the setting that shortens broker replacement. It is off by default, and we enable it only after a replacement drill in staging.
Kafka Streams: recovery time is the performance metric
For Kafka Streams, Kafka performance optimization is mostly about recovery. Under exactly-once, the expensive event is not steady-state processing. It is the restore after an error, when a task wipes its state store and rebuilds it from the changelog. KIP-892 in 4.4 adds transactional state stores. With them, only writes since the last successful commit are lost and replayed, instead of the whole store.
# Kafka Streams 4.4: transactional state stores (KIP-892), default off
processing.guarantee=exactly_once_v2
enable.transactional.statestores=true
statestore.uncommitted.max.bytes=${UNCOMMITTED_MAX_BYTES} # size the buffer against commit.interval.ms
# default.interactive.query.isolation.level controls what interactive queries can read
Measure restore time before and after from the restore listener or the restoration metrics, on a store large enough to matter. The feature is off by default, and the uncommitted buffer adds memory pressure that must be sized against the commit interval. Note also that Streams static membership in 4.4 needs 4.4 on both brokers and clients. Brokers older than 4.4 reject group.instance.id with a fatal error, which matters if you ever need to roll brokers back.
Broker defaults worth checking after the upgrade
These defaults change Kafka performance optimization results after an upgrade even when nobody edited a configuration file. Compare them with what your baseline was measured on.
| Parameter | Old default | 4.x default | Unit | Apply |
|---|---|---|---|---|
num.recovery.threads.per.data.dir | 1 | 2 | threads per log dir | Restart |
message.timestamp.after.max.ms | Long.MAX | 3600000 (1 hour) | ms | Dynamic |
remote.log.manager.thread.pool.size | 10 | 2 | threads | Restart |
linger.ms (producer) | 0 | 5 | ms | Client restart |
The timestamp change is not a speed setting, but it causes rejected produce requests from clients with skewed clocks when message.timestamp.type=CreateTime. Those requests look like a throughput drop on the dashboard.
Upgrading to Kafka 4.4 without losing the way back
A Kafka performance optimization upgrade must leave a rollback until the moment it cannot. Two 4.4 changes narrow that window. Storage directories formatted by the 4.4 kafka-storage tool are not forward-compatible (KIP-1170). And broker.id is deprecated in favour of node.id, with removal planned for 5.0.
# Kafka performance optimization upgrade gate: confirm finalized feature levels
# before and after each stage of the 4.4 rollout.
bin/kafka-features.sh --bootstrap-server "${KAFKA_BOOTSTRAP}" describe
# Only after every controller and broker runs 4.4 and the baseline tests pass:
# CONFIRMATION GATE: finalizing the release version has no supported downgrade path.
bin/kafka-features.sh --bootstrap-server "${KAFKA_BOOTSTRAP}" \
upgrade --release-version 4.4
# Validation after: the finalized metadata.version should now read 4.4-IVx
bin/kafka-features.sh --bootstrap-server "${KAFKA_BOOTSTRAP}" describe
Brokers and tools need Java 17 on the 4.x line. Clients and Kafka Streams need Java 11 or later. Confirm both before the first node rolls.
A Kafka performance optimization checklist for Kafka 4.4
We run through this list before and after every Kafka performance optimization change. Each line names the evidence that closes it.
| Check | Evidence |
|---|---|
Every producer sets linger.ms, batch.size and compression.type explicitly | Client config inventory |
| Baseline p99 produce latency and consumer lag recorded on the current version | Five-run medians with spread |
| Page-cache read hit ratio and write amplification scraped (KIP-1291) | Prometheus series per broker |
All consumer groups on group.protocol=consumer | No classic-protocol warnings in client logs (4.3+) |
| Share group lag alerting in place before share groups carry production traffic | Alert fires in a staging test |
Tiered storage upload lag checked before setting remote.copy.lag.ms | Remote log manager metrics |
| Streams restore time measured with and without transactional state stores | Restoration metrics on a production-sized store |
| Release version finalized only after binaries and baselines pass | kafka-features.sh describe output in the change record |
Kafka performance optimization FAQ
What is the most important Kafka performance optimization setting after upgrading to Kafka 4.x?
For most estates it is the producer linger.ms default, which moved from 0 to 5 milliseconds in Apache Kafka 4.0. Latency-sensitive producers that relied on the old default now wait up to 5 ms to fill a batch. Set linger.ms explicitly in every producer and measure p99 produce latency before and after.
Does Apache Kafka 4.4 make Kafka faster?
Not in the headline sense. For Kafka performance optimization, the 4.4 release improves recovery and visibility rather than raw throughput: transactional state stores shorten Kafka Streams recovery under exactly-once, delayed upload reduces tiered storage duplication, and new Linux I/O metrics make page-cache behaviour measurable. Treat any throughput change as something to measure on your own workload.
Are Kafka share groups ready for production?
The Apache Kafka 4.2 release announcement describes share groups as production-ready, 4.3 added further hardening, and 4.4 adds a dead-letter queue once share.version is upgraded to 2. Before adopting them, confirm that your lag alerting reads the share partition lag metrics, because consumer group lag alerts do not cover share groups.
How should we benchmark Kafka performance optimization changes?
Change one setting per run, fix the record size and key distribution, use warmup records, repeat each configuration at least five times and report the median with its spread. Kafka 4.4 adds --key-distribution, --record-key-range and --random-seed to kafka-producer-perf-test so runs are reproducible.
Where we can help
We run this test plan, the upgrade gates and the 24×7 operations behind them for Apache Kafka estates on self-managed clusters and managed services. If you want a second set of eyes on a 4.x upgrade, a latency regression or a Kafka performance optimization review, see our Kafka consulting and support practice, or our 24/7 emergency DBA coverage for production incidents. The primary sources for every version claim above are the Apache Kafka 4.4.0 release announcement and the Kafka 4.4 upgrade notes.
Test every change on your own workload in staging before applying it to production, and keep a tested disaster recovery posture.
Running this in production?
MinervaDB provides Data Engineering Consulting, Data Analytics Platform Engineering and Kafka Consulting with 24x7 coverage and a 15-minute S1 response. Talk to an engineer.
