Kafka Support

MinervaDB Kafka support, consulting and real-time streaming performance engineering

Kafka Support, Consulting and 24/7 Performance Engineering

MinervaDB operates, tunes and rescues Apache Kafka event-streaming platforms for teams that cannot tolerate a stalled pipeline. This page is deliberately technical. It explains how the distributed commit log behaves under load, which configuration knobs actually move latency and throughput, how to size hardware, and how our engineers diagnose Kafka failures in production.

If you are comparing Apache Kafka support vendors, treat the sections below as a checklist of what a competent partner should already know before touching your cluster. Everything here comes from incident work rather than marketing copy: broker saturation at 03:00, consumer groups trapped in a rebalance loop, silent data loss after an unclean leader election, and cost blowouts caused by nothing more exotic than a badly chosen partition count.

Contents

Reference Architecture of a Production Kafka Platform

A healthy Kafka deployment is layered. Publishers never write to storage directly, subscribers never depend on a single node, and cluster metadata lives in its own replicated quorum. The Kafka topology below is the shape we deploy, review and support most often.

     PUBLISHERS                KAFKA BROKER TIER (N nodes)               SUBSCRIBERS
 +----------------+        +----------------------------+        +-----------------+
 | App services   |        |  broker-1        broker-2  |        | Stream jobs     |
 | CDC / Debezium |  --->  |  [P0 L][P1 F]   [P0 F][P1 L]  --->  | Flink, Streams  |
 | Edge + mobile  |  mTLS  |  [P2 F][P3 L]   [P2 L][P3 F]        | Sink connectors |
 | Log shippers   |        |          broker-3          |        | OLAP + search   |
 +----------------+        |  [P0 F][P1 F][P2 F][P3 F]  |        +-----------------+
         |                 +-------------+--------------+                 |
         |                               | metadata RPC                   |
         |                  +------------v-------------+                  |
         |                  | KRaft controller quorum  |                  |
         |                  |  3 or 5 voters, Raft log |                  |
         |                  +--------------------------+                  |
         |                                                                |
         +----------------- schema registry + governance ------------------+

   L = partition leader        F = in-sync follower        P0..P3 = partitions

Three design rules govern every Kafka cluster we build. First, the metadata quorum is sized for an odd number of voters so a majority always exists; three voters tolerate one failure, five tolerate two. Second, replicas of the same partition are spread across fault domains using broker.rack, so a rack, availability zone or hypervisor loss never removes a majority of a partition’s replicas. Third, the client tier is isolated from storage decisions by a schema contract, which lets us evolve payloads without redeploying every consumer.

We also insist on separating workload classes onto different listeners and, where volume justifies it, different broker pools. Mixing a bursty clickstream with a low-latency payments topic on the same disks is the fastest route to unpredictable tail latency.

The Kafka Distributed Log: Partitions, Segments and Offsets

A Kafka topic is not a queue. It is a partitioned, append-only log, and each partition is an ordered, immutable sequence of records addressed by a monotonically increasing offset. Ordering is guaranteed inside a partition and nowhere else. That single property is the most frequently misunderstood aspect of the platform and the root cause of most “events arrived out of order” tickets we are asked to investigate.

topic "payments", 3 partitions, append-only, offsets never reused

partition-0  | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |  <-- append at log end
partition-1  | 0 | 1 | 2 | 3 | 4 | 5 | 6 |
partition-2  | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
                                  ^
                          consumer group commits
                          offset 5 for partition-2

on-disk layout of partition-0
  /var/lib/data/payments-0/
      00000000000000000000.log         records, closed segment
      00000000000000000000.index       offset  -> byte position
      00000000000000000000.timeindex   timestamp -> offset
      00000000000000368421.log         ACTIVE segment (never deleted)
      leader-epoch-checkpoint          fencing data for truncation
      partition.metadata

Segments roll when they exceed segment.bytes or segment.ms, and only closed segments are eligible for deletion or compaction. This is why a topic with a seven-day retention window and a one-gigabyte segment size can still hold data that is far older than seven days: the active segment has not rolled yet. We tune segment size against retention granularity rather than leaving both at defaults.

Keying, partition assignment and hot partitions

The default Kafka partitioner hashes the record key with murmur2 and takes the modulus of the partition count. Two operational consequences follow. Records sharing a key always land on the same partition and therefore preserve relative order, and increasing the partition count permanently breaks that mapping for existing keys. Partition counts are a one-way door, so we plan them during design review instead of discovering the problem during a migration.

Symptom Likely cause Remedy we apply
One partition 20x the size of its peers Low-cardinality or skewed key, for example tenant_id where one tenant dominates Composite key, salted key, or custom partitioner with explicit routing
Lag concentrated on a single consumer Hot partition pinned to one group member Re-key the topic, split the tenant into a dedicated topic
Throughput plateau despite adding consumers Consumers exceed partition count, so extras idle Raise partitions with a planned re-key, or fan out downstream
Ordering violations after a scale-up Partition count changed, hash modulus moved Dual-write migration to a new topic with the final partition count

As a planning heuristic we keep partitions per Kafka broker in the low thousands, allow at least one partition per unit of intended consumer parallelism, and leave 30–40 percent headroom for future growth. Every partition costs an open file handle set, a replication fetch slot and a slice of controller metadata, so more is not free.

Kafka Replication, In-Sync Replicas and Durability Guarantees

Kafka durability is decided by the interaction of three settings: the replication factor, min.insync.replicas, and the producer acknowledgement mode. Getting any one of them wrong silently converts a “no data loss” design into a best-effort one.

partition P0   replication.factor = 3   min.insync.replicas = 2

  leader   broker-1  | 0 1 2 3 4 5 6 7 8 |   LEO = 9   HW = 7
  follower broker-2  | 0 1 2 3 4 5 6 7 |     LEO = 8   in ISR
  follower broker-3  | 0 1 2 3 4 5 6 7 |     LEO = 8   in ISR
  follower broker-4  | 0 1 2 3 |             LEO = 4   OUT of ISR (lagging)
                             ^
        High Watermark (HW): highest offset replicated to every
        ISR member. Consumers can only read up to the HW, so a
        record is never visible before it is durably replicated.

        LEO = Log End Offset (next offset to be written)

  replica falls out of ISR when it has not fetched within
  replica.lag.time.max.ms  (default 30s)
acks Write is acknowledged when Loss window Typical fit
0 The client hands bytes to the socket Everything in flight Disposable telemetry only
1 The leader writes to its own page cache Leader crash before replication Metrics, logs, low-value events
all Every ISR member has the record Only a simultaneous loss of all ISR members Payments, ledgers, CDC, audit

With acks=all and min.insync.replicas=2 on a three-replica partition, the cluster tolerates one broker failure with no loss and no availability impact, and it deliberately rejects writes with NOT_ENOUGH_REPLICAS if a second replica also fails. That rejection is a feature: it trades availability for correctness at the exact moment the guarantee would otherwise be violated.

We keep unclean.leader.election.enable=false on every business-critical Kafka topic. Enabling it allows an out-of-sync replica to become leader, which truncates committed records and produces the hardest class of data-loss incident to explain after the fact. Leader epochs and the leader-epoch-checkpoint file exist precisely to make truncation deterministic, and they only work if unclean elections stay disabled.

The Kafka Producer Path and Write-Side Tuning

Most throughput complaints we receive are not Kafka broker problems at all. They are client problems: unbatched sends, a single producer instance shared by hundreds of threads, or a compression codec chosen without measuring CPU cost. Understanding the write path makes the fix obvious.

send(record)
    |
    v
 serializer  (Avro / Protobuf / JSON + schema id)
    |
    v
 partitioner  (murmur2(key) % partitions, or sticky when key is null)
    |
    v
 RecordAccumulator            per-partition batch buffers
 +-----------------------------------------------+
 | P0: [ rec rec rec ]   16 KB batch.size        |
 | P1: [ rec rec ]                               |
 | P2: [ rec ]                                   |
 +-----------------------------------------------+
    |  flush trigger: batch full OR linger.ms elapsed
    v
 Sender thread (one per client instance)
    |  max.in.flight.requests.per.connection = 5
    |  enable.idempotence = true  -> sequence numbers per partition
    v
 broker: validate -> append to page cache -> (fsync by OS)
    |
    |  acks=all: wait until min.insync.replicas have fetched
    v
 ack  (or retriable error -> retry with same sequence number)
Setting Default What we usually set Why
batch.size 16 KB 64–256 KB Fewer, larger requests cut per-request overhead dramatically
linger.ms 0 5–50 ms Buys batching at a bounded, predictable latency cost
compression.type none zstd or lz4 zstd for storage and network savings, lz4 when CPU is the constraint
enable.idempotence true true Removes duplicates caused by retries without a transaction
max.in.flight.requests.per.connection 5 5 with idempotence Ordering is still preserved because sequences are checked
buffer.memory 32 MB 128–512 MB Absorbs bursts instead of blocking the calling thread
delivery.timeout.ms 120 s Aligned to business SLA Caps total time a record can sit in retry limbo

Kafka compression is applied per batch, not per record, which is why linger.ms=0 and compression together often deliver worse ratios than no compression at all. We also standardise on Kafka broker-side compression.type=producer so the cluster never spends CPU recompressing what the client already compressed.

Kafka Consumer Groups, Rebalancing and Lag

A Kafka consumer group is a coordination protocol, not just a label. The group coordinator assigns partitions, tracks liveness through heartbeats, and stores committed offsets in the internal __consumer_offsets topic. Rebalances are where availability is usually lost.

group "fraud-scoring"    topic "payments" (6 partitions)

  BEFORE scale-out                AFTER adding member C3
  C1 -> P0 P1 P2                  C1 -> P0 P1
  C2 -> P3 P4 P5                  C2 -> P2 P3
                                  C3 -> P4 P5

  EAGER protocol (legacy)
    join -> REVOKE ALL -> compute -> assign -> resume
            |______ stop-the-world pause for the whole group ______|

  COOPERATIVE-STICKY protocol (what we deploy)
    join -> compute -> revoke ONLY the partitions that move
         -> second rebalance assigns them -> others never stop

  liveness:  session.timeout.ms      (heartbeat thread)
  progress:  max.poll.interval.ms    (application processing time)

Two Kafka client timeouts are routinely confused. session.timeout.ms governs the background heartbeat and detects a dead process. max.poll.interval.ms governs how long the application may spend processing a batch before the coordinator assumes it is stuck. A slow database write inside the poll loop triggers the second, not the first, and the resulting eviction looks like a network problem to the untrained eye. The fix is usually to lower max.poll.records and move blocking work off the poll thread.

Kafka consumer lag is the arithmetic difference between the partition log end offset and the group’s committed offset. We alert on the derivative rather than the absolute value, because a batch job legitimately sits at high lag while a steadily climbing lag on a real-time topic is an emergency.

lag(partition) = log_end_offset - committed_offset
time_to_drain  = lag / (consume_rate - produce_rate)      when consume > produce

  healthy      lag  ~~~~~~~~~~~~~~~~  flat, small, sawtooth
  falling behind    lag  /////////     monotonic climb  -> page someone
  stalled           lag  ---------     flat but large   -> stuck consumer
  rebalance storm   lag  ////      repeated spikes   -> timeout tuning

Kafka KRaft Metadata Quorum Versus ZooKeeper

Modern Kafka clusters replace the external coordination service with a built-in Raft quorum that stores metadata in a dedicated internal log. For teams still running the older architecture, migration is one of the most common engagements we are asked to plan and execute.

   LEGACY (ZooKeeper)                       KRaft
 +-------------------+                +---------------------------+
 | ZK 1  ZK 2  ZK 3  |                | controller-1  (Raft leader)|
 +---------+---------+                | controller-2  (voter)      |
           ^  znodes: topics,         | controller-3  (voter)      |
           |  ISR, configs, ACLs      +-------------+-------------+
           |                                        | __cluster_metadata
 +---------+---------+                +-------------v-------------+
 | brokers + one     |                | brokers replay the        |
 | active controller |                | metadata log locally and  |
 +-------------------+                | cache it in memory        |
                                      +---------------------------+

  failover: seconds to minutes  |  failover: sub-second
  practical partition ceiling:  |  practical ceiling an order of
  tens of thousands             |  magnitude higher
  two security models to manage |  one security model

The operational win for Kafka operations is not merely fewer moving parts. Because every broker holds an up-to-date replica of the metadata log, controller failover no longer requires reloading state from an external store, so recovery time becomes largely independent of cluster size. Our Kafka migration runbook covers voter sizing, dual-write bridge mode, rollback checkpoints, and validation of every ACL and dynamic config before the old coordination layer is decommissioned.

Kafka Transactions and Exactly-Once Semantics

Exactly-once is achievable, but only for the read-process-write pattern inside Kafka, and only when producers, consumers and the state store all participate. We spend a lot of time correcting the belief that setting one flag delivers it end to end.

  consume -> transform -> produce, atomically

  1. initTransactions()      producer.id + epoch fenced by coordinator
  2. beginTransaction()
  3. send(outputTopic, ...)  markers written to partitions
  4. sendOffsetsToTransaction(consumedOffsets, groupId)
  5. commitTransaction()
        |
        v
  transaction coordinator writes COMMIT marker to __transaction_state
  and control records to every touched partition

  reader side: isolation.level = read_committed
    |A A A|C|  -> visible        (C = commit marker)
    |B B B|X|  -> filtered out   (X = abort marker)

  Last Stable Offset (LSO) <= High Watermark: read_committed
  consumers stop at the LSO, so one long transaction blocks
  visibility for every later record in that partition.

The practical cost of Kafka transactions is throughput and latency: commit markers add round trips, and a hung transaction pins the last stable offset, which stalls all read_committed readers on that partition until transaction.timeout.ms expires. We therefore keep transaction scope small, measure the overhead against the alternative of idempotent writes with downstream deduplication, and only recommend full transactional pipelines where the business genuinely cannot tolerate duplicates.

Kafka Retention, Compaction and Tiered Storage

Kafka storage policy is where cost control lives. Two cleanup policies exist and they solve different problems: time or size based deletion for event streams, and key-based compaction for changelog or state topics.

  cleanup.policy = delete            cleanup.policy = compact
  drop whole segments once           retain the latest value per key
  retention.ms / retention.bytes
  is exceeded

  BEFORE compaction
    | k1=v1 | k2=v1 | k1=v2 | k3=v1 | k2=v2 | k1=v3 | k4=null |
                       ^ superseded          ^ superseded

  AFTER compaction (tail rewritten, offsets preserved, gaps allowed)
    | k3=v1 | k2=v2 | k1=v3 |            (k4=null tombstone removed
                                          after delete.retention.ms)

Kafka log compaction never touches the active segment, runs on the cleaner threads governed by log.cleaner.threads and min.cleanable.dirty.ratio, and preserves original offsets, which is why a compacted topic legitimately shows offset gaps. A cleaner that cannot keep up is a common hidden cause of disks filling on a topic that “should” be bounded.

Tiered storage

   BROKER LOCAL DISK (hot)              OBJECT STORE (warm/cold)
 +--------------------------+        +---------------------------+
 | active + recent segments |  --->  | offloaded closed segments |
 | NVMe, low latency reads  |  copy  | S3 / GCS / Azure Blob     |
 | local.retention.ms       |        | retention.ms (long tail)  |
 +-----------+--------------+        +-------------+-------------+
             |                                     |
             |    consumer reads recent data       |
             +------------------> broker <---------+
                        remote fetch for historical replay

  effect: local disk sized for the working set, not for retention;
  cluster rebalance and broker replacement become far faster because
  there is much less local data to move.

Kafka tiered storage changes capacity planning fundamentally. Local disk is sized for the hot working set while long retention moves to object storage at a fraction of the cost, and broker replacement stops being a multi-hour data-shuffling exercise. The trade-off is higher and less predictable latency for historical reads, so we validate replay performance against real backfill jobs before recommending it.

Kafka Capacity Planning Arithmetic

We refuse to size Kafka clusters by intuition. The model below is deliberately simple and has survived contact with production many times.

  ingress_raw       = events_per_sec * avg_event_bytes
  ingress_on_wire   = ingress_raw / compression_ratio
  replication_load  = ingress_on_wire * (replication_factor - 1)
  broker_egress     = ingress_on_wire * consumer_group_count
                      + replication_load

  daily_storage     = ingress_on_wire * 86400 * replication_factor
  retained_storage  = daily_storage * retention_days * (1 + headroom)

  partitions_min    = max( target_throughput / per_partition_throughput ,
                           required_consumer_parallelism )

  WORKED EXAMPLE
  200,000 events/sec, 1.2 KB average, zstd ratio 4:1, RF=3,
  3 consumer groups, 7 day retention, 40% headroom

  ingress_raw      = 240   MB/s
  ingress_on_wire  =  60   MB/s
  replication_load = 120   MB/s
  broker_egress    = 300   MB/s   (fleet aggregate)
  daily_storage    = 60 MB/s * 86400 * 3     =  15.5 TB/day
  retained_storage = 15.5 * 7 * 1.4          = 152   TB
  -> 9 brokers with 20 TB usable NVMe each, 25 GbE, or the same
     ingest with ~2 days local retention plus tiered storage.

Notice that Kafka replication and fan-out, not raw ingest, dominate network sizing. A cluster that comfortably accepts 60 MB/s of writes can still saturate its network interfaces once three replicas and three consumer groups are accounted for. This is the calculation most often skipped in self-designed deployments.

Kafka Broker, JVM and Operating-System Tuning Reference

Layer Parameter Guidance
Broker num.network.threads / num.io.threads Network threads near core count; I/O threads at roughly two times the number of data directories
Broker num.replica.fetchers Raise to 4–8 on high-partition clusters so followers keep pace and the ISR stays stable
Broker socket.send.buffer.bytes 1 MB or more on high bandwidth-delay-product links, especially cross-region replication
Broker log.flush.interval.messages Leave unset; rely on replication for durability rather than synchronous flush
JVM Heap size 6–12 GB is almost always correct; the page cache, not the heap, serves reads
JVM Collector G1 with a 20 ms pause target, or ZGC on very large heaps; alert on pauses above 100 ms
OS vm.swappiness 1, so the page cache is never swapped out under memory pressure
OS vm.dirty_ratio / dirty_background_ratio Lower the background ratio to smooth writeback spikes and avoid latency cliffs
OS File descriptors At least 100,000; every segment and connection consumes handles
Storage Filesystem XFS with noatime; separate data directories per physical device, never RAID 5
Network MTU and offload Consistent MTU end to end; verify that segmentation offload is enabled on all hosts

The single most valuable Kafka tuning rule on this list is heap sizing. Zero-copy transfer from the page cache is what makes the log fast, so memory handed to the JVM is memory taken away from the mechanism that actually serves your consumers.

Kafka Observability: The Metrics That Actually Matter

Dashboards with two hundred panels are not observability. These are the Kafka signals we wire up first on every engagement, because each one maps directly to a decision.

Signal Source Why it matters Action threshold
UnderReplicatedPartitions Broker JMX Replicas are behind, so durability is degraded right now Any non-zero value sustained beyond a minute
OfflinePartitionsCount Controller JMX Partitions have no leader and are unavailable Page immediately on any non-zero value
ActiveControllerCount Controller JMX Must sum to exactly one across the cluster Zero or greater than one is a split-brain risk
IsrShrinksPerSec Broker JMX Flapping replicas usually mean disk or network saturation Repeated shrink and expand cycles
RequestQueueTimeMs p99 Broker JMX Distinguishes queueing delay from real work Rising queue time with flat local time means thread starvation
RequestHandlerAvgIdlePercent Broker JMX Direct measure of I/O thread saturation Below 30 percent means add threads or brokers
Consumer group lag Admin API The only true measure of business freshness Alert on sustained positive slope, not absolute value
Log flush latency p99 Broker JMX Detects a failing or throttled disk before it takes a broker down Deviation from the fleet baseline
GC pause duration JVM Long pauses cause spurious ISR shrinks and session timeouts Any pause beyond 100 ms
Produce error rate by type Client metrics NOT_ENOUGH_REPLICAS and timeouts have completely different fixes Any sustained non-zero rate

We instrument Kafka clients as aggressively as brokers. Server-side dashboards that look perfect while a producer quietly buffers and drops records are one of the most dangerous blind spots in streaming operations, and client metrics are the only place that failure is visible.

Kafka Multi-Region Topologies and Disaster Recovery

There is no single correct cross-region Kafka design, only trade-offs between recovery point objective, recovery time objective, write latency and cost. We choose between three patterns.

 (1) ACTIVE / PASSIVE with asynchronous replication
   REGION A (write)                     REGION B (standby)
   [cluster] --- MirrorMaker 2 ---> [cluster]
   RPO: seconds to minutes    RTO: minutes (offset translation needed)
   cheapest, simplest, small but real data-loss window

 (2) ACTIVE / ACTIVE with prefixed remote topics
   REGION A  <===== bidirectional MM2 =====>  REGION B
   orders            a.orders / b.orders           orders
   RPO: seconds     RTO: near zero for reads
   requires consumers that understand remote topic naming
   and application-level conflict handling

 (3) STRETCH CLUSTER across three low-latency zones
   ZONE 1        ZONE 2        ZONE 3
   [brokers] === [brokers] === [brokers]     one logical cluster
   rack-aware replica placement, min.insync.replicas = 2
   RPO: zero     RTO: automatic failover
   requires sub-10 ms inter-zone latency; not viable across continents

Whichever Kafka pattern is chosen, the recovery plan is only real if it is exercised. Our Kafka disaster-recovery engagements include scripted failover drills, consumer offset translation verification, and a documented decision tree for the human on call. We also verify that schema registry state, connector configurations and ACLs are replicated, because a cluster that comes up without them is not actually a recovered service.

Kafka Security Hardening: mTLS, SASL and ACLs

  CLIENT                          BROKER
    |  1. TLS handshake, both sides present certificates
    |----------------------------------------------->|
    |  2. SASL authentication (SCRAM-SHA-512, GSSAPI,
    |     or OAUTHBEARER against your identity provider)
    |----------------------------------------------->|
    |  3. Principal extracted -> mapped via
    |     ssl.principal.mapping.rules
    |  4. Authorizer checks ACLs
    |     (principal, operation, resource, host)
    |<-----------------------------------------------|
    |  5. Allowed operations only; quotas applied per principal

  listeners:
    INTERNAL  :9092  broker-to-broker, mTLS
    EXTERNAL  :9093  clients, TLS + SASL
    CONTROLLER:9094  metadata quorum only, never client reachable

  at rest: full-disk or volume encryption, plus payload-level
  encryption for regulated fields the platform must not read.

Least privilege in Kafka is enforced with explicit allow rules per principal and resource, wildcard grants are removed, and quotas cap the damage any single misbehaving client can do. We treat the controller listener as strictly internal, run regular ACL drift audits, and keep credential rotation scripted so it is a routine change rather than a project.

Kafka Failure Modes and How We Diagnose Them

What you observe Underlying cause we usually find Resolution
Producer timeouts under normal load Request queue saturation, or a leader on a broker with a degraded disk Add I/O threads, move leadership, replace the device
Endless rebalance loop max.poll.interval.ms exceeded by slow downstream calls Reduce max.poll.records, make processing asynchronous, adopt cooperative-sticky assignment
Lag grows only on some partitions Key skew creating a hot partition Re-key, salt the key, or isolate the noisy tenant
Disk fills despite short retention Log cleaner starved, or a huge active segment that never rolled Add cleaner threads, tune segment size and dirty ratio
Records disappear after a node restart Unclean leader election combined with acks=1 Disable unclean election, move to acks=all with min.insync.replicas=2
Consumers stall at a fixed offset A hung transaction pinning the last stable offset Lower transaction.timeout.ms, fence the zombie producer, shrink transaction scope
Cross-region replication falls behind Undersized socket buffers on a high-latency link Enlarge send and receive buffers, increase fetcher parallelism
Periodic latency spikes every few minutes Garbage collection pauses or page-cache writeback storms Right-size the heap, tune dirty ratios, tune the collector
Schema evolution breaks a downstream job No compatibility policy enforced in the registry Enforce backward compatibility in CI and gate deployments on it

Our Kafka diagnostic method is consistent: establish whether the constraint is client-side, network, storage or coordination before changing anything. Configuration changes made without that classification are how a one-hour incident becomes a three-day one.

MinervaDB Kafka Support Catalogue and Service Levels

MinervaDB Kafka support services: architecture review, performance tuning and 24/7 operations

1. Architecture design and design review

We design Kafka topic taxonomies, partition and key strategies, replica placement across fault domains, retention and compaction policy, and the metadata quorum layout. Existing deployments get a written review with prioritised, quantified findings rather than a generic best-practice list.

2. Performance engineering

Throughput and tail-latency work across the whole Kafka path: producer batching and compression, broker thread pools and file systems, replication fetcher parallelism, consumer poll-loop design, and stream-processing state stores. We benchmark before and after so the improvement is a number, not an opinion.

3. 24/7 consultative support and incident response

Follow-the-sun Kafka coverage from engineers who operate distributed logs for a living. We join your incident channel, drive root-cause analysis, and deliver a written post-incident report with concrete preventive changes.

4. Migrations and upgrades

Kafka version upgrades, coordination-layer migration to the Raft-based metadata quorum, self-managed to cloud moves, cloud to self-managed repatriation for cost reasons, and cross-cloud replication cutovers. Every plan includes rollback checkpoints and validation gates.

5. Pipeline and integration engineering

Change-data-capture ingestion into Kafka, connector development and hardening, stream processing with Flink or the Streams library, and delivery into analytical stores. See our related work on data strategy and analytics and vector data engineering for how streaming fits a wider platform.

6. Cost optimisation

Right-sizing Kafka brokers, tiered storage adoption, compression and retention policy tuning, partition rationalisation, and elimination of redundant replication traffic. Reducing infrastructure spend by a third without touching durability guarantees is a common outcome.

7. Security, compliance and governance

Kafka authentication and authorisation design, ACL rationalisation, encryption in transit and at rest, quota policy, audit logging, and evidence packages for regulated environments.

Tier Coverage First response Best for
Advisory Business hours, scheduled review sessions 1 business day Teams building their first streaming platform
Production 24 x 7 for severity 1 and 2 30 minutes for severity 1 Revenue-bearing real-time pipelines
Mission critical 24 x 7 x 365, named engineers, quarterly game days 15 minutes for severity 1 Payments, trading, fraud detection, telemetry at scale
Project Fixed-scope engagement By agreement Migrations, upgrades, audits, capacity studies

How an engagement starts

  WEEK 0        discovery: Kafka topology, versions, topics, clients, SLOs
     |
  WEEK 1        instrumented baseline: throughput, p99 latency, lag,
     |          ISR stability, storage growth, cost per event
     |
  WEEK 2        findings report: ranked issues, effort vs impact,
     |          quantified risk for each durability gap
     |
  WEEK 3-4      remediation in staging, then controlled production
     |          rollout with rollback checkpoints at every step
     |
  ONGOING       24/7 coverage, monthly performance review,
                capacity forecast, quarterly failover drill

Frequently Asked Questions

How many partitions should a Kafka topic have?

Enough to meet your throughput target and to allow the consumer parallelism you need, plus roughly 30 to 40 percent headroom. Because increasing the count reshuffles key-to-partition mapping, we size it at design time using the arithmetic in the capacity section rather than adjusting it later.

Do we still need ZooKeeper with Kafka?

New Kafka clusters should be built on the Raft-based metadata quorum, and existing ZooKeeper-backed clusters should have a dated migration plan. The benefits are faster controller failover, a much higher practical partition ceiling and one security model instead of two.

Can we get exactly-once delivery end to end?

Atomic read-process-write is achievable within Kafka using transactions and read_committed isolation. Once data leaves for an external system, you need either an idempotent sink or a transactional sink; otherwise the correct design is at-least-once delivery with downstream deduplication.

What causes most Kafka production incidents?

In our experience, client misconfiguration and partition skew cause more outages than Kafka broker failures. Defaults that are safe for a laptop are frequently wrong for a cluster handling hundreds of thousands of events per second.

Should we self-manage Kafka or use a managed service?

It depends on data-gravity, latency requirements, compliance constraints and the true total cost at your volume. We model both options with your real numbers, and we support either choice; our value is engineering judgement, not a licence resale margin.

Do you work alongside our existing Kafka team?

Yes. Most engagements are collaborative: we embed with your engineers, transfer knowledge deliberately, and leave behind runbooks, dashboards and tuned configurations that your team owns.

Talk to MinervaDB About Your Kafka Platform

Whether you need a second opinion on a Kafka architecture, an urgent hand on a live incident, or continuous 24/7 coverage for a revenue-critical pipeline, our Kafka engineers are available now. Bring us your topology, your metrics and your worst-performing topic, and we will tell you exactly what we would change and why.

Please contact us to schedule a technical consultation.

Contact MinervaDB for Kafka support and streaming data infrastructure consulting

Further Reading

MinervaDB Inc. — your trusted partner in real-time data streaming, performance engineering and mission-critical data infrastructure.