Kafka Health Check: Brokers, Clients and Canaries

A kafka health check has to answer one of two unrelated questions, and most implementations quietly answer the wrong one. Either you are asking whether the cluster can serve reads and writes, or you are asking whether this producer or consumer process is still doing its job. The probes for those two are not interchangeable.

Broker health and client health are different questions

Broker health is about the cluster: are the brokers reachable, is metadata current, are partitions replicated. Client health is about one process: is its connection alive, is the poll loop turning, is it committing offsets. A probe that answers one tells you nothing about the other.

Question	Where the probe runs	What it should look at
Can the cluster serve traffic?	outside the client	broker metadata, under-replicated partitions
Is this process working?	inside the client	connection state, last poll time
Can a message get through?	a dedicated canary	produce and consume round trip
Can anyone reach the listener?	an external probe	TCP connect to the advertised port

Mixing them is how a service reports healthy while its consumer has been wedged for an hour.

What does a TCP check on port 9092 actually prove?

That something is listening. Nothing else. The broker can be up and refusing your credentials, out of disk, missing the topic you need, or advertising a hostname your client cannot resolve. All of those leave the socket opening cleanly.

That does not make the port check useless. It is the cheapest way to detect a firewall change, a dead box or a listener that never bound, and it is the only check available from outside the network. Treat it as necessary and nowhere near sufficient.

For a real protocol handshake without writing code, the shipped CLI does the job.

kafka-broker-api-versions.sh --bootstrap-server broker-1:9092

That command negotiates versions with the broker and fails loudly on an auth or TLS problem. It is a better smoke test than nc -z and still cheap enough to run from a deploy pipeline.

Checking brokers with the admin API

Cluster metadata is the right broker-side signal. Fetching it proves the client can connect, authenticate and read the topic layout, which is most of what a producer needs before it sends anything.

from confluent_kafka.admin import AdminClient

admin = AdminClient({"bootstrap.servers": "broker-1:9092,broker-2:9092"})
meta = admin.list_topics(timeout=5.0)

if not meta.brokers:
    raise RuntimeError("no brokers in metadata")
if "orders" not in meta.topics:
    raise RuntimeError("orders topic missing")

Keep the timeout short and the assertions specific. Asserting that the topic your service reads from exists catches a whole class of deploy mistakes, and it costs one metadata request. The Kafka protocol documentation describes what that request carries if you want to know the cost precisely, and the confluent-kafka-python client reference documents the admin methods used above.

Which signals should a kafka health check watch?

Two numbers matter more than any probe result. Under-replicated partitions tell you the cluster has lost redundancy, usually because a broker is down or falling behind. Consumer lag tells you a consumer group is not keeping up with its input. Both are published as JMX metrics, listed in the Kafka monitoring documentation.

Lag is the one people get wrong. It is a throughput signal, not a liveness signal, and wiring it to a restart makes things worse. A restarting consumer rebalances, loses its position in the batch and falls further behind. Alert a human on lag, and let a probe handle liveness.

The distinction maps cleanly onto the four golden signals: lag is saturation, under-replicated partitions are a form of error, and neither is latency.

A health route your orchestrator can use

Client services need an HTTP endpoint because that is what orchestrators and load balancers speak. The endpoint should report internal state, not go and poke the cluster on every request.

For a consumer, the useful field is how long ago the poll loop last returned. Most clients expose it, and in the Java client it is the last-poll-seconds-ago metric. If that number exceeds your max.poll.interval.ms budget, the consumer is wedged and a restart is the right answer.

We tested a naive version of this first: a route that called list_topics on every request. Under a readiness probe at one-second intervals it added a steady stream of metadata requests to the cluster for no benefit. Cache the cluster answer, refresh it on a timer, and let the route read a variable.

The canary topic and what it costs

Round-tripping a message through a dedicated topic is the only kafka health check that proves end-to-end delivery. It is also the most expensive one you can run, in three ways.

It needs produce and consume ACLs, which is a real security conversation in a locked-down cluster. It adds partitions and traffic that grow with your probe interval. And it can pass while the topics your application actually uses are unavailable, because a healthy canary topic says nothing about a mis-replicated business topic.

We prefer running it rarely and deliberately. Once a minute at most, as a separate monitor with its own alert route. Never as the readiness probe that decides whether traffic arrives.

Watching the listener from outside

Everything above runs inside the network, on a host that already has credentials and routes. None of it proves a client in another VPC or another region can open a connection to the advertised listener.

A tcp check against 9092 from outside answers that. An http check against the client service's health route answers the other half. Run both from more than one location, because a security group change often breaks one path and leaves the rest working.

Uptime Room probes from six regions as of September 2026, with the source addresses listed for allowlisting at the probe IPs page. The trade-offs of transport-level checks are covered in our post on monitoring TCP and UDP ports.

Common mistakes

Restarting on lag. The rebalance costs more than the backlog did. Page someone instead, and reserve restarts for a poll loop that has genuinely stopped.

Probing the cluster on every readiness request. At a one-second interval across fifty pods, the probe becomes a meaningful load source. Cache it.

Treating a canary pass as an all-clear. A dedicated health topic with one partition and one replica stays green through failures that take your real topics offline. Read what a check result actually means before you trust one.

A useful kafka health check picks its question first. Cluster metadata and under-replicated partitions for the brokers, poll-loop age for the clients, and one external tcp check so a network fault cannot hide behind green pods. The free plan covers 10 monitors at a one-minute interval as of September 2026, enough for a listener and a health route, so start free.