Kubernetes Health Check: Probes, Timing, Blind Spots

A kubernetes health check is a probe the kubelet runs on a schedule against a container, and the cluster acts on the result without asking you. Three probe kinds exist, they have different consequences, and the timing fields decide how long a broken pod keeps its endpoint. Getting the numbers wrong is more common than getting the syntax wrong.

Three kubernetes health check probes, three consequences

Each probe answers a different question and triggers a different action. That is the whole design, and it is why copying one probe's config into another causes trouble.

A startup probe runs first and gates the other two. While it is still failing, the liveness and readiness probes are disabled entirely. That is what makes it the correct fix for an application needing ninety seconds to warm a cache. A liveness probe failure restarts the container. A readiness probe failure removes the pod's address from every Service that selects it, and nothing restarts.

So liveness is about a process that has wedged and cannot recover. Readiness is about a pod that is alive and temporarily unable to serve. Those are not the same event and they should not read the same endpoint.

Four handlers, and when each fits

The probe body picks one handler. The probe concept page documents all four.

httpGet is the default choice for anything that speaks HTTP. A 2xx or 3xx response passes and everything else fails, and you can set custom headers and an HTTPS scheme. tcpSocket just opens a connection, which suits databases and brokers with no HTTP surface. grpc calls the standard gRPC health service on a port you name.

exec runs a command inside the container and fails on a non-zero exit. It is the most flexible handler and the most expensive, because it forks a process on every period. The task page for configuring probes has a worked example of each.

We prefer httpGet wherever it is available. An exec probe on a one-second period across a large deployment is a measurable CPU cost, and it is the option people reach for out of habit.

What do the timing fields actually control?

They multiply into a detection window, and the defaults are slower than most people assume.

Field	Default	Effect
initialDelaySeconds	0	grace before the first probe
periodSeconds	10	gap between probes
timeoutSeconds	1	how long a probe may take
failureThreshold	3	consecutive failures before action
successThreshold	1	consecutive passes to recover

At the defaults, a container that hangs takes about thirty seconds to restart. A pod that stops being ready keeps its Service endpoint for the same thirty seconds, which means real user requests land on it during that window. Tighten periodSeconds on the readiness probe if that matters more than probe overhead.

Watch timeoutSeconds in particular. One second is short for an endpoint that touches a database, and a probe that times out counts as a failure. That is how a slow dependency turns into a restart loop.

A Deployment with all three probes

Here is a manifest that separates the three jobs properly.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments
spec:
  replicas: 3
  selector:
    matchLabels: { app: payments }
  template:
    metadata:
      labels: { app: payments }
    spec:
      containers:
        - name: api
          image: registry.example.com/payments:1.9.2
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 5
            failureThreshold: 24
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2

Two routes, not one. /healthz returns 200 from memory and touches nothing external. /readyz checks the database and the cache, and returns 503 when either is unavailable. The startup probe allows two minutes of boot time at 5 seconds times 24 attempts, so no initialDelaySeconds guesswork is needed anywhere.

The cascading restart

Point a liveness probe at a shared dependency and you have built a fleet-wide outage trigger. When the database stalls for forty seconds, every replica fails liveness at the same moment. Every container restarts, and now there is a cold start problem stacked on a database problem.

Readiness has no such failure mode, because removing endpoints is reversible and cheap. That asymmetry is the reason liveness endpoints should be boring. The Reddit thread that ranks on this query is mostly people who learned it the hard way.

There is a subtler version of the same trap. A readiness probe that checks a downstream API drops every pod from the Service when that API is slow. Your own app goes offline for a fault you do not own. Check dependencies you cannot serve without, and report the rest as degraded rather than unready.

Checking the cluster, not just the pods

Pod probes say nothing about nodes. A kubernetes health check at the workload layer cannot see a disk filling up underneath it. For that layer, kubectl is the tool.

kubectl get nodes -o wide
kubectl describe node worker-3 | grep -A6 Conditions
kubectl get pods -A --field-selector=status.phase!=Running

Node conditions cover Ready, MemoryPressure, DiskPressure and PIDPressure, and the node reference defines each one. A node under disk pressure evicts pods that were perfectly healthy, and no container probe will explain why. Treat node conditions as the saturation signal described in the four golden signals.

What every probe inside the cluster misses

The kubelet probes a container from the node it runs on. It never traverses your ingress controller, your load balancer, your TLS termination or your DNS record. Every one of those can break while all three probes stay green.

We tested this on a single-node cluster by deleting the Ingress object. Every pod stayed Ready, every probe passed, and an external http check against the public hostname failed on the next interval. The in-cluster view was correct and completely beside the point.

An outside check fixes the gap by resolving the real name and completing a real handshake. Run it from several places, because an ingress or routing fault often affects one region only. Uptime Room runs probes in six regions as of September 2026, and you can pull the egress ranges to allowlist from our probe address reference. Our guide to blackbox monitoring explains why that outside-in view is a separate discipline rather than a duplicate of your probes.

Common mistakes

One endpoint for liveness and readiness. It forces you to choose between restarting on a dependency blip and never detecting a wedged process. Write two routes.

Padding initialDelaySeconds instead of using a startup probe. A 120-second delay also delays detection of a genuine hang by 120 seconds. The startup probe exists for exactly this.

Assuming Ready means reachable. Readiness controls Service endpoints, not ingress rules or certificates. Check the public URL separately, the way our notes on what a check result means describe.

A working kubernetes health check setup uses three probes with three separate jobs. Give it timing that matches your real boot and failure windows, and a liveness endpoint with no dependencies in it. Add one external check on the public hostname so an ingress fault cannot hide behind green pods. The free plan covers 10 monitors at a one-minute interval as of September 2026, so start free and watch the edge too.