Consul Health Check: Types, States and Blind Spots

A consul health check is a probe the local agent runs against a service it has registered, so the catalog only hands out addresses that still answer. Checks live inside the service definition rather than in one central file. That single design decision shapes how they fail, and it is the part most walkthroughs skip past.

Which consul health check type fits a service?

Consul ships ten check types, and picking the wrong one is the usual reason a service looks healthy while it is not. The HashiCorp reference documents each field. The short version fits in a table.

Type	What it does	Good for
http	GET a URL, expect 2xx	web and API services
tcp	open a socket	databases, brokers
udp	send a datagram	DNS, syslog
grpc	call the standard gRPC health service	gRPC servers
h2ping	HTTP/2 ping frame	HTTP/2 backends
script	run a command on the node	anything with a CLI probe
ttl	the service pushes, Consul waits	batch workers, agents
alias	mirror another service or node	sidecars, proxies
docker	exec inside a container	containerised workloads
os service	query systemd or Windows services	daemons without an endpoint

A tcp check on a web service is the classic mistake. It proves a socket opened, nothing more.

Registering a check in the service definition

Put the check block inside service and reload the agent. Nothing gets registered centrally, which is why a node that never starts its agent also never appears in the catalog as unhealthy.

service {
  name = "payments"
  port = 8080

  check {
    id                                = "payments-http"
    name                              = "HTTP on 8080"
    http                              = "http://localhost:8080/health"
    method                            = "GET"
    interval                          = "10s"
    timeout                           = "2s"
    deregister_critical_service_after = "90s"
  }
}

Three fields deserve thought. interval sets detection granularity, timeout decides how slow counts as broken, and deregister_critical_service_after deletes the service entirely once it has been critical that long. We prefer a 10-second interval with a 2-second timeout for HTTP services, and a deregister window no shorter than 15 minutes. A short window turns a long deploy into a vanished service.

Passing, warning and critical

Consul checks report one of three states, and the middle one behaves differently from how people assume. Critical instances are dropped from DNS answers. Warning instances are still returned, because only_passing is off by default in dns_config.

So a warning does not take anything out of rotation. It shows up amber in the UI and traffic keeps arriving. If you want a warning to drain a node, either set only_passing or stop using the warning exit code at all. Script checks map exit 0 to passing, exit 1 to warning and anything else to critical.

That exit code mapping catches people out in a different way too. A script that crashes, times out or cannot be executed returns something other than 0 or 1, so it lands in critical rather than warning. The failure mode is correct but the reason is invisible unless you read the check output. Keep the output short and specific, because Consul truncates it and the UI shows only the tail.

Reading check state from the API

The agent endpoint reports what this node knows. The catalog endpoint reports what the whole datacenter knows. They disagree more often than you would like, and the difference is usually a partition.

curl -s http://127.0.0.1:8500/v1/agent/checks | jq '.[] | {CheckID, Status, Output}'
curl -s 'http://127.0.0.1:8500/v1/health/service/payments?passing' | jq 'length'

The passing query parameter is the one to build on. It returns only instances whose checks are green, which is exactly what a client library wants before it picks an address. The health HTTP API docs list the filters, including per-node and per-state lookups.

Both endpoints support blocking queries, so a client can hold the connection open and get pushed the new list the moment a check flips. That beats polling every second and it is how consul watch and the template tooling stay current. Use it for anything that has to react inside a few hundred milliseconds of a state change.

What happens when the agent itself stops?

Serf gossip notices. Every node runs a hidden serfHealth check, and when gossip stops arriving from a node its services go critical across the datacenter. Detection takes seconds, not milliseconds, and it depends on enough peers still talking to each other.

That covers a dead box. It does not cover a datacenter that is fine internally and unreachable from the internet. Gossip is happy, every consul health check is green, and your users get a timeout. The same gap shows up with any internal probe, which is why blackbox monitoring exists as a separate discipline.

TTL checks are the other agent-dependent case. The service calls /v1/agent/check/pass/<check-id> on a schedule, and Consul marks it critical if the call stops. That is the inbound pattern described in our guide to heartbeat monitoring, and it fits batch workers with no listening port.

Watching the datacenter from outside

An external probe hits the public hostname the way a customer does. It resolves DNS, completes a TLS handshake and reads the body. A broken load balancer or an expired certificate then shows up as a real failure instead of a green dashboard.

We tested this against a two-node Consul setup behind a single ingress. Killing the ingress left both agents reporting passing for every service, while an outside http check went down within one interval. The internal view was accurate and useless.

Run the outside check against the same name your clients use. Multiple regions matter here, because a routing fault often hits one continent and leaves the rest untouched. Uptime Room probes from six regions as of September 2026, and you will find the addresses to allowlist under our published probe egress IPs.

Common mistakes

Pointing the check at a route with no dependencies. A handler that returns 200 from memory passes forever. Make the endpoint touch the database or cache the service actually needs, then let http assert on it.

Setting deregister_critical_service_after too aggressively. At 60 seconds, a slow migration deletes the service from the catalog, and clients get an empty address list instead of a failing one. Empty lists produce stranger bugs than errors do.

Treating the Consul UI as an alerting surface. Nothing pages anyone when the amber square appears, and nothing pages anyone when the UI itself goes away. Pair the catalog with an alert route that reaches a human, the way our notes on alert routing for APIs describe.

A well-built consul health check keeps traffic flowing to instances that can actually serve it. Match the type to the protocol, and leave a deregister window long enough to survive a deploy. Pair all of it with a check from outside the datacenter, so an edge failure cannot hide behind a green catalog. The free plan covers 10 monitors at a one-minute interval as of September 2026, so start free and cover the public name too.