MySQL Health Check: The Two Probes That Lie

Most of the trouble with a mysql health check comes from two probes that report success against a server you cannot actually use. One is documented behaviour in mysqladmin. The other is a temporary server the official Docker image runs during initialisation. Both produce a green container and a broken application.

Why does mysqladmin ping succeed with the wrong password?

Because the question it answers is narrower than the one you meant to ask. The mysqladmin reference states it outright. The exit status is 0 whenever the server is running, and an access denied error still counts as running. A server that refuses your credentials is not the same as a server that is down.

That distinction is correct for its purpose and wrong for yours. If your health probe uses ping, then a rotated password, a dropped grant or a wrong hostname in the connection string all read as healthy.

# exits 0, prints "mysqld is alive"
mysqladmin -h 127.0.0.1 -u app -pWRONG ping

# exits 1, which is the answer you wanted
mysql -h 127.0.0.1 -u app -pWRONG -e 'SELECT 1' appdb

Run a real query as the user your application uses, against the database your application uses. Anything less tests the socket rather than the service.

The temporary server that makes a first boot lie

The official image runs a temporary server while it executes the initialisation scripts. That lets seed SQL apply before the real server accepts traffic. On current versions the temporary instance listens on the local socket only, with networking disabled.

A probe that connects over the socket can therefore pass, let a dependent container start, and then watch the database disappear while initialisation finishes. This is discussed at length in docker-library/mysql issue 930, which is still the clearest write-up of the problem.

The fix is one flag. Force TCP with -h 127.0.0.1, which the temporary server does not answer, and a generous start_period so the wait does not exhaust your retries.

A Compose mysql health check that does not lie

Three requirements. Use TCP rather than the socket, run a real query, and give the check its own credentials rather than the application's.

services:
  db:
    image: mysql:8.4
    environment:
      MYSQL_DATABASE: appdb
      MYSQL_ROOT_PASSWORD_FILE: /run/secrets/mysql_root
    healthcheck:
      test: ["CMD-SHELL", "mysql -h 127.0.0.1 -u healthz -p\"$$HEALTHZ_PASSWORD\" -e 'SELECT 1' appdb"]
      interval: 10s
      timeout: 5s
      retries: 12
      start_period: 60s
  api:
    build: .
    depends_on:
      db:
        condition: service_healthy

The doubled $$ escapes the variable so Compose passes it through to the container rather than expanding it on your machine. Naming the database at the end matters too, since it forces the server to resolve the schema and surfaces a missing grant that a bare SELECT 1 would hide.

We tested the socket version of this against a cold volume. It reported healthy eighteen seconds in, the API container started, and the connection was refused four seconds later when the temporary server shut down.

A least privilege user for the check

The probe needs to connect and run one trivial statement. It does not need to read your customer table.

CREATE USER 'healthz'@'%' IDENTIFIED BY 'a-long-random-string';
GRANT USAGE ON *.* TO 'healthz'@'%';

-- only if the check names a database, which it should
GRANT SELECT ON appdb.* TO 'healthz'@'%';

USAGE is the no-privileges privilege. It permits a connection and lets SELECT 1 return, which is enough for liveness. Adding SELECT on the one schema lets the check name the database and prove the grant chain works, without exposing anything else.

Keep this user out of your application's credential rotation. A probe that fails at the same moment as the application tells you nothing extra.

The application route, and the flag most checks forget

A container probe and an application route fail differently, so run both. The route should borrow a connection from the pool the application actually uses, rather than opening its own, because pool exhaustion is invisible otherwise.

Then add the check that separates MySQL from other databases. After a failover, a promoted replica or a misconfigured node can sit with read_only or super_read_only enabled. It accepts connections, answers every SELECT, and rejects every write.

-- returns 0 when writes are accepted, 1 when they are not
SELECT @@read_only AS ro, @@super_read_only AS sro;

Return 200 with both values at zero on a node you expect to write to. Return 503 otherwise, and put the reason in the body. A 503 carrying {"mysql":"read_only"} saves the on-call engineer from rediscovering it at three in the morning.

Watching it from outside

Both layers so far run inside your network, so neither fires when the host, the security group or the region is the failure. Point an external check at the application route rather than at port 3306, because the route exercises credentials, the pool and the read-only flag.

curl -X POST https://uptimeroom.com/api/v1/monitors \
  -H "Authorization: Bearer urm_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "app readiness",
    "type": "api",
    "target": "https://example.com/readyz",
    "intervalSec": 60,
    "regions": ["london", "los-angeles"],
    "expectedStatus": [200],
    "keyword": "\"mysql\":\"ok\"",
    "keywordMatch": "contains",
    "alertAfterCycles": 2
  }'

If a tcp monitor on 3306 is genuinely needed, allowlist the ranges on the probe IPs page instead of opening the port to the internet. We prefer two regions with agreement required, so one bad network path never pages anyone. The same layering applies to Postgres and, with a different failure mode, to MongoDB.

What only a session inside MySQL can see

Replication lag is the big one. SHOW REPLICA STATUS reports Seconds_Behind_Source, and a replica an hour behind is an incident that every primary-side probe calls healthy.

The same is true of Threads_connected against max_connections, the InnoDB buffer pool hit rate, and any query that has been running for minutes. Keep those in a scheduled diagnostic job rather than in the probe, and have the job ping an inbound heartbeat so its own silent failure gets noticed.

Common mistakes

Treating mysqladmin ping as a mysql health check. It answers a question about the process, not about your access to it.

Connecting over the socket in a container probe. It matches the temporary initialisation server and produces a healthy container with no database behind it.

Running the check as root. A probe with full privileges is an unnecessary credential in an unnecessary place. USAGE plus one SELECT grant is enough.

Ignoring read_only after a failover. Reads succeed, writes fail, and a probe that only runs a SELECT reports everything as fine.

Letting the probe interval outrun max_connections. Every check consumes a connection. A ten-second container probe, an application route and several external regions all add up, and on a small instance under load that arithmetic starts to matter.

Putting an expensive query in the probe. A COUNT(*) over a large table turns your health endpoint into a load source every ten seconds. Keep it to SELECT 1 and the two read-only flags.

A mysql health check that holds up has three cheap pieces. Use a TCP-forced Compose probe with a least privilege user, an application route that checks the read-only flags, and one external check in front of both. That combination removes the two lies the default setup tells you. The free plan covers 10 monitors at a 1-minute interval as of September 2026, so add the outer layer free and move to the 30-second intervals when it matters.