Prometheus Blackbox Exporter: Setup and Blind Spots

The Prometheus blackbox exporter probes endpoints from the outside and turns the result into scrapeable metrics. It speaks HTTP, HTTPS, DNS, TCP, ICMP and gRPC, and it answers one question per probe: did this target respond correctly, and how long did each phase take. Getting it running takes ten minutes. Getting it to tell you something true takes a little longer.

What the Prometheus blackbox exporter measures

Unlike a normal exporter, this one does not expose metrics about itself. It takes a target as a query parameter, probes it, and returns metrics about that probe. The project README lists the full set; these are the ones worth alerting or graphing on.

Metric	What it gives you
probe_success	1 or 0, the only binary in the set
probe_duration_seconds	Total probe time
probe_http_status_code	The status the target returned
probe_http_duration_seconds{phase}	Per-phase timing: resolve, connect, tls, processing, transfer
probe_ssl_earliest_cert_expiry	Certificate expiry as a Unix timestamp
probe_dns_lookup_time_seconds	Resolution time alone

Most tutorials stop at the first row. The per-phase histogram is the one that pays for itself, because it separates "your application is slow" from "the TLS handshake is slow" without any guesswork.

A quick way to use it: graph probe_http_duration_seconds summed by phase for a single instance and watch which band grows. A widening tls band usually means an extra intermediate certificate or an OCSP fetch. A widening resolve band points at your resolver rather than at anything you deployed. A processing band that grows on its own is the only one that is really your application's fault.

Certificate expiry deserves its own treatment too. probe_ssl_earliest_cert_expiry is an absolute timestamp, not a countdown, so every useful query subtracts time() from it and divides by 86400 to get days. Graph that number and you get a sawtooth that should reset every 60 days under automated renewal. A sawtooth that stops resetting is a renewal job that died quietly weeks ago.

Running it

Start the container on its default port and give it a config file.

docker run -d --name blackbox-exporter \
  -p 9115:9115 \
  -v "$PWD/blackbox.yml:/config/blackbox.yml" \
  prom/blackbox-exporter:latest \
  --config.file=/config/blackbox.yml

A minimal blackbox.yml defines two modules. One does a plain 2xx check, the other asserts on the response body.

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      preferred_ip_protocol: ip4
  http_health_json:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      fail_if_body_not_matches_regexp:
        - '"status"\s*:\s*"ok"'

That second module matters more than it looks. A 200 response carrying an error payload passes http_2xx happily, and every option is documented in CONFIGURATION.md.

Wiring it into Prometheus

This is where people get stuck. Prometheus does not scrape the target directly. It scrapes the exporter and passes the target along as a parameter. That indirection needs the relabel dance below.

scrape_configs:
  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com
          - https://api.example.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Read those three rules top to bottom. The target becomes the target parameter, that parameter becomes the instance label so your graphs are readable, and only then is the real scrape address rewritten to the exporter. The multi-target exporter guide walks through the same pattern in more depth, and the exporter serves a small debug page on port 9115 showing the last few probes.

Two alert rules cover most of the value:

groups:
  - name: blackbox
    rules:
      - alert: EndpointDown
        expr: probe_success == 0
        for: 2m
      - alert: CertExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
        for: 1h

The for: 2m is doing real work. Without it a single dropped packet pages someone.

Why does the Prometheus blackbox exporter miss user-facing outages?

Because it runs where you run it, and that is almost always next to the thing it checks. An exporter in the same cluster shares the node pool, the service mesh, the internal resolver and the egress path with its target. It is testing a network path that no customer ever uses.

The failure mode is worse than inaccurate. If the cluster loses its load balancer, or the region's networking degrades, the exporter goes down with the service and the alert never fires. In our testing that is the single most common gap in an otherwise well-built setup: the graphs look perfect right up to the moment they stop being collected at all.

There are three honest fixes, in increasing order of effort. Run the exporter in a different cluster from the target. Run several copies in different regions and federate or remote-write them into one Prometheus. Or keep this setup for internal targets and put an independent vantage point in front of the user-facing ones.

Adding an outside vantage point

The second and third options overlap more than they look, because the hard part is not the probe. It is running probes in several regions, agreeing on what counts as down, and keeping the alerting path off the infrastructure being measured.

That is the trade-off to weigh. Self-hosting gives you full control of the module config and keeps the data in your own time series database. A hosted check gives you regions you do not operate and an alert path that survives your own outage. We prefer running both, with the exporter watching internal dependencies and an external check on the handful of endpoints customers actually touch.

curl -X POST https://uptimeroom.com/api/v1/monitors \
  -H "Authorization: Bearer urm_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "public api",
    "type": "keyword",
    "target": "https://api.example.com/health",
    "intervalSec": 60,
    "regions": ["london", "singapore"],
    "keyword": "\"status\":\"ok\"",
    "expectedStatus": [200],
    "alertAfterCycles": 2
  }'

Six probe regions are available as of September 2026, with confirmation from a second region before anything is declared down. That is the same idea as for: 2m, applied across geography instead of time, and it is covered in cross-region quorum confirmation. If your endpoint sits behind an allowlist, the egress addresses are published on the probe IPs page.

Common mistakes

Probing from inside the thing you are probing. Covered above, and it is the one that actually bites during an incident.

Alerting on probe_duration_seconds with a fixed threshold. Use the per-phase metric and a percentile instead, for the reasons laid out in response time monitoring.

Leaving preferred_ip_protocol at the default. The exporter tries IPv6 first. If your target has an AAAA record pointing somewhere unmaintained, you will chase a phantom outage for an afternoon.

Ignoring certificate expiry until it fires. A 14-day window is the minimum useful lead time, and 30 days is better if a human has to file a ticket to get the renewal done.

The Prometheus blackbox exporter is the right tool for probing internal services from inside your network, and the per-phase timings make it genuinely good at that job. Its blind spot is geography and shared fate, not features. Pair it with checks that run somewhere your own cluster cannot take down. Size the targets against the objectives in your SLA and SLO work, and you can have that second vantage point running in a few minutes on the free plan.