Blackbox Monitoring: What the Outside View Sees

Blackbox monitoring watches a service from outside, with no access to its code, logs or internal counters. A prober sends the requests a real client would send, then judges the service purely on the reply it gets back. That narrow view is the point: it catches everything sitting between your server and your user.

Black box versus white box

The pair of terms comes from Google's SRE book, which splits monitoring by where the data originates. White box data comes from inside the system through instrumentation. Black box data comes from a probe that knows nothing except the address and the expected answer.

	Black box	White box
Vantage point	Outside the system	Inside the process
Typical signal	Status code, latency, certificate expiry	Queue depth, heap usage, error counters
Answers	Is it broken for users right now?	Why is it breaking?
Needs code changes	No	Usually yes
Sees DNS and routing faults	Yes	No

Neither one replaces the other. A team running only white box metrics will have green dashboards during a DNS outage, because the application is perfectly healthy and simply unreachable.

What does a probe actually send?

A probe speaks the real protocol rather than a summary of it. An HTTP probe opens a connection, completes the TLS handshake, sends a request and reads the response. Each stage produces a number, which is why an HTTP-family check reports a timing waterfall: DNS resolution, TCP connect, TLS handshake, time to first byte, then content download.

That breakdown is the reason the outside view earns its keep. When a page gets slow, the waterfall tells you whether the delay arrived in the resolver, the handshake or your own application code. Probes for TCP, ICMP, DNS and certificate expiry follow the same shape with a different payload.

Why blackbox monitoring runs from outside your network

Running the prober next to the thing it probes removes most of the value. A checker on the same host, in the same VPC or behind the same load balancer shares every dependency with its target. When the shared piece fails, both go quiet together and nobody gets paged.

In our testing the failures worth catching all lived in that shared layer. One was an expired intermediate certificate. One was a regional resolver handing back stale records. The third was a load balancer health check that kept passing while the origin returned 503 to real traffic. None of the three are visible from inside the box.

Where does the Prometheus blackbox exporter fit?

The blackbox exporter is the standard self-hosted prober, covering HTTP, HTTPS, DNS, TCP, ICMP and gRPC. You define modules in YAML, then point Prometheus at the exporter with the target as a parameter. A minimal module from its configuration reference looks like this:

modules:
  http_2xx_checkout:
    prober: http
    timeout: 5s
    http:
      valid_status_codes: [200]
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      method: GET
      fail_if_body_not_matches_regexp:
        - "\"status\"\\s*:\\s*\"ok\""
      preferred_ip_protocol: ip4

It works well and costs nothing in licence fees. The catch is the one above: you now run a prober, and something has to watch the prober. Most teams solve that with a hosted checker, a second exporter in another region, or both.

What can an outside view never tell you?

It cannot see saturation, and it cannot explain a failure. A probe reports that the request took 4 seconds; it has no idea the connection pool was exhausted. Queue depth, garbage collection pauses and per-endpoint error counters live inside the process and need instrumentation there.

It is also blind to anything a probe cannot reach. Internal cron jobs, batch pipelines and admin-only routes need a different pattern, usually an inbound heartbeat. The four golden signals map neatly onto this divide, with latency and errors on the outside and saturation firmly on the inside.

Turning the idea into running checks

The practical version is a small set of probes, each from more than one vantage point, with confirmation before anyone gets paged. Six probe regions are available as of September 2026, and the egress addresses are published on the probe IPs page so you can allowlist them. Requiring agreement from a second region is what separates a real incident from one slow network path, an idea covered in cross-region quorum confirmation.

Blackbox monitoring is the cheapest reliable answer to "is it up for users", and it stays honest because it has no privileged information. Pair it with in-process metrics for the why, and keep the probes far enough away that they fail independently. Ten monitors at a 1-minute interval are included on the free plan as of September 2026, which is enough to cover a small production surface.