Linux Monitoring Tools: The Four Tiers, and What Each Sees

Linux monitoring tools fall into four tiers, and most roundups mix them up. A terminal process viewer, a per-host agent, a cluster-wide aggregator and an external prober answer different questions and are not substitutes for each other. Pick one from each tier you actually need, rather than the highest-rated thing on a list.

What do linux monitoring tools actually cover?

Each tier sees a different slice of the machine, and the slice determines what it can tell you during an outage. The table below is the shortest honest summary of the field.

Tier	Runs	Sees	Typical tools
Terminal	On demand, over SSH	This second, this box	top, htop, btop, iostat, journalctl
Agent	Continuously, on the box	History for one host	Netdata, node_exporter, Cockpit
Aggregator	Centrally, scraping agents	Every host at once	Prometheus, Zabbix, Checkmk, LibreNMS
External	Off your network	What a user gets	HTTP, TCP and ping probes

A tool from the top of that table cannot answer a question from the bottom. If the box is unreachable, nothing that runs on the box is going to tell you.

Tier one: the terminal

This is where you go once you already know something is wrong. htop gives you processes sorted by whatever is scarce, and btop is the same idea with better graphs. Neither keeps history, so their value is entirely in the sixty seconds after you log in.

The numbers worth knowing are further down. vmstat 1 shows the run queue and swap activity, iostat -x 1 shows per-device await and utilisation, and ss -s shows socket states without the ceremony of netstat. Red Hat's system monitoring tools chapter is still the clearest reference for what each one reports.

vmstat 1 5          # run queue, swap in/out, context switches
iostat -x 1 3       # per-device %util and await
journalctl -p err -b --no-pager | tail -20

Jeff Geerling's console roundup covers the rest of the family, including iotop and nvtop. We prefer keeping this tier small. Two commands you know cold beat six you have to look up while a service is down.

Tier two: the agent on the box

An agent turns the terminal snapshot into a time series. Netdata collects at one-second granularity and serves its own dashboard, which makes it the fastest way to get a graph out of a single server. Prometheus node_exporter collects nothing on its own but exposes a metrics endpoint for something else to scrape.

Cockpit sits between the two. It is a web console rather than a metrics pipeline, and it happens to show CPU, memory and disk while you are in there restarting a unit. That is a genuinely different use case from either of the above.

The trade-off at this tier is retention against footprint. A per-second agent storing locally is wonderful for a week and unhelpful for a quarter, which is what the next tier is for.

Tier three: the aggregator

Once you have more than about five hosts, you want one place that knows about all of them. Prometheus scrapes exporters and stores the series; Grafana draws them; Alertmanager routes the alerts. Zabbix and Checkmk bundle all three jobs into one product, which is why they keep winning sysadmin threads on machines that are not cloud-native.

Aggregator	Model	Best fit
Prometheus and Grafana	Pull, scrape exporters	Containers, ephemeral hosts
Zabbix	Agent plus templates	Mixed estates, network gear
Checkmk	Auto-discovery	Fast setup, low tuning appetite
LibreNMS	SNMP-first	Switches, routers, appliances

If you are moving off an older stack rather than starting fresh, the migration questions are covered in Nagios alternatives and Zabbix alternatives. Those posts are about leaving a product. This one is about assembling a set.

What all three tiers miss

Every tool above runs inside your network, and that is the blind spot. A kernel panic, a full disk on the monitoring host, a bad firewall rule or an expired edge certificate can each take the service down. In the first two cases the dashboards stop reporting. In the last two they stay green while nobody can reach you. The general form of this argument is in black box monitoring.

The fourth tier is an external probe. It checks the service the way a user reaches it, from somewhere your infrastructure does not control.

curl -X POST https://uptimeroom.com/api/v1/monitors \
  -H "Authorization: Bearer urm_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "name": "app edge",
    "type": "http",
    "target": "https://example.com/healthz",
    "intervalSec": 60,
    "regions": ["london", "sao-paulo"],
    "expectedStatus": [200]
  }'

Six probe regions are available as of September 2026. Host firewalls that drop unrecognised traffic will fail the check before it reaches your service. Each region's source addresses are published on the probe IPs page, so allowlist them first and save yourself debugging a check that was never let through.

Cron jobs and timers need a different shape

Backups, log rotation, certificate renewal and nightly imports fail by not running. No tool that polls a host can see an absence. That needs an inbound check instead: the job reports success, and a missing report is the alert.

0 3 * * * /usr/local/bin/backup.sh && \
  curl -fsS -m 10 https://uptimeroom.com/api/heartbeat/YOUR_TOKEN

The same line works from a systemd timer's ExecStartPost. We tested a deliberately broken unit by making the script exit non-zero, and because && short-circuits, no ping arrived and the monitor opened an incident after the grace period elapsed. The longer treatment is in heartbeat monitoring.

Which linux monitoring tools fit which job?

Start from the question you are trying to answer, not from the tool. If the question is "why is this box slow right now", tier one is enough and everything else is overhead. If it is "was it slow last Tuesday", you need an agent. If it is "which of my forty hosts is slow", you need an aggregator. If it is "did customers notice", you need the outside view.

Most small teams end up with htop plus one agent plus an external checker, and never install an aggregator at all. That is a reasonable place to stop.

Common mistakes

Monitoring the monitoring host with itself. A Prometheus server that has stopped scraping reports no failures. It also stopped scraping the rule that would have told you. Something off the box has to watch it, and a heartbeat from the scrape loop is the cheapest version.

Paging on CPU. High CPU on a batch host is the machine doing its job. Page on symptoms a user can feel, and graph the resource numbers for the investigation afterwards, which is the split described in golden signals.

Installing four agents and reading none of them. Every agent is a package to patch and a port to secure. An unread dashboard is a liability rather than coverage.

The useful way to choose between linux monitoring tools is to write down the four questions above and mark the ones you currently cannot answer. Most teams find the gap is the last one, because every tool they installed runs on the hardware they are worried about. Closing it costs one HTTP check and one heartbeat, which is two of the ten monitors a free account allows as of September 2026.