Kubernetes Monitoring Tools: Four Questions, Three Sources
Kubernetes monitoring tools split by which question they answer, not by rank. Here is what metrics-server, cAdvisor and kube-state-metrics each see.
Most kubernetes monitoring tools answer one of four questions, and the ranked lists rarely say which. A cluster needs to know whether its objects match what you declared and what its containers are consuming. It also needs to know whether the control plane is answering and whether anyone outside can reach the service. Four questions, three data sources, and a lot of half-empty dashboards.
What a cluster actually needs answered
Start from the question rather than the product. Each of the four has a different source of truth, and a tool that is excellent at one is usually blind to another. That blindness is rarely documented on the tool's own page, which is why comparing kubernetes monitoring tools feature by feature goes nowhere.
| Question | Where the answer lives | What usually provides it |
|---|---|---|
| Is the object in the state I declared? | the API server's object state | kube-state-metrics |
| What is this container consuming? | the kubelet on each node | cAdvisor, scraped by Prometheus or an agent |
| Is the control plane answering? | the API server endpoint | your provider's console, or a check from outside |
| Can a user reach the service? | outside the cluster | an external probe |
Rows one and two are covered exhaustively by every shortlist you will read. Rows three and four are where clusters actually fail quietly, and they are covered by almost nobody.
The three data sources everyone conflates
This is the part worth getting right, because installing the wrong component is how a dashboard ends up half empty. Three things emit numbers in a cluster and they do not overlap as much as their names suggest.
metrics-server fetches resource metrics from the kubelets and exposes them through the Metrics API. The resource metrics pipeline documentation is explicit that this API "only offers the minimum CPU and memory metrics to enable automatic scaling using HPA and / or VPA". It keeps no history and it is not a monitoring backend.
cAdvisor is bundled into the kubelet and exposes per-container counters on the /metrics/resource and /stats endpoints. This is what Prometheus and most vendor agents actually scrape, and it is where your CPU and memory series come from.
kube-state-metrics listens to the API server and generates metrics about the state of objects. Its README is clear that it "is not focused on the health of the individual Kubernetes components". The focus is the objects inside instead, such as deployments, nodes and pods. It emits no resource usage whatsoever.
| Component | Emits | Answers | History |
|---|---|---|---|
| metrics-server | current CPU and memory per pod and node | will the autoscaler act | none, current value only |
| kubelet and cAdvisor | per-container resource counters | what is this container using | whatever your database keeps |
| kube-state-metrics | desired versus ready replicas, phases, reasons | did the rollout finish | whatever your database keeps |
Why is kubectl top empty when Grafana looks fine?
Because they read different pipelines. kubectl top calls the Metrics API, which only exists if metrics-server is installed and its API service is available. Prometheus scrapes the kubelet directly and never touches that API, so a cluster can have complete dashboards and a completely broken kubectl top.
kubectl get apiservice v1beta1.metrics.k8s.io
kubectl -n kube-system get deploy metrics-server
On self-managed clusters the usual cause is certificates. The kubelet's serving certificate is not signed by the cluster CA, metrics-server refuses the connection, and the API service reports as not available. The common fix is to run metrics-server with --kubelet-insecure-tls, which works and is worth recording somewhere, because it is a security decision rather than a typo.
Events are the signal the graphs never show
CrashLoopBackOff, OOMKilled, FailedScheduling, ImagePullBackOff and Evicted are not metrics. They are object state and events, and no amount of CPU graphing surfaces them. A pod that cannot be scheduled draws a flat line at zero, which looks exactly like a pod that is idle.
kubectl get events --sort-by=.lastTimestamp -A | tail -20
kubectl get pods -A --field-selector=status.phase!=Running
Events also expire. The kube-apiserver flag --event-ttl defaults to one hour, so an event nobody scraped has simply gone by the time you look. That is the strongest practical argument for kube-state-metrics. The series kube_pod_container_status_last_terminated_reason keeps the OOMKill at your retention rather than the cluster's.
This is also where most kubernetes monitoring tools quietly differ. Some scrape events, some scrape only the metrics endpoints, and the difference does not show up until you go looking for last night's evictions and find nothing.
What a managed control plane will not tell you
On EKS, GKE and AKS the control plane belongs to the provider. There is no node to log into, no etcd to scrape, and scheduler internals are not yours. You get whatever the provider exposes through its own logging service, and that is usually switched off until someone turns it on after an incident.
There is one check that works on every cluster regardless. Request the API server endpoint from outside and see whether it answers. No agent, no permissions, no install. If that endpoint is unreachable then nothing inside the cluster can help you, and nothing inside the cluster is going to tell you either. The same reasoning applied to servers and networks generally is worked through in our look at how infrastructure data gets collected.
Four categories of kubernetes monitoring tools
| Category | Typical stack | Where it runs | Blind spot |
|---|---|---|---|
| In-cluster open source | Prometheus, kube-state-metrics, Grafana, Alertmanager | in the cluster | shares fate with the cluster |
| Vendor agent | a DaemonSet shipping data to a SaaS backend | agent inside, data outside | billing tends to track pod churn |
| Provider native | Container Insights, Cloud Monitoring, Azure Monitor | the provider's side | stops at the provider's boundary |
| External probe | scheduled checks against the ingress hostname | entirely outside | no view of pod internals |
We prefer the unglamorous combination of rows one and four. Prometheus with kube-state-metrics gives depth that nothing outside can match, and one external check gives you something that keeps working when the cluster does not. The probe design behind that second half is covered in our Kubernetes health check walkthrough.
When the alert path lives inside the cluster
Alertmanager is a pod. So is Prometheus, so is Grafana, and so is whatever ingress controller carries the webhook out to Slack. An alert path assembled entirely from workloads on the cluster cannot report on the cluster's own node pool, ingress or DNS failing.
We tested this by scaling an ingress controller to zero on a staging cluster. Every in-cluster dashboard stayed green, because every pod really was healthy and nothing had restarted. Nothing outside could reach the service for eleven minutes, and no alert fired from inside.
curl -X POST https://uptimeroom.com/api/v1/monitors \
-H "Authorization: Bearer urm_live_..." \
-H "Content-Type: application/json" \
-d '{"name":"cluster ingress","type":"http","target":"https://app.example.com/healthz","intervalSec":60,"regions":["singapore","london"],"alertAfterCycles":2}'
Two regions and two failed cycles keeps a single bad network path from paging anyone, and the argument for that rule is set out in cross-region quorum confirmation. Six probe regions are available as of September 2026, with every egress address published on the probe IPs page for clusters that filter inbound traffic.
Common mistakes
Installing kube-state-metrics and expecting kubectl top to work. They are separate pipelines. You need metrics-server for the Metrics API and kube-state-metrics for object state, and most clusters want both.
Alerting on pod restarts by themselves. A rolling Deployment is a wave of restarts. Alert on the termination reason and on ready replicas falling below the desired count instead.
Running the only alert path inside the cluster it watches. At minimum, one check from outside should be able to raise a human when everything in there is unreachable.
The useful way to compare kubernetes monitoring tools is by which of the four questions each one answers, not by position in somebody's ranked list. Take kube-state-metrics for object state, the kubelet for resource use, your provider for the control plane, and something outside for reachability. That last piece needs one monitor and a hostname, which fits comfortably in the free tier as of September 2026.
FAQ
There is no single one, because four different questions need four different sources. kube-state-metrics answers object state, the kubelet's cAdvisor answers container resource use, your cloud provider answers for a managed control plane, and only a probe from outside answers whether a user can reach the service. Most clusters end up running at least two of these.
metrics-server pulls CPU and memory from the kubelets and exposes them through the Metrics API for autoscaling and kubectl top, keeping no history. kube-state-metrics listens to the API server and generates metrics about object state, such as desired versus ready replicas. It emits no resource usage, so the two are complements rather than alternatives.
Because metrics-server is missing or its APIService is not ready. Check with kubectl get apiservice v1beta1.metrics.k8s.io. On self-managed clusters the usual cause is a kubelet serving certificate that the cluster CA did not sign, which makes metrics-server refuse the connection. Prometheus is unaffected, since it scrapes the kubelet directly.
Not with resource graphs. Those are object state and events, so a pod that cannot be scheduled looks identical to an idle one on a CPU chart. Kubernetes events expire after the kube-apiserver --event-ttl, which defaults to one hour, so scrape kube_pod_container_status_last_terminated_reason from kube-state-metrics to keep the history.
Not if Prometheus and Alertmanager are pods on that cluster. When the node pool, ingress controller or cluster DNS fails, the alert path fails with it, and every dashboard stays green because the pods really are healthy. One scheduled check from outside the cluster covers that case and needs nothing installed inside it.
Related reading
- Kubernetes Health Check: Probes, Timing, Blind Spots - A kubernetes health check has three forms and three consequences. Here are the handlers, the timing arithmetic, and the restart storm to avoid.
- Infrastructure Monitoring: The Three Collection Methods - Infrastructure monitoring is defined by how data is collected, not which metrics you chart. Here is which layer each method reaches and which it cannot.
- Cross-Region Quorum: How Confirmation Stops False Alerts - Cross-region quorum means a target is only called down once probes in more than one region agree. Here is the voting rule, its cost and its limits.