Why container memory usage is doubled in cAdvisor metrics?

by

That’s because cAdvisor takes these values from cgroups. The structure of cgroups looks like a tree, where there are branches for each pod, and every pod has child cgroups for each container in it. This is how it looks (systemd-cgls):

├─kubepods
│ ├─podb0c98680-4c6d-4788-95ef-0ea8b43121d4
│ │ ├─799e2d3f0afe0e43d8657a245fe1e97edfdcdd00a10f8a57277d310a7ecf4364
│ │ │ └─5479 /bin/node_exporter --path.rootfs=/host --web.listen-address=0.0.0.0:9100
│ │ └─09ce1040f746fb497d5398ad0b2fabed1e4b55cde7fb30202373e26537ac750a
│ │   └─5054 /pause

The resource value for each cgroup is a cumulative for all its children. That’s how you got memory utilization doubled, you just summarized the total pod consumption with each container in it.

If you execute those queries in Prometheus, you would notice the duplicated values:

{pod="cluster-autoscaler-58b9c77456-krl5m"} 59076608
{container="POD",pod="cluster-autoscaler-58b9c77456-krl5m"} 708608
{container="cluster-autoscaler",pod="cluster-autoscaler-58b9c77456-krl5m"}  58368000

The first one is the parent cgroup. As you see, it has no container label. The two others in this example are the pause container and the actual application. Combining their values you will get the value of the parent cgroup:

>>> 708608 + 58368000 == 59076608
True

There are multiple ways to fix the problem. For example, you can exclude metrics without container name by using container!="" label filter.

Another (more difficult) way to solve this is to drop the cumulative metrics in metric_relabel_configs (prometheus.yml). I.e. you can write a relabeling rule that will drop metrics without a container name. Be careful with this one, you may accidentally drop all non-cadvisor metrics.

More Related Contents:

  1. Restart pods when configmap updates in Kubernetes?
  2. pod has unbound PersistentVolumeClaims
  3. How do I access the Kubernetes api from within a pod container?
  4. Expose port in minikube
  5. How kubectl port-forward works?
  6. chown: changing ownership of ‘/data/db’: Operation not permitted
  7. What’s the best way to share/mount one file into a pod?
  8. Mount local directory into pod in minikube
  9. Invalid x509 certificate for kubernetes master
  10. Can a PVC be bound to a specific PV?
  11. How to merge kubectl config file with ~/.kube/config?
  12. Connect to local database from inside minikube cluster
  13. Is it possible to Autoscale Akka
  14. Minikube expose MySQL running on localhost as service
  15. Allow scheduling of pods on Kubernetes master?
  16. Adding insecure registry in containerd
  17. Namespace “stuck” as Terminating, How I removed it
  18. no matches for kind “Deployment” in version “extensions/v1beta1”
  19. How to rolling restart pods without changing deployment yaml in kubernetes?
  20. How to schedule pods restart
  21. Restart container within pod
  22. What is a headless service, what does it do/accomplish, and what are some legitimate use cases for it?
  23. How to get Kubernetes cluster name from K8s API
  24. Ingress configuration for k8s in different namespaces
  25. How to access/expose kubernetes-dashboard service outside of a cluster?
  26. 413 error with Kubernetes and Nginx ingress controller
  27. Kubernetes PVC with ReadWriteMany on AWS
  28. Kubernetes Persistent Volume Access Modes: ReadWriteOnce vs ReadOnlyMany vs ReadWriteMany
  29. Listing all resources in a namespace
  30. Renew kubernetes pki after expired

Leave a Comment