How can I ‘join’ two metrics in a Prometheus query?

You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity): ( max(consul_health_service_status{status=”critical”}) by (service_name,status,node) == 1 ) + on(service_name,node) group_left(env) ( 0 * consul_service_tags ) The important part here is the operation + on(service_name,node) group_left(env): the + is “abused” as a join operator (fine … Read more

Why container memory usage is doubled in cAdvisor metrics?

That’s because cAdvisor takes these values from cgroups. The structure of cgroups looks like a tree, where there are branches for each pod, and every pod has child cgroups for each container in it. This is how it looks (systemd-cgls): ├─kubepods │ ├─podb0c98680-4c6d-4788-95ef-0ea8b43121d4 │ │ ├─799e2d3f0afe0e43d8657a245fe1e97edfdcdd00a10f8a57277d310a7ecf4364 │ │ │ └─5479 /bin/node_exporter –path.rootfs=/host –web.listen-address=0.0.0.0:9100 │ │ … Read more