support disk usage metrics for containerd factory by chengjoey · Pull Request #3502 · google/cadvisor

chengjoey · 2024-03-19T15:07:56Z

query rootfs dir from contianerd status and get disk usage stat by rootfs dir /run/containerd/io.containerd.runtime.v2.task/${contianerid}/rootfs

set hasFileSystem to true for containerd when included disk usage metrics
added common fs handler for container rootfs

k8s-ci-robot · 2024-03-19T15:08:07Z

Hi @chengjoey. Thanks for your PR.

I'm waiting for a google member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

chengjoey · 2024-03-19T15:09:17Z

container_fs_usage_bytes metrics miss pod/container label when useing containerd

iwankgb · 2024-03-31T17:31:40Z


 	var fsStats info.FsStats
-	addDiskStats(fileSystems, nil, &fsStats)
+	fs.AddDiskStats(fileSystems, nil, &fsStats)


This test should be moved to fs module.

iwankgb · 2024-03-31T17:31:46Z


 	// Act
-	addDiskStats(fileSystems, &fsInfo, &fsStats)
+	fs.AddDiskStats(fileSystems, &fsInfo, &fsStats)


iwankgb · 2024-03-31T17:34:40Z

+	}
+	configStr := r.Info["config"]
+	config := make(map[string]interface{})
+	if err := json.Unmarshal([]byte(configStr), &config); err != nil {


Could you unmarshal config into a struct? It will be easier to grasp what values are interesting.

The original definition of config is here

I extracted the struct in config.go to container/containerd/config

iwankgb · 2024-03-31T17:35:34Z

/ok-to-test

chengjoey · 2024-04-02T14:59:58Z

/retest

silence8013 · 2024-05-06T02:47:06Z

Will this commit continue? Looking forward to it~

duanhongyi · 2024-05-11T10:27:38Z

I am looking forward to this feature merging.

jianxiaoguo · 2024-05-11T13:51:29Z

I eagerly anticipate the completion of this feature.

EamonZhang · 2024-05-13T01:32:20Z

This feature is particularly important and I'm looking forward to it

wsszh · 2024-05-24T11:07:07Z

Hi, I think the disk usage of a container should include the usage of log files in /var/log/pods.

chengjoey · 2024-05-25T06:28:12Z

Hi, I think the disk usage of a container should include the usage of log files in /var/log/pods.

hi @wsszh , The disk usage calculation of contianerd is similar to that of docker and podman. I think that several container factories currently do not include /var/log/pods in the calculation. This may be because cadvisor is more about container-level calculations rather than pods. It is also difficult to know which directory should be calculated because it is not fixed in /var/log/pods. although I think it should be calculated, the specific directory of the log cannot be obtained from the info interface of cri.

chengjoey · 2024-05-27T13:24:15Z

Hi, I think the disk usage of a container should include the usage of log files in /var/log/pods.

if a container should include the usage of log files, I would like to submit another PR to modify the container runtimes such as podman, docker, containerd, etc. This PR only does usage metric for containerd

kxbin · 2024-12-19T07:33:06Z

great work，Hope to speed up to merge in master

melikeiremguler · 2024-12-30T10:27:42Z

Hi, will this pull request solve this problem ?

chengjoey · 2024-12-30T15:09:54Z

Hi, will this pull request solve this problem ?

@melikeiremguler , Yes, this PR is to solve the issue of 2785.

gigilee · 2025-03-24T20:21:36Z

Is this PR going to be merged?

Imlosing · 2025-04-24T08:09:50Z

Looking forward to the early merge.

Imlosing · 2025-04-24T08:10:50Z

Looking forward to the early merge.

Or is there any alternative solution now?

kam1kaze · 2025-05-05T07:25:22Z

@iwankgb How could we speed up merge process of this code? We are using it on production and it works perfect.

Imlosing · 2025-05-13T06:34:30Z

@iwankgb How could we speed up merge process of this code? We are using it on production and it works perfect.

@kam1kaze I also merged this PR and tried to use it, but I found that the actual value collected was greater than the writable layer capacity. Have you encountered this problem?

kamikaze · 2025-07-29T14:58:35Z

please

desert3agle · 2025-11-01T19:00:00Z

is it still in work or has been abandoned?

ttc0419 · 2025-11-11T07:05:12Z

@iwankgb could you take a look again?

till · 2025-12-14T13:28:51Z

@dims maybe you can help review this? And re-open the ticket

dims · 2025-12-14T14:54:54Z

@till if/when it gets rebased/updated the PR author can reopen this PR

query rootfs dir from contianerd status and get disk usage stat by rootfs dir `/run/containerd/io.containerd.runtime.v2.task/${contianerid}/rootfs` Signed-off-by: joey <zchengjoey@gmail.com>

dims · 2025-12-15T02:34:39Z

 # Since github.com/google/cadvisor/cmd is a submodule, we must build from inside that directory
 pushd cmd > /dev/null
-  go build ${GO_FLAGS} -ldflags "${ldflags}" -o "${output_file}" "${repo_path}/cmd"
+  go mod tidy && go build ${GO_FLAGS} -ldflags "${ldflags}" -o "${output_file}" "${repo_path}/cmd"


please remove the extra go mod tidy

dims · 2025-12-15T02:36:45Z

 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gotest.tools/v3 v3.0.2 h1:kG1BFyqVHuQoVQiR1bWGnfz/fmHvvuiSPIV7rvl360E=
 gotest.tools/v3 v3.0.2/go.mod h1:3SzNCllyD9/Y+b5r9JIKQ474KzkZyqLqEfYqMsX94Bk=
+k8s.io/cri-api v0.34.3 h1:zFdQSHZuQlQXesw9ncjQRUyDpvLng/84Q4qLKd8x2zE=


hmm, we really should not be creating circular dependencies!

This circular dependency refers to the fact that k8s(kubelet) already depends on cAdvisor, so should cAdvisor not depend on the packages in the k8s repository?

correct! it took us a few years to straighten out the dependencies.

dims · 2025-12-15T02:37:23Z

 }

+func (c *client) RootfsDir(ctx context.Context) (string, error) {
+	r, err := c.runtimeService.Status(ctx, &runtimeapi.StatusRequest{Verbose: true})


please find another way. we can't be creating circular dependencies

github-actions · 2026-03-16T05:06:59Z

This PR is stale because it has been open 90 days with no activity. This PR will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

mindw · 2026-04-05T09:05:40Z

still relevant.

k8s-ci-robot added the needs-ok-to-test label Mar 19, 2024

This was referenced Mar 19, 2024

[containerd-cri]Set HasFilesystem to true #3253

Closed

container_fs_usage_bytes metrics miss pod/container label #3495

Closed

iwankgb requested changes Mar 31, 2024

View reviewed changes

k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels Mar 31, 2024

chengjoey force-pushed the feat/containerd-disk-usage branch from 68c7bbf to 5f1986c Compare April 1, 2024 02:37

chengjoey force-pushed the feat/containerd-disk-usage branch 7 times, most recently from 1b85ff8 to 9e7a939 Compare August 14, 2024 15:13

Imlosing mentioned this pull request May 28, 2025

[BUG] #3502: The container_fs_usage_bytes metric is too large, including the size of the container's read-only layer #3691

Open

elfkuzco mentioned this pull request Oct 28, 2025

Measure and report all tasks resources usage openzim/zimfarm#1423

Closed

george-angel mentioned this pull request Dec 11, 2025

Disk usage metrics for containerd #2785

Closed

support disk usage metrics for containerd factory

578c27f

query rootfs dir from contianerd status and get disk usage stat by rootfs dir `/run/containerd/io.containerd.runtime.v2.task/${contianerid}/rootfs` Signed-off-by: joey <zchengjoey@gmail.com>

chengjoey force-pushed the feat/containerd-disk-usage branch from 9e7a939 to 578c27f Compare December 15, 2025 02:30

dims reviewed Dec 15, 2025

View reviewed changes

ttc0419 mentioned this pull request Dec 15, 2025

Support pod/container level container_fs_usage_bytes #3777

Open

github-actions Bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 16, 2026

github-actions Bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 6, 2026

Conversation

chengjoey commented Mar 19, 2024

Uh oh!

k8s-ci-robot commented Mar 19, 2024

Uh oh!

chengjoey commented Mar 19, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iwankgb commented Mar 31, 2024

Uh oh!

chengjoey commented Apr 2, 2024

Uh oh!

silence8013 commented May 6, 2024

Uh oh!

duanhongyi commented May 11, 2024

Uh oh!

jianxiaoguo commented May 11, 2024

Uh oh!

EamonZhang commented May 13, 2024

Uh oh!

wsszh commented May 24, 2024

Uh oh!

chengjoey commented May 25, 2024

Uh oh!

chengjoey commented May 27, 2024

Uh oh!

kxbin commented Dec 19, 2024

Uh oh!

melikeiremguler commented Dec 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chengjoey commented Dec 30, 2024

Uh oh!

gigilee commented Mar 24, 2025

Uh oh!

Imlosing commented Apr 24, 2025

Uh oh!

Imlosing commented Apr 24, 2025

Uh oh!

kam1kaze commented May 5, 2025

Uh oh!

Imlosing commented May 13, 2025

Uh oh!

kamikaze commented Jul 29, 2025

Uh oh!

desert3agle commented Nov 1, 2025

Uh oh!

ttc0419 commented Nov 11, 2025

Uh oh!

till commented Dec 14, 2025

Uh oh!

dims commented Dec 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chengjoey Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Mar 16, 2026

Uh oh!

mindw commented Apr 5, 2026

Uh oh!

melikeiremguler commented Dec 30, 2024 •

edited

Loading

chengjoey Dec 15, 2025 •

edited

Loading