docker driver loses BuildKit job for large bake target sets

[tg-freigmbh]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

bakes matrix build has very poor performance. If you create > 100 targets using a matrix bake will spend > 1 min in
=> => reading docker-bake.hcl 235B / 235B
This will ultimatively cause the build to fail with an error message like
ERROR: target 58-1: failed to receive status: rpc error: code = NotFound desc = no such job xtcwi97x2fguo5z68ffeg6v5a

I havent eveluated if the matrix feature is the culprit here or if this always happens if there is a nontrivial amount of targets.

Expected behaviour

Better performance, or the build should at least not fail

Actual behaviour

After about 3 minutes the build fails:
ERROR: target 58-1: failed to receive status: rpc error: code = NotFound desc = no such job xtcwi97x2fguo5z68ffeg6v5a

Buildx version

github.com/docker/buildx v0.33.0-desktop.1 7f91f038ac14cbf5c4b2a6b76470860814424da1

Docker info

Client:
 Version:    29.4.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  agent: Docker AI Agent Runner (Docker Inc.)
    Version:  v1.43.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-agent.exe
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.20.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-ai.exe
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.33.0-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v5.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.47
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.3.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-desktop.exe
  dhi: CLI for managing Docker Hardened Images (Docker Inc.)
    Version:  v0.0.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dhi.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.31
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  v0.40.3
    Path:     C:\Program Files\Docker\cli-plugins\docker-mcp.exe
  model: Docker Model Runner (Docker Inc.)
    Version:  v1.1.33
    Path:     C:\Program Files\Docker\cli-plugins\docker-model.exe
  offload: Docker Offload (Docker Inc.)
    Version:  v0.5.82
    Path:     C:\Program Files\Docker\cli-plugins\docker-offload.exe
  pass: Docker Pass Secrets Manager Plugin (beta) (Docker Inc.)
    Version:  v0.0.25
    Path:     C:\Program Files\Docker\cli-plugins\docker-pass.exe
  sandbox: Docker Sandbox (Docker Inc.)
    Version:  v0.12.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sandbox.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.20.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 58
  Running: 48
  Paused: 0
  Stopped: 10
 Images: 396
 Server Version: 29.4.0
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: docker.com/gpu=webgpu
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.87.2-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 31.28GiB
 Name: docker-desktop
 ID: 370dca8d-7ed4-4efd-8b7a-cb8b1d592524
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
  ::1/128
 Registry Mirrors:
  https://docker.freigmbh.de/
 Live Restore Enabled: false
 Default Address Pools:
   Base: 192.168.128.0/17, Size: 26
 Firewall Backend: iptables

Builders list

default             docker                                   
 \_ default          \_ default         running   v0.29.0    linux/amd64 (+3), linux/arm64, linux/arm (+2), linux/ppc64le, (2 more)
desktop-linux*      docker                                   
 \_ desktop-linux    \_ desktop-linux   running   v0.29.0    linux/amd64 (+3), linux/arm64, linux/arm (+2), linux/ppc64le, (2 more)

Configuration

target default {
  name= "${a}${b}-${c}"
  tags = ["${a}${b}:latest${c}"]
  dockerfile-inline = "from scratch"
  matrix ={
    a=[1,2,3,4,5,6,7,8,9,10]
 #   b= [1]
 #   c= [1]
    b=[1,2,3,4,5,6,7,8,9,10]
    c=[1,2,3]
  }
}

Build logs

Additional info

No response

[crazy-max]

That looks legit, but right now it overlaps pretty heavily with #3486. The interesting failure here is not really: "reading docker-bake.hcl took a long time", because that progress line is just the local bake definition loading step. The useful signal is the later:

failed to receive status: rpc error: code = NotFound desc = no such job ...

Can you please reduce this to a raw buildx bake repro and attach the exact outputs below? That will tell us whether this is a Bake matrix scaling bug, a BuildKit job lifecycle bug under load, or just a Desktop-specific failure mode.

Please try this minimal repro first.

target "default" {
  name = "${a}${b}-${c}"
  tags = ["${a}${b}:latest${c}"]
  dockerfile-inline = "FROM scratch"

  matrix = {
    a = [1,2,3,4,5,6,7,8,9,10]
    b = [1,2,3,4,5,6,7,8,9,10]
    c = [1,2,3]
  }
}

Then run these commands and paste the full output.

docker buildx version
docker version
docker info
docker buildx inspect --bootstrap
docker buildx bake --print
docker buildx -D bake --progress=plain

It would also help a lot if you bisect the matrix size and report the smallest failing case. For example, please try 10x10x1, 10x10x2, 10x10x3, and then smaller combinations until you find the first size that fails reliably.

If this only fails with many targets, please say whether it still fails when using a non-Desktop builder, for example a docker-container builder instead of the default docker driver.

If you want to rule out the default docker driver, please try the same repro with a separate docker-container builder.

Example:

docker buildx create --name repro-container --driver docker-container --use
docker buildx inspect --bootstrap
docker buildx bake --print
docker buildx -D bake --progress=plain

If you want to switch back afterwards:

docker buildx use default

If you want to remove the test builder afterwards:

docker buildx rm repro-container

Please include the output of:

docker buildx ls
docker buildx inspect --bootstrap

for both the default builder and the docker-container builder, and say clearly whether the failure reproduces in both cases or only with one of them.

Right now my guess is that this is the same class of issue as #3486, but we need the raw buildx bake repro and debug output to prove that.

[tg-freigmbh]

Files collected using the requested repro using default/desktop-linux driver

bootstrap.log
dversion.log
info.log
plain.log
print.log
version.log

[tg-freigmbh]

Files collected using the requested repro using docker-container driver
Build takes about 80s to start, but failure is not reproducible (works as expected, but slow)

bootstrap.log
ls.log
plain.log
print.log

[tg-freigmbh]

I tried switching back back to default to reavaluate:

Thorsten Arbeitsplat@DESKTOP-EQRL39B MINGW64 /g/development/bake-bug (master)
$ docker buildx use default
ERROR: run `docker context use default` to switch to default context

Is that expected?
After switching back using context use default: rpc failure is still reproducible

[tg-freigmbh]

So I converted the matrix to a an explicit file (no matrix feature):
Results are about the same: its slow but succeeds using the docker-container-driver.
Its slow and fails using the desktop driver.
So there are probably 2 separate tasks:

1. bake should be faster when there are many targets (in hcl files?)
2. the desktop driver should not fail if bake is a bit slower

From my experimentation:
100 targets work (but also slow),
200-300 targets do not work.

But might be down to individual cpu performance

docker-bake.explicit.txt

[crazy-max]

I tried switching back back to default to reavaluate:

Thorsten Arbeitsplat@DESKTOP-EQRL39B MINGW64 /g/development/bake-bug (master)
$ docker buildx use default
ERROR: run `docker context use default` to switch to default context

Is that expected? After switching back using context use default: rpc failure is still reproducible

Ah for this in your case as you're on WSL it would be:

docker buildx use desktop-linux

Files collected using the requested repro

This is much better, thanks. The latest repro narrows the issue a lot.

The key observations now are that the failure reproduces with raw buildx bake, it still reproduces after expanding the matrix into an explicit bake file, and it only fails with the default docker / Desktop path while docker-container is slow but succeeds.

That means this no longer looks matrix-specific. The matrix feature is just one way to generate a large number of targets. The correctness bug here appears to be that the docker driver loses the job under a larger Bake target set and returns:

failed to receive status: rpc error: code = NotFound desc = no such job ...

The rough threshold you found, around 100 works and 200-300 fails, is also useful.

So I would reframe this as a driver-specific correctness issue under larger Bake target counts, with a separate performance concern on top.

This sounds very close to the same class of failure as #3486, but the new repro here is cleaner because it isolates the default docker driver versus docker-container.

So I think the next step is to focus on why the docker driver path is dropping the BuildKit job for larger Bake target sets. I think the docker driver exposes an engine-side BuildKit bug/limit.

Build takes about 80s to start, but failure is not reproducible (works as expected, but slow)

For docker-container, tune BuildKit instead of using the embedded driver. BuildKit supports max-parallelism in daemon config. A typical config for the container driver is:

[worker.oci]
  max-parallelism = 16

Then create the builder with that config:

docker buildx create \
  --name large-bake \
  --driver docker-container \
  --buildkitd-config /path/to/buildkitd.toml \
  --use --bootstrap

If you want to you can create a default config file in buildx config (~/.docker/buildx/buildkitd.default.toml) so it's looked up when creating builders: https://docs.docker.com/reference/cli/docker/buildx/create/#buildkitd-config

That is a mitigation for throughput and resource pressure. It's not the root fix for the docker driver bug though.