buildx imagetools issue with gitlab registries

[JKrehling]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

I am getting an error in docker:29 images when doing

docker buildx imagetools create
unknown: Schema 2 manifest not supported by client

This does not happen in docker:28

Expected behaviour

I would expect to not get this schema 2 error.
It didn't happen for me with local docker so I don't know what exactly is going on with docker buildx that has been included in the docker base images.

Actual behaviour

Problem:

With the docker:28 based image I can do this

 docker run -it --entrypoint sh docker:28
docker buildx version
github.com/docker/buildx v0.29.1 a32761aeb3debd39be1eca514af3693af0db334b
/usr/local/bin/docker-entrypoint.sh sh

docker login registry.gitlab.com/registry/myregistry -u username
password

docker buildx imagetools create "docker.elastic.co/elasticsearch/elasticsearch:8.18.8" --tag registry.gitlab.com/registry/myregistry/elasticsearch:8.18.8

If I switch the image from docker:28 to docker:29

docker buildx version

/usr/local/bin/docker-entrypoint.sh sh

docker login registry.gitlab.com/registry/myregistry -u username
password

github.com/docker/buildx v0.30.1 9e66234aa13328a5e75b75aa5574e1ca6d6d9c01

docker buildx imagetools create "docker.elastic.co/elasticsearch/elasticsearch:8.18.8" --tag registry.gitlab.com/registry/myregistry/elasticsearch:8.18.8
[+] Building 2.3s (1/1) FINISHED                                                                                                                                                                                 
 => ERROR [internal] pushing registry.gitlab.com/registry/myregistry/elasticsearch:8.18.8                                                                                            2.3s
------                                                                                                                                                                                                           
 > [internal] pushing registry.gitlab.com/registry/myregistry/elasticsearch:8.18.8:                                                                                                       
0.000 copying sha256:24650e8f5783f04b1859c6e32d11ddbf5daa84e4ce491ccf92fac848f251f37a from docker.elastic.co/elasticsearch/elasticsearch:8.18.8 to registry.gitlab.com/registry/myregistry/elasticsearch:8.18.8                                                                                                                                                                                            
0.000 copying sha256:ce62ac650a4ebe653b0fb41f20cfd5f80a7ae188560ad709947b272d73150ba8 from docker.elastic.co/elasticsearch/elasticsearch:8.18.8 to registry.gitlab.com/registry/myregistry/elasticsearch:8.18.8
------
ERROR: unexpected status from GET request to https://docker.elastic.co/v2/elasticsearch/elasticsearch/manifests/sha256-d413f19aaeec97c924161c405f12a2c907552c640d7239766f05b17ea94382f1: 400 Bad Request
unknown: Schema 2 manifest not supported by client

Buildx version

github.com/docker/buildx v0.30.1 9e66234

Docker info

docker info
Client:
 Version:    29.1.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.30.1
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.0.1
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose

Builders list

docker buildx ls
NAME/NODE   DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*                      error

Configuration

docker run -it --entrypoint sh docker:29

its in the public image

Build logs

Additional info

No response

[thaJeztah]

Error comes from here in the registry code; https://github.com/distribution/distribution/blob/40594bd98e6d6ed993b5c6021c93fdf96d2e5851/registry/handlers/manifests.go#L202

Which checks for "Accept" headers being sent, so possibly the mediaType for schema 2 is missing; https://github.com/distribution/distribution/blob/40594bd98e6d6ed993b5c6021c93fdf96d2e5851/registry/handlers/manifests.go#L104

That said (I'd have to check the logic) ISTR some of the logic in that registry code was overly restrictive, and (e.g.) didn't detect wildcards.

[JKrehling]

I haven't seen any way of getting more debug logging out of this but if you know of anything that would help I can run it for sure.

[thaJeztah]

I gave it a VERY quick try to see if I could reproduce; the good thing is that it also reproduces with a local registry, which may be useful for debugging.

Run a local registry (I picked "v2" for now, as the v3 one is rather noisy with OTEL);

docker run -d --name registry -e REGISTRY_LOG_LEVEL=debug -p 5002:5000 registry:2

Try to push to the registry;

docker buildx imagetools create "docker.elastic.co/elasticsearch/elasticsearch:8.18.8" --tag localhost:5002/registry/myregistry/elasticsearch:8.18.8
[+] Building 43.7s (1/1) FINISHED
 => ERROR [internal] pushing localhost:5002/registry/myregistry/elasticsearch:8.18.8                           43.7s
------
 > [internal] pushing localhost:5002/registry/myregistry/elasticsearch:8.18.8:
0.000 copying sha256:24650e8f5783f04b1859c6e32d11ddbf5daa84e4ce491ccf92fac848f251f37a from docker.elastic.co/elasticsearch/elasticsearch:8.18.8 to localhost:5002/registry/myregistry/elasticsearch:8.18.8
0.001 copying sha256:ce62ac650a4ebe653b0fb41f20cfd5f80a7ae188560ad709947b272d73150ba8 from docker.elastic.co/elasticsearch/elasticsearch:8.18.8 to localhost:5002/registry/myregistry/elasticsearch:8.18.8
------
ERROR: unexpected status from GET request to https://docker.elastic.co/v2/elasticsearch/elasticsearch/manifests/sha256-d413f19aaeec97c924161c405f12a2c907552c640d7239766f05b17ea94382f1: 400 Bad Request
unknown: Schema 2 manifest not supported by client

Logs are attached, but I still need to dig through them to see if anything stands out clearly w.r.t. headers that are sent (not sure if they are printed in the logs).

docker logs registry

logs3.log

[thaJeztah]

Overall, I think one part of the issue is that the registry ignores */*, so if an exact content-type is missing, it assumes the client doesn't accept it. It looks like the containerd code does set some defaults if missing, which should include the "usual" media-types, but it also looks like it won't do so if an accept header is already present, so possibly something changed there;

buildx/vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go

Lines 161 to 173 in 44945d7

	resolveHeader := http.Header{}
	if _, ok := options.Headers["Accept"]; !ok {
	// set headers for all the types we support for resolution.
	resolveHeader.Set("Accept", strings.Join([]string{
	images.MediaTypeDockerSchema2Manifest,
	images.MediaTypeDockerSchema2ManifestList,
	ocispec.MediaTypeImageManifest,
	ocispec.MediaTypeImageIndex, "*/*",
	}, ", "))
	} else {
	resolveHeader["Accept"] = options.Headers["Accept"]
	delete(options.Headers, "Accept")
	}

edit: Also did a quick check with a container to debug headers / requests made;

docker run --rm -d --name debugger -p 127.0.0.1:5002:8080 mendhak/http-https-echo
docker buildx imagetools create "docker.elastic.co/elasticsearch/elasticsearch:8.18.8" --tag localhost:5002/registry/myregistry/elasticsearch:8.18.8
# [+] Building 1.6s (1/1) FINISHED
#  => [internal] pushing localhost:5002/registry/myregistry/elasticsearch                             1.6s
# INFO: registry did not send a Docker-Content-Digest header

Then checking logs;

docker logs debugger | grep 'accept":'
        "accept": "application/vnd.docker.container.image.v1+json, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.distribution.manifest.v2+json, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.container.image.v1+json, */*",
        "accept": "application/vnd.docker.image.rootfs.diff.tar.gzip, */*",
        "accept": "application/vnd.docker.distribution.manifest.v2+json, */*",
        "accept": "application/vnd.docker.distribution.manifest.list.v2+json, */*",