pkg.pr.new preview installs unnecesssary binaries for all OS and CPU

[notramo]

Originally reported at biomejs/biome#5645, but a Biome developer pointed out that it's a bug in pkg.pr.new

[Aslemammad]

hmm, i see, how does bun know which os to install for?

because we do not do anything different, we just upload and serve the tarball. is there any info required for a package manager to identify which specific files to download?

[notramo]

There are os and cpu fields in the package.json, see this: https://www.npmjs.com/package/@rollup/rollup-linux-x64-musl?activeTab=code

This mechanism works with the Rollup package, as the Bun package manager correctly installs only the packages meant for the OS and CPU. It's not possible to configure the libc, so both the musl and glibc versions are downloaded, but the OS and CPU are configurable.

[Aslemammad]

i still don't quite get what pkg.pr.new is not doing that needs to be done so this behaviour works well on our side.

[notramo]

Maybe it's not providing the necessary metadata to Bun. When I install the @biomejs/biome@beta release from NPM (bun add --dev @biomejs/biome@beta), the Bun package manager only downloads the relevant package, but when I install the preview version of the same package from pkg.pr.new, it downloads all OS and CPU combinations. I found that the os and cpu fields present in the preview releases too (inspecting inside node_modules/), so I guess that some package dependency resolution metadata is not exposed on the pkg.pr.new server.

[Aslemammad]

oh yea we do not do any metadata in pkg.pr.new, since it's just a tarball link.

[Aslemammad]

and i do not think we can do provide the metadate, i do not have that much information on that topic. do you know anyway we can do it?

[notramo]

The @biomejs/biome package from NPM is correctly installed, and only the relevant packages are downloaded. So, it should be possible in pkg.pr.new too, if there are no fundamental differences. Is pkg.pr.new fundamentally different from NPM?

[Aslemammad]

@AmirSa12 can you look into why this happens and what's different from our side that does not let this optimization happen?

[nojaf]

Do you know if this issue is on our side or specific to Bun? I'm experiencing the same problem. I opened an issue at oven-sh/bun#19415 a while ago.

[RiskyMH]

Yeah the problem is with npm registry you can get the metadata before actually downloading (https://registry.npmjs.com/@biomejs/cli-linux-x64 which includes versions.<version>.os) whereas pkg.pr.new works by just giving the tgz bundle (no registry api part).

Other package managers may check the package.json after downloading and save extra info to the lockfile, but Bun must be only respecting the proper api response.

[Aslemammad]

I think it's the metadata thing @RiskyMH, let me look if we can add it.

[RiskyMH]

I don't think you can do this without making an actual proper registry that would need to be added to .npmrc/bunfig.toml