Add dune pkg binary release workflow

[gridbugs]

When a tag is pushed, a github action will build the project using dune package management and release the binaries under the tag on github.

[avsm]

This is very exciting to see! I left some questions on the details of the GH action, and here are some high level workflow questions that it would be useful to document:

• How do we maintain the dune.lock dir? Is there a way to refresh a particular package or all of them?
• How does this interact with opam (i.e. our existing development workflow). Will the existence of a dune.lock interface with opam switches if dune is invoked from within one?
• How do we do development that ignore the lock files (e.g. to test newer packages that are in opam-repo?)
• How do we stash stuff in the GH actions cache so the compiler isnt built every time; is this something that setup-dune does?
• Does dune-release still work as before to create distribution tarballs to release new versions of this to opam-repository?

Looking forward to trying this out in more detail once I understand the intended workflow!

[avsm]

What is this environment variable for? What happens if we remove it?

[gridbugs]

It just prints a log of packages being downloaded/built. I put it here to get early feedback from the github action since by default dune doesn't print very much. Now that the action is working it can be removed.

[avsm]

Can this just use actions/upload-artifact@v4 instead? See https://github.com/actions/upload-artifact?tab=readme-ov-file#inputs ins

[gridbugs]

It depends on what sort of release you're doing. I'm trying to demonstrate releasing an archive with pre-built binaries, including making it available on the project's "Releases" page. But if you just want the archive to be uploaded as a workflow artifact and made available to other workflows, then actions/upload-artifact can be used instead. Let me know if you'd like me to change this workflow to the latter.

[avsm]

That makes sense; the upload-artifact is actually quite useful as it can run on PRs as well, and let branched-binaries-with-a-new-feature be quickly downloadable to try out. Not directly applicable here as this workflow on runs on push, but this could be generalised to work on pull_request too.

[patricoferris]

Thanks @gridbugs, I'm sure there's a lot of unseen effort here :))

How do we do development that ignore the lock files (e.g. to test newer packages that are in opam-repo?)

I was also wondering that, in particular also with pin-depends and different OCaml compiler versions ?

[gridbugs]

How do we maintain the dune.lock dir? Is there a way to refresh a particular package or all of them?

At the moment it's only possible to update all the packages (by renunning dune pkg lock). However since the lockdir contains a separate file for each package in the solution, it should be pretty easy to use git to get more fine grained control over which dependencies are updated (ie. just commit the new lockfiles for the packages you want to update).

How does this interact with opam (i.e. our existing development workflow). Will the existence of a dune.lock interface with opam switches if dune is invoked from within one?

If a lockdir is present, commands like dune build will use the lockdir instead of an opam switch to resolve dependencies, even if dune was run from an opam switch (opam is still the recommended way of installing dune after all, so dune will almost always be invoked from an opam switch).

How do we do development that ignore the lock files (e.g. to test newer packages that are in opam-repo?)

Pass the flag --ignore-lock-dir to commands like dune build to force dune to ignore the lockdir and resolve dependencies within the opam switch.

How do we stash stuff in the GH actions cache so the compiler isnt built every time; is this something that setup-dune does?

setup-dune just installs dune, but not the compiler. Dune installs the compiler as a mostly regular opam package. Caching the directories _build/_private/default/.pkg and ~/.cache/dune/toolchains keyed with something like hashFiles("dune.lock") may work, but these directories are implementation details and so subject to change and github caching isn't a use case that's officially supported yet by dune pkg. I'm curious if this approach will actually work in practice so I'm trying it out, and I'll let you know what I learn.

Does dune-release still work as before to create distribution tarballs to release new versions of this to opam-repository?

Yep, dune-release works as before. When dune generates opam files for packages it always builds with dune build -p <package> ..., which ignores the lockdir, so the presence of a lockdir won't affect the packaging of a package for opam.

[gridbugs]

A correction to my suggestion for caching artifacts in github's cache: You need to cache the directories _build and ~/.cache/dune rather than the directories I suggested originally. This way the entire state of dune's cache ends up in github's cache. This will mean that each time the lockdir changes, dependencies (including the compiler) are built from scratch, but subsequent builds with the same lockdir will start from a state where all dependencies are already built. I'll update the github action in this PR with an example of my proposed caching strategy.

[gridbugs]

I've updated this PR to include an example of using actions/cache to cache dependencies, avoiding the need to build the compiler on every run. This has involved adding an additional workflow that runs on pushes to the default branch to populate the cache, since workflows running on tags can only access artifacts cached from workflows running on the default branch.

[gridbugs]

As requested in https://github.com/tarides/in-and-out/issues/25, I've added an example of producing statically-linked executables on linux. I've validated the statically-linked exe by running it on nixos and a stock alpine install where exes dynamically linked against glibc do not work.

[avsm]

Just a quick comment on this: (will try this out properly later, am on the road right now!)

@gridbugs wrote:

If a lockdir is present, commands like dune build will use the lockdir instead of an opam switch to resolve dependencies, even if dune was run from an opam switch (opam is still the recommended way of installing dune after all, so dune will almost always be invoked from an opam switch).

This really does need an option to disable the lockfile usage. The problem here is that we're blocking the existing dev workflow by checking in lockfiles, and none of the old opam based scripts will work any more. In practise, this will mean I'll need to rm the lockdir for local development, which will leave a giant diff in my working trees.

I'm not sure I have an actual answer to this; I just wanted to observe that a smoother transition would be useful before I can check in lockfiles to an existing project. I'm fine with that option being off by default, of course (or hidden behind a dune-workspace option so it could be a profile?)

[avsm]

I just saw the --ignore-lock-dir comment. Would it be possible to have this part of a dune-workspace spec, so we can switch profiles and have this work along with other options like warnings flags and so on? It's not a big deal if not possible, but would be neater from a dev workflow perspective (same CLI, different DUNE_PROFILE env variables)

[gridbugs]

I just saw the --ignore-lock-dir comment. Would it be possible to have this part of a dune-workspace spec, so we can switch profiles and have this work along with other options like warnings flags and so on? It's not a big deal if not possible, but would be neater from a dev workflow perspective (same CLI, different DUNE_PROFILE env variables)

I think there's a way to make that work. I'll come up with a workflow and document it in the dune documentation and link it here.

[gridbugs]

Here's documentation on how to continue using an opam-based workflow, but switch to a dune package management workflow by setting an environment variable or passing an argument to dune: https://github.com/ocaml/dune/blob/main/doc/howto/use-opam-alongside-dune-package-management.rst

[avsm]

Thanks @gridbugs. I don't see (pkg) mentioned in https://dune.readthedocs.io/en/latest/reference/dune-workspace/index.html -- is this a new option?

[avsm]

Is this really working to build a fully statically linked binary? Isn't there a missing musl here, since glibc doesn't support static linking?

[dra27]

I think it's more that glibc is recommended for static linking because parts of it don't work properly - a scary halfway house that it works sometimes?

I'm not sure how to test it, but if the switch contained ocaml.5.4.0 and ocaml-option-musl and specified -static here, then I think you'd have a musl-based static binary (@gridbugs test release at https://github.com/gridbugs/container-image/releases/tag/2025-11-21.0 is definitely static according to ldd)

[avsm]

glibc unfortunately degrades very badly if you statically link it despite ldd showing no dependencies. This is because various pieces of it (nss, resolv, iconv) can call dlopen and expect to find .so files around. In my experience musl or equivalent is the only safe way to statically link since it's explicitly supported there.

[dra27]

I was hoping that if any of those things could possibly be done, then the program would fail to link statically… but perhaps my faith is misplaced 🙈

Regardless, completely agree that linking with musl is the way to go…

[gridbugs]

Fair enough! For personal projects that build with dune package management I usually make statically-linked binaries by building in an alpine docker container with musl installed since the turnaround times for debugging are short (I can test locally) and since there's no glibc on the system at all I know it won't sneak into my build unexpectedly. Would that be an acceptable solution here or would it be better to keep building the project directly in a github action but start linking it with muslc?

[avsm]

An Alpine version is fine, but the problem (as always) is that the binary needs to be tested on the host distro it's going to be run on, and there are some considerations around static-pie to take into account. This thread is a good start about how to do it in opam: https://discuss.ocaml.org/t/generating-static-and-portable-executables-with-ocaml/8405

[gridbugs]

I've added a docker-based action that statically links image with muslc. The action tries running image --help on the ubuntu host, and I've also tried running it on my nixos machine. Also according to ldd:

$ ldd image
        not a dynamic executable

[avsm]

ocaml/dune#12761 (comment)

[gridbugs]

Thanks @gridbugs. I don't see (pkg) mentioned in https://dune.readthedocs.io/en/latest/reference/dune-workspace/index.html -- is this a new option?

Yes it's a new option.

[avsm]

I see; so the minimum dune required for all this is actually the dev branch. That would have been good to know before starting all the tests, but I know now!

EDIT: to clarify, I've discovered the feature is in dune 3.20, but broken (ocaml/dune#12761 (comment)), hence the dev version of dune is required. This puts a practical limit of dune 3.21 before this PR can be merged.

[gridbugs]

the minimum dune required for all this is actually the dev branch

The dev branch is required for the (pkg ...) stanza but the github actions added in this PR use dune 3.20.2.