Developer Insights: Making the OCaml Compiler Cacheable for Dune Package Management

We have recently updated Dune to take advantage of the ‘relocatable OCaml’ support that landed in OCaml 5.5!

The goal of the Dune build system is to provide a streamlined way for developers to manage their OCaml projects. It takes the many setup and customisation options of OCaml and opam and provides a nice interface that simplifies user workflows. In the same vein, the Dune package management (Dune pkg) project aims to extend Dune’s capabilities to manage and build your opam repository packages for you as well.

This post will take you behind the scenes of the rather complex technical work required to allow Dune to treat the OCaml compiler like any other package, meaning it’s cacheable and movable without necessitating a rebuild. This change means Dune will no longer rebuild the compiler from scratch, seemingly at ‘random’, which should mean a lot fewer headaches and frustrations for users.

Definitions: What do We Mean by Relocatable OCaml?

As a quick aside before we delve into what it means for Dune, let’s define ‘relocatable OCaml’. Prior to the 5.5 update, the OCaml compiler was hard-coded to a specific location and could not be cloned and moved to a new one without being rebuilt from sources.

Now, the compiler can be cloned and moved, which is beneficial not only because of the time and energy it no longer wastes rebuilding itself, but also because it allows developers to reliably distribute pre-compiled binaries and create more easily reproducible builds.

Why is Relocatable OCaml Good for Dune?

OCaml compilation allows users to get very specific and granular about how they want their projects set up. This is great for users who want fine-grained control, but frustrating for the many developers for whom that level of detail is irrelevant. The manual path involves using multiple tools to manage all of your dependencies, with many points where things can break.

The idea behind Dune package management is that it handles the complexity for you. You tell it what you want to use, and it installs and builds it. One of Dune’s core features is the cacheability and reproducibility of builds: it can store your builds in a cache and restore them deterministically.

As you can imagine, an OCaml compiler that wasn’t relocatable significantly disrupted this workflow. Since it couldn’t be cached, its design and Dune pkg’s design were fundamentally incompatible. Numerous workarounds were tried, but ultimately didn’t prevent problems, namely that Dune would rebuild the compiler from scratch whenever it detected changes. This was a lengthy process, and users wondered why Dune would decide to rebuild the compiler and make them wait.

With a relocatable compiler, this pain goes away. At least, it does now, after our team has made compatibility possible!

The Road to a Relocatable Compiler in Dune

Getting Started

The first step for our engineers, Ali Caglayan and Ambre Suhamy, was to investigate how relocatable OCaml worked under the hood, including compiler cloning between switches. For the 5.5 update itself, relatively little was needed beyond checking a version bound and updating the compiler-detection heuristics so that a relocatable compiler would be treated as a normal opam package rather than being routed through the toolchains mechanism. Issue #13229 tracked this initial work.

The OCaml compiler uses some of the most complex opam packaging techniques out of any opam package, and Ali and Ambre needed to understand how they would work, or rather where they did not, with Dune pkg. They let Dune attempt the build and observed what broke: mostly variables that were never supported, like %{build-id}%.These were, for Dune, obscure features, some of which it had never encountered before. Ali and Ambre needed to interpret some things anew, for example, opam’s string interpolation, and adjust some of the old ways they had been interpreted before.

Lots of Fixes

This detailed investigation paid off. For example, opam’s string interpolation %{ocaml-system:installed?system:}% is a ternary conditional operator that checks if ocaml-system is installed and gives a string value of ”system” if true and “” if false. Curiously, there was a typo that left off the “:”, which read like an if … then …; conditional, which defaulted to the empty string. Then, since opam swallows errors that happen during parsing and defaults to the empty string, this became the unintentional semantics for the expression.

It was a rather puzzling problem for our team investigating it on Dune’s side, since the spec made no mention of this operator. After discussing the issue with opam maintainers, it turned out to be a typo and was fixed in a subsequent pull request. After the fix, Dune worked as expected again!

Translation and the Lockfile

The way Dune pkg works is by running a solver to find a package solution, and then translating all the opam build instructions into instructions that Dune can understand. This ‘lock file’ serves as a kind of cache for the translation but also allows consumers of the lock file to avoid having to consult the opam repository. This is especially useful for CI systems since they only need their package sources.

The team had to make sure that Dune pkg was able to interpret opam’s build instructions as actions that it can run, whether they are in a lock file or not. That meant ensuring that every variable the package management system had previously encountered was handled correctly, and adding support for those it had never seen before. This process led to several simplifications in the lock directory’s structure. There's still more to do here, but the result is solid enough that we can ship OCaml 5.5 support in Dune pkg.

One Last Hurdle: OCamlbuild and Symlink Support

The OCaml compiler wasn’t the only package that we had to update. OCamlbuild is a build system that predates Dune and is still used by some actively-maintained packages. OCamlbuild was also recently patched to be relocatable in an effort to make Dune pkg useful for as many projects as possible. However, another problem presented itself: OCamlbuild sources contained a symbolic link pointing to a directory. Directory symlinks are not supported by the Dune engine, and so Dune was constantly rejecting OCamlbuild upon fetching it.

Instead of adding full support for directory symlinks, which would have taken a lot of time, we decided on a compromise. Ambre figured out how to resolve the directory symlinks at fetch time, before the build begins. Explore the solution in greater detail in PR #13792.

Backwards Compatibility and Future Plans

Thanks to David Allsopp’s hard work to backport relocatable OCaml as far back as OCaml 4.08, our team could implement backwards compatibility for Dune pkg as well. Ali and Ambre introduced a mechanism that allows Dune to automatically overlay the upstream opam repository with the backported compiler versions, maintained based on David’s opam repository fork. They updated the compiler-detection heuristics to allow any relocatable compiler (whether it’s an OCaml 5.5+ or an older backported version) to bypass the toolchains mechanism and be treated as a normal opam package.

Our team did not remove the toolchains mechanism since they wanted to keep supporting OxCaml. The OxCaml branch does not yet support a relocatable compiler, but once it does, we can remove the toolchains entirely to simplify the implementation and make maintenance easier.

The Outcome

Ultimately, all of the fixes and effort have paid off. When your Dune cache is enabled, and you use package management, any project that has built the compiler now has it cached, and any other project (or the same after deleting _build) will retrieve this cache.

The team also decided that the relocatable compiler is the default behaviour, so that users do not have to find and enable the relocatable compiler feature. This way, the compiler just behaves like a normal package, and the users don’t have to know why or how it works – it just does!

The end result is faster builds, less frustration, and (given how often the compiler was being unnecessarily rebuilt) hopefully some saved energy along the way.

Until Next Time

Have you tested Dune package management with the new relocatable compiler? Let us know about your experience and whether you’ve saved any time when not waiting for the compiler to rebuild!

Connect with us on Bluesky, Mastodon, Threads, and LinkedIn or sign up for our mailing list to stay updated on our latest projects. You can also connect with other OCaml users on Discuss to share your experience and feedback. We look forward to hearing your thoughts!