Revisiting the NetBSD build system

I recently picked up an embedded project in which I needed to build a highly customized full system image with minimal boot times. As I explored my options, I came to the conclusion that NetBSD, the often-forgotten BSD variant, was the best viable choice for my project.

One reason for this choice is NetBSD’s build system. Once you look and get past the fact that it feels frozen in time since 2002, you realize it is still one of the most advanced build systems you can find for an OS. And it shows: the NetBSD build system allows you to build the full OS from scratch, on pretty much any host POSIX platform, while targeting any hardware architecture supported by NetBSD. All without root privileges.

Another reason for this choice is that NetBSD was my daily workhorse for many years and I’m quite familiar with its internals, which is useful knowledge to quickly achieve the goals I have in mind. In fact, I was a NetBSD Developer with capital D: I had commit access to the project from about 2002 through 2012 or so, and I have just revived my account in service of this project. jmmv@ is back!

So, strap onto your seats and let’s see how today’s NetBSD build system looks like and what makes it special. I’ll add my own critique at the end, because it ain’t perfect, but overall it continues to deliver on its design goals set in the late 1990s.

The basics

The NetBSD build system is powerful and featureful, but it’s also arcane as it’s based on a combination of the BSD variant of make and shell scripts. Just peek through the files under src/share/mk/, the directory that contains the bulk of the infrastructure, to see what I mean.

As a user of the build system, however, you rarely interact with make directly. Instead, you use the build.sh script located at the top of the source tree. This script provides a user-friendly interface to most operations you may want to do, and abstracts away the intricacies of the targets that coordinate the build of the system and the configuration that controls it.

The structure of the command is to pass high-level “goals” to build.sh as arguments, which indicate the operations to perform. In its most simple form, all you need to do to build a full system distribution targeting the architecture of the host is:

./build.sh tools release

But hey, I promised you can trivially cross-build too, right? Sure, let’s compile the system for a Raspberry Pi with a 64-bit chip, produce the USB image that we can write to an SD card, and do everything as an unprivileged user:

./build.sh -U -a aarch64 -m evbarm tools release disk-image

That’s it, really. That’s all it takes.

We must dig deeper though, so let’s look at some of those “goals” to see what tools means and understand how a release is put together without root privileges.

The toolchain

The very first step of any build.sh invocation is to generate the toolchain used to build the rest of the system. This is true of any build, including those that target the host machine, because this ensures that the build is independent of the host’s state. In particular, this avoids the situation where you might have to upgrade certain components before building, which was/is common in other BSDs.

And you have seen this prerequisite step in the previous section, by the way: all sample build.sh invocations I showed you included tools as the first goal. Now, you don’t have to provide tools to every invocation of the script: as soon as you have built a toolchain, you can reuse it in subsequent invocations.

Building a toolchain with build.sh is incredibly handy on its own, as you can produce cross-build toolchains and use them for other purposes outside of building NetBSD itself. Zig’s build system has often been praised for this reason, but NetBSD’s has nothing to envy. For example, if we do this:

./build.sh -a aarch64 -m evbarm -T ~/tools tools

We end up with a cross-build C and C++ toolchain under ~/tools/ that targets NetBSD running on ARM 64 bits. And what goes into the toolchain, you ask?

Directory listing of tools as produced by the command above and its bin subdirectory.

In this listing, you can see a bunch of binaries prefixed with aarch64--netbsd-. These are all part of the C and C++ toolchain. The rest of the tools, prefixed with nb, are NetBSD-specific tools required during the build. These are programs that are part of a normal NetBSD installation and would be available without the nb prefix if we were building on a NetBSD host, but remember, the build system supports any POSIX host OS. Take nbsed as an example: yes, all POSIX hosts provide a sed tool, but its syntax varies among systems so the NetBSD build system isolates itself from those differences by compiling sed as a host tool and using that throughout.

One special tool from this listing is nbmake-evbarm. This is not the binary for make (which is itself stored as nbmake). This is a shell script that captures all settings provided to build.sh and then invokes nbmake with those, and this script is useful when you want to manually rebuild portions of the tree. Not something you would want to use as an “end user” of the build, but something you definitely will want to use as a NetBSD developer.

Build structure

Let’s explore the source tree a bit, which is the prime example of a monorepo in an open source project:

Directory listing of the source tree, its bin and bin/ls subdirectories, and the content of bin/ls/Makefile.

In this picture, you can see first the content of the top-level directory of the source tree. It all looks pretty simple: there are various subdirectories, such as bin, lib, or usr.bin, that roughly track the structure of the installed system; there is the build.sh script that I previously described; and there is a Makefile as well. Looking into one subdirectory, like bin, we see another Makefile and many more subdirectories, one per tool installed onto /bin/.

Knowing this directory-based structure, we can use the nbmake-evbarm wrapper script I mentioned earlier to operate on just a portion of the monorepo. Focusing on the ls example shown in the screenshot, we could build and upgrade this piece of the system on its own by doing:

$ cd ~/src/bin/ls
$ ~/tools/bin/nbmake-evbarm all
... console noise ...
$ sudo ~/tools/bin/nbmake-evbarm install
... console noise ...
$ █

Another extra detail to highlight from the screenshot is that NetBSD’s Makefiles are mostly declarative. Each Makefile defines a bunch of variables to specify what is being built and, at the end, includes one of the many bsd.*.mk files that pull in the build logic. Among these, we have bsd.prog.mk to build one program, bsd.lib.mk to build one static/shared library, and bsd.subdir.mk to recurse into subdirectories. Importantly, the general design is to build just one item per directory—although I myself broke this rule when I added bsd.tests.mk to build tests because splitting them into subdirectories would have added too much noise to the tree.

This declarative design is interesting because it maps well to the foundations of modern build systems like Bazel. In fact, the design of the NetBSD build system is what fueled my interest in build systems, influenced the design of my own Buildtool, and made me like Bazel Blaze as soon as I first saw it in 2008.

The destdir

In order to produce the structure of the final installation, the build system uses the “destdir” concept. A destdir is a staging location where built files are installed, but paths to this staging location are not used within the artifacts produced by the build. This idea exists in other build systems such as GNU Automake and is pretty much a necessity to build multiple pieces of software together before installing them or to package software without root privileges.

Imagine that you want to build a library, say libm (the math library), and a tool that uses it, say bc (the calculator). libm typically goes into /lib/ so we cannot just build and install it in place: for one, we may “break” the existing system if the new version happens to be backwards-incompatible; for another, we may be targeting a different architecture so we cannot just replace /lib/libm.so with an incompatible version.

The destdir comes to the rescue. We first build libm as if it would be installed into /lib/. However, during installation, we prefix all file copy operations with the destdir. In this way, we build a separate “system root”, say /tmp/destdir/lib/, that contains the newly-built libm.so. After that, we build bc and point it to the libm that’s in /tmp/destdir/lib/, but… we have a problem: we can’t allow the /tmp/destdir/ path to appear anywhere inside the bc binary because this directory is transient. To fix this, we must separate build paths from runtime paths during the build: when we build bc, we tell the linker to look for libraries under /tmp/destdir/lib/ via the -L flag, and we also tell the linker that, at runtime, libraries will be available in /lib/ via the --rpath flag (which stands for runtime path).

As you can imagine, the NetBSD build system heavily relies on this idea and, after a distribution build (implied by the release goal I showed earlier):

./build.sh -D /tmp/testdir/ distribution

we end up with a destdir that contains all system files laid out exactly as they need to be installed.

In fact, if you run the build as root and target the host system (where the host is NetBSD), the destdir can serve as the target of a chroot. So, if you do:

chroot /tmp/destdir

you essentially can enter the freshly-built system. This may or may not work, however: the newly-built binaries might require new kernel features, which is likely true if you are building a more modern NetBSD release from an older release or if you are tracking NetBSD-current. And this obviously won’t work if you are cross-building.

Distribution media

The destdir serves as a staging area but it does not represent the final artifacts of the build. To put the destdir to use, we either have to “copy” the staging area onto the host to perform an in-place upgrade, or we need to build distribution media.

The former case of an in-place upgrade is tricky because it requires issuing manual post-installation steps, so I’m not going to describe it here. But the latter case of producing distribution media is trivial. For example, we can do:

./build.sh release

to produce the release “sets” for the system from the contents of the destdir, or we can do:

./build.sh iso-image install-image live-image

to create various types of installation media (a bootable CD, a bootable USB image, a live system image…) from the contents of the destdir as well.

The release sets are an interesting thing to discuss because they form the core of a NetBSD distribution. You see: NetBSD ships as a collection of tarballs, and installing NetBSD amounts to simply unpacking those tarballs onto a file system and performing a few post-installation configuration steps.

Content of the binary/sets directory of the NetBSD/amd64 distribution.

Now, the way these tarballs are produced from the destdir is by leveraging mtree, a really cool tool that is not known in Linux land. The purpose of this tool is to compare a textual “golden” representation of a directory against the actual contents of the directory, and highlight where they might differ.

BSD systems use mtree to describe how the installed system looks like and, as you can imagine, NetBSD is no exception. The NetBSD build system uses mtree files to ensure the destdir contents match expectations, and also uses the mtree “manifests” to “bucketize” the files from the destdir into the individual release sets. You can find these golden manifests in the src/distrib/sets/lists/ directory.

Unprivileged builds

These mtree files are also critical for another very important feature: namely, the ability to build the whole NetBSD system as an unprivileged user. This, to me, is one of the most impressive features of this build system: you can produce the full build, including disk images, without ever running sudo or using weird intercept tools like Debian’s fakeroot.

Here is how this works. When building in unprivileged mode (enabled via the -U flag to build.sh), the build system produces a METALOG file under the destdir. This file looks like this:

$ grep '^\./sbin' ~/destdir/METALOG | head -n 5
./sbin type=dir uname=root gname=wheel mode=0755
./sbin/amrctl type=file uname=root gname=wheel mode=0555 size=72120 time=1735119460.0 sha256=d6f474441dc98648a4e8ec633dd76fc3349c85a0af9830ce8eb6566e94291fdb
./sbin/apmlabel type=file uname=root gname=wheel mode=0555 size=72360 time=1735119460.0 sha256=731cf433328bd14c6d3f322ef60fa92eb9eaa2725baa0cb52253b50743d9d324
./sbin/atactl type=file uname=root gname=wheel mode=0555 size=73512 time=1735119461.0 sha256=74a15335c8715513ac50f615a8e906319df79f76dad601bc688ae214b3e41673
./sbin/badsect type=file uname=root gname=wheel mode=0555 size=72328 time=1735119461.0 sha256=76e34649bf100fe490befb2a4ec58455a7710a665dd59bcbbd8c672a6086bde8
$ █

Every line of this file maps a file system entry (a directory, a file, a device…) stored in the destdir to its properties, including ownership information and permissions. These entries are generated from metadata encoded in the Makefiles whenever the build system places a new file under the destdir via the install command (another nice tool often unknown to Linux users). The METALOG is the key that allows building media images without root privileges.

If you think about it, media images are simply files with an internal structure that represents disk partitions, file systems, and metadata. Because they are simply files, there is no need to have root access nor to make the host’s passwd file contain all users represented by entries in these file systems. Traditionally, OS builds have needed root because it’s easier to leverage the kernel’s virtual devices and file system implementations, but there is not inherent reason for that to be the only choice. All the work can be done in user space, and that’s precisely what NetBSD does.

Now, go back and revisit the screenshot above that showed the toolchain contents. You’ll notice tools like nbmakefs (the tool to format a file system) and nbgpt (the tool to create a GPT partitioning scheme). These tools are part of the toolchain because they are needed to generate installation media, and these tools know how to read the METALOG in order to embed the right permissions and special file modes into the built images. All without ever becoming root.

sysbuild

Now, as simple and powerful as build.sh might be, I find it cumbersome for day to day use if you want to customize any of its default settings. It is not uncommon to end up running build.sh with invocations like:

./build.sh -O ../obj -T ../tools -U -u -V MKDEBUG=no -V MKGCC=no distribution

which the official documentation describes as “golden invocations” and I have no desire to type or even remember.

This is what drove me to write sysbuild: a layer of abstraction over build.sh and cvs that coordinates updating the source tree and building it. The tool even integrates with cron trivially, providing a mechanism to keep NetBSD-current installations up-to-date.

sysbuild is driven by configuration “profiles” which allow you to customize the paths and settings of a build in just one place and then puts them to use with a trivial command. For example, with a configuration file like the following stored in ~/.sysbuild/rpi.conf:

BUILD_ROOT="${HOME}/netbsd"
BUILD_TARGETS="tools sets disk-image"
CVSROOT=:ext:anoncvs@cvs.NetBSD.org:/cvsroot
MACHINES=amd64
RELEASEDIR="${BUILD_ROOT}/release"
SRCDIR="${BUILD_ROOT}/src"
UPDATE_SOURCES=no

We can simply run:

sysbuild -c rpi build

to update the NetBSD source tree to the latest version, ensure that the tools are up-to-date, and produce the USB disk images for a Raspberry Pi.

Deficiencies

Not everything about the NetBSD build system is rosy though.

The thing that differentiates a good build system from a “meh” one for me personally is the behavior of incremental builds and, in particular, two aspects of these:

• First, incremental builds need to do minimal work, especially when there is “nothing to do”. The NetBSD build system is a recursive make one (which comes with its own set of problems), so it does not do minimal work. On my 72-core machine, it takes about 3 minutes to run through a build.sh release invocation that does nothing. This is OK for end users looking to upgrade their running machine, but it is painful because it makes iterating on system changes difficult. As a developer, you end up needing to know how to surgically rebuild individual subdirectories using the nbmake-<arch> wrappers I described earlier, and manually track dependencies across those.

• Second, incremental builds must always deliver correct results without having to do a make clean in between. But that’s generally not true for make-based build systems—and NetBSD’s is no exception. Generally, an incremental build after a git pull (sorry, a cvs update) will work fine, but sometimes it won’t. And if you start playing with build-time switches (things like MKDEBUG), then you are out of luck and must resort to a make clean to “switch configurations”.

And there are other problems. Running a parallel build on a system with many cores sometimes leads to spurious build failures because the interdependencies between components are not always precisely specified (it’s really difficult to be correct with make). And the build is inefficient: of those 3 minutes I mentioned earlier, you can see that most of the time is wasted by make recursing through directories and discovering there is nothing to do, whereas other times, make “chokes” on way too many C++ compiles at once that lead to out of memory situations.

It has been my dream since the publication of Bazel as open source to have a Bazel-based build of NetBSD. I think Bazel is the perfect build system for such a project because it’d deliver correct and efficient incremental builds to NetBSD’s monorepo, and it would save tons of resources when running on many-core machines. Except for the fact that it’s written in Java, so it’d be a really odd choice for such project. Maybe Buck 2 would be suitable. Anyway, one can only dream…

But why?

Why I’m looking at this at all again, after years of not touching NetBSD? I said it in the opening: I’m working on a new embedded project for which NetBSD is the greatest fit. I could tell you what it is about but it’s easier to just show you:

OK, fine, in words: I am building a minimal system that boots straight into EndBASIC with quick build times and low overhead. You’ll have to wait a bit more to get your hands on this though, as I’m still ironing out various details and want to end up providing a pre-built “box” with the right hardware and software combination.

I’m also becoming super tempted to migrate NetBSD’s build to Bazel to make my own life easier in this journey. This is a monumental task… but I’m not sure that it’d be crazy to tackle the minimum subset of NetBSD that I need for this minimal disk image and port only those portions to Bazel. The results might impress some and then want to help the effort. Right?

In the meantime, I encourage you to read through the comprehensive Building the system portion of The NetBSD guide, and to play with building a NetBSD image straight from your Linux machine. You may like it.