Kyua graduates

The story behind the FreeBSD and NetBSD testing frameworks

After years of inactivity, the Kyua project has graduated as an open source citizen and has a new home under the FreeBSD umbrella!

But uh… wait, what is Kyua and why is this exciting? To resolve confusion and celebrate this milestone, I’d like to revisit what Kyua is, how it came to be, why I stopped working on it for a while, why that was a problem for FreeBSD—and, indirectly, NetBSD—and how Kyua being free software has helped keep it alive.

The birth of ATF

Let’s begin by talking about Kyua’s predecessor: the Automated Testing Framework (ATF).

In 2005, I was one of the lucky pioneers of the Google Summer of Code (GSoC) program. During that summer, I wrote the tmpfs file system for NetBSD and my development process went something like this on an iBook G3:

1. Modify the kernel.

2. Build and install the updated kernel.

3. Reboot the laptop.

4. Mount the file system.

5. Manually validate new functionality and bug fixes.

6. Witness the machine crash.

7. Force-reboot.

8. Rinse and repeat.

Needless to say, this was a painful process. Using kernel modules lowered the pain a tiny little when the changes I made didn’t crash but, in general, they did crash. Documentation was scarce and I didn’t know what I was doing!

In an attempt to make the validation step less painful and to ensure that each change I made was a “net positive” instead of regressing functionality, I got what-I-think-was my first real exposure to automated testing. I had to automate steps 4 and 5 in the flow above, and so I wrote ad-hoc scripts to do so.

My GSoC mentors, Luke Mewburn and Bill Studenmund, pointed me at the src/tests directory of the NetBSD source tree which contained a bunch of tests for the system. Some of these were for user space tools and others exercised portions of the kernel. The existing machinery could probably be reused to automate my manual regression testing steps, so I had to try to integrate with it.

Machinery is… a strong word given how rudimentary src/tests was. The main issue with this infrastructure was that the tests didn’t have a real runner: they were all small programs with unique command-line interfaces glued together by a collection of Makefiles. There was no guarantee that these tests built nor that running a make test would run them all because many tests were known to be broken. And what’s worse: some of the existing tests crashed the kernel due to known bugs.

I started envisioning the need for a principled test framework and wanted to create one, but I needed the time to do so and school didn’t give me much of that. So, in 2006, I applied once again to GSoC but, while I got in, I chose to take a detour and help write the beginnings of the Boost.Process library. The reason for this choice was primarily to learn Win32 internals but a good side-effect was that I got to experience a more advanced test framework.

And then, in 2007, I wanted to create this automation project as my bachelor’s degree graduation project… but I couldn’t find a professor that would back the idea. (A pity because I believe it would have been a great fit for the operating systems group.) So I did what I knew how to do: I applied and got into GSoC once again—this time to develop a testing framework for NetBSD.

The idea was to create libraries to write tests with a unified CLI interface in either C, C++, or shell, and then provide a unified runner that would execute these tests in an isolated manner and without access to the source tree. And thus the Automated Testing Framework (ATF) was born.

Problems with ATF

ATF was fine for a while. We got a large portion of the old tests in NetBSD resurrected, fixed, and hooked into the ATF infrastructure. A major contribution was Andreas Gustafsson’s Anita: an automation system to create a VM, install NetBSD on it via the interactive installer, and run the whole battery of tests inside of it.

Anita and ATF were the beginnings of something akin to a CI system for a full operating system in the open source world. As far as I know, Linux did not have anything similar—and it couldn’t have had anything similar due to the distributed nature of its development—nor did the other BSDs. Windows did have this, of course, as I could glance from the many articles in The Old New Thing describing backwards compatibility testing.

This was great but, over time, ATF showed signs of suboptimal design. It had at least two problems that were really hard to fix, namely:

• Its design (remember running atf-run | atf-report?) prevented the execution of tests in parallel, and concurrent test execution is crucial for performance. The Anita test runs took just too long for a good CI turnaround.

• ATF expected tests to be written using the atf-c, atf-c++, and atf-sh libraries—but developers wanted to implement and reuse tests without porting them to these libraries. In particular, people wanted to create simpler test programs and wanted to support programs that emitted the more-widespread Test Anything Protocol (TAP) popularized by Perl test harnesses.

Aside from these problems, ATF also lacked a critical feature that Anita had to supply on its own: a data store to track historical test reports and an interface to browse through them. This feature was a necessity for a CI system so, ideally, it had to live within ATF to not be stuck inside a web server. (“Web apps everywhere” was not a thing pre-2010.) To compound the problem, I got an internship at Google in 2008 and joined full-time in 2009, which let me experience Blaze and its ecosystem for test results tracking and reporting first-hand—and I wanted those features for NetBSD.

I concluded, maybe wrongly, that ATF needed a rewrite.

Kyua arrives

So, in 2010, I set to create a brand new execution engine. At that point, the target was to rewrite just the portion of ATF that executed tests: that is, the atf-run | atf-report pipeline. I did not want to recreate the ATF libraries. The idea was to:

• Create a generic test runner that could integrate multiple test protocols, with the ATF-based tests being the first use case. The unwritten goal was to let the ATF libraries dwindle at some point in the future in favor of better libraries (like GoogleTest for C++ tests or my own shtk for shell tests).

• Support a “tests database” to store the results of all test executions. This database was going to allow generating reports in multiple formats and permit running queries to look at historical trends.

• Support parallel test execution via a more modular runner design. (If you know the history behind Kyua, you’ll notice that this feature only appeared very late in the game. It was a mistake to delay it and I’m not sure why I did that.)

Thus, in November of 2010, Kyua was born as a “testing framework for infrastructure software”. It took some work to get to the point where Kyua could run most of the NetBSD test suite unmodified, but by July 2011, I reached this milestone and launched Kyua 0.1.

Which led to frustration: ATF was deeply ingrained in NetBSD and Kyua wasn’t well-received for various reasons. One was that Kyua was a larger C++ project than ATF and C++ was (and still is) frowned upon in that community. Another was the fact that Kyua’s reports were harder to manage and required significantly more disk space, and Andreas wasn’t ready to bubble up his CI hardware costs. There were other concerns, all pretty reasonable.

I kept working on Kyua to try to resolve most of the concerns raised and I believe I addressed all of those that could be addressed. Unfortunately, this took time, and I had grown demotivated to pursue another “sell” of the project. But then…

Kyua meets FreeBSD

The FreeBSD community had been paying attention. They needed to improve their CI story as well, and they did have much more interest in building a test suite due to the needs of a few corporate-backed projects. I’m not exactly sure who I talked to other than Enji Cooper, George N. Neville, and Ed Maste, but by 2013, I had gotten a FreeBSD commit bit and I was set to integrate Kyua—not ATF!—into FreeBSD.

One thing I remember was how the FreeBSD project offered me three dedicated beast machines in a colo so that I could set them up for CI runs. This felt very different from the NetBSD approach: FreeBSD had resources—understandably so, as NetBSD has always been underrated—and they were ready to invest them into the shiny-new thing. I took the bait and worked on that for a while. (Fun fact: I used a PowerMac G5 for this work.)

By December of 2013, I shipped the initial CI system for FreeBSD. This was a really exciting moment because all efforts I had put behind Kyua finally paid off: Kyua was the driver for what I think is the most important operating system project in the open source community outside the Linux monoculture.

The future of the project was tainted though: I was still employed by Google… which made me keep experiencing Blaze and its ecosystem day after day. You see: Blaze is often touted as a build system, but its secret sauce lies in it being a test system. Blaze is a test runner that knows how to precisely build and run the minimum set of tests impacted by a code change. And this matters.

Precise test selection doesn’t make a big difference for the run-off-the-mill projects you see in GitHub, but for monorepo-style projects—and the BSDs are such projects—this becomes a requirement to run a CI system with quick turnaround times. By this point, I saw Kyua as a “local maximum” because I did not see how to make this possible in combination with the native BSDs’ build systems.

Stagnation

No matter what I secretly thought, FreeBSD was bought into ATF and Kyua, and they—Enji in particular—had larger plans for the system. The FreeBSD developers really wanted to integrate TAP-style tests, needed to set up a full CI system with many more configurations and architectures than I had set up, and needed better reporting capabilities for failures.

The problem was that I owned the critical software pieces of this infrastructure, and this was a problem for various reasons: I was too busy with work and two little kids; I had stopped using any BSD on a daily basis; I did not see a bright future for Kyua as it was designed nor implemented; and to make things worse, contributing to Kyua required signing the Google CLA because I still worked for Google (which never made any sense to me but I didn’t have a say in the matter).

So, for years, Kyua went pretty much unmaintained. FreeBSD kept building local patches and occasionally sent PRs to my upstream project, but I dreaded dealing with them. Maintaining open source projects is stressful if only because of this guilt that you end up with when you don’t have the time to deal with contributions in a timely manner. The more time the situation lasts, the worse it gets.

It took until 2020 to convince myself that ATF and Kyua needed attention and that I was not fit to support them. I had to let go off these projects and I needed to clear my mind for other cool stuff. So… I emailed the FreeBSD project owners and offered them to take ownership. If they forked the project and owned it, any FreeBSD committer could easily gain access to the project to do what was best for FreeBSD. And if they owned the fork, they wouldn’t have to worry about the CLA part of the situation because I wouldn’t be involved any longer.

Things didn’t make the progress I desired though. The FreeBSD project owners had concerns about officially forking these projects into FreeBSD because it cloud give the wrong impression and alienate other consumers like NetBSD. Kyua and ATF were not FreeBSD-only projects so it was probably better for them to stand alone. The discussions died off and stayed like that…

Rebirth

… until the end of 2023 when Li-Wen Hsu resurrected the email thread I had started. Kyua had become a critical piece of the FreeBSD quality infrastructure and it being almost-abandoned was starting to be problematic. He asked if we could move forward with the project ownership transfer idea—and I was at the ready.

And so that’s what happened!

On January 17th, 2024, the right buttons were pushed and… Kyua (along with ATF and Lutok) all became owned by the FreeBSD GitHub organization. Since then, I’ve seen a flurry of PRs reviewed and merged into the project, which makes me happy, really, because the people that care now have the chance to do what’s best for the project.

Plus, you know what? Even if Kyua has some “design flaws” because it was not designed to be exactly like Blaze, it does have some other interesting ideas that make it a good-enough fit for FreeBSD. In particular, one design principle behind ATF and Kyua was to allow running a test suite without having access to the source code.

Actually, I think this was a killer idea in the original design of ATF. The motivation for “running tests without the source” was to allow installing a system from scratch and then let the end user run the tests to verify that the system worked correctly on their specific software/hardware combination. The BSDs have limited resources and cannot do the sorts of testing that e.g. Microsoft does with Windows so, by pushing the tests “to the edge”, users could fill that gap.

And to illustrate this novel-at-the-time idea, let’s take a tour.

Kyua in FreeBSD 14

Kyua is part of the FreeBSD base system and, as such, it is installed under /usr/bin/kyua and is available out of the box.

Similarly, the test suite is available under /usr/tests if you choose to unpack tests.txz during the system installation. Peeking under it, we find:

/usr/tests$ ls -F
Kyuafile     conftest.py  lib/         sys/
README       etc/         libexec/     usr.bin/
__init__.py  examples/    local@       usr.sbin/
atf_python/  games/       sbin/
bin/         gnu/         secure/
cddl/        include/     share/
/usr/tests$ █

We can use kyua list to see what kind of test coverage we have. This command prints one test case name per line in its default output format, so we can easily count how many tests there are:

/usr/tests$ kyua list | wc -l
    8246
/usr/tests$ █

8000+ tests. Not bad! Let’s peek into a tiny sample by focusing on the tests for dd(1):

/usr/tests$ kyua list bin/dd
bin/dd/dd2_test:max_seek
bin/dd/dd2_test:seek_overflow
bin/dd/dd2_test:sigint_open
bin/dd/dd2_test:sigint_read
bin/dd/dd_test:io
bin/dd/dd_test:length
bin/dd/dd_test:seek
/usr/tests$ █

And let’s run those!

/usr/tests$ time kyua -v parallelism=$(nproc) test bin/dd
bin/dd/dd2_test:seek_overflow  ->  passed  [0.042s]
bin/dd/dd2_test:max_seek  ->  skipped: tmpfs can't create arbitrarily large sparse files  [0.044s]
bin/dd/dd_test:length  ->  skipped: fdescfs is not mounted on /dev/fd  [0.024s]
bin/dd/dd_test:seek  ->  passed  [0.048s]
bin/dd/dd_test:io  ->  passed  [0.071s]
bin/dd/dd2_test:sigint_read  ->  passed  [3.077s]
bin/dd/dd2_test:sigint_open  ->  passed  [3.079s]

Results file id is usr_tests.20240728-161942-793100
Results saved to /home/jmmv/.kyua/store/results.usr_tests.20240728-161942-793100.db

7/7 passed (0 failed)
kyua -v parallelism=$(nproc) test bin/dd  0.17s user 0.33s system 15% cpu 3.198 total
/usr/tests$ █

Easy peasy. We can run tests for any part of the system right from the /usr/tests hierarchy without having had to build anything nor needing access to the source tree. As mentioned earlier, you can use this feature to validate that FreeBSD works on your specific machine given that device drivers are of varying quality (something that’s true on any operating system) and may result in different behavior. And you can use this feature to validate that the system continues to work as you perform upgrades over time.

Outside of your machine, though, we can also look at the CI system based on Jenkins. For that, head to https://ci.freebsd.org/ and peek around. Notice the tens of different jobs configured to build and test FreeBSD under various architectures, releases, and build settings.

As an example, look at the recent run 25361 of FreeBSD-main-amd64-test, and notice how we can inspect the same test results we saw on the command line. Do so by clicking on bin.dd and finding the results for the same tests we ran earlier. This showcases Kyua’s ability to generate a junit.xml test report—an output format that was precisely written to integrate with Jenkins for the FreeBSD CI system.

---

And that’s it.

The morale of the story is two-fold. First, free software can remain alive and kicking even if the original authors lose interest. And second… do you see how powerful it is to have a CI system for a whole OS? This is something Linux can only dream of because of its inherent distributed development nature. Yes, there is the Linux Test Project, but because Linux is “just a kernel”, this is what this project can test. There is nothing that can validate that the kernel works in combination with all other pieces that form a Linux distribution in a uniform and cohesive manner.

In any case, congrats grads. Kyua has a bright future ahead.