ssh-agent broken in tmux? I've got you!

A re-introduction of the ssh-agent-switcher tool and its recent changes

A little over two years ago, I wrote an article titled SSH agent forwarding and tmux done right. In it, I described how SSH agent forwarding works—a feature that lets a remote machine use the credentials stored in your local ssh-agent instance—and how using a console multiplexer like tmux or screen often breaks it.

In that article, I presented the ssh-agent-switcher: a program I put together in a few hours to fix this problem. In short, ssh-agent-switcher exposes an agent socket at a stable location (/tmp/ssh-agent.${USER?} by default) and proxies all incoming credential requests to the transient socket that the sshd server creates on a connection basis.

In this article, I want to formalize this project by presenting its first actual release, 1.0.0, and explain what has changed to warrant this release number. I put effort into creating this formal release because ssh-agent-switcher has organically gained more interest than I imagined as it is solving a real problem that various people have.

Some background

When I first wrote ssh-agent-switcher, I did so to fix a problem I was having at work: we were moving from local developer workstations to remote VMs, we required SSH to work on the remote VMs for GitHub access, and I kept hitting problems with the ssh-agent forwarding feature breaking because I’m an avid user of tmux.

To explain the problem to my peers, I wrote the aforementioned article and prototyped ssh-agent-switcher after-hours to demonstrate a solution. At the end of the day, the team took a different route for our remote machines but I kept using this little program on my personal machines.

Because of work constraints, I had originally written ssh-agent-switcher in Go and I had used Bazel as its build system. I also used my own shtk library to quickly write a bunch of integration tests and, because of the Bazel requirement, I even wrote my first ruleset, rules_shtk, to make it possible.

The program worked, but due to the apparent lack of interest, I considered it “done” and what you found in GitHub was a code dump of a little project I wrote in a couple of free evenings.

New OpenSSH naming scheme

Recently, however, ssh-agent-switcher stopped working on a Debian testing machine I run and I had to fix it. Luckily, someone had sent a bug report describing what the problem was: OpenSSH 10.1 had changed the location where sshd creates the forwarding sockets and even changed their naming scheme, so ssh-agent-switcher had to adapt.

Fixing this issue was straightforward, but doing so made me have to “touch” the ssh-agent-switcher codebase again and I got some interest to tweak it further.

The Rust rewrite

As I wanted to modernize this program, one thing kept rubbing me the wrong way: I had originally forced myself to use Go because of potential work constraints. As these requirements never became relevant and I “needed to write some code” to quench some stress, I decided to rewrite the program in Rust. Why, you ask? Just because I wanted to. It’s my code and I wanted to have fun with it, so I did the rewrite.

Which took me into a detour. You see: while command line parsing in Rust CLI apps is a solved problem, I had been using the ancient getopts crate in other projects of mine out of inertia. Using either library requires replicating some boilerplate across apps that I don’t like, so… I ended up cleaning up that “common code” as well and putting it into a new crate aptly-but-oddly-named getoptsargs. Take a look and see if you find it interesting… I might write a separate article on it.

Doing this rewrite also made me question the decision to use Bazel (again imposed by constraints that never materialized) for this simple tool: as much as I like the concepts behind this build system and think it’s the right choice for large codebases, it was just too heavy for a trivial program like ssh-agent-switcher. So… I just dropped Bazel and wrote a Makefile—which you’d think isn’t necessary for a pure Rust project but remember that this codebase includes shell tests too.

Daemonization support

With the Rust rewrite done, I was now on a path to making ssh-agent-switcher a “real project” so the first thing I wanted to fix were the ugly setup instructions from the original code dump.

Here is what the project README used to tell you to write into your shell startup scripts:

if [ ! -e "/tmp/ssh-agent.${USER}" ]; then
    if [ -n "${ZSH_VERSION}" ]; then
        eval ~/.local/bin/ssh-agent-switcher 2>/dev/null "&!"
    else
        ~/.local/bin/ssh-agent-switcher 2>/dev/null &
        disown 2>/dev/null || true
    fi
fi
export SSH_AUTH_SOCK="/tmp/ssh-agent.${USER}"

Yikes. You needed shell-specific logic to detach the program from the controlling session so that it didn’t stop running when you logged out, as that would have made ssh-agent-switcher suffer from the exact same problems as regular sshd socket handling.

The solution to this was to make ssh-agent-switcher become a daemon on its own with proper logging and “singleton” checking via PID file locking. So now you can reliably start it like this from any shell:

~/.local/bin/ssh-agent-switcher --daemon 2>/dev/null || true
export SSH_AUTH_SOCK="/tmp/ssh-agent.${USER}"

I suppose you could make systemd start and manage ssh-agent-switcher automatically with a per-user socket trigger without needing the daemonization support in the binary per se… but I do care about more than just Linux and so assuming the presence of systemd is not an option.

Going async

With that done, I felt compelled to fix a zero-day TODO that kept causing trouble for people: a fixed-size buffer used to proxy requests between the SSH client and the forwarded agent. This limitation caused connections to stall if the response from the ssh-agent contained more keys than fit in the buffer.

The workaround had been to make the fixed-size buffer “big enough”, but that was still insufficient for some outlier cases and came with the assumption that the messages sent over the socket would fit in the OS internal buffers in one go as well. No bueno.

Fixing this properly required one of the following:

• adding threads to handle reads and writes over two sockets in any order,

• dealing with the annoying select/poll family of system calls, or

• using an async runtime and library (tokio) to deal with the event-like nature of proxying data between two network connections.

People dislike async Rust for some good reasons, but async is the way to get to the real fearless concurrency promise. I did not fancy managing threads by hand, and I did not want to deal with manual event handling… so async it was.

And you know what? Switching to async had two immediate benefits:

• Handling termination signals with proper cleanup became straightforward. The previous code had to install a signal handler and deal with potential races in the face of blocking system calls by doing manual polling of incoming requests, which isn’t good if you like power efficiency. Using tokio made this trivial and in a way that I more easily trust is correct.

• I could easily implement the connection proxying by using an event-driven loop and not having to reason about threads and their terminating conditions. Funnily enough, after a couple of hours of hacking, I felt proud of the proxying algorithm and the comprehensive unit tests I had written so I asked Gemini for feedback, and… while it told me my code was correct, it also said I could replace it all with a single call to a tokio::io primitive! Fun times. I still don’t trust AI to write much code for me, but I do like it a lot to perform code reviews.

Even with tokio in the picture and all of the recent new features and fixes, the Rust binary of ssh-agent-switcher is still smaller (by 100KB or so) than the equivalent Go one and I trust its implementation more.

The 1.0.0 formal release

Knowing that various people had found this project useful over the last two years, I decided to conclude this sprint by creating an actual “formal release” of ssh-agent-switcher.

Formal releases require:

• Documentation, which made me write a manual page.

• A proper installation process, which made me write a traditional make install-like script because cargo install doesn’t support installing supporting documents.

• A tag and release number, which many people forget about doing these days but are critical if you want the code to be packaged in upstream OSes.

And with that, ssh-agent-switcher 1.0.0 went live on Christmas day of 2025. pkgsrc already has a package for it; what is your OS waiting for? 😉