Demystifying secure NFS
My lab notes on the arduous process of setting up NFSv4 with Kerberos across a Synology NAS and various Linux and FreeBSD clients.
I recently got a Synology DS923+ for evaluation purposes which led me to setting up NFSv4 with Kerberos. I had done this about a year ago with FreeBSD as the host, and going through this process once again reminded me of how painful it is to secure an NFS connection.
You see, Samba is much easier to set up, but because NFS is the native file sharing protocol of Unix systems, I felt compelled to use it instead. However, if you opt for NFSv3 (the “easy default”), you are left with a system that has zero security: traffic travels unencrypted and unsigned, and the server trusts the client when the client asserts who is who. Madness for today’s standards. Yet, when you look around, people say “oh, but NFSv3 is fine if you trust the network!” But seriously, who trusts the network in this day and age?
You have to turn to NFSv4 and combine it with Kerberos for a secure file sharing option. And let me tell you: the experience of setting these up and getting things to work is horrible, and the documentation out there is terrible. Most documents are operating-system specific so they only tell you what works when a specific server and a specific client talk to each other. Other documents just assume, and thus omit, various important details of the configuration.
So. This article is my recollection of “lab notes” on how to set this whole thing up along with the necessary background to understand NFSv4 and Kerberos. My specific setup involes the Synology DS923+ as the NFSv4 server; Fedora, Debian, and FreeBSD clients; and the supporting KDC on a pfSense (or FreeBSD) box.
NFSv3’s insecurity
NFSv3, or usually just NFS, is a protocol from the 1980s—and it shows. In broad terms, NFSv3 exposes the inodes of the underlying file system to the network. This became clear to me when I implemented tmpfs for NetBSD in 2005 and realized that a subset of the APIs I had to support were in service of NFS. This was… a mind-blowing realization. “Why would tmpfs, a memory file system, need NFS-specific glue?”, I thought, and then I learned the bad stuff.
Anyhow. Now that you know that NFSv3 exposes inodes, you may understand why sharing a directory over NFSv3 is an all-or-nothing option for the whole file system. Even if you can configure mountd
to export a single directory via /etc/exports
, malicious clients can craft NFS RPCs that reference inodes outside of the shared directory. Which means… they get free access to the whole file system and explains why system administrators used to put NFS-exported shares in separate partitions, mounted under /export/
, in an attempt to isolate access to subsets of data.
To make things worse, NFSv3 has no concept of security. A client can simply assert that a request comes from UID 1000 and the server will trust that the client is really operating on behalf of the server’s UID 1000. Which means: a malicious client can pretend to be any user that exists on the server and gain access to any file in the exported file system. Which then explains why the maproot
option exists as an attempt to avoid impersonating root… but only root. Crazy talk.
All in all, NFSv3 may still be OK if you really trust the network, if you compartmentalize the exported file system, and if you are sharing inconsequential stuff. But can you trust the network? Maybe you can if you are using a P2P link, but otherwise… it is really, really risky and I do not want to do that.
How is NFSv4 better?
NFSv4, despite having the same NFS name as NFSv3, is a completely different protocol. Here are two main differences:
NFSv4 operates on the basis of usernames, not UIDs. Each request to the server contains a username and the server is responsible for translating that username to a local UID while verifying access permissions.
NFSv4 operates at the path level, not the inode level. Each request to the server contains the path of the file to operate on and thus the server can apply access control based on those.
Take these two differences together and NFSv4 can implement secure access to file systems. Because the server sees usernames and paths, the server can first verify that a user is who they claim to be. And because the server can authenticate users, it can then authorize accesses at the path level.
That said, if all you have is NFSv4, you only get the AUTH_SYS
security level, which is… the same as having no security at all. In this mode, the server trusts the client and assumes that user X on the client maps exactly to user X on the server, which is almost the same as NFSv3 did.
The real security features of NFSv4 come into play when it’s paired with Kerberos. When Kerberos is in the picture, you get to choose from the following security levels for each network share:
krb5
: Requires requests to be authenticated by Kerberos, which is good to ensure only trusted users access what they should but offers zero “on-the-wire” security. Traffic flows unsigned and unencrypted, so an attacker could tamper with the data and slurp it before it reaches the client.krb5i
: Builds onkrb5
to offer integrity checks on all data. Basically, all packets on the network are signed but not encrypted. This prevents spoofing packets but does not secure the data against prying eyes.krb5p
: Builds onkrb5
to offer encrypted data on the wire. This prevents tampering with the network traffic and also avoids anyone from seeing what’s being transferred.
Sounds good? Yeah, but unfortunately, Kerberos and its ecosystem are… complicated.
Kerberos 101
Kerberos is an authentication broker. Its goal is to detach authentication decisions between a client machine and a service running on a second machine, and move that responsibility to a third machine—the Kerberos Domain Controller (KDC). Consequently, the KDC is a trusted entity between the two machines that try to communicate with each other.
All the machines that interact with the KDC form a realm (AKA a domain, but not a DNS domain). Each machine needs an /etc/krb5.conf
file that describes which realms the machine belongs to and who the KDC for each realm is.
The actors that exist within the realm are the principals. The KDC maintains the authoritative list of principals and their authentication keys (passwords). These principals represent:
Users, which have names of the form
<username>@REALM
. There has to be one of these principals for every person (or role) that interacts with the system.Machines, which have names of the form
host/<machine>.<domain>@REALM
. There has to be one of these principals for every server, and, depending on the service, the clients may need one too as is the case for NFSv4.Services, which have names of the form
<service>/<machine>.<domain>@REALM
. Some services like NFSv4 require one of these, in which case<service>
isnfs
, but others like SSH do not.
Let’s say Alice wants to log into the Kerberos-protected SSH service running on SshServer from a client called LinuxLaptop, all within the EXAMPLE.ORG
Kerberos realm.
(Beware that the description below is not 100% accurate. My goal is for you to understand the main concepts so that you can operate a Kerberos realm.)
First, Alice needs to obtain a Ticket-Granting-Ticket (TGT) if she doesn’t have a valid one yet. This ticket is issued by the KDC after authenticating Alice with her password, and allows Alice to later obtain service-specific tickets without having to provide her password again. For this flow:
Alice issues a login request to the KDC from the client LinuxLaptop by typing
kinit
(or using other tools such as a PAM module). This request carries Alice’s password.The KDC validates Alice’s authenticity by checking her password against the KDC’s database and issues a TGT. The TGT is encrypted with the KDC’s key and includes an assertion of who Alice is and how long the ticket is valid for.
The client LinuxLaptop stores the TGT on disk. Alice can issue
klist
to see the ticket:
linux-laptop$ klist
Credentials cache: FILE:/tmp/krb5cc_1001
Principal: alice@EXAMPLE.ORG
Issued Expires Principal
Oct 26 09:04:57 2024 Oct 26 19:05:01 2024 krbtgt/EXAMPLE.ORG@EXAMPLE.ORG
linux-laptop$ █
The TGT, however, is not sufficient to access a service. When Alice wants to access the Kerberos-protected SSH service running on the SshServer machine, Alice needs a ticket that’s specific to that service. For this flow:
Alice sends a request to the Ticket-Granting-Service (KDS) and asks for a ticket to SshServer. This request carries the TGT.
The TGS (which lives in the KDC) verifies who the TGT belongs to and verifies that it’s still valid. If so, the TGS generates a ticket for the service. This ticket is encrypted with the service’s secret key and includes details on who Alice is and how long the ticket is valid for.
The client LinuxLaptop stores the service ticket on disk. As before, Alice can issue
klist
to see the ticket:
linux-laptop$ klist
Credentials cache: FILE:/tmp/krb5cc_1001
Principal: alice@EXAMPLE.ORG
Issued Expires Principal
Oct 26 09:04:57 2024 Oct 26 19:05:01 2024 krbtgt/EXAMPLE.ORG@EXAMPLE.ORG
Oct 26 09:05:11 2024 Oct 26 19:05:01 2024 host/ssh-server.example.org@EXAMPLE.ORG
linux-laptop$ █
At this point, all prerequisite Kerberos flows have taken place. Alice can now initiate the connection to the SSH service:
Alice sends the login request from the LinuxLaptop client to the SshServer server and presents the service/host-specific ticket that was granted to her earlier on.
The SshServer server decrypts the ticket with its own key, extracts details of who the request is from, and verifies that they are correct. This happens without talking to the KDC and is only possible because SshServer trusts the KDC via a pre-shared key.
The SSH service on SshServer decides if Alice has SSH access as requested and, if so, grants such access.
Note these very important details:
The KDC is only involved in the ticket issuance process. Once the client has a service ticket, all interactions between the client and the server happen without talking to the KDC. This is essential to not make the KDC a bottleneck in the communication.
Each host/service and the KDC have unique shared keys that are known by both the host/service and the KDC. These shared keys are created when registering the host or service principals and are copied to the corresponding machines as part of their initial setup. These keys live in machine-specific
/etc/krb5.keytab
files.Kerberos does authentication only, not authorization. The decision to grant Alice access to the SSH service in Think is made by the service itself, not Kerberos, after asserting that Alice is truly Alice.
As you can imagine, the KDC must be protected with the utmost security measures. If an attacker can compromise the KDC’s locally-stored database, they will get access to all shared keys so they can impersonate any user against any Kerberos-protected service in the network. That’s why attackers try to breach into an Active Directory (AD) service as soon as they infiltrate a Microsoft network because… AD is a KDC.
Setting up the KDC
Enough theory. Let’s get our hands dirty and follow the necessary steps to set up a KDC.
The KDC’s needs are really modest. Per the discussion above, the KDC isn’t in the hot data path of any service so the number of requests it receives are limited. Those requests are not particularly complex to serve either: at most, there is some CPU time to process cryptographic material but no I/O involved, so for a small network, any machine will do.
In my particular case, I set up the KDC in my little pfSense box as it is guaranteed to be almost-always online. This is probably not the best of ideas security-wise, but… it’s sufficient for my paranoia levels. Note that most of the steps below will work similarly on a FreeBSD box, but if you are attempting that, please go read FreeBSD’s official docs on the topic instead. Those docs are one of the few decent guides on Kerberos out there.
Here are the actors that will appear throughout the rest of this article. I’m using the real names of my setup here because, once again, these are my lab notes:
MEROH.NET
: The name of the Kerberos realm.jmmv
: The user on the client machine wanting access to the NFSv4 share. The UID is irrelevant.router.meroh.net
: The pfSense box running the KDC.nas.meroh.net
: The Synology DS923+ NAS acting as the NFSv4 server.think.meroh.net
: A FreeBSD machine that will act as a Kerberized SSH server for testing purposes and an NFSv4 client. (It’s a ThinkStation, hence its name.)x1nano.meroh.net
: A Linux machine that will act as an NFSv4 client. While in reality this is running Fedora, I’ll use this hostname interchangeably for Fedora and Debian.
Knowing all actors, we can set up the KDC. The first step is to create the krb5.conf
for the KDC which tells the system which realm the machine belongs to. You’ll have to open up SSH access to the machine via the web interface to perform these steps.
Here is the minimum content you need:
[libdefaults]
default_realm = MEROH.NET
[realms]
MEROH.NET = {
kdc = router.meroh.net
admin_server = router.meroh.net
}
[domain_realm]
.meroh.net = MEROH.NET
With that, you should be able to start the kdc
service, which is responsible for the KDC. All documentation you find out there will tell you to also start kadmind
, but if you don’t plan to do administer the KDC from another machine (why would you?), then you don’t need this service.
pfSense’s configuration is weird because of the read-only nature of its root partition, so to do this, you have to edit the /cf/conf/config.xml
file stored in NVRAM and add this line right before the closing </system>
tag:
<shellcmd>service kdc start</shellcmd>
If you were to set this up on a FreeBSD host instead of pfSense, you would modify /etc/rc.conf
instead and add:
kdc_enable=YES
Then, from the root shell on either case:
kdc# service kdc start
kdc# █
It is now a good time to ensure that every machine involved in the realm has a DNS record and that reverse DNS lookups work. Failure to do this will cause problems later on when attempting to mount the NFSv4 shares, and clearing those errors won’t be trivial because of caching at various levels.
Creating principals
Once the KDC is running, we must create principals for the hosts, the NFSv4 service, and the users that will be part of the realm. The client host and service principals aren’t always necessary though: SSH doesn’t require them, but NFSv4 does.
To create the principals, we need access the KDC’s administrative console. Given that the KDC isn’t configured yet, we can only gain such access by running kadmin -l
on the KDC machine directly (the pfSense shell), which bypasses the networked kadmind
service that we did not start.
Start kadmin -l
and initialize the realm:
kdc# kadmin -l
kadmin> init MEROH.NET
... answer questions with defaults ...
kadmin> █
Next, create principals for the users that will be part of the realm:
kadmin> add jmmv
... answer questions with defaults ...
... but enter the desired user password ...
kadmin> █
Then, create principals for the hosts (server and clients, but not the KDC) and the NFSv4 service:
kadmin> add --random-key host/think.meroh.net
... answer questions with defaults ...
kadmin> add --random-key host/x1nano.meroh.net
... answer questions with defaults ...
kadmin> add --random-key host/nas.meroh.net
... answer questions with defaults ...
kadmin> add --random-key nfs/nas.meroh.net
... answer questions with defaults ...
kadmin> █
And finally, extract the host and service credentials into the machine-specific keytab files. Note that, for the servers, we extract both the host and any service principals they need, but for the client, we just extract the host principal. We do not export any user principals:
kadmin> ext_keytab --keytab=think.keytab host/think.meroh.net
kadmin> ext_keytab --keytab=x1nano.keytab host/x1nano.meroh.net
kadmin> ext_keytab --keytab=nas.keytab host/nas.meroh.net nfs/nas.meroh.net
kadmin> █
You now need to copy each extracted keytab file to the corresponding machine and name it /etc/krb5.keytab
. (We’ll do this later on the Synology NAS via its web interface.) This file is what contains the shared key between the KDC and the host and is what allows the host to verify the authenticity of KDC tickets without having to contact the KDC. Make sure to protect it with chmod 400 /etc/krb5.keytab
so that nobody other than root can read it.
If scp
is unsuitable or hard to use from the KDC to the client machines (as is my case because I restrict SSH access to the KDC to one specific machine), you can use the base64
command to print out a textual representation of the keytab and use the local clipboard to carry it to a shell session on the destination machine.
Initial client setup
At this point, the realm should be functional but we need to make the clients become part of the realm. We also need to install all necessary tools, like kinit
, which aren’t present by default on some systems:
On Debian:
Run
apt install krb5-user nfs-common
.Follow the prompts that the
krb5-user
installer shows to configure the realm and the address of the KDC. This will auto-create/etc/krb5.conf
with the right contents so you don’t have to do anything else.
On Fedora:
Run
dnf install krb5-workstation
.Edit the system-provided
/etc/krb5.conf
file to register the realm and its settings. Use the file content shown above for the KDC as the template, or simply replace all placeholders forexample.org
andEXAMPLE.ORG
with the name of your DNS domain and realm.
On FreeBSD:
Create the
/etc/krb5.conf
file from scratch in the same way we did for the KDC.
All set! But… do you trust that you did the right thing everywhere? We could go straight into NFSv4, but due to the many pitfalls in its setup, I’d suggest you verify your configuration using a simpler service like SSH.
To do this, modify the SSH server’s (aka think
’s configuration) /etc/ssh/sshd_config
file and add GSSAPIAuthentication yes
so that it can leverage Kerberos for authentication. Restart the SSH service and give it a go: run kinit
on the client (x1nano
) and then see how ssh think
works without typing a password anymore.
But… GSSAPIAuthentication
? What’s up with the cryptic name?
GSS-API stands for Generic Security Services API and is the interface that programs use to communicate with the Kerberos implementation on the machine. GSS-API is not always enabled by default for a service, and the way you enable it is service-dependent. As you saw above, all we had to do for SSH was modify the sshd_config
file… but for other services, you may need to take extra steps on the server and/or the client.
And, guess what, NFSv4 is weird on this topic. Not only we need service-specific principals for NFS, but we also need the gssd
daemon to be running on the server and the client machines. This is because NFSv4 is typically implemented inside the kernel, but not Kerberos, so the kernel needs a mechanism to “call into” Kerberos. And the kernel needs to do this to map kernel-level UIDs (a Unix kernel doesn’t know anything about usernames) to Kerberos principals and vice-versa—and that’s precisely what gssd
offers. So:
On the Synology NAS:
Do nothing. The system handles
gssd
by itself.
On Linux:
You shouldn’t have to do anything if you correctly created the prerequisite
/etc/krb5.keytab
early enough, but make sure the service is running withsystemctl status rpc-gssd.service
(and know that this command only shows useful diagnostic logs when run as root).Run
systemctl start rpc-gssd.service
if the service isn’t running.
On FreeBSD:
Add
gssd_enable=YES
to/etc/rc.conf
.Run
service gssd start
.
Exporting NFSv4 services
It’s time to deal with NFSv4, so let’s start by configuring the server on the NAS.
The Synology Disk Station Manager (DSM) interface—the web UI for the NAS—is… interesting. As you might expect, it is a web-based interface but… it pretends to be a desktop environment in the browser, which I find overkill and unnecessary. But it’s rather cool in its own way.
The first step is to enable the NFS service. Roughly follow these steps, which are illustrated in the picture just above:
Open the File Services tab of the Control Panel.
In the NFS tab, set NFSv4 as the Minimum NFS protocol.
Click on Advanced Settings and, in the panel that opens, enter the Kerberos realm under the NFSv4 domain option.
Click on Kerberos Settings and, in the panel that opens, select Import and then upload the keytab file that we generated earlier on for the NAS. This should populate the
host
andnfs
principals in the list.Finish and save all settings.
That should be all to enable NFSv4 file serving.
Then, we need to expose shared folders over NFSv4, and we have to do this for every folder we want to share. Assuming you want to share the homes
folder as shown in the picture just above:
Open the Shared Folder tab of the Control Panel.
Select the folder you want to share (in our case,
homes
), and click Edit.In the NFS Permissions tab, click either Create or Edit to enter the permissions for every host client that should have access to the share.
Fill the NFS rule details. In particular, enter the hostname of the client machine, enable the Allow connections from non-privileged ports option, and select the Security level you desire.
In my case, I want krb5p
and krb5p
only so that’s the only option I enable. But your risk profile and performance needs may be different, so experiment and see what works best for you.
Mounting NFSv4 from Linux
Now that the server is ready and we have dealt with the GSS-API prerequisites, we can start mounting NFSv4 on the clients.
On Linux, things are pretty simple. We can mount the file system with:
x1nano# sudo mount nas:/volume1/homes /shared
x1nano# █
Or persist the entry in /etc/fstab
if we want to:
nas:/volume1/homes /shared nfs sec=krb5p 0 0
And then we should be able to list its contents assuming we’ve got a valid TGT for the current user (run kinit
if it doesn’t work):
x1nano$ ls -l /shared
total 0K
drwxrwxrwx 1 nobody users 0 Sep 27 21:11 admin
drwxrwxrwx 1 jmmv users 0 Nov 2 20:41 jmmv
drwxrwxrwx 1 nobody users 0 Oct 8 16:59 manager
x1nano$ █
Easy peasy, right? But wait… why do all directories have 777
permissions?
This is rather unfortunate and I’m not sure why the Synology does this. Logging onto the DS923+ via SSH, I inspected the shared directory and realized that it has various ACLs in place to control access to the directories, but somehow, the traditional Unix permissions are all 777
indeed. Not great.
I used chmod
to fix the permissions for all directories to 755
and things seem to be OK, but that doesn’t give me a lot of comfort because I do not know if the DSM will ever undo my changes or if I might have broken something.
There might be one more problem though, which I did not encounter on Debian clients but that showed up later in Fedora and FreeBSD clients:
x1nano$ ls -l /shared
total 0K
drwxr-xr-x 1 nobody nogroup 0 Sep 27 21:11 admin
drwxr-xr-x 1 nobody nogroup 0 Nov 2 20:41 jmmv
drwxr-xr-x 1 nobody nogroup 0 Oct 8 16:59 manager
x1nano$ █
Note how all entries are owned by nobody:nogroup
which is… not correct. Yet the right permissions are in effect: accessing the jmmv
directory is only possible by the jmmv
user as expected. Which means that the user mapping between Kerberos principals and local users is working correctly on the server… but not on the client, where stat
isn’t returning the right information.
I do not yet know why this issue happens, especially because I see no material differences between my Fedora and Debian configurations.
Mounting NFSv4 from FreeBSD
We now have the Linux clients running just fine so it is time to pivot to FreeBSD. If we try a similar “trivial” mount command, we get an error:
think# mount -t nfs nas:/volume1/homes /shared
mount_nfs: nmount: /shared: Permission denied
think# █
The error is pretty… unspecific. It took me quite a bit of trial and error to realize that I had to specify -t nfsv4
for it to attempt a NFSv4 connection and not NFSv3 (unlike Linux, whose mount
command attempts the highest possible version first and then falls back to older versions):
think# mount -t nfs -o nfsv4 nas:/volume1/homes /shared
mount_nfs: nmount: /shared, wrong security flavor
think# █
OK, progress. Now this complains that the security flavor we request is wrong. Maybe we just need to be explicit and also pass sec=krb5p
as an argument:
think# mount -t nfs -o nfsv4,sec=krb5p nas:/volume1/homes /shared
mount_nfs: nmount: /shared, wrong security flavor
think# █
Wait, what? The mount operation still fails? This was more puzzling and also took a fair bit of research to figure out because logs on the client and on the server were just insufficient to see the problem.
The reason for the failure is that we are trying to mount the share as root but… we don’t have a principal for this user so root cannot obtain an NFSv4 service ticket to contact the NAS. So… do we need to create a principal for root
? No! We do not need to provide user credentials when mounting an NFSv4 share (unlike what you might be used to with Windows shares).
What Kerberized NFSv4 needs during the mount operation is a host ticket: the NFSv4 server checks if the client machine is allowed to access the server and, if so, exposes the file system to it. This is done using the client’s host principal. Once the file system is mounted, however, all operations against the share carry the ticket of the user requesting the operation.
Knowing this, we need to “help” FreeBSD and tell it that it must use the host’s principal when mounting the share. Why this isn’t the default, I don’t know, particularly because non-root users are not allowed to mount file systems in the default configuration. Anyhow. The gssname=host
option rescues us:
think# mount -t nfs -o nfsv4,sec=krb5p,gssname=host nas:/volume1/homes /shared
think# █
Which finally allows the mount operation to succeed. We should persist all this knowledge into an /etc/fstab
entry like this one:
nas:/volume1/homes /shared nfs rw,nfsv4,gssname=host,sec=krb5p 0 0
Verifying encryption
Color me skeptical, but everything I described above seems convoluted and fragile, so I did not trust that my setup was sound. Consequently, I wanted to verify that the traffic on the network was actually encrypted.
To verify this, I installed Wireshark and ran a traffic capture against the NAS with host nas
as the filter. Then, from the client, I created a text file on the shared folder and then read it. Inspecting the captured packets confirmed that the traffic is indeed flowing in encrypted form. I could not find the raw file content anywhere in the whole trace (but I could when using anything other than krb5p
).
And, as a final test, I tried to mount the network share without krb5p
and confirmed that this was not possible:
# mount -t nfs -o nfsv4,gssname=host,sec=krb5i nas:/volume1/homes /shared
mount_nfs: nmount: /shared, wrong security flavor
# █
All good! I think…
Open questions
That’s about it. But I still have a bunch of unanswered questions from this setup:
Kerberos claims to be an authentication system only, not an authorization system. However, the protocol I described above separates the TGT from the TGS, and this separation makes it sound like Kerberos could also implement authorization policies. Why doesn’t it do these?
The fact that Fedora and FreeBSD show
nobody
for file ownership even when they seems to do the right thing when talking to the NFSv4 server sound like a bug either in the code or in my configuration. Which is it?Having to type
kinit
after logging into the machine is annoying. I remember that, back at Google when we used Kerberos and NFS—those are long gone days—the right tickets would be granted after logging in or unlocking a workstation. This must have been done with the Kerberos PAM modules… but I haven’t gotten them to do this yet and I’m not sure why.The fact that the shared directories created by the Synology NAS have 777 permissions seems wrong. Why is it doing that? And does anything break if you manually tighten these permissions?
And the most important question of all: is this all worth it? I’m tempted to just use password-protected Samba shares and call it a day. I still don't trust that the setup is correct, and I still encounter occasional problems here and there.
If you happen to have answers to any of the above or have further thoughts, please drop a note in the comments section. And…
Credit and disclaimers: the DS923+ and the 3 drives it contains that I used for throughout this article were provided to me for free by Synology for evaluation purposes in exchange for blogging about the NAS. The content in this article is not endorsed has not been reviewed by them.
> The fact that Fedora and FreeBSD show nobody for file ownership even when they seems to do the right thing when talking to the NFSv4 server
At least in my production Kerberized NFSv4 environment, this indicates rpc.idmapd isn't running. You will need it on _both_ the server and client. A simple "Domain = your.domain.here", "Local-Realms = your.domain.here" will suffice. Older kernels require a reboot before this config takes effect.
Re: files that are misattributed to nobody/nogroup, I've seen this before (on Debian, as it happens).
The problem came from an unexpected source: /etc/hosts. Originally my machines each had a hosts entry such as:
a.b.c.d myhost myhost.mydomain
But I learned that uid/gid resolution on the NFS client worked only if the qualified name comes first:
a.b.c.d myhost.mydomain myhost
Unfortunately it's been a while since this discovery, so I don't have a reference to whatever docs/bugs/etc. clued me in to this. Hope it works for you!