| Message ID | 20220110184419.27665-1-richard@nod.at (mailing list archive) |
|---|---|
| Headers | show |
| Series | Dealing with NFS re-export and cross mounts | expand |
On Mon, Jan 10, 2022 at 07:44:16PM +0100, Richard Weinberger wrote: > Currently when re-exporting a NFS share the NFS cross mount feature does > not work [0]. > This RFC patch series outlines an approach to address the problem. > > Crossing mounts does not work for two reasons: > > 1. As soon the NFS client (on the re-exporting server) sees a different > filesystem id, it installs an automount. That way the other filesystem > will be mounted automatically when someone enters the directory. > But the cross mount logic of KNFS does not know about automount. > The three patches in this series address the problem and teach both KNFSD > and the exportfs logic of NFS to deal with automount. > > 2. When KNFSD detects crossing of a mount point, it asks rpc.mountd to install > a new export for the target mount point. Beside of authentication rpc.mountd > also has to find a filesystem id for the new export. Is the to be exported > filesystem a NFS share, rpc.mountd cannot derive a filesystem id from it and > refuses to export. In the logs you’ll see error such as: > mountd: Cannot export /srv/nfs/vol0, possibly unsupported filesystem or fsid= required > To deal with that I changed rpc.mountd to use an arbitrary fsid. > Since this is a gross hack we need to agree on an approach to derive filesystem > ids for NFS mounts. The toughest problem to deal with is reboot of the re-export server. If you want this to work across reboots, then you need to pick an fsid that will be the same across reboots. Also, you need to deal with getting an fsid for a filesystem that isn't mounted yet. That's because, if you reboot while a client is using /srv/nfs/vol0, when you come back up, the client *isn't* going to look up the path /srv/nfs/vol0 again--it's just going to give you a filehandle for some object under there, and you're going to have to figure out what to do with that. Simplest might be recording the fsid's you use in an on-disk database. knfsd makes an upcall to rpc.mountd each time it encounters a new fsid, so maybe that'd mean you could do all the management of that database in rpc.mountd and minimize required kernel patches. Maybe a last-resort option would be just to not support reboot of the re-export server. That's already what we do for locking. I'm not happy about that, and have some vague ideas how it might be fixed, but not anything that's likely to be done soon. Then I think a random fsid might be OK. I believe fsids can be up to 32 bits so there's effectively no chance of collisions. But, I can't remember, can those nfs automounts expire? An export that looks idle from the server's point of view might still be in use by a client, so we can't drop that mount and then get stuck returning ESTALE when the client does eventually try to use it. --b. > > rpc.mountd could: > a) re-use the fsid from the original NFS server. > Beside of requesting this information, the problem with that approach is > that the original fsid might conflict with an existing export. > b) derive the fsid from stat->st_dev. > c) allocate a free fsid. > > One use case to consider is load balancing. When multiple NFS servers re-export > a NFS mount, they need to use the same fsid for crossed mounts. > So I'm a little puzzled which approach is best. What do you think? > > Known issues: > - Only tested with NFSv3 (both server and client) so far. > > [0] https://marc.info/?l=linux-nfs&m=161653016627277&w=2 > > Richard Weinberger (3): > NFSD: Teach nfsd_mountpoint() auto mounts > fs: namei: Allow follow_down() to uncover auto mounts > NFS: nfs_encode_fh: Remove S_AUTOMOUNT check > > fs/namei.c | 2 +- > fs/nfs/export.c | 5 ----- > fs/nfsd/vfs.c | 2 +- > 3 files changed, 2 insertions(+), 7 deletions(-) > > -- > 2.26.2
On Tue, Jan 11, 2022 at 02:43:37PM -0500, J. Bruce Fields wrote: > On Mon, Jan 10, 2022 at 07:44:16PM +0100, Richard Weinberger wrote: > > rpc.mountd could: > > a) re-use the fsid from the original NFS server. > > Beside of requesting this information, the problem with that approach is > > that the original fsid might conflict with an existing export. > > b) derive the fsid from stat->st_dev. > > c) allocate a free fsid. > > > > One use case to consider is load balancing. When multiple NFS servers re-export > > a NFS mount, they need to use the same fsid for crossed mounts. I guess if rpc.mountd kept an on-disk database of fsid's, it wouldn't be too big a deal to later enhance that with the option of a distributed database. So I'm leaning towards picking a random fsid and sticking it in a database. When you encouter a new filesystem you'd need to make sure the addition of a new entry is atomic and persistent before returning to knfsd. It'd be nice if mountd had an easy way to query the on-the-wire fsid from userspace, and then you could index entries on the fsid. Absent that, maybe just indexing on server and path would be good enough. I'm not sure how NFS's st_dev's are generated. I think they might depend on stuff that isn't necessarily the same on each boot (like the order the NFS filesystems were mounted in), so they wouldn't work. --b. > > So I'm a little puzzled which approach is best. What do you think? > > > > Known issues: > > - Only tested with NFSv3 (both server and client) so far. > > > > [0] https://marc.info/?l=linux-nfs&m=161653016627277&w=2 > > > > Richard Weinberger (3): > > NFSD: Teach nfsd_mountpoint() auto mounts > > fs: namei: Allow follow_down() to uncover auto mounts > > NFS: nfs_encode_fh: Remove S_AUTOMOUNT check > > > > fs/namei.c | 2 +- > > fs/nfs/export.c | 5 ----- > > fs/nfsd/vfs.c | 2 +- > > 3 files changed, 2 insertions(+), 7 deletions(-) > > > > -- > > 2.26.2
On Mon, Jan 10, 2022 at 07:44:16PM +0100, Richard Weinberger wrote: > Currently when re-exporting a NFS share the NFS cross mount feature does > not work [0]. > This RFC patch series outlines an approach to address the problem. > > Crossing mounts does not work for two reasons: > > 1. As soon the NFS client (on the re-exporting server) sees a different > filesystem id, it installs an automount. That way the other filesystem > will be mounted automatically when someone enters the directory. > But the cross mount logic of KNFS does not know about automount. > The three patches in this series address the problem and teach both KNFSD > and the exportfs logic of NFS to deal with automount. > > 2. When KNFSD detects crossing of a mount point, it asks rpc.mountd to install > a new export for the target mount point. Beside of authentication rpc.mountd > also has to find a filesystem id for the new export. Is the to be exported > filesystem a NFS share, rpc.mountd cannot derive a filesystem id from it and > refuses to export. In the logs you’ll see error such as: > mountd: Cannot export /srv/nfs/vol0, possibly unsupported filesystem or fsid= required > To deal with that I changed rpc.mountd to use an arbitrary fsid. > Since this is a gross hack we need to agree on an approach to derive filesystem > ids for NFS mounts. > > rpc.mountd could: > a) re-use the fsid from the original NFS server. > Beside of requesting this information, the problem with that approach is > that the original fsid might conflict with an existing export. > b) derive the fsid from stat->st_dev. > c) allocate a free fsid. > > One use case to consider is load balancing. When multiple NFS servers re-export > a NFS mount, they need to use the same fsid for crossed mounts. > So I'm a little puzzled which approach is best. What do you think? > > Known issues: > - Only tested with NFSv3 (both server and client) so far. > > [0] https://marc.info/?l=linux-nfs&m=161653016627277&w=2 v4 testing would definitely be good, that's the case we'll care most about. --b.