diff mbox

nfs.man: document incompatibility between "bg" option and systemd.

Message ID 87tw43fgrf.fsf@notabene.neil.brown.name (mailing list archive)
State New, archived
Headers show

Commit Message

NeilBrown May 29, 2017, 10:19 p.m. UTC
Systemd does not, and will not, support "bg" correctly.
It has other, better, ways to handle "background" mounting.

Explain this.

See also https://github.com/systemd/systemd/issues/6046

Signed-off-by: NeilBrown <neilb@suse.com>
---
 utils/mount/nfs.man | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

Comments

Niels de Vos May 30, 2017, 4:47 a.m. UTC | #1
On Tue, May 30, 2017 at 08:19:16AM +1000, NeilBrown wrote:
> 
> Systemd does not, and will not, support "bg" correctly.
> It has other, better, ways to handle "background" mounting.
> 
> Explain this.
> 
> See also https://github.com/systemd/systemd/issues/6046
> 
> Signed-off-by: NeilBrown <neilb@suse.com>
> ---
>  utils/mount/nfs.man | 18 +++++++++++++++++-
>  1 file changed, 17 insertions(+), 1 deletion(-)
> 
> diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
> index cc6e992ed807..7e76492d454f 100644
> --- a/utils/mount/nfs.man
> +++ b/utils/mount/nfs.man
> @@ -372,6 +372,21 @@ Alternatively these issues can be addressed
>  using an automounter (refer to
>  .BR automount (8)
>  for details).
> +.IP
> +When
> +.B systemd
> +is used to mount the filesystems listed in
> +.IR /etc/fstab ,
> +the
> +.B bg
> +option is not supported, and may be stripped from the option list.
> +Similar functionality can be achieved by providing the
> +.B x-system.automount
> +option.  This will cause
> +.B systemd
> +to attempt to mount the filesystem when the mountpoint is first
> +accessed, rather than during system boot.  The mount still happens in
> +the "background", though in a different way.
>  .TP 1.5i
>  .BR rdirplus " / " nordirplus
>  Selects whether to use NFS v3 or v4 READDIRPLUS requests.
> @@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
>  .BR rpc.idmapd (8),
>  .BR rpc.gssd (8),
>  .BR rpc.svcgssd (8),
> -.BR kerberos (1)
> +.BR kerberos (1),
> +.BR systemd.mount (5) .
>  .sp
>  RFC 768 for the UDP specification.
>  .br
> -- 
> 2.12.2
> 

I like this, it makes it so much easier for users to find the better
alternative.

FWIW,
Reviewed-by: Niels de Vos <ndevos@redha.com>

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Michael Biebl May 30, 2017, 7:40 a.m. UTC | #2
2017-05-30 0:19 GMT+02:00 NeilBrown <neilb@suse.com>:

> +.B bg
> +option is not supported, and may be stripped from the option list.

Either systemd is updated to actually strip the bg option or not. The
documentation should reflect that.
I don't think we should be vague about this, as it would only be confusing.

Michael
NeilBrown May 30, 2017, 8:55 a.m. UTC | #3
On Tue, May 30 2017, Michael Biebl wrote:

> 2017-05-30 0:19 GMT+02:00 NeilBrown <neilb@suse.com>:
>
>> +.B bg
>> +option is not supported, and may be stripped from the option list.
>
> Either systemd is updated to actually strip the bg option or not. The
> documentation should reflect that.
> I don't think we should be vague about this, as it would only be confusing.

It depends on which version of systemd is in use.
Even if/when systemd is updated to strip the 'bg', it would not
be correct to say "and will be stripped..." as that isn't true for all
versions.
We could spell it out "and may be stripped from the option list,
depending on which version of systemd is installed", but I'm not sure
that really helps...
 "... and some versions of systemd will strip 'bg' from the option list"
 ??

Thanks,
NeilBrown
Michael Biebl May 30, 2017, 9:15 a.m. UTC | #4
2017-05-30 10:55 GMT+02:00 NeilBrown <neilb@suse.com>:
> On Tue, May 30 2017, Michael Biebl wrote:
>
>> 2017-05-30 0:19 GMT+02:00 NeilBrown <neilb@suse.com>:
>>
>>> +.B bg
>>> +option is not supported, and may be stripped from the option list.
>>
>> Either systemd is updated to actually strip the bg option or not. The
>> documentation should reflect that.
>> I don't think we should be vague about this, as it would only be confusing.
>
> It depends on which version of systemd is in use.
> Even if/when systemd is updated to strip the 'bg', it would not
> be correct to say "and will be stripped..." as that isn't true for all
> versions.
> We could spell it out "and may be stripped from the option list,
> depending on which version of systemd is installed", but I'm not sure
> that really helps...
>  "... and some versions of systemd will strip 'bg' from the option list"
>  ??

So far, no version of systemd strips bg from the option list.
If in version X systemd actually starts stripping bg, we should
explicitly mention that version in the man page imho.
Lennart Poettering May 30, 2017, 12:43 p.m. UTC | #5
On Tue, 30.05.17 08:19, NeilBrown (neilb@suse.com) wrote:

> 
> Systemd does not, and will not, support "bg" correctly.
> It has other, better, ways to handle "background" mounting.
> 
> Explain this.
> 
> See also https://github.com/systemd/systemd/issues/6046
> 
> Signed-off-by: NeilBrown <neilb@suse.com>
> ---
>  utils/mount/nfs.man | 18 +++++++++++++++++-
>  1 file changed, 17 insertions(+), 1 deletion(-)
> 
> diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
> index cc6e992ed807..7e76492d454f 100644
> --- a/utils/mount/nfs.man
> +++ b/utils/mount/nfs.man
> @@ -372,6 +372,21 @@ Alternatively these issues can be addressed
>  using an automounter (refer to
>  .BR automount (8)
>  for details).
> +.IP
> +When
> +.B systemd
> +is used to mount the filesystems listed in
> +.IR /etc/fstab ,
> +the
> +.B bg
> +option is not supported, and may be stripped from the option list.
> +Similar functionality can be achieved by providing the
> +.B x-system.automount

The option is called "x-systemd.automount", i.e. one more "d".

> +option.  This will cause
> +.B systemd
> +to attempt to mount the filesystem when the mountpoint is first
> +accessed, rather than during system boot.  The mount still happens in
> +the "background", though in a different way.

Looks good to me otherwise!

Lennart
Lennart Poettering May 30, 2017, 12:45 p.m. UTC | #6
On Tue, 30.05.17 11:15, Michael Biebl (mbiebl@gmail.com) wrote:

> 2017-05-30 10:55 GMT+02:00 NeilBrown <neilb@suse.com>:
> > On Tue, May 30 2017, Michael Biebl wrote:
> >
> >> 2017-05-30 0:19 GMT+02:00 NeilBrown <neilb@suse.com>:
> >>
> >>> +.B bg
> >>> +option is not supported, and may be stripped from the option list.
> >>
> >> Either systemd is updated to actually strip the bg option or not. The
> >> documentation should reflect that.
> >> I don't think we should be vague about this, as it would only be confusing.
> >
> > It depends on which version of systemd is in use.
> > Even if/when systemd is updated to strip the 'bg', it would not
> > be correct to say "and will be stripped..." as that isn't true for all
> > versions.
> > We could spell it out "and may be stripped from the option list,
> > depending on which version of systemd is installed", but I'm not sure
> > that really helps...
> >  "... and some versions of systemd will strip 'bg' from the option list"
> >  ??
> 
> So far, no version of systemd strips bg from the option list.
> If in version X systemd actually starts stripping bg, we should
> explicitly mention that version in the man page imho.

I figure it's up to the NFS folks to decide on this, but if I were
them I would not mention any version numbers. That just gets out of
date. Instead, document the new behaviour and indicate in the README
file that the documentation assumes systemd with a certain version is
used, and then let the packers figure out the rest, i.e. if they want
to place some kind of weak package version dependency or not.

Lennart
Steve Dickson June 6, 2017, 6:07 p.m. UTC | #7
Hello,

On 05/29/2017 06:19 PM, NeilBrown wrote:
> 
> Systemd does not, and will not, support "bg" correctly.
> It has other, better, ways to handle "background" mounting.
The only problem with this is bg mounts still work at least
up to 4.11 kernel... 

It appears there is a problem with a 4.12 kernel. The mount no 
longer errors out with ECONNREFUSED it just hangs in the 
kernel trying forever... It sounds like a bug to me but 
maybe that change was intentional.. Anna?? Trond???

So I'm a bit hesitant to commit this since not accurate, yet.

Finally, the whole idea of systemd randomly/silently 
strip off mount options is crazy... IMHO... 

Just because a concept that has been around for years
does not fix well in the systemd world it gets
rip out??? IDK... but I think we can do better than that.

Note, the 'bg' is used by clients that do want their
booting to hang by servers that are down so if the 
option is rip out, boots will start hang. This
will make it very difficult to debug since the bg
will still exist in fstab.  

Again, the whole concept of systemd messing with mounts options
is just not a good one... IMHO.. 

steved.

> 
> Explain this.
> 
> See also https://github.com/systemd/systemd/issues/6046
> 
> Signed-off-by: NeilBrown <neilb@suse.com>
> ---
>  utils/mount/nfs.man | 18 +++++++++++++++++-
>  1 file changed, 17 insertions(+), 1 deletion(-)
> 
> diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
> index cc6e992ed807..7e76492d454f 100644
> --- a/utils/mount/nfs.man
> +++ b/utils/mount/nfs.man
> @@ -372,6 +372,21 @@ Alternatively these issues can be addressed
>  using an automounter (refer to
>  .BR automount (8)
>  for details).
> +.IP
> +When
> +.B systemd
> +is used to mount the filesystems listed in
> +.IR /etc/fstab ,
> +the
> +.B bg
> +option is not supported, and may be stripped from the option list.
> +Similar functionality can be achieved by providing the
> +.B x-system.automount
> +option.  This will cause
> +.B systemd
> +to attempt to mount the filesystem when the mountpoint is first
> +accessed, rather than during system boot.  The mount still happens in
> +the "background", though in a different way.
>  .TP 1.5i
>  .BR rdirplus " / " nordirplus
>  Selects whether to use NFS v3 or v4 READDIRPLUS requests.
> @@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
>  .BR rpc.idmapd (8),
>  .BR rpc.gssd (8),
>  .BR rpc.svcgssd (8),
> -.BR kerberos (1)
> +.BR kerberos (1),
> +.BR systemd.mount (5) .
>  .sp
>  RFC 768 for the UDP specification.
>  .br
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Michael Biebl June 6, 2017, 7:57 p.m. UTC | #8
2017-06-06 20:07 GMT+02:00 Steve Dickson <SteveD@redhat.com>:
> Finally, the whole idea of systemd randomly/silently
> strip off mount options is crazy... IMHO...

Personally, I would prefer if systemd simply logged a warning/error
message but would *not* strip the bg option.
NeilBrown June 6, 2017, 9:49 p.m. UTC | #9
On Tue, Jun 06 2017, Steve Dickson wrote:

> Hello,
>
> On 05/29/2017 06:19 PM, NeilBrown wrote:
>> 
>> Systemd does not, and will not, support "bg" correctly.
>> It has other, better, ways to handle "background" mounting.
> The only problem with this is bg mounts still work at least
> up to 4.11 kernel... 

I don't think this is correct.
In the default configuration, "mount -t nfs -o bg ...."
runs for longer than 90 seconds, so systemd kill it.

A working "bg" mount would successfully mount the filesystem is the
server came back after 5 minutes. If you use current systemd in the
default configuration, it won't.

bg mounts do work sometimes, but I don't think they are reliable, and
there seems to be no interest in changing this.
Maybe the text should say "Systemd does not, and will not, reliably
support "bg" mounts...".


>
> It appears there is a problem with a 4.12 kernel. The mount no 
> longer errors out with ECONNREFUSED it just hangs in the 
> kernel trying forever... It sounds like a bug to me but 
> maybe that change was intentional.. Anna?? Trond???

Might be related to my patch
  [PATCH] SUNRPC: ensure correct error is reported by xs_tcp_setup_socket()

sent 25th May.

>
> So I'm a bit hesitant to commit this since not accurate, yet.
>
> Finally, the whole idea of systemd randomly/silently 
> strip off mount options is crazy... IMHO... 

It isn't exactly systemd, it is systemd-fstab-generator.
The options lists in /etc/fstab are not all equal.  Some
are relevant to /bin/mount, some to mount.nfs, some to the kernel.
I think /bin/mount processes the option lists before passing it
to mount.nfs.  There is no intrinsic reason that systemd-fstab-generator
shouldn't do the same thing.

>
> Just because a concept that has been around for years
> does not fix well in the systemd world it gets
> rip out??? IDK... but I think we can do better than that.

I could suggest that automount is uniformly better than bg.  Give how
long automount has been around, and how easy it is to use with systemd,
it might be time to start encouraging people to stop using the inferior
technology.

I could say that, but I'm not 100% sure.  The difference between
automount and bg is that with bg it is easy to see if the mount has
succeeded yet - just look for an empty directory.  With automount,
you'll typically get a delay at that point.  We could possibly wind down
that delay...

>
> Note, the 'bg' is used by clients that do want their
> booting to hang by servers that are down so if the 
> option is rip out, boots will start hang. This
> will make it very difficult to debug since the bg
> will still exist in fstab.

Not exactly.
Current default behaviour is that systemd will wait 90 seconds, then
kill the mount program and fail the boot.  If we strip out "bg", the
same thing will happen.

I'm OK with the patch not being applied just yet.  I think we need to
resolve this issue, but it isn't 100% clear to me what the best
resolution would be.  So I'm happy to see a conversation happening and
opinions being shared.
I'd be particularly pleased if you could double check how "bg" is
currently handled on some systemd-enabled system that you use.
Does the mount program get killed like I see?  Does boot succeed if
there is a bg mount from an unresponsive server?

Thanks,
NeilBrown


>
> Again, the whole concept of systemd messing with mounts options
> is just not a good one... IMHO.. 
>
> steved.
>
>> 
>> Explain this.
>> 
>> See also https://github.com/systemd/systemd/issues/6046
>> 
>> Signed-off-by: NeilBrown <neilb@suse.com>
>> ---
>>  utils/mount/nfs.man | 18 +++++++++++++++++-
>>  1 file changed, 17 insertions(+), 1 deletion(-)
>> 
>> diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
>> index cc6e992ed807..7e76492d454f 100644
>> --- a/utils/mount/nfs.man
>> +++ b/utils/mount/nfs.man
>> @@ -372,6 +372,21 @@ Alternatively these issues can be addressed
>>  using an automounter (refer to
>>  .BR automount (8)
>>  for details).
>> +.IP
>> +When
>> +.B systemd
>> +is used to mount the filesystems listed in
>> +.IR /etc/fstab ,
>> +the
>> +.B bg
>> +option is not supported, and may be stripped from the option list.
>> +Similar functionality can be achieved by providing the
>> +.B x-system.automount
>> +option.  This will cause
>> +.B systemd
>> +to attempt to mount the filesystem when the mountpoint is first
>> +accessed, rather than during system boot.  The mount still happens in
>> +the "background", though in a different way.
>>  .TP 1.5i
>>  .BR rdirplus " / " nordirplus
>>  Selects whether to use NFS v3 or v4 READDIRPLUS requests.
>> @@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
>>  .BR rpc.idmapd (8),
>>  .BR rpc.gssd (8),
>>  .BR rpc.svcgssd (8),
>> -.BR kerberos (1)
>> +.BR kerberos (1),
>> +.BR systemd.mount (5) .
>>  .sp
>>  RFC 768 for the UDP specification.
>>  .br
>>
Lennart Poettering June 7, 2017, 8:12 a.m. UTC | #10
On Tue, 06.06.17 14:07, Steve Dickson (SteveD@RedHat.com) wrote:

> Hello,
> 
> On 05/29/2017 06:19 PM, NeilBrown wrote:
> > 
> > Systemd does not, and will not, support "bg" correctly.
> > It has other, better, ways to handle "background" mounting.
> The only problem with this is bg mounts still work at least
> up to 4.11 kernel... 
> 
> It appears there is a problem with a 4.12 kernel. The mount no 
> longer errors out with ECONNREFUSED it just hangs in the 
> kernel trying forever... It sounds like a bug to me but 
> maybe that change was intentional.. Anna?? Trond???
> 
> So I'm a bit hesitant to commit this since not accurate, yet.
> 
> Finally, the whole idea of systemd randomly/silently 
> strip off mount options is crazy... IMHO... 
> 
> Just because a concept that has been around for years
> does not fix well in the systemd world it gets
> rip out??? IDK... but I think we can do better than that.

Well "bg" doesn't really work on systemd systems, and this was never
different, hence I think it's only fair to document that it is
incompatible with systemd. In addition, I have the suspicion it is not
used very widely, since I never actually got complaints about it.

Lennart
Lennart Poettering June 7, 2017, 8:13 a.m. UTC | #11
On Tue, 06.06.17 21:57, Michael Biebl (mbiebl@gmail.com) wrote:

> 2017-06-06 20:07 GMT+02:00 Steve Dickson <SteveD@redhat.com>:
> > Finally, the whole idea of systemd randomly/silently
> > strip off mount options is crazy... IMHO...
> 
> Personally, I would prefer if systemd simply logged a warning/error
> message but would *not* strip the bg option.

What good does that do if "bg" doesn't actually work properly?

Lennart
Steve Dickson June 7, 2017, 9:42 a.m. UTC | #12
On 06/07/2017 04:13 AM, Lennart Poettering wrote:
> On Tue, 06.06.17 21:57, Michael Biebl (mbiebl@gmail.com) wrote:
> 
>> 2017-06-06 20:07 GMT+02:00 Steve Dickson <SteveD@redhat.com>:
>>> Finally, the whole idea of systemd randomly/silently
>>> strip off mount options is crazy... IMHO...
>>
>> Personally, I would prefer if systemd simply logged a warning/error
>> message but would *not* strip the bg option.
> 
> What good does that do if "bg" doesn't actually work properly?
At the moment it works just fine... 

steved.

> 
> Lennart
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Steve Dickson June 7, 2017, 10:04 a.m. UTC | #13
On 06/07/2017 04:12 AM, Lennart Poettering wrote:
> On Tue, 06.06.17 14:07, Steve Dickson (SteveD@RedHat.com) wrote:
> 
>> Hello,
>>
>> On 05/29/2017 06:19 PM, NeilBrown wrote:
>>>
>>> Systemd does not, and will not, support "bg" correctly.
>>> It has other, better, ways to handle "background" mounting.
>> The only problem with this is bg mounts still work at least
>> up to 4.11 kernel... 
>>
>> It appears there is a problem with a 4.12 kernel. The mount no 
>> longer errors out with ECONNREFUSED it just hangs in the 
>> kernel trying forever... It sounds like a bug to me but 
>> maybe that change was intentional.. Anna?? Trond???
>>
>> So I'm a bit hesitant to commit this since not accurate, yet.
>>
>> Finally, the whole idea of systemd randomly/silently 
>> strip off mount options is crazy... IMHO... 
>>
>> Just because a concept that has been around for years
>> does not fix well in the systemd world it gets
>> rip out??? IDK... but I think we can do better than that.
> 
> Well "bg" doesn't really work on systemd systems, and this was never
> different, hence I think it's only fair to document that it is
> incompatible with systemd. 
I'm seeing it work just fine... 

/etc/fstab: nfssrv:/home/tmp /mnt/tmp nfs bg 0 0
nfssrv is down 
reboot client 
login to client 
# ps ax | grep mount
  980 ?        Ss     0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg


> In addition, I have the suspicion it is not
> used very widely, since I never actually got complaints about it.
Since it seems to be still working we probably would not hear
any complaints about it... 

steved.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Steve Dickson June 7, 2017, 10:08 a.m. UTC | #14
On 06/06/2017 05:49 PM, NeilBrown wrote:
> On Tue, Jun 06 2017, Steve Dickson wrote:
> 
>> Hello,
>>
>> On 05/29/2017 06:19 PM, NeilBrown wrote:
>>>
>>> Systemd does not, and will not, support "bg" correctly.
>>> It has other, better, ways to handle "background" mounting.
>> The only problem with this is bg mounts still work at least
>> up to 4.11 kernel... 
> 
> I don't think this is correct.
> In the default configuration, "mount -t nfs -o bg ...."
> runs for longer than 90 seconds, so systemd kill it.
I must be missing something... it seems to be working for me

# mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
mount.nfs: mount(2): Connection refused
mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
mount.nfs: backgrounding "rhel7srv:/home/tmp"
mount.nfs: mount options: "rw,bg"

# ps ax | grep mount.nfs
 2270 ?        Ss     0:00 /sbin/mount.nfs rhel7srv:/home/tmp /mnt/tmp -v -o rw,bg

> 
> A working "bg" mount would successfully mount the filesystem is the
> server came back after 5 minutes. If you use current systemd in the
> default configuration, it won't.
When I add a bg mount to fstab again... it works just fine. This
is with the latest upstream nfs-utils. 

> 
> bg mounts do work sometimes, but I don't think they are reliable, and
> there seems to be no interest in changing this.
> Maybe the text should say "Systemd does not, and will not, reliably
> support "bg" mounts...".
not reliable maybe... I'm just doing very simple testing... 
> 
> 
>>
>> It appears there is a problem with a 4.12 kernel. The mount no 
>> longer errors out with ECONNREFUSED it just hangs in the 
>> kernel trying forever... It sounds like a bug to me but 
>> maybe that change was intentional.. Anna?? Trond???
> 
> Might be related to my patch
>   [PATCH] SUNRPC: ensure correct error is reported by xs_tcp_setup_socket()
> 
> sent 25th May.
I'll take a look.. thanks!

> 
>>
>> So I'm a bit hesitant to commit this since not accurate, yet.
>>
>> Finally, the whole idea of systemd randomly/silently 
>> strip off mount options is crazy... IMHO... 
> 
> It isn't exactly systemd, it is systemd-fstab-generator.
> The options lists in /etc/fstab are not all equal.  Some
> are relevant to /bin/mount, some to mount.nfs, some to the kernel.
> I think /bin/mount processes the option lists before passing it
> to mount.nfs.  There is no intrinsic reason that systemd-fstab-generator
> shouldn't do the same thing.
OK. 
> 
>>
>> Just because a concept that has been around for years
>> does not fix well in the systemd world it gets
>> rip out??? IDK... but I think we can do better than that.
> 
> I could suggest that automount is uniformly better than bg.  Give how
> long automount has been around, and how easy it is to use with systemd,
> it might be time to start encouraging people to stop using the inferior
> technology.
Yes... bg mounts are a poor man's automount... 
> 
> I could say that, but I'm not 100% sure.  The difference between
> automount and bg is that with bg it is easy to see if the mount has
> succeeded yet - just look for an empty directory.  With automount,
> you'll typically get a delay at that point.  We could possibly wind down
> that delay...
> 
>>
>> Note, the 'bg' is used by clients that do want their
>> booting to hang by servers that are down so if the 
>> option is rip out, boots will start hang. This
>> will make it very difficult to debug since the bg
>> will still exist in fstab.
> 
> Not exactly.
> Current default behaviour is that systemd will wait 90 seconds, then
> kill the mount program and fail the boot.  If we strip out "bg", the
> same thing will happen.
Again.. I'm not seeing this 90 sec delay when I add a bg mount
to /etc/fstab.

> 
> I'm OK with the patch not being applied just yet.  I think we need to
> resolve this issue, but it isn't 100% clear to me what the best
> resolution would be.  So I'm happy to see a conversation happening and
> opinions being shared.
> I'd be particularly pleased if you could double check how "bg" is
> currently handled on some systemd-enabled system that you use.
> Does the mount program get killed like I see?  
No. after adding a bg mount to fstab and rebooting (with the server
down) I see the following mount in backgroup 
   1104 ?        Ss     0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg

> Does boot succeed if there is a bg mount from an unresponsive server?
Yes. Then when I bring up the server the mount succeeds

steved.

P.S. I'm taking some PTO today so I will not be back in the office
until later today or tomorrow... 

steved.
 
> 
> Thanks,
> NeilBrown
> 
> 
>>
>> Again, the whole concept of systemd messing with mounts options
>> is just not a good one... IMHO.. 
>>
>> steved.
>>
>>>
>>> Explain this.
>>>
>>> See also https://github.com/systemd/systemd/issues/6046
>>>
>>> Signed-off-by: NeilBrown <neilb@suse.com>
>>> ---
>>>  utils/mount/nfs.man | 18 +++++++++++++++++-
>>>  1 file changed, 17 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
>>> index cc6e992ed807..7e76492d454f 100644
>>> --- a/utils/mount/nfs.man
>>> +++ b/utils/mount/nfs.man
>>> @@ -372,6 +372,21 @@ Alternatively these issues can be addressed
>>>  using an automounter (refer to
>>>  .BR automount (8)
>>>  for details).
>>> +.IP
>>> +When
>>> +.B systemd
>>> +is used to mount the filesystems listed in
>>> +.IR /etc/fstab ,
>>> +the
>>> +.B bg
>>> +option is not supported, and may be stripped from the option list.
>>> +Similar functionality can be achieved by providing the
>>> +.B x-system.automount
>>> +option.  This will cause
>>> +.B systemd
>>> +to attempt to mount the filesystem when the mountpoint is first
>>> +accessed, rather than during system boot.  The mount still happens in
>>> +the "background", though in a different way.
>>>  .TP 1.5i
>>>  .BR rdirplus " / " nordirplus
>>>  Selects whether to use NFS v3 or v4 READDIRPLUS requests.
>>> @@ -1810,7 +1825,8 @@ such as security negotiation, server referrals, and named attributes.
>>>  .BR rpc.idmapd (8),
>>>  .BR rpc.gssd (8),
>>>  .BR rpc.svcgssd (8),
>>> -.BR kerberos (1)
>>> +.BR kerberos (1),
>>> +.BR systemd.mount (5) .
>>>  .sp
>>>  RFC 768 for the UDP specification.
>>>  .br
>>>
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Lennart Poettering June 7, 2017, 12:02 p.m. UTC | #15
On Wed, 07.06.17 06:08, Steve Dickson (SteveD@RedHat.com) wrote:

> 
> 
> On 06/06/2017 05:49 PM, NeilBrown wrote:
> > On Tue, Jun 06 2017, Steve Dickson wrote:
> > 
> >> Hello,
> >>
> >> On 05/29/2017 06:19 PM, NeilBrown wrote:
> >>>
> >>> Systemd does not, and will not, support "bg" correctly.
> >>> It has other, better, ways to handle "background" mounting.
> >> The only problem with this is bg mounts still work at least
> >> up to 4.11 kernel... 
> > 
> > I don't think this is correct.
> > In the default configuration, "mount -t nfs -o bg ...."
> > runs for longer than 90 seconds, so systemd kill it.
> I must be missing something... it seems to be working for me
> 
> # mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
> mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
> mount.nfs: mount(2): Connection refused
> mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
> mount.nfs: prog 100003, trying vers=3, prot=6
> mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
> mount.nfs: backgrounding "rhel7srv:/home/tmp"
> mount.nfs: mount options: "rw,bg"

We are talking about mounts done through /etc/fstab, i.e. the ones
systemd actually manages.

Lennart
J. Bruce Fields June 7, 2017, 4:08 p.m. UTC | #16
On Wed, Jun 07, 2017 at 06:04:12AM -0400, Steve Dickson wrote:
> 
> 
> On 06/07/2017 04:12 AM, Lennart Poettering wrote:
> > On Tue, 06.06.17 14:07, Steve Dickson (SteveD@RedHat.com) wrote:
> > 
> >> Hello,
> >>
> >> On 05/29/2017 06:19 PM, NeilBrown wrote:
> >>>
> >>> Systemd does not, and will not, support "bg" correctly.
> >>> It has other, better, ways to handle "background" mounting.
> >> The only problem with this is bg mounts still work at least
> >> up to 4.11 kernel... 
> >>
> >> It appears there is a problem with a 4.12 kernel. The mount no 
> >> longer errors out with ECONNREFUSED it just hangs in the 
> >> kernel trying forever... It sounds like a bug to me but 
> >> maybe that change was intentional.. Anna?? Trond???
> >>
> >> So I'm a bit hesitant to commit this since not accurate, yet.
> >>
> >> Finally, the whole idea of systemd randomly/silently 
> >> strip off mount options is crazy... IMHO... 
> >>
> >> Just because a concept that has been around for years
> >> does not fix well in the systemd world it gets
> >> rip out??? IDK... but I think we can do better than that.
> > 
> > Well "bg" doesn't really work on systemd systems, and this was never
> > different, hence I think it's only fair to document that it is
> > incompatible with systemd. 
> I'm seeing it work just fine... 
> 
> /etc/fstab: nfssrv:/home/tmp /mnt/tmp nfs bg 0 0
> nfssrv is down 
> reboot client 
> login to client 
> # ps ax | grep mount
>   980 ?        Ss     0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg

Right, but I think we also need to see a "systemctl status
remote-fs.target", or something, to verify whether that's the forked
background process or just the foreground process that's still hanging
up some part of the boot process (even though it's gotten far enough
along that you can log in--unless logins aren't permitted till remote
fs's are mounted, I don't know.)

--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Steve Dickson June 7, 2017, 7:48 p.m. UTC | #17
On 06/07/2017 08:02 AM, Lennart Poettering wrote:
> On Wed, 07.06.17 06:08, Steve Dickson (SteveD@RedHat.com) wrote:
> 
>>
>>
>> On 06/06/2017 05:49 PM, NeilBrown wrote:
>>> On Tue, Jun 06 2017, Steve Dickson wrote:
>>>
>>>> Hello,
>>>>
>>>> On 05/29/2017 06:19 PM, NeilBrown wrote:
>>>>>
>>>>> Systemd does not, and will not, support "bg" correctly.
>>>>> It has other, better, ways to handle "background" mounting.
>>>> The only problem with this is bg mounts still work at least
>>>> up to 4.11 kernel... 
>>>
>>> I don't think this is correct.
>>> In the default configuration, "mount -t nfs -o bg ...."
>>> runs for longer than 90 seconds, so systemd kill it.
>> I must be missing something... it seems to be working for me
>>
>> # mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
>> mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
>> mount.nfs: mount(2): Connection refused
>> mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
>> mount.nfs: prog 100003, trying vers=3, prot=6
>> mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
>> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
>> mount.nfs: backgrounding "rhel7srv:/home/tmp"
>> mount.nfs: mount options: "rw,bg"
> 
> We are talking about mounts done through /etc/fstab, i.e. the ones
> systemd actually manages.
I guess I was not clear... After adding a bg mount to fstab and
reboot, mounting a server that is not up, there is a mount in
background that looks like 

# ps ax | grep mount
 1104 ?        Ss     0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg

Looking at the remote-fs.target status:

# systemctl status remote-fs.target 
* remote-fs.target - Remote File Systems
   Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
   Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
     Docs: man:systemd.special(7)

Jun 06 12:36:51 f26.boston.devel.redhat.com systemd[1]: Reached target Remote File Systems.

It appears to be successful 

steved.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
NeilBrown June 8, 2017, 5:16 a.m. UTC | #18
On Wed, Jun 07 2017, Steve Dickson wrote:

> On 06/07/2017 08:02 AM, Lennart Poettering wrote:
>> On Wed, 07.06.17 06:08, Steve Dickson (SteveD@RedHat.com) wrote:
>> 
>>>
>>>
>>> On 06/06/2017 05:49 PM, NeilBrown wrote:
>>>> On Tue, Jun 06 2017, Steve Dickson wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> On 05/29/2017 06:19 PM, NeilBrown wrote:
>>>>>>
>>>>>> Systemd does not, and will not, support "bg" correctly.
>>>>>> It has other, better, ways to handle "background" mounting.
>>>>> The only problem with this is bg mounts still work at least
>>>>> up to 4.11 kernel... 
>>>>
>>>> I don't think this is correct.
>>>> In the default configuration, "mount -t nfs -o bg ...."
>>>> runs for longer than 90 seconds, so systemd kill it.
>>> I must be missing something... it seems to be working for me
>>>
>>> # mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
>>> mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
>>> mount.nfs: mount(2): Connection refused
>>> mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
>>> mount.nfs: prog 100003, trying vers=3, prot=6
>>> mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
>>> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
>>> mount.nfs: backgrounding "rhel7srv:/home/tmp"
>>> mount.nfs: mount options: "rw,bg"
>> 
>> We are talking about mounts done through /etc/fstab, i.e. the ones
>> systemd actually manages.
> I guess I was not clear... After adding a bg mount to fstab and
> reboot, mounting a server that is not up, there is a mount in
> background that looks like 
>
> # ps ax | grep mount
>  1104 ?        Ss     0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
>
> Looking at the remote-fs.target status:
>
> # systemctl status remote-fs.target 
> * remote-fs.target - Remote File Systems
>    Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
>    Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
>      Docs: man:systemd.special(7)
>
> Jun 06 12:36:51 f26.boston.devel.redhat.com systemd[1]: Reached target Remote File Systems.
>
> It appears to be successful 

Strange ... not for me.

I have a recent compiled-from-source upstream systemd and a very recent
upstream kernel.

I add a line to /etc/fstab

 192.168.1.111:/nowhere /mnt nfs bg 0 0

and reboot (192.168.1.111 is on a different subnet to the VM I am
testing in, but no host responds to the address).

There is a 1m30s wait while trying to mount /mnt, then boot completes
(I was wrong when I said that it didn't).

● mnt.mount - /mnt
   Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
   Active: failed (Result: timeout) since Thu 2017-06-08 11:13:52 AEST; 1min 24s ago
    Where: /mnt
     What: 192.168.1.111:/nowhere
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
  Process: 333 ExecMount=/bin/mount 192.168.1.111:/nowhere /mnt -t nfs -o bg (code=killed, signal=TERM)

Jun 08 11:12:22 debian systemd[1]: Mounting /mnt...
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mounting timed out. Stopping.
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mount process exited, code=killed status=15
Jun 08 11:13:52 debian systemd[1]: Failed to mount /mnt.
Jun 08 11:13:52 debian systemd[1]: mnt.mount: Unit entered failed state.


The /bin/mount process has been killed (SIGTERM) after the 90 second
timeout

● remote-fs.target - Remote File Systems
   Loaded: loaded (/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
  Drop-In: /run/systemd/generator/remote-fs.target.d
           └─50-insserv.conf.conf
   Active: inactive (dead)
     Docs: man:systemd.special(7)

Jun 08 11:13:52 debian systemd[1]: Dependency failed for Remote File Systems.
Jun 08 11:13:52 debian systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.


remote-fs.target has not been reached.

Because remote-fs.target is WantedBy multi-user.target, but need
RequiredBy it, boot completes.
But if anything did Require remote-fs.target, it would fail if "bg"
mounts were not mounted within 90 seconds.


Looking back over your log messages:

>>> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
>>> mount.nfs: backgrounding "rhel7srv:/home/tmp"

it appears that the fg mount attempt failed quickly (ECONNREFUSED), so it
background the process immediately.  Systemd sees that as success
(despite the fact that the filesystem isn't actually mounted) and
doesn't clean up the background processes (which is arguably a bug).

If you try to mount from a server which doesn't even return ECONNREFUSED or
EHOSTUNREACH, such that the first mount attempt takes longer than 90
seconds, then you should get the same results as me.

Your results go some way to explaining why Lennart hasn't received many
complaints, but I'm convinced that the current situation is not ideal
(because SUSE has received some complaints).

I've been pondering the possibility of making "bg" work properly with
systemd and I think I've found a promising approach.  It involves having
systemd take responsibility for the "run in the background" part.

If we get systemd-fstab-generator to translate "bg" to "retry=10000",
then "mount.nfs" will behave like the background version of
"mount.nfs -o bg".  i.e. it will retry for one month (nearly).   If there is
already a 'retry=' option, we just strip the "bg".

For this to work, we would need to add
   TimeoutSec=infinity
to the .mount unit file so that systemd doesn't kill the mount.
We would also need to add (the effect of) "nofail", so that systemd
doesn't wait for the mount to complete...
Except that the current implementation of "nofail" is faulty.
It removes the default "Before=remote-fs.mount", which has the unwanted
consequence of unmounting the filesystem too early at shutdown.

I have a solution for that too (which I'll submit a push request for
shortly).
My solution to "nofail" is to treat it much like "automount", but
instead of using an automount unit to trigger the mount, we use
a timer unit (with OnActiveSec=0).
By triggering the mount unit with a timer instead of Wanting it
directly, it gets run in a separate transaction.  This means that the
"Before=remote-fs.target" doesn't have the effect of delaying
remote-fs.target.  Before/After only order jobs within the one
transaction.
When it comes time to shutdown, remote-fs.target and the foo.mount will
be in the same transaction, so the Before= will ensure foo.mount
isn't unmounted until after remote-fs.target has been allows to finish,
which is after any services that might be using the filesystem.

So I think I've found a solution for systemd to handle "bg" nfs mounts
correctly.   I'll submit some pull requests for consideration.

Thanks,
NeilBrown
J. Bruce Fields June 8, 2017, 3:36 p.m. UTC | #19
On Thu, Jun 08, 2017 at 03:16:52PM +1000, NeilBrown wrote:
> So I think I've found a solution for systemd to handle "bg" nfs mounts
> correctly.   I'll submit some pull requests for consideration.

Out of curiosity, after that change is there still any reason you'd
recommend any new user actually use "bg" (as opposed to an automount)?

I appreciate the effort to keep existing systems working, I'm just
curious.

--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Steve Dickson June 8, 2017, 8:24 p.m. UTC | #20
On 06/08/2017 01:16 AM, NeilBrown wrote:
> On Wed, Jun 07 2017, Steve Dickson wrote:
> 
>> On 06/07/2017 08:02 AM, Lennart Poettering wrote:
>>> On Wed, 07.06.17 06:08, Steve Dickson (SteveD@RedHat.com) wrote:
>>>
>>>>
>>>>
>>>> On 06/06/2017 05:49 PM, NeilBrown wrote:
>>>>> On Tue, Jun 06 2017, Steve Dickson wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> On 05/29/2017 06:19 PM, NeilBrown wrote:
>>>>>>>
>>>>>>> Systemd does not, and will not, support "bg" correctly.
>>>>>>> It has other, better, ways to handle "background" mounting.
>>>>>> The only problem with this is bg mounts still work at least
>>>>>> up to 4.11 kernel... 
>>>>>
>>>>> I don't think this is correct.
>>>>> In the default configuration, "mount -t nfs -o bg ...."
>>>>> runs for longer than 90 seconds, so systemd kill it.
>>>> I must be missing something... it seems to be working for me
>>>>
>>>> # mount -vvv -o bg rhel7srv:/home/tmp /mnt/tmp
>>>> mount.nfs: trying text-based options 'bg,vers=4.1,addr=172.31.1.60,clientaddr=172.31.1.58'
>>>> mount.nfs: mount(2): Connection refused
>>>> mount.nfs: trying text-based options 'bg,addr=172.31.1.60'
>>>> mount.nfs: prog 100003, trying vers=3, prot=6
>>>> mount.nfs: trying 172.31.1.60 prog 100003 vers 3 prot TCP port 2049
>>>> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
>>>> mount.nfs: backgrounding "rhel7srv:/home/tmp"
>>>> mount.nfs: mount options: "rw,bg"
>>>
>>> We are talking about mounts done through /etc/fstab, i.e. the ones
>>> systemd actually manages.
>> I guess I was not clear... After adding a bg mount to fstab and
>> reboot, mounting a server that is not up, there is a mount in
>> background that looks like 
>>
>> # ps ax | grep mount
>>  1104 ?        Ss     0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
>>
>> Looking at the remote-fs.target status:
>>
>> # systemctl status remote-fs.target 
>> * remote-fs.target - Remote File Systems
>>    Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
>>    Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
>>      Docs: man:systemd.special(7)
>>
>> Jun 06 12:36:51 f26.boston.devel.redhat.com systemd[1]: Reached target Remote File Systems.
>>
>> It appears to be successful 
> 
> Strange ... not for me.
> 
> I have a recent compiled-from-source upstream systemd and a very recent
> upstream kernel.
> 
> I add a line to /etc/fstab
> 
>  192.168.1.111:/nowhere /mnt nfs bg 0 0
> 
> and reboot (192.168.1.111 is on a different subnet to the VM I am
> testing in, but no host responds to the address).
> 
> There is a 1m30s wait while trying to mount /mnt, then boot completes
> (I was wrong when I said that it didn't).
> 
> ● mnt.mount - /mnt
>    Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
>    Active: failed (Result: timeout) since Thu 2017-06-08 11:13:52 AEST; 1min 24s ago
>     Where: /mnt
>      What: 192.168.1.111:/nowhere
>      Docs: man:fstab(5)
>            man:systemd-fstab-generator(8)
>   Process: 333 ExecMount=/bin/mount 192.168.1.111:/nowhere /mnt -t nfs -o bg (code=killed, signal=TERM)
> 
> Jun 08 11:12:22 debian systemd[1]: Mounting /mnt...
> Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mounting timed out. Stopping.
> Jun 08 11:13:52 debian systemd[1]: mnt.mount: Mount process exited, code=killed status=15
> Jun 08 11:13:52 debian systemd[1]: Failed to mount /mnt.
> Jun 08 11:13:52 debian systemd[1]: mnt.mount: Unit entered failed state.
> 
> 
> The /bin/mount process has been killed (SIGTERM) after the 90 second
> timeout
> 
> ● remote-fs.target - Remote File Systems
>    Loaded: loaded (/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
>   Drop-In: /run/systemd/generator/remote-fs.target.d
>            └─50-insserv.conf.conf
>    Active: inactive (dead)
>      Docs: man:systemd.special(7)
> 
> Jun 08 11:13:52 debian systemd[1]: Dependency failed for Remote File Systems.
> Jun 08 11:13:52 debian systemd[1]: remote-fs.target: Job remote-fs.target/start failed with result 'dependency'.
> 
> 
> remote-fs.target has not been reached.
I'm seeing this now... the server has to be up to cause this hang.

> 
> Because remote-fs.target is WantedBy multi-user.target, but need
> RequiredBy it, boot completes.
> But if anything did Require remote-fs.target, it would fail if "bg"
> mounts were not mounted within 90 seconds.
> 
> 
> Looking back over your log messages:
> 
>>>> mount.nfs: portmap query failed: RPC: Remote system error - Connection refused
>>>> mount.nfs: backgrounding "rhel7srv:/home/tmp"
> 
> it appears that the fg mount attempt failed quickly (ECONNREFUSED), so it
> background the process immediately.  Systemd sees that as success
> (despite the fact that the filesystem isn't actually mounted) and
> doesn't clean up the background processes (which is arguably a bug).
No... Systemd is doing the right thing in this case... Letting
bg mounts work... 

> 
> If you try to mount from a server which doesn't even return ECONNREFUSED or
> EHOSTUNREACH, such that the first mount attempt takes longer than 90
> seconds, then you should get the same results as me.
This is clearly a bug in systemd! The 'bg' mounts work with the server
down but hangs when the server is up. How can't this be a bug in systemd?? 

Why is systemd even looking at or interpreting the mount options??? 
This clear an overreach of systemd, IMHO. Then just to blindly rip
them out... WOW!

I'm only assuming... but it appears, do to this overreach, that systemd
actually think it know how to mount a file system better than the 
actual mount command tailor for particular that file system and
if it doesn't like an mount option it just going to rip it
out! Again, I hope this is not the case because if its... 
that's just crazy!!! 

> 
> Your results go some way to explaining why Lennart hasn't received many
> complaints, but I'm convinced that the current situation is not ideal
> (because SUSE has received some complaints).
It just proves bg mounts are being used.

> 
> I've been pondering the possibility of making "bg" work properly with
> systemd and I think I've found a promising approach.  It involves having
> systemd take responsibility for the "run in the background" part.
> 
> If we get systemd-fstab-generator to translate "bg" to "retry=10000",
> then "mount.nfs" will behave like the background version of
> "mount.nfs -o bg".  i.e. it will retry for one month (nearly).   If there is
> already a 'retry=' option, we just strip the "bg".
> 
> For this to work, we would need to add
>    TimeoutSec=infinity
> to the .mount unit file so that systemd doesn't kill the mount.
> We would also need to add (the effect of) "nofail", so that systemd
> doesn't wait for the mount to complete...
> Except that the current implementation of "nofail" is faulty.
> It removes the default "Before=remote-fs.mount", which has the unwanted
> consequence of unmounting the filesystem too early at shutdown.
> 
> I have a solution for that too (which I'll submit a push request for
> shortly).
> My solution to "nofail" is to treat it much like "automount", but
> instead of using an automount unit to trigger the mount, we use
> a timer unit (with OnActiveSec=0).
> By triggering the mount unit with a timer instead of Wanting it
> directly, it gets run in a separate transaction.  This means that the
> "Before=remote-fs.target" doesn't have the effect of delaying
> remote-fs.target.  Before/After only order jobs within the one
> transaction.
> When it comes time to shutdown, remote-fs.target and the foo.mount will
> be in the same transaction, so the Before= will ensure foo.mount
> isn't unmounted until after remote-fs.target has been allows to finish,
> which is after any services that might be using the filesystem.
> 
> So I think I've found a solution for systemd to handle "bg" nfs mounts
> correctly.   I'll submit some pull requests for consideration.
Neil, your an excellent engineer so I'm sure you will craft 
something very cool...  but I truly feel you are breaking one 
of the NFS communities long standing golden rule... We never add 
fix to client that fixes a bug in the server... Fix the server... 

That is the case here, IMHO...  There is a bug in systemd that is not
letting a mount command succeed... It's clearly their bug, 
let them fix it... Again, why systemd looking at mount options
is just mind boggling... 

steved.
 
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Steve Dickson June 8, 2017, 8:34 p.m. UTC | #21
On 06/07/2017 12:08 PM, J. Bruce Fields wrote:
> On Wed, Jun 07, 2017 at 06:04:12AM -0400, Steve Dickson wrote:
>> # ps ax | grep mount
>>   980 ?        Ss     0:00 /sbin/mount.nfs nfssrv:/home/tmp /mnt/tmp -o rw,bg
> 
> Right, but I think we also need to see a "systemctl status
> remote-fs.target", or something, to verify whether that's the forked
> background process or just the foreground process that's still hanging
> up some part of the boot process (even though it's gotten far enough
> along that you can log in--unless logins aren't permitted till remote
> fs's are mounted, I don't know.)
It succeeds... 

# systemctl status remote-fs.target 
* remote-fs.target - Remote File Systems
   Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled; vendor preset: enabled)
   Active: active since Tue 2017-06-06 12:36:51 EDT; 12min ago
     Docs: man:systemd.special(7)

Jun 06 12:36:51 f26 systemd[1]: Reached target Remote File Systems.

The reason being, as Neil pointed out, the mount.nfs gets the 
ECONNREFUSED right away because the server is down. So a
child is quickly forked that continues to try the mount... 
Basically sneaking around systemd back... Which is hard
to do... these day 8-)  

steved.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
NeilBrown June 8, 2017, 9:54 p.m. UTC | #22
On Thu, Jun 08 2017, J. Bruce Fields wrote:

> On Thu, Jun 08, 2017 at 03:16:52PM +1000, NeilBrown wrote:
>> So I think I've found a solution for systemd to handle "bg" nfs mounts
>> correctly.   I'll submit some pull requests for consideration.
>
> Out of curiosity, after that change is there still any reason you'd
> recommend any new user actually use "bg" (as opposed to an automount)?

Me? Recommend?  Who would listen?  Who would even hear?
For the last few years I've been recommending that automount should
be used for *all* NFS mounts at every opportunity.  I think I've had two
opportunities.

But no, I would not recommend "bg".  I would recommend automount and
then when they reported problems,  I would help fix them.

I would be much happier recommending automount if it were easier.
Setting up /etc/auto.direct with automountd is fairly easy, but you need
to actually enable it but modifying auto.master or auto.master.d, which
is slightly annoying.

systemd does make it easier to do direct mounts, but it is ugly.
You need to include "comment=systemd.automount" or "x-systemd.automount"
in /etc/fstab instead of just "automount" or "ondemand".  I understand
exactly why they did that and I cannot fault the logic.  But it still
looks clumsy.

With systemd, you cannot divorce the timeout that an application has to
wait when accessing the mountpoint while the server is down, from the
timeout imposed on the mount program.  i.e., mount cannot keep trying
in the background. - that could be useful if you want really-short
timeouts... at least it seems to me that they should be separate.

The timeout is configured differently if mounting from a device, or
mounting from anything else such as NFS.  The first uses
x-systemd.device-timeout.  The other needs x-systemd.mount-timeout.


But I'm ranting... I should probably shut up and send patches.
A generator for /etc/fstab.auto??

>
> I appreciate the effort to keep existing systems working, I'm just
> curious.

Compatibility with existing practice is certainly the main driver.

Thanks,
NeilBrown
NeilBrown July 4, 2017, 10:20 p.m. UTC | #23
On Tue, May 30 2017, NeilBrown wrote:

> Systemd does not, and will not, support "bg" correctly.
> It has other, better, ways to handle "background" mounting.

For those who aren't closely watching systemd development, a
patch was recently accepted which causes systemd to work correctly with
NFS bg mounts.  So the above "and will not" was, happily, not correct.

So this patch to the nfs documentation is no longer relevant.

Thanks everyone,

NeilBrown
Steve Dickson July 10, 2017, 3:26 p.m. UTC | #24
Hey Neil,

On 07/04/2017 06:20 PM, NeilBrown wrote:
> On Tue, May 30 2017, NeilBrown wrote:
> 
>> Systemd does not, and will not, support "bg" correctly.
>> It has other, better, ways to handle "background" mounting.
> 
> For those who aren't closely watching systemd development, a
> patch was recently accepted which causes systemd to work correctly with
> NFS bg mounts.  So the above "and will not" was, happily, not correct.
Could you please post a pointer to the thread?

tia,

steved.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
NeilBrown July 10, 2017, 10:56 p.m. UTC | #25
On Mon, Jul 10 2017, Steve Dickson wrote:

> Hey Neil,
>
> On 07/04/2017 06:20 PM, NeilBrown wrote:
>> On Tue, May 30 2017, NeilBrown wrote:
>> 
>>> Systemd does not, and will not, support "bg" correctly.
>>> It has other, better, ways to handle "background" mounting.
>> 
>> For those who aren't closely watching systemd development, a
>> patch was recently accepted which causes systemd to work correctly with
>> NFS bg mounts.  So the above "and will not" was, happily, not correct.
> Could you please post a pointer to the thread?

The main commmit is
 https://github.com/systemd/systemd/commit/65e1dee7dcf1668c25c32f0238c935708dbffbcf

The link in the title leads to
  https://github.com/systemd/systemd/pull/6103
which is the discussion of the pull request.
The "Fixes" link at the bottom leads to
 https://github.com/systemd/systemd/issues/6046
which is an 'issue' that started out as "we should warn if 'bg' is used"
but became "here is a fix so bg works".

I've been wondering if we should add text to nfs.5 suggesting that
automount is often a good match for NFS, and can particularly be useful
where "bg" is currently used.  Not sure what to do about timeouts
though.
systemd currently waits for the mount command to fail, or kills it after
90 seconds.  It doesn't return an error to accesses of the mount point
(typically ENODEV) until mount exits.  I think I would prefer a shorter
timeout before the error, but a longer timeout before killing mount.
But I'm not really sure.  Does anyone have opinions?

Thanks,
NeilBrown
diff mbox

Patch

diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
index cc6e992ed807..7e76492d454f 100644
--- a/utils/mount/nfs.man
+++ b/utils/mount/nfs.man
@@ -372,6 +372,21 @@  Alternatively these issues can be addressed
 using an automounter (refer to
 .BR automount (8)
 for details).
+.IP
+When
+.B systemd
+is used to mount the filesystems listed in
+.IR /etc/fstab ,
+the
+.B bg
+option is not supported, and may be stripped from the option list.
+Similar functionality can be achieved by providing the
+.B x-system.automount
+option.  This will cause
+.B systemd
+to attempt to mount the filesystem when the mountpoint is first
+accessed, rather than during system boot.  The mount still happens in
+the "background", though in a different way.
 .TP 1.5i
 .BR rdirplus " / " nordirplus
 Selects whether to use NFS v3 or v4 READDIRPLUS requests.
@@ -1810,7 +1825,8 @@  such as security negotiation, server referrals, and named attributes.
 .BR rpc.idmapd (8),
 .BR rpc.gssd (8),
 .BR rpc.svcgssd (8),
-.BR kerberos (1)
+.BR kerberos (1),
+.BR systemd.mount (5) .
 .sp
 RFC 768 for the UDP specification.
 .br