diff mbox series

Boot time improvement with systemd and nfs-utils

Message ID CAL5u83HS=nurJ=r0tJU8ZqAXXkvu9-vWZpbVWoKALNh22WdKnw@mail.gmail.com (mailing list archive)
State New, archived
Headers show
Series Boot time improvement with systemd and nfs-utils | expand

Commit Message

Hackintosh 5 Jan. 4, 2021, 11:32 a.m. UTC
rpc-statd-notify is causing a 10 second hang on my system during boot
due to an unwanted dependency on network-online.target. This
dependency isn't needed anyway, because rpc-statd-notify (sm-notify)
will wait for the network to come online if it isn't already (up to 15
minutes, so no risk of timeout that would be avoided by systemd)
=============================================
From c90bd7e701c2558606907f08bf27ae9be3f8e0bf Mon Sep 17 00:00:00 2001
From: Hackintosh 5 <git@hack5.dev>
Date: Sat, 2 Jan 2021 14:28:30 +0000
Subject: [PATCH] systemd: network-online.target is not needed for
 rpc-statd-notify.service

Commit 09e5c6c2 changed the After line for rpc-statd-notify to change
network.target to network-online.target, which is incorrect, because
sm-notify has a default timeout of 15 minutes, which is longer than
the timeout for network-online.target. In other words, the dependency
on network-online.target is useless and delays system boot by ~10
seconds.
---
 systemd/rpc-statd-notify.service | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comments

Chuck Lever III Jan. 4, 2021, 12:27 p.m. UTC | #1
Hello, thanks for your report.

The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.


> On Jan 4, 2021, at 6:32 AM, Hackintosh Five <hackintoshfive@gmail.com> wrote:
> 
> rpc-statd-notify is causing a 10 second hang on my system during boot
> due to an unwanted dependency on network-online.target. This
> dependency isn't needed anyway, because rpc-statd-notify (sm-notify)
> will wait for the network to come online if it isn't already (up to 15
> minutes, so no risk of timeout that would be avoided by systemd)
> =============================================
> From c90bd7e701c2558606907f08bf27ae9be3f8e0bf Mon Sep 17 00:00:00 2001
> From: Hackintosh 5 <git@hack5.dev>
> Date: Sat, 2 Jan 2021 14:28:30 +0000
> Subject: [PATCH] systemd: network-online.target is not needed for
> rpc-statd-notify.service
> 
> Commit 09e5c6c2 changed the After line for rpc-statd-notify to change
> network.target to network-online.target, which is incorrect, because
> sm-notify has a default timeout of 15 minutes, which is longer than
> the timeout for network-online.target. In other words, the dependency
> on network-online.target is useless and delays system boot by ~10
> seconds.
> ---
> systemd/rpc-statd-notify.service | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/systemd/rpc-statd-notify.service
> b/systemd/rpc-statd-notify.service
> index aad4c0d2..8a40e862 100644
> --- a/systemd/rpc-statd-notify.service
> +++ b/systemd/rpc-statd-notify.service
> @@ -1,8 +1,8 @@
> [Unit]
> Description=Notify NFS peers of a restart
> DefaultDependencies=no
> -Wants=network-online.target
> -After=local-fs.target network-online.target nss-lookup.target
> +Wants=network.target
> +After=local-fs.target network.target nss-lookup.target
> 
> # if we run an nfs server, it needs to be running before we
> # tell clients that it has restarted.
> -- 
> 2.29.2

--
Chuck Lever
Hackintosh 5 Jan. 4, 2021, 12:51 p.m. UTC | #2
Hi, thanks for the fast reply

I have never even used nfs and I'm not a systemd expert, so I'm not at
all sure this interpretation is correct, but here goes. I only removed
the dependency from rpc.statd.notify, not rpc.statd. I didn't remove
the `After=nfs-server` line, and for nfs-server to be up,
network-online must be up first (there's an After requirement in the
nfs-server unit). So if the nfs-server is enabled, the
rpc-statd-notify will order itself after the server is up, which
depends on the network. That means that, if there is a server, the
server must be up before it sends notifications, so it will have the
right hostname. This only improves boot speed on nfs clients, where
nfs-client.target pulls in rpc-statd-notify.service.


On Mon, Jan 4, 2021 at 12:27 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>
> Hello, thanks for your report.
>
> The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.
>
>
> > On Jan 4, 2021, at 6:32 AM, Hackintosh Five <hackintoshfive@gmail.com> wrote:
> >
> > rpc-statd-notify is causing a 10 second hang on my system during boot
> > due to an unwanted dependency on network-online.target. This
> > dependency isn't needed anyway, because rpc-statd-notify (sm-notify)
> > will wait for the network to come online if it isn't already (up to 15
> > minutes, so no risk of timeout that would be avoided by systemd)
> > =============================================
> > From c90bd7e701c2558606907f08bf27ae9be3f8e0bf Mon Sep 17 00:00:00 2001
> > From: Hackintosh 5 <git@hack5.dev>
> > Date: Sat, 2 Jan 2021 14:28:30 +0000
> > Subject: [PATCH] systemd: network-online.target is not needed for
> > rpc-statd-notify.service
> >
> > Commit 09e5c6c2 changed the After line for rpc-statd-notify to change
> > network.target to network-online.target, which is incorrect, because
> > sm-notify has a default timeout of 15 minutes, which is longer than
> > the timeout for network-online.target. In other words, the dependency
> > on network-online.target is useless and delays system boot by ~10
> > seconds.
> > ---
> > systemd/rpc-statd-notify.service | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/systemd/rpc-statd-notify.service
> > b/systemd/rpc-statd-notify.service
> > index aad4c0d2..8a40e862 100644
> > --- a/systemd/rpc-statd-notify.service
> > +++ b/systemd/rpc-statd-notify.service
> > @@ -1,8 +1,8 @@
> > [Unit]
> > Description=Notify NFS peers of a restart
> > DefaultDependencies=no
> > -Wants=network-online.target
> > -After=local-fs.target network-online.target nss-lookup.target
> > +Wants=network.target
> > +After=local-fs.target network.target nss-lookup.target
> >
> > # if we run an nfs server, it needs to be running before we
> > # tell clients that it has restarted.
> > --
> > 2.29.2
>
> --
> Chuck Lever
>
>
>
Chuck Lever III Jan. 4, 2021, 12:54 p.m. UTC | #3
Hi-

> On Jan 4, 2021, at 7:51 AM, Hackintosh Five <hackintoshfive@gmail.com> wrote:
> 
> Hi, thanks for the fast reply
> 
> I have never even used nfs and I'm not a systemd expert, so I'm not at
> all sure this interpretation is correct, but here goes. I only removed
> the dependency from rpc.statd.notify, not rpc.statd.

Same problem exists for sm-notify.


> I didn't remove
> the `After=nfs-server` line, and for nfs-server to be up,
> network-online must be up first (there's an After requirement in the
> nfs-server unit). So if the nfs-server is enabled, the
> rpc-statd-notify will order itself after the server is up, which
> depends on the network.

IIRC sm-notify runs on clients too. That's why the dependency is
on the network and not on nfs-server.


> That means that, if there is a server, the
> server must be up before it sends notifications, so it will have the
> right hostname. This only improves boot speed on nfs clients, where
> nfs-client.target pulls in rpc-statd-notify.service.
> 
> 
> On Mon, Jan 4, 2021 at 12:27 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>> 
>> Hello, thanks for your report.
>> 
>> The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.
>> 
>> 
>>> On Jan 4, 2021, at 6:32 AM, Hackintosh Five <hackintoshfive@gmail.com> wrote:
>>> 
>>> rpc-statd-notify is causing a 10 second hang on my system during boot
>>> due to an unwanted dependency on network-online.target. This
>>> dependency isn't needed anyway, because rpc-statd-notify (sm-notify)
>>> will wait for the network to come online if it isn't already (up to 15
>>> minutes, so no risk of timeout that would be avoided by systemd)
>>> =============================================
>>> From c90bd7e701c2558606907f08bf27ae9be3f8e0bf Mon Sep 17 00:00:00 2001
>>> From: Hackintosh 5 <git@hack5.dev>
>>> Date: Sat, 2 Jan 2021 14:28:30 +0000
>>> Subject: [PATCH] systemd: network-online.target is not needed for
>>> rpc-statd-notify.service
>>> 
>>> Commit 09e5c6c2 changed the After line for rpc-statd-notify to change
>>> network.target to network-online.target, which is incorrect, because
>>> sm-notify has a default timeout of 15 minutes, which is longer than
>>> the timeout for network-online.target. In other words, the dependency
>>> on network-online.target is useless and delays system boot by ~10
>>> seconds.
>>> ---
>>> systemd/rpc-statd-notify.service | 4 ++--
>>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>> 
>>> diff --git a/systemd/rpc-statd-notify.service
>>> b/systemd/rpc-statd-notify.service
>>> index aad4c0d2..8a40e862 100644
>>> --- a/systemd/rpc-statd-notify.service
>>> +++ b/systemd/rpc-statd-notify.service
>>> @@ -1,8 +1,8 @@
>>> [Unit]
>>> Description=Notify NFS peers of a restart
>>> DefaultDependencies=no
>>> -Wants=network-online.target
>>> -After=local-fs.target network-online.target nss-lookup.target
>>> +Wants=network.target
>>> +After=local-fs.target network.target nss-lookup.target
>>> 
>>> # if we run an nfs server, it needs to be running before we
>>> # tell clients that it has restarted.
>>> --
>>> 2.29.2
>> 
>> --
>> Chuck Lever
>> 
>> 
>> 

--
Chuck Lever
Hackintosh 5 Jan. 4, 2021, 4:03 p.m. UTC | #4
I see. Does rpc-statd-notify HAVE to start before nfs-client? If not,
perhaps a one-off timer unit with no delay could be made so that the
startup of rpc-statd-notify doesn't block the boot process, while
still running after network-online?

On Mon, Jan 4, 2021 at 1:26 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>
> The problem is not in sm-notify itself, it's in the C library functions. The system's DNS resolver configuration is set during network startup. When a process first attempts a DNS query, it retrieves the system DNS configuration as it is at that moment, and keeps that configuration until the process exits. If sm-notify starts before the system's DNS resolver is configured, then it simply doesn't work because it can't perform DNS queries correctly.
>
>
> On Jan 4, 2021, at 8:13 AM, Hackintosh Five <hackintoshfive@gmail.com> wrote:
>
> Yep, sm-notify is Wanted by nfs-utils.target, and hence clients. But I can't see anywhere in the source of lm-notify where being offline would make a difference (beyond triggering a retry). If such a function does exist and I missed it, perhaps it could be moved to the end of the program (so the hostname is calculated only when the network is available, or whatever)
>
> On Mon, 4 Jan 2021, 12:54 Chuck Lever, <chuck.lever@oracle.com> wrote:
>>
>> Hi-
>>
>> > On Jan 4, 2021, at 7:51 AM, Hackintosh Five <hackintoshfive@gmail.com> wrote:
>> >
>> > Hi, thanks for the fast reply
>> >
>> > I have never even used nfs and I'm not a systemd expert, so I'm not at
>> > all sure this interpretation is correct, but here goes. I only removed
>> > the dependency from rpc.statd.notify, not rpc.statd.
>>
>> Same problem exists for sm-notify.
>>
>>
>> > I didn't remove
>> > the `After=nfs-server` line, and for nfs-server to be up,
>> > network-online must be up first (there's an After requirement in the
>> > nfs-server unit). So if the nfs-server is enabled, the
>> > rpc-statd-notify will order itself after the server is up, which
>> > depends on the network.
>>
>> IIRC sm-notify runs on clients too. That's why the dependency is
>> on the network and not on nfs-server.
>>
>>
>> > That means that, if there is a server, the
>> > server must be up before it sends notifications, so it will have the
>> > right hostname. This only improves boot speed on nfs clients, where
>> > nfs-client.target pulls in rpc-statd-notify.service.
>> >
>> >
>> > On Mon, Jan 4, 2021 at 12:27 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>> >>
>> >> Hello, thanks for your report.
>> >>
>> >> The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.
>> >>
>> >>
>> >>> On Jan 4, 2021, at 6:32 AM, Hackintosh Five <hackintoshfive@gmail.com> wrote:
>> >>>
>> >>> rpc-statd-notify is causing a 10 second hang on my system during boot
>> >>> due to an unwanted dependency on network-online.target. This
>> >>> dependency isn't needed anyway, because rpc-statd-notify (sm-notify)
>> >>> will wait for the network to come online if it isn't already (up to 15
>> >>> minutes, so no risk of timeout that would be avoided by systemd)
>> >>> =============================================
>> >>> From c90bd7e701c2558606907f08bf27ae9be3f8e0bf Mon Sep 17 00:00:00 2001
>> >>> From: Hackintosh 5 <git@hack5.dev>
>> >>> Date: Sat, 2 Jan 2021 14:28:30 +0000
>> >>> Subject: [PATCH] systemd: network-online.target is not needed for
>> >>> rpc-statd-notify.service
>> >>>
>> >>> Commit 09e5c6c2 changed the After line for rpc-statd-notify to change
>> >>> network.target to network-online.target, which is incorrect, because
>> >>> sm-notify has a default timeout of 15 minutes, which is longer than
>> >>> the timeout for network-online.target. In other words, the dependency
>> >>> on network-online.target is useless and delays system boot by ~10
>> >>> seconds.
>> >>> ---
>> >>> systemd/rpc-statd-notify.service | 4 ++--
>> >>> 1 file changed, 2 insertions(+), 2 deletions(-)
>> >>>
>> >>> diff --git a/systemd/rpc-statd-notify.service
>> >>> b/systemd/rpc-statd-notify.service
>> >>> index aad4c0d2..8a40e862 100644
>> >>> --- a/systemd/rpc-statd-notify.service
>> >>> +++ b/systemd/rpc-statd-notify.service
>> >>> @@ -1,8 +1,8 @@
>> >>> [Unit]
>> >>> Description=Notify NFS peers of a restart
>> >>> DefaultDependencies=no
>> >>> -Wants=network-online.target
>> >>> -After=local-fs.target network-online.target nss-lookup.target
>> >>> +Wants=network.target
>> >>> +After=local-fs.target network.target nss-lookup.target
>> >>>
>> >>> # if we run an nfs server, it needs to be running before we
>> >>> # tell clients that it has restarted.
>> >>> --
>> >>> 2.29.2
>> >>
>> >> --
>> >> Chuck Lever
>> >>
>> >>
>> >>
>>
>> --
>> Chuck Lever
>>
>>
>>
>
> --
> Chuck Lever
>
>
>
Chuck Lever III Jan. 4, 2021, 6 p.m. UTC | #5
I'm not aware of a similar reason why the notify step would need to
block the boot process. Maybe Type=forking is wrong, but I thought
sm-notify was a forking/daemonizing type of utility.


> On Jan 4, 2021, at 11:03 AM, Hackintosh Five <hackintoshfive@gmail.com> wrote:
> 
> I see. Does rpc-statd-notify HAVE to start before nfs-client? If not,
> perhaps a one-off timer unit with no delay could be made so that the
> startup of rpc-statd-notify doesn't block the boot process, while
> still running after network-online?
> 
> On Mon, Jan 4, 2021 at 1:26 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>> 
>> The problem is not in sm-notify itself, it's in the C library functions. The system's DNS resolver configuration is set during network startup. When a process first attempts a DNS query, it retrieves the system DNS configuration as it is at that moment, and keeps that configuration until the process exits. If sm-notify starts before the system's DNS resolver is configured, then it simply doesn't work because it can't perform DNS queries correctly.


--
Chuck Lever
Hackintosh 5 Jan. 4, 2021, 6:53 p.m. UTC | #6
The issue isn't with the forking type, that's certainly correct (since
it does indeed fork). The problem is that systemd is putting a
dependency between nfs-client (required by multi-user.target) and
rpc-statd-notify (which requires network-online), resulting in gdm
waiting for network-online. The only workaround I was able to make
work was to create a new timer unit which simply launches sm-notify
after 1 second. nfs-client can start the timer unit, which then
*asynchronously* starts sm-notify, meaning that sm-notify gets to keep
its dependency on network-online. Patch with that method will be sent
in a moment (if git send-email decides to work)



On Mon, Jan 4, 2021 at 6:00 PM Chuck Lever <chuck.lever@oracle.com> wrote:
>
> I'm not aware of a similar reason why the notify step would need to
> block the boot process. Maybe Type=forking is wrong, but I thought
> sm-notify was a forking/daemonizing type of utility.
>
>
> > On Jan 4, 2021, at 11:03 AM, Hackintosh Five <hackintoshfive@gmail.com> wrote:
> >
> > I see. Does rpc-statd-notify HAVE to start before nfs-client? If not,
> > perhaps a one-off timer unit with no delay could be made so that the
> > startup of rpc-statd-notify doesn't block the boot process, while
> > still running after network-online?
> >
> > On Mon, Jan 4, 2021 at 1:26 PM Chuck Lever <chuck.lever@oracle.com> wrote:
> >>
> >> The problem is not in sm-notify itself, it's in the C library functions. The system's DNS resolver configuration is set during network startup. When a process first attempts a DNS query, it retrieves the system DNS configuration as it is at that moment, and keeps that configuration until the process exits. If sm-notify starts before the system's DNS resolver is configured, then it simply doesn't work because it can't perform DNS queries correctly.
>
>
> --
> Chuck Lever
>
>
>
diff mbox series

Patch

diff --git a/systemd/rpc-statd-notify.service
b/systemd/rpc-statd-notify.service
index aad4c0d2..8a40e862 100644
--- a/systemd/rpc-statd-notify.service
+++ b/systemd/rpc-statd-notify.service
@@ -1,8 +1,8 @@ 
 [Unit]
 Description=Notify NFS peers of a restart
 DefaultDependencies=no
-Wants=network-online.target
-After=local-fs.target network-online.target nss-lookup.target
+Wants=network.target
+After=local-fs.target network.target nss-lookup.target

 # if we run an nfs server, it needs to be running before we
 # tell clients that it has restarted.