diff mbox

[2/3] watchdog: control hard lockup detection default

Message ID 1406196811-5384-3-git-send-email-drjones@redhat.com (mailing list archive)
State New, archived
Headers show

Commit Message

Andrew Jones July 24, 2014, 10:13 a.m. UTC
From: Ulrich Obergfell <uobergfe@redhat.com>

In some cases we don't want hard lockup detection enabled by default.
An example is when running as a guest. Introduce

  watchdog_enable_hardlockup_detector(bool)

allowing those cases to disable hard lockup detection. This must be
executed early by the boot processor from e.g. smp_prepare_boot_cpu,
in order to allow kernel command line arguments to override it, as
well as to avoid hard lockup detection being enabled before we've
had a chance to indicate that it's unwanted. In summary,

  initial boot:					default=enabled
  smp_prepare_boot_cpu
    watchdog_enable_hardlockup_detector(false):	default=disabled
  cmdline has 'nmi_watchdog=1':			default=enabled

The running kernel still has the ability to enable/disable at any
time with /proc/sys/kernel/nmi_watchdog us usual. However even
when the default has been overridden /proc/sys/kernel/nmi_watchdog
will initially show '1'. To truly turn it on one must disable/enable
it, i.e.
  echo 0 > /proc/sys/kernel/nmi_watchdog
  echo 1 > /proc/sys/kernel/nmi_watchdog

This patch will be immediately useful for KVM with the next patch
of this series. Other hypervisor guest types may find it useful as
well.

Signed-off-by: Ulrich Obergfell <uobergfe@redhat.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 include/linux/nmi.h |  9 +++++++++
 kernel/watchdog.c   | 45 +++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 52 insertions(+), 2 deletions(-)

Comments

Paolo Bonzini July 24, 2014, 10:46 a.m. UTC | #1
Il 24/07/2014 12:13, Andrew Jones ha scritto:
> 
> The running kernel still has the ability to enable/disable at any
> time with /proc/sys/kernel/nmi_watchdog us usual. However even
> when the default has been overridden /proc/sys/kernel/nmi_watchdog
> will initially show '1'. To truly turn it on one must disable/enable
> it, i.e.
>   echo 0 > /proc/sys/kernel/nmi_watchdog
>   echo 1 > /proc/sys/kernel/nmi_watchdog

Why is it hard to make this show the right value? :)

Paolo

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Ulrich Obergfell July 24, 2014, 11:18 a.m. UTC | #2
> ----- Original Message -----
> From: "Paolo Bonzini" <pbonzini@redhat.com>
> To: "Andrew Jones" <drjones@redhat.com>, linux-kernel@vger.kernel.org, kvm@vger.kernel.org
> Cc: uobergfe@redhat.com, dzickus@redhat.com, akpm@linux-foundation.org, mingo@redhat.com
> Sent: Thursday, July 24, 2014 12:46:11 PM
> Subject: Re: [PATCH 2/3] watchdog: control hard lockup detection default
>
>Il 24/07/2014 12:13, Andrew Jones ha scritto:
>> 
>> The running kernel still has the ability to enable/disable at any
>> time with /proc/sys/kernel/nmi_watchdog us usual. However even
>> when the default has been overridden /proc/sys/kernel/nmi_watchdog
>> will initially show '1'. To truly turn it on one must disable/enable
>> it, i.e.
>>   echo 0 > /proc/sys/kernel/nmi_watchdog
>>   echo 1 > /proc/sys/kernel/nmi_watchdog
>
> Why is it hard to make this show the right value? :)
>
> Paolo

'echo 1 > /proc/sys/kernel/nmi_watchdog' enables both - hard lockup and
soft lockup detection. watchdog_enable_all_cpus() starts a 'watchdog/N'
thread for each CPU. If the kernel runs on a bare metal system where the
processor does not have a PMU, or when perf_event_create_kernel_counter()
returns failure to watchdog_nmi_enable(), or when the kernel runs as a
guest on a hypervisor that does not emulate a PMU, then the 'watchdog/N'
threads are still active for soft lockup detection. Patch 2/3 essentially
makes watchdog_nmi_enable() behave in the same way as if -ENOENT would
have been returned by perf_event_create_kernel_counter(). This is then
reported via a console message.

  NMI watchdog: disabled (cpu0): hardware events not enabled

It's hard say what _is_ 'the right value' (because lockup detection is
then enabled 'partially'), regardless of whether patch 2/3 is applied
or not.

Regards,

Uli
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Paolo Bonzini July 24, 2014, 11:26 a.m. UTC | #3
Il 24/07/2014 13:18, Ulrich Obergfell ha scritto:
>>> >> The running kernel still has the ability to enable/disable at any
>>> >> time with /proc/sys/kernel/nmi_watchdog us usual. However even
>>> >> when the default has been overridden /proc/sys/kernel/nmi_watchdog
>>> >> will initially show '1'. To truly turn it on one must disable/enable
>>> >> it, i.e.
>>> >>   echo 0 > /proc/sys/kernel/nmi_watchdog
>>> >>   echo 1 > /proc/sys/kernel/nmi_watchdog
>> >
>> > Why is it hard to make this show the right value? :)
>> >
>> > Paolo
> 'echo 1 > /proc/sys/kernel/nmi_watchdog' enables both - hard lockup and
> soft lockup detection. watchdog_enable_all_cpus() starts a 'watchdog/N'
> thread for each CPU. If the kernel runs on a bare metal system where the
> processor does not have a PMU, or when perf_event_create_kernel_counter()
> returns failure to watchdog_nmi_enable(), or when the kernel runs as a
> guest on a hypervisor that does not emulate a PMU, then the 'watchdog/N'
> threads are still active for soft lockup detection. Patch 2/3 essentially
> makes watchdog_nmi_enable() behave in the same way as if -ENOENT would
> have been returned by perf_event_create_kernel_counter(). This is then
> reported via a console message.
> 
>   NMI watchdog: disabled (cpu0): hardware events not enabled
> 
> It's hard say what _is_ 'the right value' (because lockup detection is
> then enabled 'partially'), regardless of whether patch 2/3 is applied
> or not.

But this means that it is not possible to re-enable softlockup detection
only.  I think that should be the effect of echo 0 + echo 1, if
hardlockup detection was disabled by either the command line or patch 3.

Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Ulrich Obergfell July 24, 2014, 11:44 a.m. UTC | #4
>----- Original Message -----
>From: "Paolo Bonzini" <pbonzini@redhat.com>
>To: "Ulrich Obergfell" <uobergfe@redhat.com>
>Cc: "Andrew Jones" <drjones@redhat.com>, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, dzickus@redhat.com, akpm@linux-foundation.org, >mingo@redhat.com
>Sent: Thursday, July 24, 2014 1:26:40 PM
>Subject: Re: [PATCH 2/3] watchdog: control hard lockup detection default
>
>Il 24/07/2014 13:18, Ulrich Obergfell ha scritto:
>>>> >> The running kernel still has the ability to enable/disable at any
>>>> >> time with /proc/sys/kernel/nmi_watchdog us usual. However even
>>>> >> when the default has been overridden /proc/sys/kernel/nmi_watchdog
>>>> >> will initially show '1'. To truly turn it on one must disable/enable
>>>> >> it, i.e.
>>>> >>   echo 0 > /proc/sys/kernel/nmi_watchdog
>>>> >>   echo 1 > /proc/sys/kernel/nmi_watchdog
>>> >
>>> > Why is it hard to make this show the right value? :)
>>> >
>>> > Paolo
>> 'echo 1 > /proc/sys/kernel/nmi_watchdog' enables both - hard lockup and
>> soft lockup detection. watchdog_enable_all_cpus() starts a 'watchdog/N'
>> thread for each CPU. If the kernel runs on a bare metal system where the
>> processor does not have a PMU, or when perf_event_create_kernel_counter()
>> returns failure to watchdog_nmi_enable(), or when the kernel runs as a
>> guest on a hypervisor that does not emulate a PMU, then the 'watchdog/N'
>> threads are still active for soft lockup detection. Patch 2/3 essentially
>> makes watchdog_nmi_enable() behave in the same way as if -ENOENT would
>> have been returned by perf_event_create_kernel_counter(). This is then
>> reported via a console message.
>> 
>>   NMI watchdog: disabled (cpu0): hardware events not enabled
>> 
>> It's hard say what _is_ 'the right value' (because lockup detection is
>> then enabled 'partially'), regardless of whether patch 2/3 is applied
>> or not.
>
> But this means that it is not possible to re-enable softlockup detection
> only.  I think that should be the effect of echo 0 + echo 1, if
> hardlockup detection was disabled by either the command line or patch 3.
>
> Paolo

The idea was to give the user two options to override the effect of patch 3/3.
Either via the kernel command line ('nmi_watchdog=') at boot time or via /proc
('echo 0' + 'echo 1') when the system is up and running.

Regards,

Uli
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Paolo Bonzini July 24, 2014, 11:45 a.m. UTC | #5
Il 24/07/2014 13:44, Ulrich Obergfell ha scritto:
> > But this means that it is not possible to re-enable softlockup detection
> > only.  I think that should be the effect of echo 0 + echo 1, if
> > hardlockup detection was disabled by either the command line or patch 3.
>
> The idea was to give the user two options to override the effect of patch 3/3.
> Either via the kernel command line ('nmi_watchdog=') at boot time or via /proc
> ('echo 0' + 'echo 1') when the system is up and running.

I think the kernel command line is enough; another alternative is to
split the nmi_watchdog /proc entry in two.

Paolo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Ulrich Obergfell July 24, 2014, 12:02 p.m. UTC | #6
>----- Original Message -----
>From: "Paolo Bonzini" <pbonzini@redhat.com>
>To: "Ulrich Obergfell" <uobergfe@redhat.com>
>Cc: "Andrew Jones" <drjones@redhat.com>, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, dzickus@redhat.com, akpm@linux-foundation.org, >mingo@redhat.com
>Sent: Thursday, July 24, 2014 1:45:47 PM
>Subject: Re: [PATCH 2/3] watchdog: control hard lockup detection default
>
>Il 24/07/2014 13:44, Ulrich Obergfell ha scritto:
>> > But this means that it is not possible to re-enable softlockup detection
>> > only.  I think that should be the effect of echo 0 + echo 1, if
>> > hardlockup detection was disabled by either the command line or patch 3.
>>
>> The idea was to give the user two options to override the effect of patch 3/3.
>> Either via the kernel command line ('nmi_watchdog=') at boot time or via /proc
>> ('echo 0' + 'echo 1') when the system is up and running.
>
> I think the kernel command line is enough; another alternative is to
> split the nmi_watchdog /proc entry in two.
>
> Paolo

The current behaviour (without the patch) already allows a user to disable
NMI watchdog at boot time ('nmi_watchdog=0') and enable it explicitly when
the system is up and running ('echo 0' + 'echo 1'). I think it would be
more consistent with this behaviour and more intuitive if we would give
the user the option to override the effect of patch 3/3 via /proc. By
'intuitive' I mean that the user says: 'I _want_ this to be enabled'.

Regards,

Uli

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Ulrich Obergfell July 25, 2014, 8:32 a.m. UTC | #7
> ----- Original Message -----
> From: "Andrew Jones" <drjones@redhat.com>
> To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org
> Cc: uobergfe@redhat.com, dzickus@redhat.com, pbonzini@redhat.com, akpm@linux-foundation.org, mingo@redhat.com
> Sent: Thursday, July 24, 2014 12:13:30 PM
> Subject: [PATCH 2/3] watchdog: control hard lockup detection default

[...]

> The running kernel still has the ability to enable/disable at any
> time with /proc/sys/kernel/nmi_watchdog us usual. However even
> when the default has been overridden /proc/sys/kernel/nmi_watchdog
> will initially show '1'. To truly turn it on one must disable/enable
> it, i.e.
>   echo 0 > /proc/sys/kernel/nmi_watchdog
>   echo 1 > /proc/sys/kernel/nmi_watchdog

[...]

> @@ -626,15 +665,17 @@ int proc_dowatchdog(struct ctl_table *table, int write,
>          * disabled. The 'watchdog_running' variable check in
>          * watchdog_*_all_cpus() function takes care of this.
>          */
> -        if (watchdog_user_enabled && watchdog_thresh)
> +        if (watchdog_user_enabled && watchdog_thresh) {
> +                watchdog_enable_hardlockup_detector(true);
>                  err = watchdog_enable_all_cpus(old_thresh != watchdog_thresh);
> -        else
> +        } else

[...]


I just realized a possible issue in the above part of the patch:

If we would want to give the user the option to override the effect of patch 3/3
via /proc, I think proc_dowatchdog() should enable hard lockup detection _only_
in case of a state transition from 'NOT watchdog_running' to 'watchdog_running'.
                                                          |
   if (watchdog_user_enabled && watchdog_thresh) {        | need to add this
       if (!watchdog_running) <---------------------------'
           watchdog_enable_hardlockup_detector(true);
       err = watchdog_enable_all_cpus(old_thresh != watchdog_thresh);
   } else
       ...

The additional 'if (!watchdog_running)' would _require_ the user to perform the
sequence of commands

   echo 0 > /proc/sys/kernel/nmi_watchdog
   echo 1 > /proc/sys/kernel/nmi_watchdog

to enable hard lockup detection explicitly.

I think changing the 'watchdog_thresh' while 'watchdog_running' is true should
_not_ enable hard lockup detection as a side-effect, because a user may have a
'sysctl.conf' entry such as

   kernel.watchdog_thresh = ...

or may only want to change the 'watchdog_thresh' on the fly.

I think the following flow of execution could cause such undesired side-effect.

   proc_dowatchdog
     if (watchdog_user_enabled && watchdog_thresh) {

         watchdog_enable_hardlockup_detector
           hardlockup_detector_enabled = true

         watchdog_enable_all_cpus
           if (!watchdog_running) {
               ...
           } else if (sample_period_changed)
                      update_timers_all_cpus
                        for_each_online_cpu
                            update_timers
                              watchdog_nmi_disable
                              ...
                              watchdog_nmi_enable

                                watchdog_hardlockup_detector_is_enabled
                                  return true

                                enable perf counter for hard lockup detection

Regards,

Uli
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Andrew Jones July 25, 2014, 11:25 a.m. UTC | #8
On Fri, Jul 25, 2014 at 04:32:55AM -0400, Ulrich Obergfell wrote:
> > ----- Original Message -----
> > From: "Andrew Jones" <drjones@redhat.com>
> > To: linux-kernel@vger.kernel.org, kvm@vger.kernel.org
> > Cc: uobergfe@redhat.com, dzickus@redhat.com, pbonzini@redhat.com, akpm@linux-foundation.org, mingo@redhat.com
> > Sent: Thursday, July 24, 2014 12:13:30 PM
> > Subject: [PATCH 2/3] watchdog: control hard lockup detection default
> 
> [...]
> 
> > The running kernel still has the ability to enable/disable at any
> > time with /proc/sys/kernel/nmi_watchdog us usual. However even
> > when the default has been overridden /proc/sys/kernel/nmi_watchdog
> > will initially show '1'. To truly turn it on one must disable/enable
> > it, i.e.
> >   echo 0 > /proc/sys/kernel/nmi_watchdog
> >   echo 1 > /proc/sys/kernel/nmi_watchdog
> 
> [...]
> 
> > @@ -626,15 +665,17 @@ int proc_dowatchdog(struct ctl_table *table, int write,
> >          * disabled. The 'watchdog_running' variable check in
> >          * watchdog_*_all_cpus() function takes care of this.
> >          */
> > -        if (watchdog_user_enabled && watchdog_thresh)
> > +        if (watchdog_user_enabled && watchdog_thresh) {
> > +                watchdog_enable_hardlockup_detector(true);
> >                  err = watchdog_enable_all_cpus(old_thresh != watchdog_thresh);
> > -        else
> > +        } else
> 
> [...]
> 
> 
> I just realized a possible issue in the above part of the patch:
> 
> If we would want to give the user the option to override the effect of patch 3/3
> via /proc, I think proc_dowatchdog() should enable hard lockup detection _only_
> in case of a state transition from 'NOT watchdog_running' to 'watchdog_running'.
>                                                           |
>    if (watchdog_user_enabled && watchdog_thresh) {        | need to add this
>        if (!watchdog_running) <---------------------------'
>            watchdog_enable_hardlockup_detector(true);
>        err = watchdog_enable_all_cpus(old_thresh != watchdog_thresh);
>    } else
>        ...
> 
> The additional 'if (!watchdog_running)' would _require_ the user to perform the
> sequence of commands
> 
>    echo 0 > /proc/sys/kernel/nmi_watchdog
>    echo 1 > /proc/sys/kernel/nmi_watchdog
> 
> to enable hard lockup detection explicitly.
> 
> I think changing the 'watchdog_thresh' while 'watchdog_running' is true should
> _not_ enable hard lockup detection as a side-effect, because a user may have a
> 'sysctl.conf' entry such as
> 
>    kernel.watchdog_thresh = ...
> 
> or may only want to change the 'watchdog_thresh' on the fly.
> 
> I think the following flow of execution could cause such undesired side-effect.
> 
>    proc_dowatchdog
>      if (watchdog_user_enabled && watchdog_thresh) {
> 
>          watchdog_enable_hardlockup_detector
>            hardlockup_detector_enabled = true
> 
>          watchdog_enable_all_cpus
>            if (!watchdog_running) {
>                ...
>            } else if (sample_period_changed)
>                       update_timers_all_cpus
>                         for_each_online_cpu
>                             update_timers
>                               watchdog_nmi_disable
>                               ...
>                               watchdog_nmi_enable
> 
>                                 watchdog_hardlockup_detector_is_enabled
>                                   return true
> 
>                                 enable perf counter for hard lockup detection
> 
> Regards,
> 
> Uli

Nice catch. Looks like this will need a v2. Paolo, do we have a
consensus on the proc echoing? Or should that be revisited in the v2 as
well?

drew
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/include/linux/nmi.h b/include/linux/nmi.h
index 447775ee2c4b0..72aacf4e3d539 100644
--- a/include/linux/nmi.h
+++ b/include/linux/nmi.h
@@ -17,11 +17,20 @@ 
 #if defined(CONFIG_HAVE_NMI_WATCHDOG) || defined(CONFIG_HARDLOCKUP_DETECTOR)
 #include <asm/nmi.h>
 extern void touch_nmi_watchdog(void);
+extern void watchdog_enable_hardlockup_detector(bool val);
+extern bool watchdog_hardlockup_detector_is_enabled(void);
 #else
 static inline void touch_nmi_watchdog(void)
 {
 	touch_softlockup_watchdog();
 }
+static inline void watchdog_enable_hardlockup_detector(bool)
+{
+}
+static inline bool watchdog_hardlockup_detector_is_enabled(void)
+{
+	return true;
+}
 #endif
 
 /*
diff --git a/kernel/watchdog.c b/kernel/watchdog.c
index c985a21926545..34eca29e28a4c 100644
--- a/kernel/watchdog.c
+++ b/kernel/watchdog.c
@@ -63,6 +63,25 @@  static unsigned long soft_lockup_nmi_warn;
 static int hardlockup_panic =
 			CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE;
 
+static bool hardlockup_detector_enabled = true;
+/*
+ * We may not want to enable hard lockup detection by default in all cases,
+ * for example when running the kernel as a guest on a hypervisor. In these
+ * cases this function can be called to disable hard lockup detection. This
+ * function should only be executed once by the boot processor before the
+ * kernel command line parameters are parsed, because otherwise it is not
+ * possible to override this in hardlockup_panic_setup().
+ */
+void watchdog_enable_hardlockup_detector(bool val)
+{
+	hardlockup_detector_enabled = val;
+}
+
+bool watchdog_hardlockup_detector_is_enabled(void)
+{
+	return hardlockup_detector_enabled;
+}
+
 static int __init hardlockup_panic_setup(char *str)
 {
 	if (!strncmp(str, "panic", 5))
@@ -71,6 +90,14 @@  static int __init hardlockup_panic_setup(char *str)
 		hardlockup_panic = 0;
 	else if (!strncmp(str, "0", 1))
 		watchdog_user_enabled = 0;
+	else if (!strncmp(str, "1", 1) || !strncmp(str, "2", 1)) {
+		/*
+		 * Setting 'nmi_watchdog=1' or 'nmi_watchdog=2' (legacy option)
+		 * has the same effect.
+		 */
+		watchdog_user_enabled = 1;
+		watchdog_enable_hardlockup_detector(true);
+	}
 	return 1;
 }
 __setup("nmi_watchdog=", hardlockup_panic_setup);
@@ -451,6 +478,15 @@  static int watchdog_nmi_enable(unsigned int cpu)
 	struct perf_event_attr *wd_attr;
 	struct perf_event *event = per_cpu(watchdog_ev, cpu);
 
+	/*
+	 * Some kernels need to default hard lockup detection to
+	 * 'disabled', for example a guest on a hypervisor.
+	 */
+	if (!watchdog_hardlockup_detector_is_enabled()) {
+		event = ERR_PTR(-ENOENT);
+		goto handle_err;
+	}
+
 	/* is it already setup and enabled? */
 	if (event && event->state > PERF_EVENT_STATE_OFF)
 		goto out;
@@ -465,6 +501,7 @@  static int watchdog_nmi_enable(unsigned int cpu)
 	/* Try to register using hardware perf events */
 	event = perf_event_create_kernel_counter(wd_attr, cpu, NULL, watchdog_overflow_callback, NULL);
 
+handle_err:
 	/* save cpu0 error for future comparision */
 	if (cpu == 0 && IS_ERR(event))
 		cpu0_err = PTR_ERR(event);
@@ -610,11 +647,13 @@  int proc_dowatchdog(struct ctl_table *table, int write,
 		    void __user *buffer, size_t *lenp, loff_t *ppos)
 {
 	int err, old_thresh, old_enabled;
+	bool old_hardlockup;
 	static DEFINE_MUTEX(watchdog_proc_mutex);
 
 	mutex_lock(&watchdog_proc_mutex);
 	old_thresh = ACCESS_ONCE(watchdog_thresh);
 	old_enabled = ACCESS_ONCE(watchdog_user_enabled);
+	old_hardlockup = watchdog_hardlockup_detector_is_enabled();
 
 	err = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
 	if (err || !write)
@@ -626,15 +665,17 @@  int proc_dowatchdog(struct ctl_table *table, int write,
 	 * disabled. The 'watchdog_running' variable check in
 	 * watchdog_*_all_cpus() function takes care of this.
 	 */
-	if (watchdog_user_enabled && watchdog_thresh)
+	if (watchdog_user_enabled && watchdog_thresh) {
+		watchdog_enable_hardlockup_detector(true);
 		err = watchdog_enable_all_cpus(old_thresh != watchdog_thresh);
-	else
+	} else
 		watchdog_disable_all_cpus();
 
 	/* Restore old values on failure */
 	if (err) {
 		watchdog_thresh = old_thresh;
 		watchdog_user_enabled = old_enabled;
+		watchdog_enable_hardlockup_detector(old_hardlockup);
 	}
 out:
 	mutex_unlock(&watchdog_proc_mutex);