From patchwork Mon Aug 6 03:40:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 10556465 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C027D13BB for ; Mon, 6 Aug 2018 03:41:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4CB65290BE for ; Mon, 6 Aug 2018 03:41:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3DD23290C1; Mon, 6 Aug 2018 03:41:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id AD089290BE for ; Mon, 6 Aug 2018 03:41:15 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 23C3721F492; Sun, 5 Aug 2018 20:41:15 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from mx1.suse.de (mx2.suse.de [195.135.220.15]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 3C72621F465 for ; Sun, 5 Aug 2018 20:41:13 -0700 (PDT) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 3499CACC5; Mon, 6 Aug 2018 03:41:12 +0000 (UTC) From: NeilBrown To: James Simmons , Lustre Development List Date: Mon, 06 Aug 2018 13:40:59 +1000 Message-ID: <87d0uwnovo.fsf@notabene.neil.brown.name> MIME-Version: 1.0 Subject: [lustre-devel] lustre: RFC patch to restore watchdog functionality X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP Lustre in drivers/staging lost the watchdog functionality some time ago Commit: 5d4450c41fe3 ("staging/lustre: fix build error on non-x86 platforms") This is an attempt to provide the same core functionality in a simpler way. It compiles, but I haven't tested it. I don't plan to pursue this further at the moment, but as watchdogs have been under discussion lately I thought it might be interesting to some. NeilBrown From: NeilBrown Date: Mon, 6 Aug 2018 13:37:40 +1000 Subject: [PATCH] lustre: add watchdog for ptlrpc service threads. If a ptlrpc thread takes longer than expected, it can be used to get a stack trace to see where the problem is. This patch restores this functionality by using a delayed work struct to report a back trace of the thread if it ever fires. Signed-off-by: NeilBrown --- drivers/staging/lustre/lustre/include/lustre_net.h | 10 +-- drivers/staging/lustre/lustre/ptlrpc/service.c | 87 ++++++++++++++++++---- 2 files changed, 76 insertions(+), 21 deletions(-) diff --git a/drivers/staging/lustre/lustre/include/lustre_net.h b/drivers/staging/lustre/lustre/include/lustre_net.h index 361b8970368e..3eccec7c7dcc 100644 --- a/drivers/staging/lustre/lustre/include/lustre_net.h +++ b/drivers/staging/lustre/lustre/include/lustre_net.h @@ -1283,18 +1283,14 @@ struct ptlrpc_thread { */ unsigned int t_id; /** - * service thread pid + * service thread */ + struct task_struct *t_task; pid_t t_pid; /** * put watchdog in the structure per thread b=14840 - * - * Lustre watchdog is removed for client in the hope - * of a generic watchdog can be merged in kernel. - * When that happens, we should add below back. - * - * struct lc_watchdog *t_watchdog; */ + struct delayed_work t_watchdog; /** * the svc this thread belonged to b=18582 */ diff --git a/drivers/staging/lustre/lustre/ptlrpc/service.c b/drivers/staging/lustre/lustre/ptlrpc/service.c index 55f68b9b3818..4d508a691e89 100644 --- a/drivers/staging/lustre/lustre/ptlrpc/service.c +++ b/drivers/staging/lustre/lustre/ptlrpc/service.c @@ -1949,6 +1949,73 @@ ptlrpc_at_check(struct ptlrpc_service_part *svcpt) return svcpt->scp_at_check; } +/* + * If a thread spends too long, we want to know about it, + * so we set up a delayed work item as a watchdog. + * If it fires, we display a stack trace of the delayed thread, + * providing we aren't rate-limited + */ +#ifdef CONFIG_STACKTRACE +#define MAX_STACK_TRACE_DEPTH 64 +static void lc_watchdog_fire(struct work_struct *w) +{ + struct ptlrpc_thread *thread = container_of(w, struct ptlrpc_thread, + t_watchdog.work); + struct stack_trace trace; + unsigned long *entries; + unsigned int i; + static DEFINE_RATELIMIT_STATE(_rs, + DEFAULT_RATELIMIT_INTERVAL, + DEFAULT_RATELIMIT_BURST); + + pr_emerg("ptlrpc thread %d blocked for too long\n", thread->t_pid); + + if (!__ratelimit(&_rs)) + return; + + entries = kmalloc_array(MAX_STACK_TRACE_DEPTH, sizeof(*entries), + GFP_KERNEL); + if (!entries) + return; + + trace.nr_entries = 0; + trace.max_entries = MAX_STACK_TRACE_DEPTH; + trace.entries = entries; + trace.skip = 0; + + pr_warn("Stack trace of blocked thread\n"); + save_stack_trace_tsk(thread->t_task, &trace); + + for (i = 0; i < trace.nr_entries; i++) + pr_warn("[<0>] %pB\n", (void *)entries[i]); + + kfree(entries); +} +#else +static void lc_watchdog_fire(struct work_struct *w) +{ + struct ptlrpc_thread *thread = container_of(w, struct ptlrpc_thread, + lc_watchdog.work); + pr_emerg("ptlrpc thread %d blocked for too long\n", thread->t_pid); + +} +#endif + +static inline void lc_watchdog_init(struct delayed_work *work, int time) +{ + INIT_DELAYED_WORK(work, lc_watchdog_fire); + schedule_delayed_work(work, time); +} + +static inline void lc_watchdog_disable(struct delayed_work *work) +{ + cancel_delayed_work_sync(work); +} +static inline void lc_watchdog_touch(struct delayed_work *work, int time) +{ + mod_delayed_work(system_wq, work, time); +} + /** * requests wait on preprocessing * user can call it w/o any lock but need to hold @@ -1972,9 +2039,7 @@ ptlrpc_wait_event(struct ptlrpc_service_part *svcpt, { /* Don't exit while there are replies to be handled */ - /* XXX: Add this back when libcfs watchdog is merged upstream - lc_watchdog_disable(thread->t_watchdog); - */ + lc_watchdog_disable(&thread->t_watchdog); cond_resched(); @@ -2001,10 +2066,8 @@ ptlrpc_wait_event(struct ptlrpc_service_part *svcpt, if (ptlrpc_thread_stopping(thread)) return -EINTR; - /* - lc_watchdog_touch(thread->t_watchdog, + lc_watchdog_touch(&thread->t_watchdog, ptlrpc_server_get_timeout(svcpt)); - */ return 0; } @@ -2024,6 +2087,7 @@ static int ptlrpc_main(void *arg) struct lu_env *env; int counter = 0, rc = 0; + thread->t_task = current; thread->t_pid = current->pid; unshare_fs_struct(); @@ -2104,10 +2168,8 @@ static int ptlrpc_main(void *arg) /* wake up our creator in case he's still waiting. */ wake_up(&thread->t_ctl_waitq); - /* - thread->t_watchdog = lc_watchdog_add(ptlrpc_server_get_timeout(svcpt), - NULL, NULL); - */ + lc_watchdog_init(&thread->t_watchdog, + ptlrpc_server_get_timeout(svcpt)); spin_lock(&svcpt->scp_rep_lock); list_add(&rs->rs_list, &svcpt->scp_rep_idle); @@ -2163,10 +2225,7 @@ static int ptlrpc_main(void *arg) } } - /* - lc_watchdog_delete(thread->t_watchdog); - thread->t_watchdog = NULL; - */ + lc_watchdog_disable(&thread->t_watchdog); out_srv_fini: /*