From patchwork Mon Aug 9 18:09:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 12427085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.0 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7DA4C4338F for ; Mon, 9 Aug 2021 18:09:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BBA4460F02 for ; Mon, 9 Aug 2021 18:09:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232288AbhHISJq (ORCPT ); Mon, 9 Aug 2021 14:09:46 -0400 Received: from mail.kernel.org ([198.145.29.99]:52252 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230243AbhHISJq (ORCPT ); Mon, 9 Aug 2021 14:09:46 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 80DFB60F02 for ; Mon, 9 Aug 2021 18:09:25 +0000 (UTC) Subject: [PATCH v1 1/4] SUNRPC: Add a /sys/kernel/debug/fail_sunrpc/ directory From: Chuck Lever To: linux-nfs@vger.kernel.org Date: Mon, 09 Aug 2021 14:09:24 -0400 Message-ID: <162853256480.4752.1219884794743002339.stgit@klimt.1015granger.net> In-Reply-To: <162853242223.4752.16344468003771993974.stgit@klimt.1015granger.net> References: <162853242223.4752.16344468003771993974.stgit@klimt.1015granger.net> User-Agent: StGit/1.1 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org This directory will contain a set of administrative controls for enabling error injection for kernel RPC consumers. Signed-off-by: Chuck Lever Reported-by: kernel test robot --- net/sunrpc/debugfs.c | 20 ++++++++++++++++++++ net/sunrpc/fail.h | 17 +++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100644 net/sunrpc/fail.h diff --git a/net/sunrpc/debugfs.c b/net/sunrpc/debugfs.c index 56029e3af6ff..7a9065b6a058 100644 --- a/net/sunrpc/debugfs.c +++ b/net/sunrpc/debugfs.c @@ -8,7 +8,9 @@ #include #include #include + #include "netns.h" +#include "fail.h" static struct dentry *topdir; static struct dentry *rpc_clnt_dir; @@ -297,6 +299,22 @@ static const struct file_operations fault_disconnect_fops = { .release = fault_release, }; +struct fail_sunrpc_attr fail_sunrpc = { + .attr = FAULT_ATTR_INITIALIZER, +}; + +#if IS_ENABLED(CONFIG_FAULT_INJECTION_DEBUG_FS) +static void fail_sunrpc_init(void) +{ + fault_create_debugfs_attr("fail_sunrpc", NULL, + &fail_sunrpc.attr); +} +#else +static inline void fail_sunrpc_init(void) +{ +} +#endif + void __exit sunrpc_debugfs_exit(void) { @@ -321,4 +339,6 @@ sunrpc_debugfs_init(void) debugfs_create_file("disconnect", S_IFREG | 0400, rpc_fault_dir, NULL, &fault_disconnect_fops); + + fail_sunrpc_init(); } diff --git a/net/sunrpc/fail.h b/net/sunrpc/fail.h new file mode 100644 index 000000000000..96a46eff94e4 --- /dev/null +++ b/net/sunrpc/fail.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2021, Oracle. All rights reserved. + */ + +#ifndef _NET_SUNRPC_FAIL_H_ +#define _NET_SUNRPC_FAIL_H_ + +#include + +struct fail_sunrpc_attr { + struct fault_attr attr; +}; + +extern struct fail_sunrpc_attr fail_sunrpc; + +#endif /* _NET_SUNRPC_FAIL_H_ */ From patchwork Mon Aug 9 18:09:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 12427087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17E4AC4338F for ; Mon, 9 Aug 2021 18:09:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F2DC560F02 for ; Mon, 9 Aug 2021 18:09:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232625AbhHISJw (ORCPT ); Mon, 9 Aug 2021 14:09:52 -0400 Received: from mail.kernel.org ([198.145.29.99]:52272 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230243AbhHISJw (ORCPT ); Mon, 9 Aug 2021 14:09:52 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 7AA0B60F02 for ; Mon, 9 Aug 2021 18:09:31 +0000 (UTC) Subject: [PATCH v1 2/4] SUNRPC: Server-side disconnect injection From: Chuck Lever To: linux-nfs@vger.kernel.org Date: Mon, 09 Aug 2021 14:09:30 -0400 Message-ID: <162853257078.4752.13164987567936683026.stgit@klimt.1015granger.net> In-Reply-To: <162853242223.4752.16344468003771993974.stgit@klimt.1015granger.net> References: <162853242223.4752.16344468003771993974.stgit@klimt.1015granger.net> User-Agent: StGit/1.1 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Disconnect injection stress-tests the ability for both client and server implementations to behave resiliently in the face of network instability. A file called /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect enables administrators to turn off server-side disconnect injection while allowing other types of sunrpc errors to be injected. So far there are no others. The default setting is that server-side disconnect injection is enabled (ignore=false). Signed-off-by: Chuck Lever Reported-by: kernel test robot Reported-by: kernel test robot --- net/sunrpc/debugfs.c | 9 +++++++-- net/sunrpc/fail.h | 2 ++ net/sunrpc/svc.c | 6 ++++++ 3 files changed, 15 insertions(+), 2 deletions(-) diff --git a/net/sunrpc/debugfs.c b/net/sunrpc/debugfs.c index 7a9065b6a058..654e513afff7 100644 --- a/net/sunrpc/debugfs.c +++ b/net/sunrpc/debugfs.c @@ -306,8 +306,13 @@ struct fail_sunrpc_attr fail_sunrpc = { #if IS_ENABLED(CONFIG_FAULT_INJECTION_DEBUG_FS) static void fail_sunrpc_init(void) { - fault_create_debugfs_attr("fail_sunrpc", NULL, - &fail_sunrpc.attr); + struct dentry *dir; + + dir = fault_create_debugfs_attr("fail_sunrpc", NULL, + &fail_sunrpc.attr); + + debugfs_create_bool("ignore-server-disconnect", S_IFREG | 0600, dir, + &fail_sunrpc.ignore_server_disconnect); } #else static inline void fail_sunrpc_init(void) diff --git a/net/sunrpc/fail.h b/net/sunrpc/fail.h index 96a46eff94e4..302daa1fea8b 100644 --- a/net/sunrpc/fail.h +++ b/net/sunrpc/fail.h @@ -10,6 +10,8 @@ struct fail_sunrpc_attr { struct fault_attr attr; + + bool ignore_server_disconnect; }; extern struct fail_sunrpc_attr fail_sunrpc; diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index 35b549c147a2..332628da0372 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -31,6 +31,8 @@ #include +#include "fail.h" + #define RPCDBG_FACILITY RPCDBG_SVCDSP static void svc_unregister(const struct svc_serv *serv, struct net *net); @@ -1507,6 +1509,10 @@ svc_process(struct svc_rqst *rqstp) struct svc_serv *serv = rqstp->rq_server; u32 dir; + if (!fail_sunrpc.ignore_server_disconnect && + should_fail(&fail_sunrpc.attr, 1)) + svc_xprt_deferred_close(rqstp->rq_xprt); + /* * Setup response xdr_buf. * Initially it has just one page From patchwork Mon Aug 9 18:09:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 12427089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.0 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD63AC4320A for ; Mon, 9 Aug 2021 18:09:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AA4AC60F02 for ; Mon, 9 Aug 2021 18:09:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232764AbhHISJ6 (ORCPT ); Mon, 9 Aug 2021 14:09:58 -0400 Received: from mail.kernel.org ([198.145.29.99]:52316 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230243AbhHISJ6 (ORCPT ); Mon, 9 Aug 2021 14:09:58 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 6E43861004 for ; Mon, 9 Aug 2021 18:09:37 +0000 (UTC) Subject: [PATCH v1 3/4] SUNRPC: Move client-side disconnect injection From: Chuck Lever To: linux-nfs@vger.kernel.org Date: Mon, 09 Aug 2021 14:09:36 -0400 Message-ID: <162853257675.4752.9147452655276556317.stgit@klimt.1015granger.net> In-Reply-To: <162853242223.4752.16344468003771993974.stgit@klimt.1015granger.net> References: <162853242223.4752.16344468003771993974.stgit@klimt.1015granger.net> User-Agent: StGit/1.1 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Disconnect injection stress-tests the ability for both client and server implementations to behave resiliently in the face of network instability. Convert the existing client-side disconnect injection infrastructure to use the kernel's generic error injection facility. The generic facility has a richer set of injection criteria. Signed-off-by: Chuck Lever --- include/linux/sunrpc/xprt.h | 18 ------------ net/sunrpc/debugfs.c | 64 +------------------------------------------ net/sunrpc/fail.h | 1 + net/sunrpc/xprt.c | 8 +++++ 4 files changed, 11 insertions(+), 80 deletions(-) diff --git a/include/linux/sunrpc/xprt.h b/include/linux/sunrpc/xprt.h index c8c39f22d3b1..b15c1f07162d 100644 --- a/include/linux/sunrpc/xprt.h +++ b/include/linux/sunrpc/xprt.h @@ -288,7 +288,6 @@ struct rpc_xprt { const char *address_strings[RPC_DISPLAY_MAX]; #if IS_ENABLED(CONFIG_SUNRPC_DEBUG) struct dentry *debugfs; /* debugfs directory */ - atomic_t inject_disconnect; #endif struct rcu_head rcu; const struct xprt_class *xprt_class; @@ -502,21 +501,4 @@ static inline int xprt_test_and_set_binding(struct rpc_xprt *xprt) return test_and_set_bit(XPRT_BINDING, &xprt->state); } -#if IS_ENABLED(CONFIG_SUNRPC_DEBUG) -extern unsigned int rpc_inject_disconnect; -static inline void xprt_inject_disconnect(struct rpc_xprt *xprt) -{ - if (!rpc_inject_disconnect) - return; - if (atomic_dec_return(&xprt->inject_disconnect)) - return; - atomic_set(&xprt->inject_disconnect, rpc_inject_disconnect); - xprt->ops->inject_disconnect(xprt); -} -#else -static inline void xprt_inject_disconnect(struct rpc_xprt *xprt) -{ -} -#endif - #endif /* _LINUX_SUNRPC_XPRT_H */ diff --git a/net/sunrpc/debugfs.c b/net/sunrpc/debugfs.c index 654e513afff7..8c3d52df982f 100644 --- a/net/sunrpc/debugfs.c +++ b/net/sunrpc/debugfs.c @@ -16,8 +16,6 @@ static struct dentry *topdir; static struct dentry *rpc_clnt_dir; static struct dentry *rpc_xprt_dir; -unsigned int rpc_inject_disconnect; - static int tasks_show(struct seq_file *f, void *v) { @@ -237,8 +235,6 @@ rpc_xprt_debugfs_register(struct rpc_xprt *xprt) /* make tasks file */ debugfs_create_file("info", S_IFREG | 0400, xprt->debugfs, xprt, &xprt_info_fops); - - atomic_set(&xprt->inject_disconnect, rpc_inject_disconnect); } void @@ -248,57 +244,6 @@ rpc_xprt_debugfs_unregister(struct rpc_xprt *xprt) xprt->debugfs = NULL; } -static int -fault_open(struct inode *inode, struct file *filp) -{ - filp->private_data = kmalloc(128, GFP_KERNEL); - if (!filp->private_data) - return -ENOMEM; - return 0; -} - -static int -fault_release(struct inode *inode, struct file *filp) -{ - kfree(filp->private_data); - return 0; -} - -static ssize_t -fault_disconnect_read(struct file *filp, char __user *user_buf, - size_t len, loff_t *offset) -{ - char *buffer = (char *)filp->private_data; - size_t size; - - size = sprintf(buffer, "%u\n", rpc_inject_disconnect); - return simple_read_from_buffer(user_buf, len, offset, buffer, size); -} - -static ssize_t -fault_disconnect_write(struct file *filp, const char __user *user_buf, - size_t len, loff_t *offset) -{ - char buffer[16]; - - if (len >= sizeof(buffer)) - len = sizeof(buffer) - 1; - if (copy_from_user(buffer, user_buf, len)) - return -EFAULT; - buffer[len] = '\0'; - if (kstrtouint(buffer, 10, &rpc_inject_disconnect)) - return -EINVAL; - return len; -} - -static const struct file_operations fault_disconnect_fops = { - .owner = THIS_MODULE, - .open = fault_open, - .read = fault_disconnect_read, - .write = fault_disconnect_write, - .release = fault_release, -}; - struct fail_sunrpc_attr fail_sunrpc = { .attr = FAULT_ATTR_INITIALIZER, }; @@ -313,6 +258,8 @@ static void fail_sunrpc_init(void) debugfs_create_bool("ignore-server-disconnect", S_IFREG | 0600, dir, &fail_sunrpc.ignore_server_disconnect); + debugfs_create_bool("ignore-client-disconnect", S_IFREG | 0600, dir, + &fail_sunrpc.ignore_client_disconnect); } #else static inline void fail_sunrpc_init(void) @@ -332,18 +279,11 @@ sunrpc_debugfs_exit(void) void __init sunrpc_debugfs_init(void) { - struct dentry *rpc_fault_dir; - topdir = debugfs_create_dir("sunrpc", NULL); rpc_clnt_dir = debugfs_create_dir("rpc_clnt", topdir); rpc_xprt_dir = debugfs_create_dir("rpc_xprt", topdir); - rpc_fault_dir = debugfs_create_dir("inject_fault", topdir); - - debugfs_create_file("disconnect", S_IFREG | 0400, rpc_fault_dir, NULL, - &fault_disconnect_fops); - fail_sunrpc_init(); } diff --git a/net/sunrpc/fail.h b/net/sunrpc/fail.h index 302daa1fea8b..e5d3e80fac0c 100644 --- a/net/sunrpc/fail.h +++ b/net/sunrpc/fail.h @@ -12,6 +12,7 @@ struct fail_sunrpc_attr { struct fault_attr attr; bool ignore_server_disconnect; + bool ignore_client_disconnect; }; extern struct fail_sunrpc_attr fail_sunrpc; diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c index fb6db09725c7..fbbc38771c65 100644 --- a/net/sunrpc/xprt.c +++ b/net/sunrpc/xprt.c @@ -56,6 +56,7 @@ #include "sunrpc.h" #include "sysfs.h" +#include "fail.h" /* * Local variables @@ -855,6 +856,13 @@ xprt_init_autodisconnect(struct timer_list *t) queue_work(xprtiod_workqueue, &xprt->task_cleanup); } +static void xprt_inject_disconnect(struct rpc_xprt *xprt) +{ + if (!fail_sunrpc.ignore_client_disconnect && + should_fail(&fail_sunrpc.attr, 1)) + xprt->ops->inject_disconnect(xprt); +} + bool xprt_lock_connect(struct rpc_xprt *xprt, struct rpc_task *task, void *cookie) From patchwork Mon Aug 9 18:09:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 12427091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F4C9C4338F for ; Mon, 9 Aug 2021 18:09:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 86D0D61004 for ; Mon, 9 Aug 2021 18:09:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232875AbhHISKE (ORCPT ); Mon, 9 Aug 2021 14:10:04 -0400 Received: from mail.kernel.org ([198.145.29.99]:52336 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230243AbhHISKE (ORCPT ); Mon, 9 Aug 2021 14:10:04 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 6539A61004 for ; Mon, 9 Aug 2021 18:09:43 +0000 (UTC) Subject: [PATCH v1 4/4] SUNRPC: Add documentation for the fail_sunrpc/ directory From: Chuck Lever To: linux-nfs@vger.kernel.org Date: Mon, 09 Aug 2021 14:09:42 -0400 Message-ID: <162853258270.4752.4445102254929246927.stgit@klimt.1015granger.net> In-Reply-To: <162853242223.4752.16344468003771993974.stgit@klimt.1015granger.net> References: <162853242223.4752.16344468003771993974.stgit@klimt.1015granger.net> User-Agent: StGit/1.1 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org Signed-off-by: Chuck Lever --- Documentation/fault-injection/fault-injection.rst | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/Documentation/fault-injection/fault-injection.rst b/Documentation/fault-injection/fault-injection.rst index f47d05ed0d94..4a25c5eb6f07 100644 --- a/Documentation/fault-injection/fault-injection.rst +++ b/Documentation/fault-injection/fault-injection.rst @@ -24,6 +24,10 @@ Available fault injection capabilities injects futex deadlock and uaddr fault errors. +- fail_sunrpc + + injects kernel RPC client and server failures. + - fail_make_request injects disk IO errors on devices permitted by setting @@ -151,6 +155,20 @@ configuration of fault-injection capabilities. default is 'N', setting it to 'Y' will disable failure injections when dealing with private (address space) futexes. +- /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect: + + Format: { 'Y' | 'N' } + + default is 'N', setting it to 'Y' will disable disconnect + injection on the RPC client. + +- /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect: + + Format: { 'Y' | 'N' } + + default is 'N', setting it to 'Y' will disable disconnect + injection on the RPC server. + - /sys/kernel/debug/fail_function/inject: Format: { 'function-name' | '!function-name' | '' }