From patchwork Thu Jul 25 02:44:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 11057839 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B851313B1 for ; Thu, 25 Jul 2019 02:44:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A67E6287C2 for ; Thu, 25 Jul 2019 02:44:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9AE21288AA; Thu, 25 Jul 2019 02:44:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4BB1F287C2 for ; Thu, 25 Jul 2019 02:44:20 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id E36754C3F71; Wed, 24 Jul 2019 19:44:18 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 464FA21FB77 for ; Wed, 24 Jul 2019 19:44:13 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 6DE461005266; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 64CD92DB; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown , Shaun Tancheff Date: Wed, 24 Jul 2019 22:44:00 -0400 Message-Id: <1564022647-17351-2-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> References: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 1/8] lustre: seq: make seq_proc_write_common() safer X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Andreas Dilger Don't allow seq_proc_write_common() to specify arbitrary ranges, since this can permanently corrupt the sequence controller and/or sequnece server. That would allow duplicate FID allocation, or possibly prevent any new files to be created or servers to be added to the filesystem. Instead, limit the sequence range that can be written via /proc to a subset of the sequence range currently allocated to that node. Add the "clear" keyword to allow dropping the entire local sequence and force a new one to be fetched from the sequence server. WC-bug-id: https://jira.whamcloud.com/browse/LU-3642 Lustre-commit: 05f69f5ee20eeffcc26f643333cedcfb53ba6669 Signed-off-by: Andreas Dilger Reviewed-on: http://review.whamcloud.com/7123 Reviewed-by: Jinshan Xiong Reviewed-by: Alex Zhuravlev Reviewed-by: James Simmons Reviewed-by: Oleg Drokin --- fs/lustre/fid/lproc_fid.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/fs/lustre/fid/lproc_fid.c b/fs/lustre/fid/lproc_fid.c index 94869d4..e2e47df 100644 --- a/fs/lustre/fid/lproc_fid.c +++ b/fs/lustre/fid/lproc_fid.c @@ -52,14 +52,18 @@ /* Format: [0x64BIT_INT - 0x64BIT_INT] + 32 bytes just in case */ #define MAX_FID_RANGE_STRLEN (32 + 2 * 2 * sizeof(u64)) /* - * Note: this function is only used for testing, it is no safe for production - * use. + * Reduce the SEQ range allocated to a node to a strict subset of the range + * currently-allocated SEQ range. If the specified range is "clear", then + * drop all allocated sequences and request a new one from the master. + * + * Note: this function should only be used for testing, it is not necessarily + * safe for production use. */ static int ldebugfs_fid_write_common(const char __user *buffer, size_t count, struct lu_seq_range *range) { - struct lu_seq_range tmp; + struct lu_seq_range tmp = { 0, }; int rc; char kernbuf[MAX_FID_RANGE_STRLEN]; @@ -82,8 +86,6 @@ rc = sscanf(kernbuf, "[%llx - %llx]\n", (unsigned long long *)&tmp.lsr_start, (unsigned long long *)&tmp.lsr_end); - if (rc != 2) - return -EINVAL; if (!lu_seq_range_is_sane(&tmp) || lu_seq_range_is_zero(&tmp) || tmp.lsr_start < range->lsr_start || tmp.lsr_end > range->lsr_end) return -EINVAL; From patchwork Thu Jul 25 02:44:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 11057845 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ABA7513B1 for ; Thu, 25 Jul 2019 02:44:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 995A0287C2 for ; Thu, 25 Jul 2019 02:44:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8DB67288AA; Thu, 25 Jul 2019 02:44:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3A03C287C2 for ; Thu, 25 Jul 2019 02:44:28 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 1FF694C43F9; Wed, 24 Jul 2019 19:44:24 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 8F7DC21FB77 for ; Wed, 24 Jul 2019 19:44:13 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 710091005267; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 66E4C2DC; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown , Shaun Tancheff Date: Wed, 24 Jul 2019 22:44:01 -0400 Message-Id: <1564022647-17351-3-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> References: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 2/8] lustre: ptlrpc: Fix an rq_no_reply assertion failure X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Li Wei , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Li Wei An OSS had an assertion failure: LustreError: 5366:0:(ldlm_lib.c:2689:target_bulk_io()) @@@ timeout on bulk GET after 0+0s req@ffff88083a61b400 x1476486691018500/t0(4300509964) o4->8dda3382-83f8-6445-5eea-828fd59e4a06@192.168.1.116@o2ib1:0/0 lens 504/448 e 391470 to 0 dl 1408494729 ref 2 fl Complete:/4/0 rc 0/0 LustreError: 5432:0:(niobuf.c:550:ptlrpc_send_reply()) ASSERTION( req->rq_no_reply == 0 ) failed: Lustre: soaked-OST0000: Bulk IO write error with 8dda3382-83f8-6445-5eea-828fd59e4a06 (at 192.168.1.116@o2ib1), client will retry: rc -110 LustreError: 5432:0:(niobuf.c:550:ptlrpc_send_reply()) LBUG Pid: 5432, comm: ll_ost_io03_003 Call Trace: [] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [] lbug_with_loc+0x47/0xb0 [libcfs] [] ptlrpc_send_reply+0x4ec/0x7f0 [ptlrpc] [] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] [] ptlrpc_at_check_timed+0xcd5/0x1370 [ptlrpc] [] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] [] ptlrpc_main+0x12e8/0x1990 [ptlrpc] [] ? pick_next_task_fair+0xd0/0x130 [] ? schedule+0x176/0x3b0 [] ? ptlrpc_main+0x0/0x1990 [ptlrpc] [] kthread+0x96/0xa0 [] child_rip+0xa/0x20 [] ? kthread+0x0/0xa0 [] ? child_rip+0x0/0x20 The thread in tgt_brw_write() had decided not to reply by setting rq_no_reply, right before another thread tried to send an early reply for the request. WC-bug-id: https://jira.whamcloud.com/browse/LU-5537 Lustre-commit: a8d448e4cd5978c546911f98067232bcdd30b651 Signed-off-by: Li Wei Reviewed-on: http://review.whamcloud.com/11740 Reviewed-by: Andreas Dilger Reviewed-by: Johann Lombardi --- fs/lustre/ptlrpc/service.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/fs/lustre/ptlrpc/service.c b/fs/lustre/ptlrpc/service.c index a40e964..c9ab9c3 100644 --- a/fs/lustre/ptlrpc/service.c +++ b/fs/lustre/ptlrpc/service.c @@ -1098,6 +1098,16 @@ static int ptlrpc_at_send_early_reply(struct ptlrpc_request *req) reqcopy->rq_reqmsg = reqmsg; memcpy(reqmsg, req->rq_reqmsg, req->rq_reqlen); + /* + * tgt_brw_read() and tgt_brw_write() may have decided not to reply. + * Without this check, we would fail the rq_no_reply assertion in + * ptlrpc_send_reply(). + */ + if (reqcopy->rq_no_reply) { + rc = -ETIMEDOUT; + goto out; + } + LASSERT(atomic_read(&req->rq_refcount)); /** if it is last refcount then early reply isn't needed */ if (atomic_read(&req->rq_refcount) == 1) { From patchwork Thu Jul 25 02:44:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 11057837 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 173C213B1 for ; Thu, 25 Jul 2019 02:44:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 05279287C2 for ; Thu, 25 Jul 2019 02:44:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ED3F4288AA; Thu, 25 Jul 2019 02:44:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9D2F0287C2 for ; Thu, 25 Jul 2019 02:44:16 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 2517C4C339B; Wed, 24 Jul 2019 19:44:16 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id E2C9C21FEDC for ; Wed, 24 Jul 2019 19:44:13 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 73D02100537B; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 6A34A2DD; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown , Shaun Tancheff Date: Wed, 24 Jul 2019 22:44:02 -0400 Message-Id: <1564022647-17351-4-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> References: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 3/8] lustre: fld: resend seq lookup RPC if it is on LWP X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: wang di , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: wang di Because Light Weight connection might be evicted after restart, then cause inflight RPC fails, to avoid this, we need resend seq lookup RPC. remove "-f" from "stop mdt" in sanity 17m, so umount can keep the the connection, and otherwise the OSP might be evicted. WC-bug-id: https://jira.whamcloud.com/browse/LU-4571 Lustre-commit: cf7f66d87e52293535cde6e8cc7386e6c1bdfa46 Signed-off-by: wang di Reviewed-on: http://review.whamcloud.com/9106 Reviewed-by: Andreas Dilger Reviewed-by: Jinshan Xiong Reviewed-by: Niu Yawei --- fs/lustre/fld/fld_request.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/fs/lustre/fld/fld_request.c b/fs/lustre/fld/fld_request.c index 248fffa..ec45ea6 100644 --- a/fs/lustre/fld/fld_request.c +++ b/fs/lustre/fld/fld_request.c @@ -314,6 +314,7 @@ int fld_client_rpc(struct obd_export *exp, LASSERT(exp); +again: imp = class_exp2cliimp(exp); switch (fld_op) { case FLD_QUERY: @@ -329,8 +330,15 @@ int fld_client_rpc(struct obd_export *exp, op = req_capsule_client_get(&req->rq_pill, &RMF_FLD_OPC); *op = FLD_LOOKUP; - if (imp->imp_connect_flags_orig & OBD_CONNECT_MDS_MDS) + /* For MDS_MDS seq lookup, it will always use LWP connection, + * but LWP will be evicted after restart, so cause the error. + * so we will set no_delay for seq lookup request, once the + * request fails because of the eviction. always retry here + */ + if (imp->imp_connect_flags_orig & OBD_CONNECT_MDS_MDS) { req->rq_allow_replay = 1; + req->rq_no_delay = 1; + } break; case FLD_READ: req = ptlrpc_request_alloc_pack(imp, &RQF_FLD_READ, @@ -358,8 +366,19 @@ int fld_client_rpc(struct obd_export *exp, obd_get_request_slot(&exp->exp_obd->u.cli); rc = ptlrpc_queue_wait(req); obd_put_request_slot(&exp->exp_obd->u.cli); - if (rc) + if (rc != 0) { + if (rc == -EWOULDBLOCK) { + /* For no_delay req(see above), EWOULDBLOCK means the + * connection is being evicted, but this seq lookup + * should not return error, since it would cause + * unecessary failure of the application, instead + * it should retry here + */ + ptlrpc_req_finished(req); + goto again; + } goto out_req; + } if (fld_op == FLD_QUERY) { prange = req_capsule_server_get(&req->rq_pill, &RMF_FLD_MDFLD); From patchwork Thu Jul 25 02:44:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 11057849 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2D9BF746 for ; Thu, 25 Jul 2019 02:44:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1AFC0287C2 for ; Thu, 25 Jul 2019 02:44:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0E789288AA; Thu, 25 Jul 2019 02:44:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id BCE45287C2 for ; Thu, 25 Jul 2019 02:44:35 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id CBC684C44DB; Wed, 24 Jul 2019 19:44:29 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 361184C3C1F for ; Wed, 24 Jul 2019 19:44:14 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 75259100537C; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 6D8252DE; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown , Shaun Tancheff Date: Wed, 24 Jul 2019 22:44:03 -0400 Message-Id: <1564022647-17351-5-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> References: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 4/8] lustre: fld: retry fld rpc even for ESHUTDOWN X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: wang di , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: wang di when LWP is being evicted, because it is not replayable, the request might return ESHUTDOWN or EWOULDBLOCK, instead of failed, which might cause application failure, fld client will retry RPC, until the connection is being setup again or the LWP is being closed. WC-bug-id: https://jira.whamcloud.com/browse/LU-4420 Lustre-commit: d335e310d4bf490509998ddbb1824e38cff20998 Signed-off-by: wang di Reviewed-on: http://review.whamcloud.com/10285 Reviewed-by: Fan Yong Reviewed-by: Johann Lombardi --- fs/lustre/fld/fld_request.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/fs/lustre/fld/fld_request.c b/fs/lustre/fld/fld_request.c index ec45ea6..ba0ef82 100644 --- a/fs/lustre/fld/fld_request.c +++ b/fs/lustre/fld/fld_request.c @@ -367,12 +367,12 @@ int fld_client_rpc(struct obd_export *exp, rc = ptlrpc_queue_wait(req); obd_put_request_slot(&exp->exp_obd->u.cli); if (rc != 0) { - if (rc == -EWOULDBLOCK) { - /* For no_delay req(see above), EWOULDBLOCK means the - * connection is being evicted, but this seq lookup - * should not return error, since it would cause - * unecessary failure of the application, instead - * it should retry here + if (rc == -EWOULDBLOCK || rc == -ESHUTDOWN) { + /* For no_delay req(see above), EWOULDBLOCK and + * ESHUTDOWN means the connection is being evicted, + * but this seq lookup should not return error, + * since it would cause unecessary failure of the + * application, instead it should retry here */ ptlrpc_req_finished(req); goto again; From patchwork Thu Jul 25 02:44:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 11057841 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E91EB746 for ; Thu, 25 Jul 2019 02:44:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D7D47287C2 for ; Thu, 25 Jul 2019 02:44:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CBBD0288AA; Thu, 25 Jul 2019 02:44:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 85E41287C2 for ; Thu, 25 Jul 2019 02:44:21 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id C42704C438B; Wed, 24 Jul 2019 19:44:19 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 7D8544C3F10 for ; Wed, 24 Jul 2019 19:44:14 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 7AAAD100537F; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 708E09BF; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown , Shaun Tancheff Date: Wed, 24 Jul 2019 22:44:04 -0400 Message-Id: <1564022647-17351-6-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> References: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 5/8] lustre: fld: retry fld rpc until the import is closed X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Wang Di , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Wang Di Since LWP is not replayable, it should retry RPC until the import is closed, otherwise it would cause unnecessary failure of the application. WC-bug-id: https://jira.whamcloud.com/browse/LU-5312 Lustre-commit: 07d481957c16832c782bb7d0c1fc436bcb148ea6 Signed-off-by: Wang Di Reviewed-on: http://review.whamcloud.com/11039 Reviewed-by: Andreas Dilger Reviewed-by: Fan Yong Reviewed-by: Oleg Drokin --- fs/lustre/fld/fld_request.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/fs/lustre/fld/fld_request.c b/fs/lustre/fld/fld_request.c index ba0ef82..60e7105 100644 --- a/fs/lustre/fld/fld_request.c +++ b/fs/lustre/fld/fld_request.c @@ -367,12 +367,10 @@ int fld_client_rpc(struct obd_export *exp, rc = ptlrpc_queue_wait(req); obd_put_request_slot(&exp->exp_obd->u.cli); if (rc != 0) { - if (rc == -EWOULDBLOCK || rc == -ESHUTDOWN) { - /* For no_delay req(see above), EWOULDBLOCK and - * ESHUTDOWN means the connection is being evicted, - * but this seq lookup should not return error, - * since it would cause unecessary failure of the - * application, instead it should retry here + if (imp->imp_state != LUSTRE_IMP_CLOSED) { + /* Since LWP is not replayable, so it will keep + * trying unless umount happens, otherwise it would + * cause unecessary failure of the application. */ ptlrpc_req_finished(req); goto again; From patchwork Thu Jul 25 02:44:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 11057851 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C4CBE13B1 for ; Thu, 25 Jul 2019 02:44:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B37D7287C2 for ; Thu, 25 Jul 2019 02:44:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A7F59288AA; Thu, 25 Jul 2019 02:44:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 480FB287C2 for ; Thu, 25 Jul 2019 02:44:38 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id C319A4C446B; Wed, 24 Jul 2019 19:44:32 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 18CAA4C3F10 for ; Wed, 24 Jul 2019 19:44:15 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 7FEA11005393; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 739469C2; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown , Shaun Tancheff Date: Wed, 24 Jul 2019 22:44:05 -0400 Message-Id: <1564022647-17351-7-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> References: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 6/8] lustre: fld: fld client lookup should retry X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: wang di , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: wang di If FLD client lookup fails because of the remote target is shutdown (or deactive), it should retry another target, otherwise it will cause the application failure. And FLD client should stop retry if the import has been deactive. WC-bug-id: https://jira.whamcloud.com/browse/LU-6419 Lustre-commit: 3ededde903c92f8485cae0dc9f958f194ff0b140 Signed-off-by: wang di Reviewed-on: http://review.whamcloud.com/14313 Reviewed-by: Lai Siyao Reviewed-by: Fan Yong Reviewed-by: Alex Zhuravlev Reviewed-by: Oleg Drokin --- fs/lustre/fld/fld_request.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/fs/lustre/fld/fld_request.c b/fs/lustre/fld/fld_request.c index 60e7105..dfd4ae9 100644 --- a/fs/lustre/fld/fld_request.c +++ b/fs/lustre/fld/fld_request.c @@ -367,7 +367,7 @@ int fld_client_rpc(struct obd_export *exp, rc = ptlrpc_queue_wait(req); obd_put_request_slot(&exp->exp_obd->u.cli); if (rc != 0) { - if (imp->imp_state != LUSTRE_IMP_CLOSED) { + if (imp->imp_state != LUSTRE_IMP_CLOSED && !imp->imp_deactive) { /* Since LWP is not replayable, so it will keep * trying unless umount happens, otherwise it would * cause unecessary failure of the application. @@ -404,6 +404,7 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds, { struct lu_seq_range res = { 0 }; struct lu_fld_target *target; + struct lu_fld_target *origin; int rc; rc = fld_cache_lookup(fld->lcf_cache, seq, &res); @@ -415,7 +416,8 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds, /* Can not find it in the cache */ target = fld_client_get_target(fld, seq); LASSERT(target); - + origin = target; +again: CDEBUG(D_INFO, "%s: Lookup fld entry (seq: %#llx) on target %s (idx %llu)\n", fld->lcf_name, seq, fld_target_name(target), target->ft_idx); @@ -424,6 +426,23 @@ int fld_client_lookup(struct lu_client_fld *fld, u64 seq, u32 *mds, fld_range_set_type(&res, flags); rc = fld_client_rpc(target->ft_exp, &res, FLD_QUERY, NULL); + if (rc == -ESHUTDOWN) { + /* If fld lookup failed because the target has been shutdown, + * then try next target in the list, until trying all targets + * or fld lookup succeeds + */ + spin_lock(&fld->lcf_lock); + if (target->ft_chain.next == fld->lcf_targets.prev) + target = list_entry(fld->lcf_targets.next, + struct lu_fld_target, ft_chain); + else + target = list_entry(target->ft_chain.next, + struct lu_fld_target, + ft_chain); + spin_unlock(&fld->lcf_lock); + if (target != origin) + goto again; + } if (rc == 0) { *mds = res.lsr_index; From patchwork Thu Jul 25 02:44:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 11057843 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AEEF413B1 for ; Thu, 25 Jul 2019 02:44:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9D7DA287C2 for ; Thu, 25 Jul 2019 02:44:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 918C2288AA; Thu, 25 Jul 2019 02:44:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1B77B287C2 for ; Thu, 25 Jul 2019 02:44:26 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 96E2E4C447E; Wed, 24 Jul 2019 19:44:22 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id C52144C3F10 for ; Wed, 24 Jul 2019 19:44:14 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 7DB841005380; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 76FCFB45; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown , Shaun Tancheff Date: Wed, 24 Jul 2019 22:44:06 -0400 Message-Id: <1564022647-17351-8-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> References: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 7/8] lustre: tests: testcases for multiple modify RPCs feature X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Gregoire Pichon This patch creates new testcases in the Auster test suite to test the support of multiple modify RPCs in flight feature. Two new OBD_FAIL codes are added to allow several failure occurences of the reint requests or replies. This is needed because the current fail checks impose the OBD_FAIL_ONCE flag. Added testcases are : - sanity 245 - conf-sanity 90a 90b 90c 90d - replay-single 102a 102b 102c 102d This patch also removes replay-single tests 53a and 53b from the ALWAYS_EXCEPT list. WC-bug-id: https://jira.whamcloud.com/browse/LU-5319 Lustre-commit: c2d27a0f12688c0d029880919f8b002e557b540c Signed-off-by: Gregoire Pichon Reviewed-on: http://review.whamcloud.com/14861 Reviewed-by: Jian Yu Reviewed-by: Andreas Dilger --- fs/lustre/include/obd_support.h | 2 ++ fs/lustre/ldlm/ldlm_lib.c | 3 +++ 2 files changed, 5 insertions(+) diff --git a/fs/lustre/include/obd_support.h b/fs/lustre/include/obd_support.h index 3e15cac..7999ac6 100644 --- a/fs/lustre/include/obd_support.h +++ b/fs/lustre/include/obd_support.h @@ -189,6 +189,8 @@ #define OBD_FAIL_MDS_SWAP_LAYOUTS_NET 0x14f #define OBD_FAIL_MDS_HSM_ACTION_NET 0x150 #define OBD_FAIL_MDS_CHANGELOG_INIT 0x151 +#define OBD_FAIL_MDS_REINT_MULTI_NET 0x159 +#define OBD_FAIL_MDS_REINT_MULTI_NET_REP 0x15a /* layout lock */ #define OBD_FAIL_MDS_NO_LL_GETATTR 0x170 diff --git a/fs/lustre/ldlm/ldlm_lib.c b/fs/lustre/ldlm/ldlm_lib.c index 887507d..a3b8df4 100644 --- a/fs/lustre/ldlm/ldlm_lib.c +++ b/fs/lustre/ldlm/ldlm_lib.c @@ -689,6 +689,9 @@ int target_pack_pool_reply(struct ptlrpc_request *req) DEBUG_REQ(D_ERROR, req, "dropping reply"); return -ECOMM; } + if (unlikely(lustre_msg_get_opc(req->rq_reqmsg) == MDS_REINT && + OBD_FAIL_CHECK(OBD_FAIL_MDS_REINT_MULTI_NET_REP))) + return -ECOMM; if (unlikely(rc)) { DEBUG_REQ(D_NET, req, "processing error (%d)", rc); From patchwork Thu Jul 25 02:44:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 11057847 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 897C3746 for ; Thu, 25 Jul 2019 02:44:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 77A3228895 for ; Thu, 25 Jul 2019 02:44:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6BFDF288C3; Thu, 25 Jul 2019 02:44:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2BAC228895 for ; Thu, 25 Jul 2019 02:44:31 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 973C221FEDC; Wed, 24 Jul 2019 19:44:26 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp4.ccs.ornl.gov (smtp4.ccs.ornl.gov [160.91.203.40]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 61A364C3F10 for ; Wed, 24 Jul 2019 19:44:15 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp4.ccs.ornl.gov (Postfix) with ESMTP id 806F81005394; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 7A8C8B48; Wed, 24 Jul 2019 22:44:11 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown , Shaun Tancheff Date: Wed, 24 Jul 2019 22:44:07 -0400 Message-Id: <1564022647-17351-9-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> References: <1564022647-17351-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 8/8] lustre: ldlm: Don't check opcode with NULL rq_reqmsg X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jeremy Filizetti , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Jeremy Filizetti When GSS is enabled it's possible to have a NULL rq_reqmsg if a bad signature or no context is returned during the unwrap of the request. Don't check the opcode in this case. WC-bug-id: https://jira.whamcloud.com/browse/LU-7508 Lustre-commit: 3f4572caef5f25f4a9b5347b2ccf933fdad9db9c Signed-off-by: Jeremy Filizetti Reviewed-on: http://review.whamcloud.com/17414 Reviewed-by: Andreas Dilger Reviewed-by: Sebastien Buisson Reviewed-by: John L. Hammond --- fs/lustre/ldlm/ldlm_lib.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/fs/lustre/ldlm/ldlm_lib.c b/fs/lustre/ldlm/ldlm_lib.c index a3b8df4..04982b8 100644 --- a/fs/lustre/ldlm/ldlm_lib.c +++ b/fs/lustre/ldlm/ldlm_lib.c @@ -689,8 +689,12 @@ int target_pack_pool_reply(struct ptlrpc_request *req) DEBUG_REQ(D_ERROR, req, "dropping reply"); return -ECOMM; } - if (unlikely(lustre_msg_get_opc(req->rq_reqmsg) == MDS_REINT && - OBD_FAIL_CHECK(OBD_FAIL_MDS_REINT_MULTI_NET_REP))) + /* We can have a null rq_reqmsg in the event of bad signature or + * no context when unwrapping + */ + if (req->rq_reqmsg && + unlikely(lustre_msg_get_opc(req->rq_reqmsg) == MDS_REINT && + OBD_FAIL_CHECK(OBD_FAIL_MDS_REINT_MULTI_NET_REP))) return -ECOMM; if (unlikely(rc)) {