From patchwork Mon Oct 16 16:14:33 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 10009137 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A1264601E7 for ; Mon, 16 Oct 2017 16:14:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 924562861E for ; Mon, 16 Oct 2017 16:14:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8674628622; Mon, 16 Oct 2017 16:14:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CAC2D2861E for ; Mon, 16 Oct 2017 16:14:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754929AbdJPQOk (ORCPT ); Mon, 16 Oct 2017 12:14:40 -0400 Received: from mail-io0-f195.google.com ([209.85.223.195]:54360 "EHLO mail-io0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752788AbdJPQOg (ORCPT ); Mon, 16 Oct 2017 12:14:36 -0400 Received: by mail-io0-f195.google.com with SMTP id e89so8944393ioi.11 for ; Mon, 16 Oct 2017 09:14:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:user-agent:mime-version :content-transfer-encoding; bh=XZiWiA1DWEvO6QKvg9AotDazNhOy6O6uu+PHuZymf0I=; b=U0kh27nGk0Iyp6Vcg8/oqtByF2s249hcHmgSvrrH2cOvwtACqn7hjY4BddQ2zwVi5f +ckWIXnR8kFc15Tph/SD2tXgZGyUMaIk36m0zfsOU2roodesK0+E1kkfsLnrY+n2D5LR bEmJlKSQfmwwNeisyFTw964GQh6Z1RnBQ1uPfH9Opk3zMCeEIW2eGgv/Jb4HMq4ARTbs 9EomkocBYg0OubKHEBTtv9VFC9DP5uB681LgY038gsAVk2bYCk56lxYUE3uCozaGoMDT f709pNrKZtcWEx+YO2dYkpv3u0agKaJMdiYxVm4qygHVIkyGzNhsag/Ikbpgy10S98Df ZrhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :user-agent:mime-version:content-transfer-encoding; bh=XZiWiA1DWEvO6QKvg9AotDazNhOy6O6uu+PHuZymf0I=; b=CmcNbwd3qlnzesw/chKylmEBS8+UADxD/OCZ3pGmJlrYF+YD1is7rY5O8sqzNHfJaJ J9USzlwnS0gG4hjqtQL7XuxVkW1fVgq8S4uCQUaYkZTHpJMv/cHK3LDOk/tYVfz2PglP Lmm9P2XVgne/oez/GU+v6jyciHcWNqOROBjAODNgnfeNLdsX/ciXn9hIWl7Y2DL5I5J+ DMJVGl7cCweS1gKxzEtoGPsQpfPVV2rGaYwnkpuYYUs3Xhy1EVXOpmCAEiLIkZcKQz2+ 5evdQTwgB6NVN/SGLD2mjs/lULQj98aZe03xcz5rNgaYQxSVAb1e93p8sU+Qwkakju/D l+nQ== X-Gm-Message-State: AMCzsaV5gee5bIuK4dY6O59p1eNehg6i6j+uDSvTmHugAy0DL6Kuncb8 m7A942y0dax2JKxRmOb/5DCjVw== X-Google-Smtp-Source: ABhQp+SzAExCmGNrOVi0jsAoXJwSj9Y/nsSHbxdkgtu+6smvapambTFHjfW4+qppL2j7iTaEfzLSSA== X-Received: by 10.107.131.147 with SMTP id n19mr12575057ioi.87.1508170475580; Mon, 16 Oct 2017 09:14:35 -0700 (PDT) Received: from klimt.1015granger.net (c-68-46-169-226.hsd1.mi.comcast.net. [68.46.169.226]) by smtp.gmail.com with ESMTPSA id o204sm3687403ioe.63.2017.10.16.09.14.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 16 Oct 2017 09:14:34 -0700 (PDT) Subject: [PATCH] svcrdma: Preserve CB send buffer across retransmits From: Chuck Lever To: bfields@fieldses.org Cc: linux-nfs@vger.kernel.org Date: Mon, 16 Oct 2017 12:14:33 -0400 Message-ID: <20171016161133.29152.48259.stgit@klimt.1015granger.net> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP During each NFSv4 callback Call, an RDMA Send completion frees the page that contains the RPC Call message. If the upper layer determines that a retransmit is necessary, this is too soon. One possible symptom: after a GARBAGE_ARGS response an NFSv4.1 callback request, the following BUG fires on the NFS server: kernel: BUG: Bad page state in process kworker/0:2H pfn:7d3ce2 kernel: page:ffffea001f4f3880 count:-2 mapcount:0 mapping: (null) index:0x0 kernel: flags: 0x2fffff80000000() kernel: raw: 002fffff80000000 0000000000000000 0000000000000000 fffffffeffffffff kernel: raw: dead000000000100 dead000000000200 0000000000000000 0000000000000000 kernel: page dumped because: nonzero _refcount kernel: Modules linked in: cts rpcsec_gss_krb5 ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue rpcrdm a ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pc lmul crc32_pclmul ghash_clmulni_intel pcbc iTCO_wdt iTCO_vendor_support aesni_intel crypto_simd glue_helper cryptd pcspkr lpc_ich i2c_i801 mei_me mf d_core mei raid0 sg wmi ioatdma ipmi_si ipmi_devintf ipmi_msghandler shpchp acpi_power_meter acpi_pad nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c mlx4_en mlx4_ib mlx5_ib ib_core sd_mod sr_mod cdrom ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci crc32c_intel libahci drm mlx5_core igb libata mlx4_core dca i2c_algo_bit i2c_core nvme kernel: ptp nvme_core pps_core dm_mirror dm_region_hash dm_log dm_mod dax kernel: CPU: 0 PID: 11495 Comm: kworker/0:2H Not tainted 4.14.0-rc3-00001-g577ce48 #811 kernel: Hardware name: Supermicro Super Server/X10SRL-F, BIOS 1.0c 09/09/2015 kernel: Workqueue: ib-comp-wq ib_cq_poll_work [ib_core] kernel: Call Trace: kernel: dump_stack+0x62/0x80 kernel: bad_page+0xfe/0x11a kernel: free_pages_check_bad+0x76/0x78 kernel: free_pcppages_bulk+0x364/0x441 kernel: ? ttwu_do_activate.isra.61+0x71/0x78 kernel: free_hot_cold_page+0x1c5/0x202 kernel: __put_page+0x2c/0x36 kernel: svc_rdma_put_context+0xd9/0xe4 [rpcrdma] kernel: svc_rdma_wc_send+0x50/0x98 [rpcrdma] This issue exists all the way back to v4.5, but refactoring and code re-organization prevents this simple patch from applying to kernels older than v4.12. The fix is the same, however, if someone needs to backport it. Reported-by: Ben Coddington BugLink: https://bugzilla.linux-nfs.org/show_bug.cgi?id=314 Fixes: 5d252f90a800 ('svcrdma: Add class for RDMA backwards ... ') Cc: stable@vger.kernel.org # v4.12 Signed-off-by: Chuck Lever Reviewed-by: Jeff Layton --- net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) Hey Bruce- It would be great if this could go into v4.14-rc, but I can live with v4.15. Jeff's review suggested adding a comment documenting the requirement that rqst->rq_buffer is backed by a single page. However, all of the server-side transport mechanics are page-based, so I'm not sure this wouldn't be a redundant comment. Suggestions welcome. -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c index ec37ad8..1854db2 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_backchannel.c +++ b/net/sunrpc/xprtrdma/svc_rdma_backchannel.c @@ -132,6 +132,10 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma, if (ret) goto out_err; + /* Bump page refcnt so Send completion doesn't release + * the rq_buffer before all retransmits are complete. + */ + get_page(virt_to_page(rqst->rq_buffer)); ret = svc_rdma_post_send_wr(rdma, ctxt, 1, 0); if (ret) goto out_unmap; @@ -164,7 +168,6 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma, return -EINVAL; } - /* svc_rdma_sendto releases this page */ page = alloc_page(RPCRDMA_DEF_GFP); if (!page) return -ENOMEM; @@ -183,6 +186,7 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma, { struct rpc_rqst *rqst = task->tk_rqstp; + put_page(virt_to_page(rqst->rq_buffer)); kfree(rqst->rq_rbuffer); }