From patchwork Thu Apr 5 06:59:29 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kalderon, Michal" X-Patchwork-Id: 10323807 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 926196032A for ; Thu, 5 Apr 2018 06:59:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7202028DF6 for ; Thu, 5 Apr 2018 06:59:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6393F290FA; Thu, 5 Apr 2018 06:59:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 71B5528DF6 for ; Thu, 5 Apr 2018 06:59:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1750846AbeDEG7w (ORCPT ); Thu, 5 Apr 2018 02:59:52 -0400 Received: from mail-sn1nam02on0055.outbound.protection.outlook.com ([104.47.36.55]:60240 "EHLO NAM02-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750835AbeDEG7v (ORCPT ); Thu, 5 Apr 2018 02:59:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=PIybXxqp1W/22Swf7gTqCEfEP5fBRnTQo7GJGcbWFUI=; b=MQvKJjI55qsz9Wdpq0t18sDsqfr/h1ZGvxbPTNAa/4Wm/E6ks/HC80YkAwhnF2hH6scs/G68DVKlF7t/uyIQvXU2MgMfZ/d9qSQ3YGHENlIynNoGvB4P5BMBsAAT3cXxWQmbk+9jkV8UxGtk5GdXDBCKT9d7qayyLN/vn+TETEI= Received: from SN4PR0701CA0002.namprd07.prod.outlook.com (2603:10b6:803:28::12) by CY1PR07MB1512.namprd07.prod.outlook.com (2a01:111:e400:50ed::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.631.10; Thu, 5 Apr 2018 06:59:49 +0000 Received: from BY2FFO11FD016.protection.gbl (2a01:111:f400:7c0c::187) by SN4PR0701CA0002.outlook.office365.com (2603:10b6:803:28::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.653.12 via Frontend Transport; Thu, 5 Apr 2018 06:59:49 +0000 Authentication-Results: spf=fail (sender IP is 50.232.66.26) smtp.mailfrom=cavium.com; vger.kernel.org; dkim=none (message not signed) header.d=none; vger.kernel.org; dmarc=none action=none header.from=cavium.com; Received-SPF: Fail (protection.outlook.com: domain of cavium.com does not designate 50.232.66.26 as permitted sender) receiver=protection.outlook.com; client-ip=50.232.66.26; helo=CAEXCH02.caveonetworks.com; Received: from CAEXCH02.caveonetworks.com (50.232.66.26) by BY2FFO11FD016.mail.protection.outlook.com (10.1.14.148) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.20.631.7 via Frontend Transport; Thu, 5 Apr 2018 06:59:48 +0000 Received: from lb-tlvb-michal.il.qlogic.com.com (10.185.6.89) by CAEXCH02.caveonetworks.com (10.17.4.29) with Microsoft SMTP Server id 14.2.347.0; Wed, 4 Apr 2018 23:59:33 -0700 From: Michal Kalderon To: , , , CC: , Michal Kalderon , Ariel Elior Subject: [PATCH v2] RDMA/qedr: Fix wmb usage in qedr Date: Thu, 5 Apr 2018 09:59:29 +0300 Message-ID: <1522911569-13978-1-git-send-email-Michal.Kalderon@cavium.com> X-Mailer: git-send-email 1.9.3 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:50.232.66.26; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(39380400002)(346002)(396003)(39860400002)(376002)(2980300002)(1110001)(1109001)(339900001)(189003)(199004)(478600001)(69596002)(126002)(476003)(50226002)(26005)(2616005)(107886003)(186003)(86362001)(8936002)(2201001)(105606002)(4326008)(16586007)(486006)(2906002)(110136005)(316002)(6306002)(50466002)(85426001)(51416003)(6666003)(59450400001)(48376002)(54906003)(336012)(305945005)(36756003)(72206003)(356003)(8676002)(81166006)(77096007)(106466001)(5660300001)(81156014)(47776003)(104016004)(966005); DIR:OUT; SFP:1101; SCL:1; SRVR:CY1PR07MB1512; H:CAEXCH02.caveonetworks.com; FPR:; SPF:Fail; LANG:en; PTR:50-232-66-26-static.hfc.comcastbusiness.net; MX:1; A:1; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11FD016; 1:+Y1WaMiuWBG87a2Lc3n/3gdegAEO2XAMAhmzADNnFxDK8ptrLbakp1Bbrm3+gDUKItJk7gsAGF5KuTztGQtksTJQy7FZn5nloJmpzIewsbxwC4BGeChIDK7hWxpjt5Pg X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8d07ad25-9091-44d4-3d97-08d59ac2d285 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060); SRVR:CY1PR07MB1512; X-Microsoft-Exchange-Diagnostics: 1; CY1PR07MB1512; 3:r4NRPu5XkuboP+M2HFxkRXkx5Pmg53jZCexnOQTKbXSjyKxFhK60arQON1paHB+eS5RPQ3khlWe+9DRzF7Er1snwj/F1MW/6jdq3pTabt42jjlCfbLzFmsGw5r6wm1wC3W7y96snFPHth9dEOTi/P6susxQuNcVx9GmhJ9NZvxWU4aDVCx4B5reSl8iuoz6XLgVdxul7DMJcgcxIHCkdfakwVHzgTxeNCvRQLJIxqwZ54wX4zSHmJiUwnQKwxZ9E58VTXx2jxh4DdxokvN8Y7uHJq2JCRX1uud88SeUEqgf4lWbHEH+3M0wBqgW9TUQ6PuyFM9ckvFaMhDcAp8xjHgyChptENdbVJJ9rfa5VTyE=; 25:07f17m4+Ey89UUVhvOSS+29AHHuT0PWy8U7mJa+w1ry9ZkZc0lXg0X8Xs47x7ei3EZ8HKodG0IWbS2Avw3bYtq95F02NPwVuUJQrtAQh7eFlTgqbxwoMKUDqsR5T6cdjirwDFz/sehpj2aDRJu3o8P3CsY7Puvie1FUOxY+1FZS9kZmwhyCbQZ3HQHrwKVePjS/9B3vV94eHAerht4ONN2999r160buM4CTht4MpOJT+ejEZ7IMDxdFG0UAxvbG6xof5x6YA6zpru9n87iAvDiFjOlorh2+V0DWtWuYD/itWxW7HSI6vV+zDhesfXiMYWNNXakiyC+kkMfx5wCGxmw== X-MS-TrafficTypeDiagnostic: CY1PR07MB1512: X-Microsoft-Exchange-Diagnostics: 1; CY1PR07MB1512; 31:DnmuypvVRXlh98kLjwdinP+ANvcqdrV7zD69WX8n7sK2yQHONoRjN2UeREPmWvdReWHuv2fEO+mIF6fZVd6EYDZbuCf48GBkVtzIXZG5U0phSXLXWJX7xCErNDYkFUDwqulFzzXG8Tt5WKlkdX0Uh8lofzpeuFa0bpyGbrm6cWZ96syHpAyAXGJn0FtgY7YaEKe+wo5qajSb9my0wrvoGWP/EnrbNFIEKJFBHfj79wM=; 20:OCle+J4EgDgSaPfa4XKPUYaZ1TadTBhKlRfjN6MXewmJAglOCYE8x/PHLDW8JnPe4fJ1aK6R5IZT12e1tEkfIfJzZjHRui6SF+qWzBAPyBb0Z2HPvGUHcWW5j1enD8Ns/DMmIEucuR4LLSoPPWwqPtunEgcRk8dnnOA8pGxdlX0A9vMgJSAeHriuZ3kvDPzx4pce/mjwX7D9GNxUilaTvZxEE3PCX+YWrYXRXuPgtamn30gG6TzxZGxdtoT1tNXPKQ4mJm2p2efgvtVMPgQ8NJih1MdvdJyxHhB5i6Mwobeq297lvzcTf2uRha84ESQwst/ayif9M83vFAx+yYNgYkG7GNNjR/xT/eGWzGlaSGJKhntZcj7NhC4S8HZMMfzbNY7blh88loSR0nKvbFLQToqnw8bCWKvC6xKtXV7aZElT2HlWlOaC1iUA2IuGen3EnHD/6rFygMAsBH2Ac0oXyPwQc8pKhVnsbPIR+7x3B9cRuMRBk9lHOZ6MXz8/plsO X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3002001)(10201501046)(93006095)(93001095)(3231221)(944501327)(52105095)(6041310)(20161123562045)(20161123564045)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(6072148)(201708071742011); SRVR:CY1PR07MB1512; BCL:0; PCL:0; RULEID:; SRVR:CY1PR07MB1512; X-Microsoft-Exchange-Diagnostics: 1; CY1PR07MB1512; 4:ZtpyOPfzR+EEMiiR2BwnTvcmOrR9diqgmbFEx4yQE6tenOgeESZfSSLHPwvlVCZzVIj6DKX9n4IogmIjWyTl3paddlfdH0i1RwNec4FGvBp0axLggYssUEfatGrQsidBF9Zfgs7AqcubqIZ4MFnHZFXcgdtXz2s78Mw7TvoW/I1bkJMkXbtoM/jOA+WaWhOH/m4X1DEqfY0T6OtPOh1ORK0xW0jgkXOGDhk0quvXMnlt4GqhfyucmSz676BaRfcfjb9oW0XQ3WCAmp5UwC4kKg== X-Forefront-PRVS: 06339BAE63 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY1PR07MB1512; 23:LJNDwj/3YdLZLBzr9vWAbblkrq5Zl13zAK0khIgxH?= =?us-ascii?Q?MEzsNhMRDDwD8YXvBUQNvU9q1wDWqcGLOXeAr/GNG5UzXhj3RWrQR82mAa2a?= =?us-ascii?Q?qGb6RLO3wsssDvEBFb9+4VomAXZKvb84aM2fcTSWUGOQK6DbQ+oU9LtXdsTY?= =?us-ascii?Q?OhTE0eVm0HJq0duYClJtGHBZD4gZ4IlB15U4KRrhWib6YjuLWvCgOsPURzWm?= =?us-ascii?Q?9HwszBIKGzedcRJvq1dwUPRPkOgFDboC4o9Udhm27fCBCF08Gx+sxd9vk1Rr?= =?us-ascii?Q?5XGi8eYUZX3RS+On3lzfd7004pMhNL6gmjMMHoWjJkb78HXhefZJov951sHT?= =?us-ascii?Q?UMMiLoi7xZnnJbtSOYiksGKPHroDCPmVVNHpd/pyBSRM7EgV+aZbOXF06dbQ?= =?us-ascii?Q?kRINXQ/5DzdOpzlWrqVsJ+6Aq3J1U4GxyiNick+yMG69XTPwFj0cHy+0Z+wW?= =?us-ascii?Q?+L87wjw0DNqPc1Zft44z2fNbb4H/4UKoLiyB2nX3B/9j3nouBbWWk7i/h4Ox?= =?us-ascii?Q?z0xn5ATh6A9kCZ7YRqdIUu5D65+cOzkWVArXkNXxo99qQ5RJlNGguHwZWS9P?= =?us-ascii?Q?6BV8FGfQjc/Pc6CHtU6BXx5CsZjfA1JStEcL0IeflVrXjzMbt7xLcUPgSWCS?= =?us-ascii?Q?hlyi9EGb6bmt6XMRjqYFkZQoeobSLWc9Z1eW5OzdaRBSHBQs7l4KZr7o1U6G?= =?us-ascii?Q?agocwaFEZFqWUoOfPlG3TOY8lDSIXbw66WfUcS1KshHKWI8LXynbmWp5efOU?= =?us-ascii?Q?+6ESvwbZljDlCEzVEPMdta8A2wkwyWzsKvvjtVaLgPVE8hvIRaS3Npy9fm9v?= =?us-ascii?Q?6usHu4SYQFq5lTjZ1zpDoyWfP8doUIuORerm8r8qBBMJmB4GkHzAPcGI25eO?= =?us-ascii?Q?m361Jv2dT0w4juGqKnpScFvMOObeX9dWrh/uSMwB/Ezfe1noJU3VGsQ9mLSU?= =?us-ascii?Q?pggS3I2sygvOEOjBQXUmzAku30EUATm6ubH3RRI8ITsxQrl/w2izmEcF7BNc?= =?us-ascii?Q?BnvPHneHEy3UVfQ8Jfspl8CVI1++Y7wmIWxLuJ4gyglxiQkP/d6IdI4D3tSQ?= =?us-ascii?Q?rUSP+1LMe0GPObJ5M1hER61aNPDkfQMiNHQqX/Znn5wt3KGT598kqsAmqVPB?= =?us-ascii?Q?6v8bSyGXCjs+3nsJTX9A9JKKdvwluNbUoWMqaoVhSeRQqb03xP1/7O1EAqZo?= =?us-ascii?Q?T8beNhIurAnvLo=3D?= X-Microsoft-Antispam-Message-Info: UFSrp/YDMISmoSf7dsPqadKyfYE4/cG0vZvDAabAcZY9Lc/WNAoCwtJpiJoj0eKarYX1C4i7VjLsB2aNSS+VaqtcIZTfAItQ0usv0KjqBfn6owDbDY+FhJUFxFTKH+6Ftzd+7yggbjFnsDMcqGKH+Qcv61no7mktDVg5EITyxrELMTZOUWELc22Nzh4HE2Oz X-Microsoft-Exchange-Diagnostics: 1; CY1PR07MB1512; 6:NdDpr5xNLZReQv6PN2Ta9fzulbtt/RJaMsbM8tOhikw4IFRnYnXEasvziigSs8Lg0yHAvZHLUPLiNyCvJvqWGWmuQGoeTulHeRjodaISLdKYZiwEDZ/fjH2Z43wJhLt84DhM4LCTyphSkXaCvi5CzSyyF6V5WPI+JPup35Vq6pO0dXsByx4IA7bLwQup+4nSlKVHcDpVYO3o0WtmLmxll0E/+haKmIYFBHEW18Gu6z79pH5Jp0iWZ/KRkdVpAA4+R4aXIFTRh/LBV63utgXxBX5VnF+zDvWMmIGumv2HotOGlAu0ZDAEViQM4FTFgzbflxvVpAsuSxuw0HFN2NKOy40O05zZ1x3WZhZ4kULshRtXrIzm3e7iqJx0G9UJ8dRORoGQQ3dJPXFRYPItIJpp8zxMEoLfSFeHNFnibzeBSz062mbqkVSVRjNJumX7iL3qCKSz8SxrPCLeI6Gq484O9g==; 5:tXMXkUjox52bCnMFh7XwoGa9Urz4UvL+n34ZZJcdiOzdXbus7cS/DGbznfHGDhPQMQhDWDdYs8xYe3nvwEk9wq3MJx+ZaHxyyaU6DHaPnZg1KiuUm5RvWjYmfE/nJ1XiIO/6ZZBQhVrQRW/c9Xto6b79VSOS2xsM0gQ4+iaI0ko=; 24:FZHVp3pOwBxw5e8e9V+ngOtWro1rHPt1Vpozu72QASr9JiZX39j5xDWqfBKx00M0+uiGTYZ1zmKi4joxsgZpvqgebEmZ5uhmS/9204TqbQ0= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CY1PR07MB1512; 7:5MrZ7UTN5d5A7bwM/v7fMNGwZWrXRnfyEIpqDt8c9Yq1UyGg/NlAtzIUQ/FlcwZSXd5H4nMolE4iyOTMN05y20OAk4T5vP7l6eclrHUU1TtbLG3A2DSCHIijs8TrRIIFyhjHrjSUkqLceeoMd5oyhN55VdtecZ7nX1YD+2u4lmdTHzIivn5rSHBTOdJvUFMKnIgqLjrkikkce8iUKs5bGuPh6s5WdYUaJysd0BytwJ23XBKqcNqItjJIaUKlSW+V X-OriginatorOrg: cavium.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Apr 2018 06:59:48.6889 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8d07ad25-9091-44d4-3d97-08d59ac2d285 X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=711e4ccf-2e9b-4bcf-a551-4094005b6194; Ip=[50.232.66.26]; Helo=[CAEXCH02.caveonetworks.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR07MB1512 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch comes as a result of Sinan Kaya's patch: https://patchwork.kernel.org/patch/10301863/ wmb usages in qedr driver have either been removed where they were there only to order DMA accesses, and replaced with smp_wmb and comments for the places that the barrier was there for SMP reasons. Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior --- Changes from v1: - Modified the smp_wmb usage comment to be clearer --- drivers/infiniband/hw/qedr/verbs.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/hw/qedr/verbs.c b/drivers/infiniband/hw/qedr/verbs.c index 875b172..450344f 100644 --- a/drivers/infiniband/hw/qedr/verbs.c +++ b/drivers/infiniband/hw/qedr/verbs.c @@ -856,8 +856,6 @@ static inline void qedr_init_cq_params(struct qedr_cq *cq, static void doorbell_cq(struct qedr_cq *cq, u32 cons, u8 flags) { - /* Flush data before signalling doorbell */ - wmb(); cq->db.data.agg_flags = flags; cq->db.data.value = cpu_to_le32(cons); writeq(cq->db.raw, cq->db_addr); @@ -1869,7 +1867,6 @@ static int qedr_update_qp_state(struct qedr_dev *dev, */ if (rdma_protocol_roce(&dev->ibdev, 1)) { - wmb(); writel(qp->rq.db_data.raw, qp->rq.db); /* Make sure write takes effect */ mmiowb(); @@ -3256,7 +3253,14 @@ int qedr_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, * unchanged). For performance reasons we avoid checking for this * redundant doorbell. */ - wmb(); + /* qp->wqe_wr_id is accessed during qedr_poll_cq, as + * soon as we give the doorbell, we could get a completion + * for this wr, therefore we need to make sure that the + * memory is updated before giving the doorbell. + * During qedr_poll_cq, rmb is called before accessing the + * cqe. This covers for the smp_rmb as well. + */ + smp_wmb(); writel(qp->sq.db_data.raw, qp->sq.db); /* Make sure write sticks */ @@ -3343,8 +3347,14 @@ int qedr_post_recv(struct ib_qp *ibqp, struct ib_recv_wr *wr, qedr_inc_sw_prod(&qp->rq); - /* Flush all the writes before signalling doorbell */ - wmb(); + /* qp->rqe_wr_id is accessed during qedr_poll_cq, as + * soon as we give the doorbell, we could get a completion + * for this wr, therefore we need to make sure that the + * memory is update before giving the doorbell. + * During qedr_poll_cq, rmb is called before accessing the + * cqe. This covers for the smp_rmb as well. + */ + smp_wmb(); qp->rq.db_data.data.value++;