From patchwork Wed Apr 4 11:54:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kalderon, Michal" X-Patchwork-Id: 10322479 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2AD8A60318 for ; Wed, 4 Apr 2018 11:55:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E50128E40 for ; Wed, 4 Apr 2018 11:55:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1243D28E4E; Wed, 4 Apr 2018 11:55:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E7E528E40 for ; Wed, 4 Apr 2018 11:55:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751849AbeDDLy7 (ORCPT ); Wed, 4 Apr 2018 07:54:59 -0400 Received: from mail-by2nam03on0063.outbound.protection.outlook.com ([104.47.42.63]:64011 "EHLO NAM03-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751852AbeDDLy5 (ORCPT ); Wed, 4 Apr 2018 07:54:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=+fMSzvXpEh6IhZyVnJLarawSbHcBXxS+otILhh/wSc8=; b=DjlCAF8P7enxTsR8hdL4lzqSsfyr+WKe9XQWYoizGdGSZwP59JO0cgKscfmv3s4Xhit9TldFbW0JkafvUmsUVmDXtnd/GcR2KN5Lh4JY3E5t7Umms99Y3XeyhaMur7dCXJ3QqjfRSPwUPD1tsGvFcdnEmB1IKlbDldbWmYMQSkk= Received: from BYAPR07CA0001.namprd07.prod.outlook.com (2603:10b6:a02:bc::14) by CY4PR07MB3175.namprd07.prod.outlook.com (2603:10b6:903:cf::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.631.10; Wed, 4 Apr 2018 11:54:55 +0000 Received: from BY2FFO11OLC007.protection.gbl (2a01:111:f400:7c0c::164) by BYAPR07CA0001.outlook.office365.com (2603:10b6:a02:bc::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.653.12 via Frontend Transport; Wed, 4 Apr 2018 11:54:55 +0000 Authentication-Results: spf=fail (sender IP is 50.232.66.26) smtp.mailfrom=cavium.com; vger.kernel.org; dkim=none (message not signed) header.d=none; vger.kernel.org; dmarc=none action=none header.from=cavium.com; Received-SPF: Fail (protection.outlook.com: domain of cavium.com does not designate 50.232.66.26 as permitted sender) receiver=protection.outlook.com; client-ip=50.232.66.26; helo=CAEXCH02.caveonetworks.com; Received: from CAEXCH02.caveonetworks.com (50.232.66.26) by BY2FFO11OLC007.mail.protection.outlook.com (10.1.14.254) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.20.631.7 via Frontend Transport; Wed, 4 Apr 2018 11:54:55 +0000 Received: from lb-tlvb-michal.il.qlogic.com.com (10.185.6.89) by CAEXCH02.caveonetworks.com (10.17.4.29) with Microsoft SMTP Server id 14.2.347.0; Wed, 4 Apr 2018 04:54:41 -0700 From: Michal Kalderon To: , , , CC: , Michal Kalderon , Ariel Elior Subject: [PATCH] RDMA/qedr: Fix wmb usage in qedr Date: Wed, 4 Apr 2018 14:54:26 +0300 Message-ID: <1522842866-27176-1-git-send-email-Michal.Kalderon@cavium.com> X-Mailer: git-send-email 1.9.3 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:50.232.66.26; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(39380400002)(346002)(39860400002)(376002)(396003)(2980300002)(1110001)(1109001)(339900001)(189003)(199004)(316002)(107886003)(47776003)(2616005)(478600001)(104016004)(50466002)(54906003)(186003)(26005)(36756003)(48376002)(5660300001)(16586007)(356003)(1857600001)(86362001)(476003)(966005)(336012)(6306002)(85426001)(50226002)(77096007)(110136005)(126002)(8676002)(81166006)(8936002)(106466001)(4326008)(81156014)(305945005)(2201001)(72206003)(69596002)(105606002)(6666003)(51416003)(2906002)(486006)(59450400001); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR07MB3175; H:CAEXCH02.caveonetworks.com; FPR:; SPF:Fail; LANG:en; PTR:50-232-66-26-static.hfc.comcastbusiness.net; MX:1; A:1; X-Microsoft-Exchange-Diagnostics: 1; BY2FFO11OLC007; 1:7TnuWSYYXmcrxJCeqFcVQyWCUs4xBNq1Wli45/4f/LSMgy6G/ZXivMX4Nhtlu7bTzJiJzJamRXpLkSq7K0j3/JDBOCSdVey5EE9MOsX+10EvDDpfckFoHkYXBrwe33Ym X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ae3a61aa-58d8-424c-6b1e-08d59a22e230 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060); SRVR:CY4PR07MB3175; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3175; 3:bh11kAu02KwIVgQLjrBA/ynolHZYoo+jzOe+Q33/rd4rOcp82K2M/1T7iXxHo56CxDmSxxrlx0SaZ55tBPr0TRQ7ctvn8nJ9Z69B9NIxkv8B0e5myJo04AnC9FLtfJQViURzl72uJfQKKUbfg7rEfRKFjiDgynQCs3Z8qjVs2Ycn/aN5+WaaS2TFPu2LSVKkttZV65ub/8cq96ZrvqSnfsSSnLcCnRxD6vwLVNEF5KHggrWplcyVjimeRNUQg/whaC5kpf5IZGu0MoRZhqmfZYOvrOBJrWaSIOHa/whuJaDTon3xgvbIsUxV+NbwY3kyEXd0MlWI1Qj30Dy7Ytx0xyBoSlknTKG5D5DutdYVmLQ=; 25:i8+7/UK95qZeS9SskcUU5wRuzIXAuhvuGv34P2trjSqKm035bH/DBkAE5gm+uR57N242p9rnR1QcB2R0OuKcCIiY5H/Wi0tIuDPBX0VqVbcqLUUs3UxjzYO4y/71czlZWin9On7c1GG/IPlpYJrubOaM4xHbygOQlGbyiZYz8X58gU1yGWCm5juFdcVUD+lwhckNN2nHa3g9dln+phqiR8OXlCFmS5ao2em0cu44oJ2wG8JQkNgnknllt8QTxlve7qHNMMwKCvEZQMv9gV+lVW+geExipHfGRdApy5SPC7l2NMehquuun3EN8S9C7IGttz7I9PGKKcoonl12M4GWkA== X-MS-TrafficTypeDiagnostic: CY4PR07MB3175: X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3175; 31:xzRdMwRObYP/vwQbDjfUGDEkPuwLf/61o5Uu22Gf7K613EX2T9ArWihjF3ibzICtBh/DSFo/d/7RAgbB8upKGMVd5xwKowdyHloSaxh5yoGGMv7URuHrbQpU8aj/EI/jn+6qwuEI0ZAhCKKOiFSkSh4vPPtP/xs4/iWwC+4BwpE6Sn6Mujtp83BRM5Hh1IoFLM6javwlyH1Qp6392owOkeSbllXPYHQtm7TfHFIZbDU=; 20:YC9JnxItv02r0JLTTQm5tFW6/czxtcZOhN8+vv+650tOotcdxT8O0kTl+rNz9oKjen1hsJqNOY9plGhBREbpqH5WrpUp5ObAKdGIEFacIzSryAG6IxL3kfzxvkJHxIaIqTG+501hSmi4hcnacBe1RpbOb1T1o6zmVcubzie8ES1/I873X95y+z1o6L2m8BUbHBkkCfxE6IjHZ/CGz1K0MzW+BwTcs3gLf1PjumXKKk0SfKBCGvOorqK1LDotgYG0fNZwAnRao0xnuL2V20dBNAARsv3MGoBF45NZ5dZ2cB452My6oN9xotwDqTHZ7Gpb5omJVcMyrtNOarBg4PyN3g6INw0DfKbLp237Lwic1Wqsaa5zZwBUTsPuICTUQCC1L0enmN2RV9lMlO4ShI/5kf4Eb2AnEoYbuIs+Wpo6ijLpwyfaNlR2neFQQe6fMCzN8ypDHrqFpbyql+kQo/Frir0L40EwShF1NUAaiiMF47il2XQJMz1bdQMCxYDDRDDI X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3231221)(944501327)(52105095)(10201501046)(3002001)(93006095)(93001095)(6041310)(20161123562045)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123558120)(6072148)(201708071742011); SRVR:CY4PR07MB3175; BCL:0; PCL:0; RULEID:; SRVR:CY4PR07MB3175; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3175; 4:WxI1g9pXt3HUp3i/1wbvnEgudQ0Nwpr3Gme1Wm1hXyFvxFpaqlewfzVo/viauQmx+FWXrA6ftWcvx7//iMGezbihfj1BL5K7eIotePr/Wom1MDh/lcQCFBoEeqNewMAWVPX55OY7fZK7TRiZ4Q3V2Yupu77O+91i/mysqsjpumrJYKpb3LLZ0xAwLdSEuJYByGD7kryY5QI1lb3Nd0M8aJT3lbKmwSMpyYqzkEcv5AoiJyiPDEuoVb7fTQ045Cc3HzvIPDrmjJmhCm54fVTUeQ== X-Forefront-PRVS: 0632519F33 X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR07MB3175; 23:lokIeO1xYHKHTy1xfZ/JtLLfBmCgsWVNKum2VWqWi?= =?us-ascii?Q?WdJ3yvHey7Xvs3B4YjwIsfTZlRxEksUz19xHVWvD1vX13gjICWeykYRX3K6v?= =?us-ascii?Q?aZ95TmFPwpCfKb4AHtZF0uZs9eXD0OrSvAy6W/jWK/wLur7bmWVHULvtQUw0?= =?us-ascii?Q?1mF11gMXtpLGS1iOyvUq7lxo6F6/pAXa1fCsNDCUa7XUBSrg3EpRmsoVY7gP?= =?us-ascii?Q?prsBFmBWtKjANUElxn6dGqnXA0QBHILAb7PYzMcoyE5fXtVHbvcmgGHJOYbI?= =?us-ascii?Q?XOGN/olD27irMZIa9Ktm98CujMzo06+41YWiHlU1WCRWOyuY0xIJzeu8L0KX?= =?us-ascii?Q?HtDUFWbcoVhTmAekdfqndNelwhQL9YR+Tu6wlVhLXL5E0Xgi1zUtdSdY1laE?= =?us-ascii?Q?5Xehu9MYlmys8i2wcoPFp8dX3T1YXtCb2S3Ljdw2wVrzr56c7I6xvchy1uvz?= =?us-ascii?Q?nRZgSVgl+2TK/8IN0daGs/FpxHCNvAzNC+EoH5fJEU1X+ea66aFPISOlnkva?= =?us-ascii?Q?ZYvrXYH2RbntxpLoudIj4vYNwIR9l5cmJ9QHm99Y4dd2+n4xqbWIy3ET1SsK?= =?us-ascii?Q?TqdYAms3cVCU+FfG3L1XdZNlG9LlWk0Uu0vIzsVztMr+BbcQCqdXZONVSF6q?= =?us-ascii?Q?jFNqohJpjXDscGNpicIOIe4MOVJnLeS6181LG/9nIskEvHgnIW/e3jaXdeyO?= =?us-ascii?Q?MAYbd7Sb1ztXgqunLfPfWpDsbiXmY8UC0QZfF21mY/4HGg2NkaDFAWrcAMIM?= =?us-ascii?Q?pRf1Bcmp0RRiBfLRxc3N8mqQfpFZYywyw+lSAjMwQHvY/rR5MiLDuTiBzn56?= =?us-ascii?Q?V+RsgS2PoPyuNvEDw50jc22Z5RhzqGw4IVW9NKcl5Cq4cmz7v3J1Lgb63GAB?= =?us-ascii?Q?BvmhUgCJG85esX7VqEviGZSkmgOpre1aAz3MyxBOLofhoBvbiwQHxFmoYNbs?= =?us-ascii?Q?P9cu37fmYFvUrrvtIEQciQnaSRurY8L3L+I67g41BUHiqDgZpDuOw6tNGFKO?= =?us-ascii?Q?vPxmOsnJO/TtKVU772kVgwvIp5ogT1+mXX+vCwjhSvVTKbXHY2rNGyf7vxxo?= =?us-ascii?Q?JZhQYcb50DeozL0b4ED25rDABQWbpUoVdrk24/xXPObZNkf/J4lKqQojIBYq?= =?us-ascii?Q?H+RCGoj+Fp5BVVsnE4z2qGGYFx++xA8bLi0ARdaoP1gY4zxMyobzFDRdZvIl?= =?us-ascii?Q?ZM/K+TONwBBtPjHD5TxK8xCEHvbWg6GUSZB?= X-Microsoft-Antispam-Message-Info: OKQvs9IDPyzUqsnPs35s9D8GKZ4hizL3rX1e714VTj300IaD42Gw5PMXZU03u/G3F5z0opo/iUvMTQQjxMSjERn+1zpML/SPREhSVFqs2gxMWWHyohgipkZe2UKxHirl4fplO9FK45SEkyM9IOEgUOuA/ho3vtwuaw1yf3Ik/qMXYOAyKgRV9iimJ3iK6r+y X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3175; 6:kK7p6gBTWNzeZTpQ+CbVIHEo3K3bteRbH/f3n8aBN4ZHL37rhMo33j4DGlw05X9wgwym3YlJ+dAy04bYE2iUewI+aOPVDPEmzuKXXbQEqoALmHZiTE2cDKhxInU7IF8KxQ+Io3Oi1YZmxHmfAEKpXblLtdR9fvY7FXn7kHq214XNQ9kF8WSv5YP24q2nDS9/JxNeF4qfzz2EXOREUuQwWii6vQNr8p7TpkI+W/uN6nNfr5Bto8iRCxxJeaUWR2Cm8WhT2Zf6MyRlNDDNPv0vBQ/Ysa9SzmgcpwU3hZAsChsQ+lFWdCc/H7rF76OCzdgH+araPueK5MvUuD1uK5vtCMoWEM4TMRnirVnePFEuh3FOdC2gCD216kqwdcav72msu3ku9fR4vrnepu/FlHjUh3W2bo52LrdXPXFcnaGOVtbX7ymm/426jdZHmD1fYnhBuD87GlKlTEDC6RbDkiIJ5A==; 5:1144AtvPZpJcNFgFL2GSerShAw3zt2po+Y2YmCcez9T3h/Zi8iBCcm0fEofyGeNOap/23vnPnyurIAIkmCXdLvPIHM+vSelkdcqBk+va+i4Sfc1i7svWqyfKlKh3LHaWCtBPXa1pvPIO3Nnrkk7fJZ0mUzjjXVz4qJ7yZd8p1mg=; 24:5xAq8aY68vpIYTGaHx2CaqyYhbW+UrSwmKHR4J5nzmD0GisvyOuwT+7jOPs9TpsjmZ8/o5ki6sY8X7BX9DNMhG3uxKv5+8GuRJ6ma2R0ISU= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3175; 7:b4CjZHAoSdPow5gtXEf7rp0tPRN2HtjcU7hqfLy24PeckZLRNRSTtBLBp462VbwFjTbioBPrubdNruRgbtf98FDRmsv35zcSXOIy5R2qHEoxalVAK4AHaH5IcGfVnzcVMhrbJ7q+0bylJDmPzUM3TtsigdNZ3W2ufMkWER3aFUi7ma7oQm3yYG0IXNt6MhVQ86kW7dTGDkvaGKyyz3IxS2SYv4EUhRiEJ9WXD+3u4xxV2TMlavxJHQU6ZVL0px4e X-OriginatorOrg: cavium.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Apr 2018 11:54:55.4998 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ae3a61aa-58d8-424c-6b1e-08d59a22e230 X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=711e4ccf-2e9b-4bcf-a551-4094005b6194; Ip=[50.232.66.26]; Helo=[CAEXCH02.caveonetworks.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR07MB3175 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch comes as a result of Sinan Kaya's patch: https://patchwork.kernel.org/patch/10301863/ wmb usages in qedr driver have either been removed where they were there only to order DMA accesses, and replaced with smp_wmb and comments for the places that the barrier was there for SMP reasons. Signed-off-by: Michal Kalderon Signed-off-by: Ariel Elior Reviewed-by: Sinan Kaya --- drivers/infiniband/hw/qedr/verbs.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/hw/qedr/verbs.c b/drivers/infiniband/hw/qedr/verbs.c index 875b172..3868bf0 100644 --- a/drivers/infiniband/hw/qedr/verbs.c +++ b/drivers/infiniband/hw/qedr/verbs.c @@ -856,8 +856,6 @@ static inline void qedr_init_cq_params(struct qedr_cq *cq, static void doorbell_cq(struct qedr_cq *cq, u32 cons, u8 flags) { - /* Flush data before signalling doorbell */ - wmb(); cq->db.data.agg_flags = flags; cq->db.data.value = cpu_to_le32(cons); writeq(cq->db.raw, cq->db_addr); @@ -1869,7 +1867,6 @@ static int qedr_update_qp_state(struct qedr_dev *dev, */ if (rdma_protocol_roce(&dev->ibdev, 1)) { - wmb(); writel(qp->rq.db_data.raw, qp->rq.db); /* Make sure write takes effect */ mmiowb(); @@ -3256,7 +3253,8 @@ int qedr_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, * unchanged). For performance reasons we avoid checking for this * redundant doorbell. */ - wmb(); + /* qp->wqe_wr_id must be updated before completion process */ + smp_wmb(); writel(qp->sq.db_data.raw, qp->sq.db); /* Make sure write sticks */ @@ -3343,8 +3341,8 @@ int qedr_post_recv(struct ib_qp *ibqp, struct ib_recv_wr *wr, qedr_inc_sw_prod(&qp->rq); - /* Flush all the writes before signalling doorbell */ - wmb(); + /* qp->rqe_wr_id must be updated before completion process */ + smp_wmb(); qp->rq.db_data.data.value++;