From patchwork Tue Sep 22 07:56:39 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 7236111 Return-Path: X-Original-To: patchwork-linux-nfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 83EAABEEC1 for ; Tue, 22 Sep 2015 07:55:46 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 7F84D2035B for ; Tue, 22 Sep 2015 07:55:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6A03020219 for ; Tue, 22 Sep 2015 07:55:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932139AbbIVHzj (ORCPT ); Tue, 22 Sep 2015 03:55:39 -0400 Received: from mail-am1on0076.outbound.protection.outlook.com ([157.56.112.76]:61024 "EHLO emea01-am1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1757478AbbIVHzh (ORCPT ); Tue, 22 Sep 2015 03:55:37 -0400 Received: from AMXPR05CA0032.eurprd05.prod.outlook.com (10.242.9.160) by DB3PR05MB361.eurprd05.prod.outlook.com (10.141.6.141) with Microsoft SMTP Server (TLS) id 15.1.274.16; Tue, 22 Sep 2015 07:55:32 +0000 Received: from DB3FFO11FD054.protection.gbl (2a01:111:f400:7e04::132) by AMXPR05CA0032.outlook.office365.com (2a01:111:e400:803c::32) with Microsoft SMTP Server (TLS) id 15.1.274.16 via Frontend Transport; Tue, 22 Sep 2015 07:55:32 +0000 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; linux-iscsi.org; dkim=none (message not signed) header.d=none;linux-iscsi.org; dmarc=pass action=none header.from=mellanox.com; Received-SPF: Pass (protection.outlook.com: domain of mellanox.com designates 193.47.165.134 as permitted sender) receiver=protection.outlook.com; client-ip=193.47.165.134; helo=mtlcas13.mtl.com; Received: from mtlcas13.mtl.com (193.47.165.134) by DB3FFO11FD054.mail.protection.outlook.com (10.47.217.126) with Microsoft SMTP Server (TLS) id 15.1.274.4 via Frontend Transport; Tue, 22 Sep 2015 07:55:30 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38; Tue, 22 Sep 2015 10:56:40 +0300 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38 via Frontend Transport; Tue, 22 Sep 2015 10:56:40 +0300 Received: from [10.223.0.123] (10.223.0.123) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.123.3; Tue, 22 Sep 2015 10:56:40 +0300 Subject: Re: [PATCH v1 00/24] New fast registration API To: "santosh.shilimkar@oracle.com" , "Sagi Grimberg" , References: <1442482947-27785-1-git-send-email-sagig@mellanox.com> <55FDEDD5.1090105@oracle.com> <55FE7E24.20300@dev.mellanox.co.il> <5600928A.7070202@oracle.com> <560100EA.8080302@mellanox.com> CC: , "Nicholas A. Bellinger" From: Sagi Grimberg Message-ID: <560109B7.8070005@mellanox.com> Date: Tue, 22 Sep 2015 10:56:39 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <560100EA.8080302@mellanox.com> X-Originating-IP: [10.223.0.123] X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; DB3FFO11FD054; 1:IiMKUe2e7+yKv/gNkp+oHVxDX/HYflMLP4bO96fkBqjgk3KeAE7pFxfKDgV9lD26owyYB9QLGVX59iWPudcuqzpiMdhzf0dZFwMilsUuzCsAZH5mAxichtM4QbucXMxUXzCVOFtLz7qgz2n19JqHh6dOns8FxecWXtGjFzvtMFrS4uWesLeGoCc1+WzfuX4YCjKOlUSB47QhFfiRfdpOOfcNPTuNdcXBEsj8+lm/mIraNqKNAulFmVqHIIaWgx3nGbrPtPlZ/jA1j+K5FSxlG8tDsSWuVIGBdMD+9x5nhjvWEL9a2PZdQVcHfBHSulM/pzeEDmHMWhTls2zNWJAw9H5UR1tisWQXC+Yk9Ty4f5jcO/X95zU0LvFv7x5jokDIIp17TXJ6zb8YcWva5hvtuA== X-Forefront-Antispam-Report: CIP:193.47.165.134; CTRY:IL; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(2980300002)(438002)(377454003)(51444003)(479174004)(199003)(129404003)(24454002)(189002)(65816999)(86362001)(5007970100001)(99136001)(6806004)(46102003)(106466001)(33656002)(19580395003)(92566002)(68736005)(93886004)(77096005)(11100500001)(50986999)(65806001)(97736004)(5001830100001)(5001770100001)(4001540100001)(36756003)(47776003)(87936001)(65956001)(50466002)(64706001)(2501003)(80316001)(4001350100001)(5001860100001)(77156002)(2950100001)(5001920100001)(64126003)(76176999)(87266999)(54356999)(83506001)(5004730100002)(189998001)(59896002)(62966003)(23746002)(3940600001); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR05MB361; H:mtlcas13.mtl.com; FPR:; SPF:Pass; PTR:ErrorRetry; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; DB3PR05MB361; 2:tNyuf1l3lWYJY0sCvlNaMGgWPB8U/hdbWJUehC9QyKDuLU3/wGSLh8ZacEfdpdkt15GGXEoHqK1smxpBPRECEmXuuSAiXXYQiO3Z8u9ERMmYuBF9BKi0oTApeJoJJXvvwCw6Qy6arQqKG/TGLZJa9KLq0CpSNNTesJkxthxvH5E=; 3:6n2uCI2S6FECumgg5YXKG5XQjpUABsrecwv7mEY2VyGUJxtZfYNxge2yXhLlhRg7wymgC3sYDFWt79+LZ5QVeOli0L81tArPzI3Xj/xeDUG61Ghi3d65cTe1SKZGdwWF9EQ9yCsck+1pDh9/kcx3Nio1cqPUCeQ+ndswurHUnyfc/fb0zYcLCmvkUjVtbXAiqdOSF7IK1UCPXegO9m4zOX/cgUVQe6Pei2JF4vywg+a1VoXT/os8SiOPESonf9UwqABDyRSbApBQ/h4XsrclFA==; 25:gqW8ydG1QJz4GZXhjwTBGsw/CghCH0FyE8p1lnkJ5/VjpKqybtgeQ4IGG5E5AkkZ8kY2qtSv6dGEel+N1+vPVSQnLT+XwoE8X+bpzpAG8BFStRxlMSnp/9U+8EgV3SN2i//1ppDhIE2p6gBwwJLjFKUhNSI8O2hmspgviCsDU2Z5VuCLZvDJZRDWQDx8s3k9f141Nny9f9EzOgdFPtzzUf4xvhiK08E9FJBWztLWSJWW7t1wS+PIeZetDy05u/ta5iS4DjbxQMyAAHQ5G3Etcg== X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501001); SRVR:DB3PR05MB361; X-Microsoft-Exchange-Diagnostics: 1; DB3PR05MB361; 20:Ii+7ADrBlvSNUDr+PSyrR4e5JAC1BNre3T+KSfHY/oUV9J7rvlOre/IHqnSE1eTs/qD6yRw6TPDlQReIkNBYUCY80tAGHKpg6dR8zeNr6pNiGKYL8HeRmG3wKp8PsOmpdQIp6lIzmGdX5lPGKSdPNSqqndHsK0q5YvDu/RARR2JDTVIkz3/evAKXOhlxnIAfV6z15w/YFYNQtxVFS6UljO52CADGzBqdLyiH/lZjB0JZBBvHxF34lACs14FTQUdQBiYPktGAqXY8tsdAXxLNyTQdF1djz6xrxQsmKoK0p6o0USID1ifpCIdxeMcDQOK8lj4vHzYBivlD5vyhxHoQCS8Az5V+aYMXaXt3Pfdckrs0iWfd5eIZVfWK+0Osg96/C9WfdLzPtlBUarsJXuNVsYJ1PZQPQZh1WMoCtP1olVO3HOVTgFh7BpuOh3LYAPKVe/nxeWjx3GUqhd6zg4lc7Z+PBFo5LNBJzpx5XhH1S77+HRxzn6ccBRgkmA2fUAu8; 4:ih6YvHoE4m1cCHzGQMKZj9iwC04Mjzv82W8Z+cmLzVrzZZZS3sl6GIdo7ihvQkKzl9kgsVQHLhapC+HjFwi0/3QslcOzBhDHyj0Nmm1brs92Gp/pKQcwOQvzUXm+zGzXGjO15mCiBvQCVdTDmCi6F98zccu+SFulD/pjwuPMUDP58Ykcm1ggR35SWfP+Yl5dXyL5R61Ynl5jqpPfVKJ1Fo81SdVBTSPy7bfBIiIF2XI1f7EJU3Ye98I5YP8a2GYCw9Xjq63MP2Hv3QSomLYUCIWbvPlWCGZMqcKVQe42/CKbQMR424xlcKfUEUi+Ii1vPYdRe9Frr9UejE3tH6GtBkFcuv0AjAm5SXS/X2rpFIQ= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401046)(520078)(8121501046)(5005006)(3002001); SRVR:DB3PR05MB361; BCL:0; PCL:0; RULEID:; SRVR:DB3PR05MB361; X-Forefront-PRVS: 0707248B64 X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1; DB3PR05MB361; 23:xnryPfsMPTJ7xNwmxqW+WyhfcJQLi0JsjVDq0j?= =?Windows-1252?Q?WigHl6m83gQbtuPKgw/AEG+QE0qonFCIzyK8axPTVfg8x2FG91mjGuqT?= =?Windows-1252?Q?CudKoCpiMaN1hI/lkbrYvP53jZzI+HFbPlvojrNhwfY2Nq5FMMghpN3A?= =?Windows-1252?Q?Ia2MNPPeGuWyDZsO5X4nKyUMhrido2rMwyRJ8P0wH3TT87dmBA5VQnLr?= =?Windows-1252?Q?jWjn7NKDnqcIPwqZWkUHSsS/pYSk+5yaoGHQD0UpiUjBzkIF66J2yPBM?= =?Windows-1252?Q?H2otZQqmtIOnaLe38Xb106B9MvLl0UqZAEFwC16PidqiBfJFQRI5U4J0?= =?Windows-1252?Q?59XegOeTzmgAFMtlf4cwsLJxa5aygHyELu/nzY47YNf6OAUEKHVNyv7v?= =?Windows-1252?Q?ARebaoxsOWbjAYflg0iI1NIvWf2TabFr1h+Z50HLZVP+MnLrCGO9MTUu?= =?Windows-1252?Q?qxTI+3EAO8kBcPa03C/j0si9hyobMPzpPOMzR9QN4UX0pLj7B8HS00qC?= =?Windows-1252?Q?RTu6fvkfpZ9RYkfP6rJkEC5iAFmdxNxuyag7oIPe++K0VFewGXNK6mLH?= =?Windows-1252?Q?C5CDyejWv1h8YEF/bE8cDKyuVAt/9Z5g2fFBXQlYV2jZdOVqp1c8Sk1T?= =?Windows-1252?Q?NLDIKCDIy0CIIn8sj2ke73P+613BZ826KJ3EKAnD9ppyT4iSPw9NFJt1?= =?Windows-1252?Q?LV56NOh5ADyroqWYy66sNdES06+FRi/Wu/nx+vzZZsxQAif8edrdlMr6?= =?Windows-1252?Q?vImzNmMWj0SzZrtXdl0s5wyri/eR/Lwdvuh2R5WiZcSWDCaqKnW9VpFb?= =?Windows-1252?Q?LrhtXXNOlkuHLNeD7v6CctByckWMSSPyxTxSf1l3l5k82820YSh8bIY4?= =?Windows-1252?Q?FM8t0ouUOfLfiUiOYhoX9ZY1tyUQM4jflPZJfBLlHYcl8NRoMpHTdLe4?= =?Windows-1252?Q?JIiMxgrxkLs01eJX8Sii8855jb4heypeqAOZ2EHDAqPmKQvvkZDZXOeZ?= =?Windows-1252?Q?suojInzuqSjIc8gp/B/iEJ5quRHuYGfVkf4tsQrbB0B4Qxxr2BfwS0IH?= =?Windows-1252?Q?MKMNbIKMifNOZeyd4f7lqBIkq4gl2c6DEXsoPJx5SCsSaetdlvYqd7Xj?= =?Windows-1252?Q?yuzSWbFHEQP9CDTnJSlM0zvrIqXmCjtmkff4YfkaYTD3TLdBkE3r56QV?= =?Windows-1252?Q?YLQ8hKAy5nzJ1kEi8iNHhYyANrzbFacCDbGlzRzxasdyK3BlDmDLyv6Z?= =?Windows-1252?Q?oUGD3CzdZrY+VlgZMDKRWe0TIJ2ZMoLxPr4KYE7+UxIqiZCFtDm52hRH?= =?Windows-1252?Q?Xn2yL4OOrdBYKp78uFgNQ4cYTAzgaHipD4+bgNCxnHVLX1qzK8E3Ie15?= =?Windows-1252?Q?tEadq7oEgrVoFM0R1HqS4+1/5zGB4HtVIGykzS7sie52QyYG1Ime16CY?= =?Windows-1252?Q?wFZaGbMpeCVtxQNGYZ/n7SczSmy9eWQMp+54yeLqL1BSG1VBH3wGdDDl?= =?Windows-1252?Q?8Go3zv37SB/mlLKkUbrwwmUmlZpvprfmdTEqqyyrhKouSxiw=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; DB3PR05MB361; 5:gAETYY92EfH+jL8jzrV7+Cr3MQy67iERzBSpknHYj5dntyUwqvod0eiaOCaDqnW43Wkeu7ZPNldBoO7x0rk4bkK3ijxgbLhnw/VVmRGzzGbmdC627+IXIcYN86JlTRrEwK6a8YFpfvUQGioOfU+4WQ==; 24:1uolC0KA3ovXO/J2IkgBsFRWFpuCNhTN/r0nIN799m6DGsRRIlriTBBcmCf9rGT4N1JUK+L3mO4GV8suYuSU225T8qR7rJ3J4PZLwsKuWXo=; 20:9Baiyce21jglQbAkRfbkhNi9I4tLI8g2r5gWHXaGW/nz1M1G/jguDC5e2EyLOVZtV7hViJBDMyVnQuwzVpxGUg== X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Sep 2015 07:55:30.3268 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR05MB361 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 9/22/2015 10:19 AM, Sagi Grimberg wrote: >> >> As mentioned earlier, I have a WIP RDS fastreg branch [3] >> which is functional (at least I can RDMA messages across >> nodes ;-)). > > Nice! > >> So merging [2] and [3], I created [4] and applied >> a delta change based on your other patches. I saw ib_post_send >> failure with my HCA driver returning '-EINVAL'. I didn't >> debug it further but at least opcode and num_sge were set >> correctly so I shouldn't have seen it. So I did memset() >> on reg_wr which seems to have helped to fix the ib_post_send() >> failure. > > Yep - that was my fault. When converting the ULPs I optimized by > removing the memset but I forgot to set reg_wr.wr.next = NULL when > the ULP needed. This caused the driver to read a second bogus work > request. Steve just reported this as well so I'll fix that in v2. > >> >> But I got into remote access errors which tells me that I >> have messed up setup(rkey, sge setup or access flags) > > One thing that pops is that in the old API the MR was registered > with iova_start = 0 (which is probably what was sent to the peer), > but the new API the iova is implicitly sg_dma_address(&sg[0]). > > The registered MR holds these attributes in: > mr->rkey > mr->iova > mr->length > > These should be passed to a peer to perform rdma. > > Hope this helps, > Sagi. ohh, I just read the RDS 3.1 specification (for the first time..) and I noticed that RDS 3.1 header extension contains only a 32bit offset parameter. Why is that anyway? why not 64bit so it can be a valid mapped address? Also the code doesn't use it at all and always passes 0 (which is buggy if sg[0] has an offset from a page). This won't work with the proposed API as the iova is 64bit (as all other existing RDMA protocols use 64bit addresses). In any event, I'd much rather to add ib_map_mr_sg_zbva() just for RDS to use instead of polluting the API with an iova argument, but I think that the RDS spec can be updated to use 64bit offsets and align to all other RDMA protocols (it has enough space in h_exthdr which is 128bit). I was thinking of: --- Thoughts? Santosh, can you use that one instead and let us know if it resolves your issue? I think you should make sure to correctly construct the h_exthdr with: rds_rdma_make_cookie(mr->rkey, (32)mr->iova) -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index e7e0251..61fcab4 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -3033,6 +3033,21 @@ int ib_map_mr_sg(struct ib_mr *mr, unsigned int sg_nents, unsigned int page_size); +static inline int +ib_map_mr_sg_zbva(struct ib_mr *mr, + struct scatterlist *sg, + unsigned int sg_nents, + unsigned int page_size) +{ + int rc; + + rc = ib_map_mr_sg(mr, sg, sg_nents, page_size); + if (likely(!rc)) + mr->iova &= ((u64)page_size - 1); + + return rc; +} + int ib_sg_to_pages(struct ib_mr *mr, struct scatterlist *sgl, unsigned int sg_nents,