From patchwork Mon May 14 22:46:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Gurtovoy X-Patchwork-Id: 10399575 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B3A1E601F7 for ; Mon, 14 May 2018 22:47:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CC60328535 for ; Mon, 14 May 2018 22:47:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C0BFC2858D; Mon, 14 May 2018 22:47:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14E0828535 for ; Mon, 14 May 2018 22:47:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752094AbeENWrs (ORCPT ); Mon, 14 May 2018 18:47:48 -0400 Received: from mail-eopbgr40054.outbound.protection.outlook.com ([40.107.4.54]:43568 "EHLO EUR03-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752078AbeENWrr (ORCPT ); Mon, 14 May 2018 18:47:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=B2FwJu03VUhmd++LSwAZYgrd1x+FIr7VlQ2mKbpTngo=; b=gqThZCSDjG/u0gykm5LmjRi9jqK234rVELjo6D5JUbcbwUMlS4n3Vl8jSlcSUmGX3zqKPP8niVs+CBmirYOCm0eIfsPsWHKzqXTaVvYkdbTZN1dLBHcg2f6M0Lz+xcrekpbKX6jgVBfOtFO9k8BmH5fKwyydeZgbvGvmOFzoENI= Received: from HE1PR05CA0272.eurprd05.prod.outlook.com (2603:10a6:3:fc::24) by VI1PR05MB4272.eurprd05.prod.outlook.com (2603:10a6:803:40::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.755.16; Mon, 14 May 2018 22:47:44 +0000 Received: from DB5EUR03FT048.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e0a::209) by HE1PR05CA0272.outlook.office365.com (2603:10a6:3:fc::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.755.16 via Frontend Transport; Mon, 14 May 2018 22:47:43 +0000 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=mellanox.com; Received-SPF: Pass (protection.outlook.com: domain of mellanox.com designates 193.47.165.134 as permitted sender) receiver=protection.outlook.com; client-ip=193.47.165.134; helo=mtlcas13.mtl.com; Received: from mtlcas13.mtl.com (193.47.165.134) by DB5EUR03FT048.mail.protection.outlook.com (10.152.21.28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.20.735.16 via Frontend Transport; Mon, 14 May 2018 22:47:43 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.1178.4; Tue, 15 May 2018 01:47:42 +0300 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.1178.4 via Frontend Transport; Tue, 15 May 2018 01:47:42 +0300 Received: from [172.16.0.0] (172.16.0.0) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.301.0; Tue, 15 May 2018 01:46:50 +0300 Subject: Re: Cannot Connect NVMeoF At Certain NR_IO_Queues Values To: "Gruher, Joseph R" , "linux-nvme@lists.infradead.org" , "linux-rdma@vger.kernel.org" References: From: Max Gurtovoy Message-ID: <743c4b33-2bf8-7c76-95af-62043c2aaeeb@mellanox.com> Date: Tue, 15 May 2018 01:46:49 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Language: he X-Originating-IP: [172.16.0.0] X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:193.47.165.134; IPV:NLI; CTRY:IL; EFV:NLI; SFV:NSPM; SFS:(10009020)(39380400002)(376002)(39860400002)(396003)(346002)(2980300002)(438002)(51234002)(199004)(189003)(65806001)(65956001)(3846002)(59450400001)(36756003)(47776003)(106466001)(230700001)(5660300001)(16526019)(77096007)(186003)(305945005)(26005)(65826007)(76176011)(229853002)(6246003)(478600001)(67846002)(2501003)(53546011)(5890100001)(6116002)(106002)(86362001)(575784001)(110136005)(31686004)(446003)(336012)(11346002)(16576012)(58126008)(2201001)(356003)(50466002)(316002)(31696002)(64126003)(2906002)(476003)(126002)(7736002)(2616005)(8676002)(486006)(81166006)(81156014)(8936002)(7756004)(3940600001); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR05MB4272; H:mtlcas13.mtl.com; FPR:; SPF:Pass; LANG:en; PTR:mail13.mellanox.com; A:1; MX:1; X-Microsoft-Exchange-Diagnostics: 1; DB5EUR03FT048; 1:6AkIkt7f+vWdMCBSLjiPc1wlM7TPD3mHE8wdeyMyHWleciM96mCOU/MPLq4ypbdhMJFXdTg8dKa0UOIKOYxpT4MVmnu5cr7rboE9FXPjHU/wsMgwiF1Q55KqW0QoYbXN X-MS-PublicTrafficType: Email X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4608076)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:VI1PR05MB4272; X-Microsoft-Exchange-Diagnostics: 1; VI1PR05MB4272; 3:7hcCi19RcDAgH6eq4CBjhtayFchurUw3q8ysmgaXp2z0dm3K4QpcqmLnVw2DNBegTPkoHoXBpkoZoSQeHkrKaNrblSue9Q1sv8coHf9jlTdm4YZm1iCar6zBDmvUJgWBN17AktGooXDK5n4Qnx9YU0UenYYnUn/SCaotG3rx9AgPE5AEqll5EvsglTOWY/ooIrWPZ6Gzs2jWK8etObjPu1hSVLmekOFDBOoXhBBPkiwNOxKBBoydTqry9nZ8xX4ToErEfA57EQrHVu0hAaU8KxQ5M2YwL6t+LgTdNG88sZoeyMnVdXEY+nr4EHpLf9GtXSrhz+2nt0eDqD/UqtaYzulpTNY9WTiab2TjV1CY1y4=; 25:gfIK5do/d4K6M53K3EuyIklRvOMjsnKSby6czaF6PwMLfT5I8H9u4DyYPpdA9S2PvwzIefdz81+qMB2CXsV7wiabN2Hnjf8/VsARUWa3fdwFqjkVkHyi6DPAHCGSpZ9WyFEZfY+tWXLnEgWmq9kNalVVs7mqbD/cnZoOIfkdCv6EMAdhoBs2Vz4FZ1q5wgkMDVT06EDbzBG9qA8Mg1UJwIf7pusQF31G6a7b5yvWzxB/FSWtKT7fNHP0vZbp/P2IRJYtzW1atO+UdGuSnkwL51Ko/EDF2cYiEvkIOMuE0HpD9ga3pS/3rja/8wWkD/ws7XU9IJZPVDLW3eaVhu6d0A== X-MS-TrafficTypeDiagnostic: VI1PR05MB4272: X-Microsoft-Exchange-Diagnostics: 1; VI1PR05MB4272; 31:EjR/FEAu2pklfhfe3Gd6M40DDCJTCJEvWmEalIi9G5QMac0KAFpJd9AFxyvq4MCpZe4S7InfBUm5fOiByIL00BBHysBJu5WRklamCFP9xBlItcSWwhgNZXtKV386zicjWw0OzUPmSfQ6g2hgwZCqX+mT0Z03/M4TXkHAyzd4RJDq8zvrFyodTelm7k6ikI5H93Pe4jQDai2skzpQaIxxmdFw0ckyoeZAclgXbQEI4nM=; 20:N9ZA5JiirXOG8Xgu+k1PvdE1epagh1BposLhCoDo0MCCNnykJQ2Rcli9C7sx7lAsyi1MyP+Wj6CQA6zRBy/0WaGormLzi8x0CNic5eaanHZb5wfY4EpFxfLtz28K2lh/EYGS22wt1AQt6Ze94LMtsQ40GTl+rHUdBwVh5+X/5WyWZ7DIAwBuhEfkvpsMoONqAR0lXDB8g211QI7VKNPvt/+PRPsYqMMFwgz9vFHNm1O1JNtq40CzSwPBPeD8OlQu72SwzfiyvePc5HhD+Ld9/EgQ829aynM373dTGU+t/G20BC2KkrBwrmbpnnx+6S55jOgLyCWisvslIJtD8uqmBg9pb17kSlgEth8T1KyNMJmFbyISdaLcIPwnppn4K6ByGI++4+PkmcuVlcSGXrsJE1XibRiisFqfFCgGwYayFPHV1P1P7Y1gL7Bh8WznwZS90I+4+d27Cu2PSPa1E21ypl4FDXV3w8LX1b4pxQzCBRSRVy+gB5SszDY+KjeDcG8a X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(21532816269658); X-MS-Exchange-SenderADCheck: 1 X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3002001)(93006095)(93004095)(10201501046)(3231254)(944501410)(52105095)(6055026)(149027)(150027)(6041310)(20161123558120)(20161123562045)(20161123560045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:VI1PR05MB4272; BCL:0; PCL:0; RULEID:; SRVR:VI1PR05MB4272; X-Microsoft-Exchange-Diagnostics: 1; VI1PR05MB4272; 4:gFHua3fBpGwXYTkaz5w47nSxDiN3nNYHPcCgJIXw9Iu9bvwVEsY77uOHZnL7x9r1Zs9K9MEg3lDld3TMzq1SdTiFKimnHgYctPnt3H2lECcAH77s09/3tiKHDZrmT9lRSX67MQ0QXawiyvhbqCBiaAM/YnJdxminRvBxjSKg+0lvjTyot+cNL/4WjcDVvx9wYtWLRdZ5FhscKyb9kZEsH15Urkp4vXj7cQVlhaAbe9zfCOqNxDNrneOZGfwUVyixKFuXXG1mcg0a7UfHIq326OSr+F8LrJr0z+KVLXGbCWUE8Xdpk93rsQ+5eNxjDdk0 X-Forefront-PRVS: 067270ECAF X-Microsoft-Exchange-Diagnostics: =?windows-1255?Q?1; VI1PR05MB4272; 23:gzqEZYaX6+gPyTsgRVZTYYY1acAkeCSqbSNae?= =?windows-1255?Q?4K1yLVhBbN3CrYnbvySPGzHM1OsFyWVF94KnXkDmUh+DnXKdqTu1/yLS?= =?windows-1255?Q?hZWJWy7FYweDiOXMvNT6OgrxdnPRTp78eHZPOSzrr6Br4Xwdh78Z1NjD?= =?windows-1255?Q?mn6Ouk+GHaQ9XVBslQDeQLPYP9B+0I3bzvRDm8Hehw23Zxq2DTA7dCWq?= =?windows-1255?Q?/RW2+9BcOu/yY0vzh3MHWbNMZpUXTaCa8YUh5nIZ7r3YG7FeVYoGK5df?= =?windows-1255?Q?8MsSjqtT76KoFUqFWN5rDRNvbsKmXW5DU3DbNqFbQqbcySfr+B0iwwms?= =?windows-1255?Q?Vw217wMutCTm6ARZFcGwCFHH4pILZe3w/8hA00qrU+0/dvZKQfElYik2?= =?windows-1255?Q?Ib8ocIO9ek/ao3ebB5oM2HOhfKDi9dWd6wKK7EmnyVA/rKRDICvSsCOV?= =?windows-1255?Q?kPp3iMBIqm8x0IZA3Ab+wRdzmz8wJC40Djgb1k5ChboxJUjeCzfL064N?= =?windows-1255?Q?eS+SuHpaxIEekUFtiE26OcEX9e0c3zG59QM6U3zBrcxiacQ16SlTnfae?= =?windows-1255?Q?lnlzXdoCDn170o+GN3kryT8AFbmX8WQ6UfjGFyeHeM3o587pCcl8CObB?= =?windows-1255?Q?XWm3jtUZgPKDSEDsGwdtpA/0BEVFVPV/tpM0Huv/dYV2XCuglu00oQlq?= =?windows-1255?Q?MjVo3jC01NUDfUgyWAJvi81ayE3yRrl/6hflgTK0bb+vw+3Mzm4vfPsr?= =?windows-1255?Q?TcCz2kXSPVCdVJq0WFMEE+BMxUzFCTTIQrOr/h/O+M2dH4wM2arj/rtT?= =?windows-1255?Q?8ty2tIn7VvTzEP29WNJi6ZGQncY2rrkMSh0AIgjDEj2NyOs8gJfdm5af?= =?windows-1255?Q?g273f/lFyZmY6ouPF/7/yOw8RXSMuuda7swAlCfFlgnb4OhxIFZ/0N9z?= =?windows-1255?Q?PToO6m9M5nqebPz8oA52TFu7vVDpqCSvpTD8Fc6RVkPeCQTjl/JZsYFn?= =?windows-1255?Q?Uw9Dt4YIJm5ZbMnR2TmOns9UyjITr9q9rbvmassnO7QqeFbNvEPyeoPx?= =?windows-1255?Q?flQToyvG333MZD2fTrZRycoMFak3CwPzpXx/claJWfS55vYIqDBMLQ/U?= =?windows-1255?Q?RzMDgQmycMw4iyZT0R+sU5NcauIPY4mxC6LtVWA67rW6I7Bsa/iHyuCN?= =?windows-1255?Q?cPu+dLLq+Xv+o54mJId1bMz00dvslVrQ2ZN+z6dwu12BAnM1kOG5lVZu?= =?windows-1255?Q?9YnpONK2NweN/4pUVYakjghfPxc7LrgGbt8llH6TKIUKwR+uhu9uLs0U?= =?windows-1255?Q?vv2Jfs+Lrk2FkfFWuAkqeyh1Qff3Rc4s3G5bwD/+FuEvk1bUSuUkkHi2?= =?windows-1255?Q?zvkhpH9RVOEB+FYv8H5Rlx1t6Vtrxu5neXydYRifOYOXgceKoaDAnzCQ?= =?windows-1255?Q?JnEK+Rz5IHr9OfGdmoXdkKNWIPHmPpgY3nMVUhyg/4LOd0ugdcePd6lO?= =?windows-1255?Q?36tb2M=3D?= X-Microsoft-Antispam-Message-Info: LHKyjWuC+gUa3T96sgdYh0TW4Dm09z9UYe0MhjSxNbNbdhyTo/KQi5WcvtgvHE1MlaPUcj1KCgMUk5QBsd1P+CbaJnvB9Mfel3hEbQuVgILkFP/oGwp8QvRbC8766qL5l9L5RS5tn4VpcIC7VgkA11WLMTrzXBU6wUV1SA7M00rE9nE8Vz3BbTJfZiTSaMy/ X-Microsoft-Exchange-Diagnostics: 1; VI1PR05MB4272; 6:h3LccwwC6xDRH+dunANxVd8lNnPwJhXACypKDLJxCPx8gLkMGEedXx/PIWvcI93ztL3KqPRWmqlLama2sN2IQgAG+yEx38TMTmomQ9QX8iqlNkZx2dAV9ec+zJ8NCJL9MroLpnPEzeefgZCb80l38xcNQTet4B/5hrXcL1nbJf0H7+Bm9ss0PG/aZyGmxmg4B36+Rwwu3m3a2J68DbzYwRx2UERqAYYajv1J5TJtgcjbNwY3EFjdpIOoM3KlrJF3cDMqHN5Odi++0GvgodrFJwow+bKnfZDYt8PlReePvHHeLOIcbIGX2GNV+g3A3vtA6wSuDA2/YirKDg3xCngBqNokKxUCRobgQQBSFhwMicQSVDRfIaupwRg2+IEXJ2UcZyHgS1/YBzKU8xeko3E2BCq9HXKqcrFL7bXM0CTMTtrnTkLL5UpJVnOfDUk04lHKSOA1gOx+0478yJxFtTELVw==; 5:rT/NZgiHYH7tgq1zNVMUd51epcn3zFtUDPCV2tVIpMSPI/1R0oUgAJTsczZFU7b8txbvh0/AfUA5vEAnq7q4D7kwAssjOsKn0RZf6DgrO9Bolt2XHrCs9IEd0TEQVOBxY1oSkPxN3v0D2pJdx0n/i4QAa1owgG7ptHnH1qWsHdk=; 24:xug2/9dbp9+cWlKd7PhqKOfk6B6XRF/TEK4WdS6fZQdx4fCXK0aQmYFkG6tj29EaMxPJQbVmSt9gKZ+wa+RyIx8QmLeG6TOBg9zqBQD1qEY= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; VI1PR05MB4272; 7:9MqoBRdneDEAc5QgMqrI3BksXORYeDdyV0uTrBtUrl0NZFEW1/Y4Lc6kGGVPsC56snbcELbONhQevvm8XzRNMuJF2kXKq2/HUuQZUqQweHe+2OiyR2sr2M+OlVenjVv75Bjbv7xlXd66OInsadKfH8hXECc7TRzuJB4K8KEoS4FRGS4CTNCa4iQNZl41YZJcSfnpBEY6gEZWWi9zNbaQ3CGBs/RVi11AMZCfF3ngT3j7NDxwUS7cN5x01cxgBTi5 X-MS-Office365-Filtering-Correlation-Id: 400a17f1-07ef-4b8d-122d-08d5b9ecb4a5 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 May 2018 22:47:43.4274 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 400a17f1-07ef-4b8d-122d-08d5b9ecb4a5 X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR05MB4272 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Joseph, On 5/14/2018 8:46 PM, Gruher, Joseph R wrote: > I'm running Ubuntu 18.04 with the included 4.15.0 kernel and Mellanox CX4 NICs and Intel P4800X SSDs. I'm using NVMe-CLI v1.5 and nvmetcli v0.6. > > I am getting a connect failure even at a relatively moderate nr_io_queues value such as 8: > > rsa@tppjoe01:~$ sudo nvme connect -t rdma -a 10.6.0.16 -i 8 -n NQN1 > Failed to write to /dev/nvme-fabrics: Invalid cross-device link > > However, it works just fine if I use a smaller value, such as 4: > > rsa@tppjoe01:~$ sudo nvme connect -t rdma -a 10.6.0.16 -i 4 -n NQN1 > rsa@tppjoe01:~$ > > Target side dmesg from a failed attached with -i 8: > > [425470.899691] nvmet: creating controller 1 for subsystem NQN1 for NQN nqn.2014-08.org.nvmexpress:uuid:8d0ac789-9136-4275-a46c-8d1223c8fe84. > [425471.081358] nvmet: adding queue 1 to ctrl 1. > [425471.081563] nvmet: adding queue 2 to ctrl 1. > [425471.081758] nvmet: adding queue 3 to ctrl 1. > [425471.110059] nvmet_rdma: freeing queue 3 > [425471.110946] nvmet_rdma: freeing queue 1 > [425471.111905] nvmet_rdma: freeing queue 2 > [425471.382128] nvmet_rdma: freeing queue 4 > [425471.522836] nvmet_rdma: freeing queue 5 > [425471.640105] nvmet_rdma: freeing queue 7 > [425471.669427] nvmet_rdma: freeing queue 6 > [425471.670107] nvmet_rdma: freeing queue 0 > [425471.692922] nvmet_rdma: freeing queue 8 > > Initiator side dmesg from same attempt: > > [862316.209664] nvme nvme1: creating 8 I/O queues. > [862316.391411] nvme nvme1: Connect command failed, error wo/DNR bit: -16402 > [862316.406271] nvme nvme1: failed to connect queue: 4 ret=-18 IMO this issue was fixed in mlx5_core function mlx5_get_vector_affinity. It was a long discussion regarding this fix and it will be fixed again in 4.17. After the final fix, it should go to stable kernel as well. Meanwhile I can suggest a fast workaround for you if needed (or other solutions as well): -Max. --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 0f840ec..dd92cb9 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -2236,7 +2236,7 @@ static int nvme_rdma_map_queues(struct blk_mq_tag_set *set) .init_hctx = nvme_rdma_init_hctx, .poll = nvme_rdma_poll, .timeout = nvme_rdma_timeout, - .map_queues = nvme_rdma_map_queues, };