From patchwork Thu Jul 19 14:50:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Max Gurtovoy X-Patchwork-Id: 10534709 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 02171600D0 for ; Thu, 19 Jul 2018 14:51:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0340128C94 for ; Thu, 19 Jul 2018 14:51:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E9C8829C1F; Thu, 19 Jul 2018 14:51:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0291B28C94 for ; Thu, 19 Jul 2018 14:51:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731110AbeGSPfS (ORCPT ); Thu, 19 Jul 2018 11:35:18 -0400 Received: from mail-ve1eur01on0080.outbound.protection.outlook.com ([104.47.1.80]:36051 "EHLO EUR01-VE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727575AbeGSPfS (ORCPT ); Thu, 19 Jul 2018 11:35:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8LQ/GSdEACH5hQMrRpVOTSCchFfLlTAzdhC+DLdVRLo=; b=VE665022+x0eA1H1gCWBz6S8dS0FDoCfbiw1lpcUguw5paO1s4osKAwTD3jQbeIWKz0XfA0D6SEIf0wqh4VewpgSoucsfqlX3q/ibHGwJ4gvvR9GPf1hf3JdYKUTSILyNAJWiKzNxSEGu9LTfdq95HAsdyo2KjUaUmsI/SsFwzw= Received: from HE1PR05CA0116.eurprd05.prod.outlook.com (2a01:111:e400:7a1a::42) by HE1PR0501MB2345.eurprd05.prod.outlook.com (2603:10a6:3:27::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.952.21; Thu, 19 Jul 2018 14:51:40 +0000 Received: from VE1EUR03FT008.eop-EUR03.prod.protection.outlook.com (2a01:111:f400:7e09::208) by HE1PR05CA0116.outlook.office365.com (2a01:111:e400:7a1a::42) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.973.16 via Frontend Transport; Thu, 19 Jul 2018 14:51:40 +0000 Received-SPF: Pass (protection.outlook.com: domain of mellanox.com designates 193.47.165.134 as permitted sender) receiver=protection.outlook.com; client-ip=193.47.165.134; helo=mtlcas13.mtl.com; Received: from mtlcas13.mtl.com (193.47.165.134) by VE1EUR03FT008.mail.protection.outlook.com (10.152.18.75) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.20.952.17 via Frontend Transport; Thu, 19 Jul 2018 14:51:39 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.1178.4; Thu, 19 Jul 2018 17:51:37 +0300 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.1178.4 via Frontend Transport; Thu, 19 Jul 2018 17:51:37 +0300 Received: from [10.223.3.151] (10.223.3.151) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.301.0; Thu, 19 Jul 2018 17:50:50 +0300 Subject: Re: [PATCH mlx5-next] RDMA/mlx5: Don't use cached IRQ affinity mask To: Steve Wise , 'Sagi Grimberg' , 'Leon Romanovsky' CC: 'Doug Ledford' , 'Jason Gunthorpe' , 'RDMA mailing list' , "'Saeed Mahameed'" , 'linux-netdev' References: <20180716083012.15410-1-leon@kernel.org> <0cf29652-9034-6283-ef36-95de4588980f@grimberg.me> <20180716103046.GJ3152@mtr-leonro.mtl.com> <1cb63259-9fb6-59b0-3a34-0659973228ea@mellanox.com> <40d49fe1-c548-31ec-7daa-b19056215d69@mellanox.com> <243215dc-2b06-9c99-a0cb-8a45e0257077@opengridcomputing.com> <3f827784-3089-2375-9feb-b3c1701d7471@mellanox.com> <01cd01d41dce$992f4f30$cb8ded90$@opengridcomputing.com> <0834cae6-33d6-3526-7d85-f5cae18c5487@grimberg.me> <9a4d8d50-19b0-fcaa-d4a3-6cfa2318a973@mellanox.com> <02dc01d41ecd$9cc8a0b0$d659e210$@opengridcomputing.com> From: Max Gurtovoy Message-ID: Date: Thu, 19 Jul 2018 17:50:40 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <02dc01d41ecd$9cc8a0b0$d659e210$@opengridcomputing.com> Content-Language: he X-Originating-IP: [10.223.3.151] X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:193.47.165.134; IPV:NLI; CTRY:IL; EFV:NLI; SFV:NSPM; SFS:(10009020)(376002)(346002)(136003)(39860400002)(396003)(2980300002)(438002)(51234002)(189003)(199004)(43234003)(93886005)(5024004)(4326008)(3846002)(106002)(6246003)(106466001)(6116002)(229853002)(67846002)(230700001)(110136005)(54906003)(86362001)(58126008)(31696002)(316002)(16576012)(65826007)(6666003)(31686004)(14444005)(5660300001)(64126003)(50466002)(486006)(36756003)(65956001)(47776003)(2486003)(23676004)(356003)(76176011)(478600001)(65806001)(77096007)(8676002)(53546011)(6346003)(2616005)(305945005)(7736002)(16526019)(11346002)(26005)(186003)(446003)(126002)(2906002)(476003)(8936002)(81156014)(336012)(81166006)(3940600001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0501MB2345; H:mtlcas13.mtl.com; FPR:; SPF:Pass; LANG:en; PTR:mail13.mellanox.com; MX:1; A:1; X-Microsoft-Exchange-Diagnostics: 1; VE1EUR03FT008; 1:+TdfUptCWfLgIkziGUpSfYkvsYS+2oe21FKMotPRlFntopH3KOUlai69ONuh/7kjIlqbptzllBera9hlAjEJXXiTkyg5tsdcUV4hiNA0XO8zySv5HyWfXcZP7Ch8Ai4+ X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 142fc3fe-17d6-4c63-8a8a-08d5ed87229c X-Microsoft-Antispam: BCL:0; PCL:0; RULEID:(7020095)(4652040)(8989117)(5600067)(711020)(4608076)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(2017052603328)(7153060)(7193020); SRVR:HE1PR0501MB2345; X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2345; 3:Mi+/jm4rfoPyczcNnfSfT8GtIxFuOpCalsmWQCmomQZTt8/4G3blLpx4Id9aBeHt4N+KmWq1niO40pGLynUCr789xGtgIykAWxDCbBWyujesXQK63+QZVb8Wu1GXAuSfh46NtmMjMCWWBu+x68Vvt9yTn9zMB7a8JKrIuAhGElwQFSOlSqUmeaieVXlABukCE34hfqzKS9ytmN4yhqRvztpauUMjg8XD16Ho/jCxebRJyQuEoupKt/qW4VtcGTiyI5J7xRuClxrFqLhmEMiAyHR9xuIKura7yYbVvGJviSh92HnqgyshlDjsI+srwHFXCRReBJ9jT3H3kivvs/5vee2whG4sPkhVlpzYultixY0=; 25:HcLfuade6JJZ36XxtfMnBqJXXspMQM58I7oOn7d70q7ipM/W7091ZYyak5Ieb7ITeiGqVFyCGtPH0ecpLrNR4h0tGscIgpozfHhADLtbwoAikPLG4f6raIk8/KWr9MSpPxWvz2X/yYXWGz/UJ+0N/8b32kzCwIuyljC81bjEcStX8AIWiwRLFKSAZHY2/0CTkWassnP1qOY5fQiUEjGWukzT+3o7hxXUHBxnNqv92kzYo+xoAK8fQENojrUcEqtEDNLWjvxNOMCDwcxRZiX/eLVUiIJlBekMiT2mdKfZNcn5NwmQeauUxmRFfrkOVUQfhTdcJq/gXfnz3MqEduDAvA== X-MS-TrafficTypeDiagnostic: HE1PR0501MB2345: X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2345; 31:wEDLUp8/z/iiyVt85MX+uFgSo6OPtaDEcz5bwwcxNjljNWLakyEntnK8/IyBBNP7EJIgGvo9AL34+gCSU6xZNHVKcMyfU3F4arOnMmyN9tXUo50lkOH0C8nhYIStECgGSVr0wq2EJDu7jrzJmO0kS+1cQlp1BIF/1hRzqoq2U6ozV0cKCDglpHV57hje4R9/DP1I1mPoIdTJVhPk0uGPaO9z8luVpqco+18uEKe5oe8=; 20:/NRRzLxRhk3nST+KR3yPLrKW8SH25cmCvBvLtbwJm543NTAj8nQmIsYmtpWRfhEj5rU8IJXWhv+AlnGJjiWifDopHc1Ex41QdfSJJYjSQcyhTgdU2Ahn/fsgAX012lzhCgOYXEB50vsQe4WKDLw0xleATBnUOpOCsDmlHLkNEofS9xtpS5tPQFErwBgX15PWWB8krHStZ0lA3/2WoV5MB6gnB/k8+6HKwRA3nueNxAqutskpV27UAcoML/Dx3xPJMM1c6Oh/G561FQxfAvuA+Eqxgf42HfZ1/IIwQZ94hujmAuk6PRyoMXd62cCVCWIM4NnalU6ZPr/K/UnKZioE25Y4iutlmJohjlGFgipfYRJlRlZxWt4i9U1LwEEU0+AowoPJHyivXkPljNUhehbFBajewwHE4eE5RstrJzNeffSfuM5L/y25zd1B/5ZWfiOcrJrAJO+KJfnA0Ie/HIBTtB1JaK5VzKb0caEeOIDipZU3UaBUcLaLSbj39wJI4HYp X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(190756311086443)(21532816269658); X-MS-Exchange-SenderADCheck: 1 X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(3002001)(10201501046)(93006095)(93004095)(3231311)(944501410)(52105095)(6055026)(149027)(150027)(6041310)(20161123564045)(20161123562045)(20161123558120)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011)(7699016); SRVR:HE1PR0501MB2345; BCL:0; PCL:0; RULEID:; SRVR:HE1PR0501MB2345; X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2345; 4:DUSnKjPMrvJ3UzyA865J3u/J5UsfabC6bQK8SC/xQeizZFlff5tlUZ71ufFvjevPevKrLgA88WcZLdO+imyXBAGKZY3rcHd8PMRp4xPuZlUIvmu0X97SKyi0qljM4v5F0Zh8jXZmLzSIoAZB6AY1d45YIM/1JyXwGriumGN/FsTymWNya/J+a+erwoqW6IcGODqJj1xGmeq3Y1NbJNNiXCdGDsvi32Au52H9qAQq2Aw4RUC76a6hq8iud5tCXNv+DJwPNXrBOUY9EondEGD/viIlzut/7+N44UjUMl5X69bCgJ4Fz+MdL5694i1Zit2RJvJBUYv7JPbBUQse+Z/f6DKm545m55X1koSrw20PEAU= X-Forefront-PRVS: 0738AF4208 X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtIRTFQUjA1MDFNQjIzNDU7MjM6Zmhmb014aysyVVgybGxnQUEwNTd1Rmcy?= =?utf-8?B?cmYrVHR4dk5ob3BYNS92S1pUam00UEN0ZGhjOGMzWFcraVY2S0ZIeFI5K2Nu?= =?utf-8?B?NFZ1OUhTVVArd3o1TGRIbmtzYjlEVjlsK0llT1FIaG9TVitqMWxBZFBKZW1F?= =?utf-8?B?eWNUc0tzbm1ZZ3paYXA3QmVzR1Z0aGhqNEJhQXRuOHkyQ0w3cGltdVVZMDYy?= =?utf-8?B?VnpHMnh5V0JwR3FhSW8zUW9WT0ZzeGpJR3UvQTltc2s3VGNreHpqQ2dZUU0v?= =?utf-8?B?cTZLKzl0d2UydmdhdEF2MmFIWDhkd0I2aU94Y2Z4UUJPaXJEbThJY2ovdlFB?= =?utf-8?B?Z25Hc21qdXBXRVV1VERoUnZFQzdjSkVTUTRNaWY5T3FzSDlBc2FsZ3pCTEs1?= =?utf-8?B?Qmx0MDUwWXJ2dFJWeTZabGI0azVmeVZ2V0QyTzZ1QWh3VTZBNFBaREFKdGky?= =?utf-8?B?WDZ6eTdnQy9vUzI2SmYxaFlndnkxTmErS2gzZERxNkRDVVhLVGo0ME1KM3By?= =?utf-8?B?SlJQV3NBdjN5d3NkbU1YQ2F3NVRoamhiaXpnRnhCYkFyYk90SmdEeHRZS0tY?= =?utf-8?B?YStjVWhZb2N3ZHlVY1dQMmYvZFd3ZmJDeWJkMDVINzREbGhIaDNtLy9sUVNz?= =?utf-8?B?TnVycEZVTlVzWElnY25FTXdVb2xQek0rcHNSMkhIZmpPUVBHQ0RtNVhzeDVx?= =?utf-8?B?bXlVSUlqY1dSWUlLMjNHQ2k3OWF6aDRadWRNRWswZ1Q1MGRrSFd3NHI2Q3g2?= =?utf-8?B?L0xibWJsM1FweUIvZDJHR1l2YTIxVDJFQkNzNHV1QkZvRnBYT0FBRDljcUZF?= =?utf-8?B?YnZwckdoZ2k5cjNTMkNTM3ZJaFBJMFpLaDJ6YWFma0hEVGhYODYrbVJzaEJk?= =?utf-8?B?R3YrZndsQksyazIrMjExYmQxdW0yOVlWWitxSmZacW8xRmY1OFZmb3ZLbVI4?= =?utf-8?B?d0pGd0lzY1BWaFFLMURUZmpjbTJud1ZmWDczdlQ1ODVSSHBoSnpJRVo3Z1Yz?= =?utf-8?B?YXJjb3U4Z1NVOUlzaVRNbkNLclRrMHd5ZE9hMlRnbGRaVlJQMEVqeEloNHh1?= =?utf-8?B?Z3dEdE5TV3JCc1AzVG4xdDRxc0ZYcU5uMlpuSm16dkw0ZXIxOGZ6WHRUVXgz?= =?utf-8?B?OFlCaFAreFZidzlJR3hxcFBBLzFPUDhIempUVnFwcHd4WkU0bjF2ZmdyZURQ?= =?utf-8?B?YmlOVzBLRytNd2cxMXRnVVlxRG1Cczc3SXpwMk1FZFRPL2RBUDdHNmVxWnJN?= =?utf-8?B?dUxQOGoya3JUYm1GN3pDL1c1RW9rNXp0d29kTGwzVTJjZWUvclhtVVhNSHgv?= =?utf-8?B?UUJ5R3N4OXlXSU9zSWJvc2VGSlkwRE9xMERRYXFkNzFVWXB6MEtGRERTbHp1?= =?utf-8?B?VER6VVhVSUQ3WitGd1VDa0VsL0hsWldGdDN0MmVaNldpSHk3ZDJGbzIxSnNO?= =?utf-8?B?TzlYRm9rTTZJVmxZMzhCL3NWZGVZbncyOXpqblFGY1p0RWNRdjVEdlV3TkVV?= =?utf-8?B?MFBnTmRaWmJVa0Z3QjNwbUVOQ1E4N1N6U2p2SnVaRkpnN0RhaXB5Z3ZFZ2NY?= =?utf-8?B?d1pwbjFRUE4ycHpIenk0NG9Ec2dlZW9Tc1BkZmJKOE56RTFVWTBlbEtGSTAw?= =?utf-8?B?YkpPR3lUVURyQUZMUDVlbU4xdFplQVM4Q283S0lpc2ZXVFcxNXR2dUFWRjBh?= =?utf-8?B?UjlFS3dkbmsraXZ1U1VWbFhzbHpDNXdJKzFJYWlzajVGWEp4a1I4SzRGNkM4?= =?utf-8?B?WEVmVlFuZG8zakRDOW94ZzdJclEwSnR2SjVSM0Y0dFlrR0FHanBJQjBxdXMx?= =?utf-8?B?Sy9JdGJXWHpNZUxFNHBHOHJXbGRNQTlWcmYzdXNtVzBYWFVOdTlkYW1laUdi?= =?utf-8?B?UGt6L1pvS2tYNjRVbXNGQ1N6bTBwMmNTcS9ad2tEL1p0SW92aGgrYTdGaWI4?= =?utf-8?Q?ZsWW5qa4WBmVNGlQedVwXHAC/N5Jjlls=3D?= X-Microsoft-Antispam-Message-Info: 3Vaezvr0Bo3xi+DqGrWGR5ZWcX/617y1894AWrh6n4eG1PZIiKzx575PCjw3w/GZwd1c5MT+bolb6/fzmPOKdvrVqg0kkwfX/E78+5U1pQFWbrLsInoVWowKurCwFlVi3eYkHGAsLKczXb+wFAjFmcjl/R/Zys0zskJYL42ttCyc1iqPmxFHA/oElISpJFsjYtMYBhUoH0j4k4aTbTyaL0siKnLmsqb6PZ6htwA+k5N/0dEfW5I2/r9aj4pCdcjKafBoQg4muNojf4M9RvFeMkRJJc9eQhB/A3xmxarqX6kDxmyxMPh2NCizTLYSycAooTWTxaU8dRQoxEa2j+vUXLr8j3a8bVAUkUOtQyMvTGc= X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2345; 6:K97ZwVqMb9yIoL+hoxr/GmlXCD7zWRM7vw7p9PyojA9j4GUOFfVsPIZwT91acFk1dVj1CSDwm14roojzg/eO1KT33gXW5fDoZXp9zb7n6lc2WUMSpM6mbfU5+Ug2eRUVjOrZZM9Mr02yr0T3R8+BljXR4cUD+ULGKbXdtQhISybEp4jgOB+WoGJkJREZw8VRY4AAxWbO1uj2gQgGPuKKQVSVHFKl78RGKNj/ugB++JBkpdaTxM9hZx75hGQTdBY3knCCHaZWgQ6nmQtbj3I2qmihpuTTrvuLoiN6DaAoohWnEAWtjSaIn/0ZSk7xV3UYKE8TXheEWckOQbA+OUaTwkKeG2u6N816ia4/8yc/U0S0tYe3/AACti9qYTYSCNzr0Wp+qVjC38EFNowQHtOpyx2Xb5WDw1vNxKwr76poqrjG3XIla8h9kNh+tdoYv5vg5qyJR3jYcRW2x5NK7Z6owg==; 5:RcjFxI8nj/FrUf+JyjHgaSjwNvMLXZt5Hnb/mvzQA1niSt04k3Cw1BOlGNU/sDHxJk44OlC4ymi9OM8nqBgXplJbcPWO3DZghr7CqJ6aXYF2IfF5WjAV3i463nSKyPQPSHoBuuENL8S4Zz9IqaJMEQgMhBouG4/fBdDilXqXzEw=; 24:04Tbvmzr8xt9Gh8L+ELdSCual181RKFq3mscMWSOgaZbbRQTosSfcqkLTf2SaOj6DUVF3ddyfZ07T0aGfn96yG4S1hsHorgqKMi5TAzMpZY= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2345; 7:BSwFQMZ8D2zHbguedz7/e2VUbRfXtNpqUTOWUOqKCtya1S4um5XpmnBpFjvFpv+XkL6E2R8kPaGB3QXmgxdV4lROZPT4BbYIDI8aFWr3LhoyXuSoo3D14DN98uRfs5VE0L068pqHkYpTPF1RbLvTrygqflCXiVdHPYlQAHmcEVedHnvrZrE7k9x2T3aBNUtHwjfAKDg/ix+p8EiNpchZWeEpJEBaEqUpzHlc9ptfnnLb9+HbaO9v6Lzh/HFHsLgP X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2018 14:51:39.6564 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 142fc3fe-17d6-4c63-8a8a-08d5ed87229c X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0501MB2345 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 7/18/2018 10:29 PM, Steve Wise wrote: > >> >> On 7/18/2018 2:38 PM, Sagi Grimberg wrote: >>> >>>>> IMO we must fulfil the user wish to connect to N queues and not reduce >>>>> it because of affinity overlaps. So in order to push Leon's patch we >>>>> must also fix the blk_mq_rdma_map_queues to do a best effort >> mapping >>>>> according the affinity and map the rest in naive way (in that way we >>>>> will *always* map all the queues). >>>> >>>> That is what I would expect also. For example, in my node, where >>>> there are >>>> 16 cpus, and 2 numa nodes, I observe much better nvmf IOPS >> performance by >>>> setting up my 16 driver completion event queues such that each is >>>> bound to a >>>> node-local cpu. So I end up with each nodel-local cpu having 2 queues >>>> bound >>>> to it. W/O adding support in iw_cxgb4 for ib_get_vector_affinity(), >>>> this >>>> works fine. I assumed adding ib_get_vector_affinity() would allow >>>> this to >>>> all "just work" by default, but I'm running into this connection failure >>>> issue. >>>> >>>> I don't understand exactly what the blk_mq layer is trying to do, but I >>>> assume it has ingress event queues and processing that it trying to align >>>> with the drivers ingress cq event handling, so everybody stays on the >>>> same >>>> cpu (or at least node). But something else is going on. Is there >>>> documentation on how this works somewhere? >>> >>> Does this (untested) patch help? >> >> I'm not sure (I'll test it tomorrow) because the issue is the unmapped >> queues and not the cpus. >> for example, if the affinity of q=6 and q=12 returned the same cpumask >> than q=6 will not be mapped and will fail to connect. >> > > Attached is a patch that applies cleanly for me. It has problems if vectors have affinity to more than 1 cpu: > > [ 2031.988881] iw_cxgb4: comp_vector 0, irq 203 mask 0xff00 > [ 2031.994706] iw_cxgb4: comp_vector 1, irq 204 mask 0xff00 > [ 2032.000348] iw_cxgb4: comp_vector 2, irq 205 mask 0xff00 > [ 2032.005992] iw_cxgb4: comp_vector 3, irq 206 mask 0xff00 > [ 2032.011629] iw_cxgb4: comp_vector 4, irq 207 mask 0xff00 > [ 2032.017271] iw_cxgb4: comp_vector 5, irq 208 mask 0xff00 > [ 2032.022901] iw_cxgb4: comp_vector 6, irq 209 mask 0xff00 > [ 2032.028514] iw_cxgb4: comp_vector 7, irq 210 mask 0xff00 > [ 2032.034110] iw_cxgb4: comp_vector 8, irq 211 mask 0xff00 > [ 2032.039677] iw_cxgb4: comp_vector 9, irq 212 mask 0xff00 > [ 2032.045244] iw_cxgb4: comp_vector 10, irq 213 mask 0xff00 > [ 2032.050889] iw_cxgb4: comp_vector 11, irq 214 mask 0xff00 > [ 2032.056531] iw_cxgb4: comp_vector 12, irq 215 mask 0xff00 > [ 2032.062174] iw_cxgb4: comp_vector 13, irq 216 mask 0xff00 > [ 2032.067817] iw_cxgb4: comp_vector 14, irq 217 mask 0xff00 > [ 2032.073457] iw_cxgb4: comp_vector 15, irq 218 mask 0xff00 > [ 2032.079102] blk_mq_rdma_map_queues: set->mq_map[0] queue 0 vector 0 > [ 2032.085621] blk_mq_rdma_map_queues: set->mq_map[1] queue 1 vector 1 > [ 2032.092139] blk_mq_rdma_map_queues: set->mq_map[2] queue 2 vector 2 > [ 2032.098658] blk_mq_rdma_map_queues: set->mq_map[3] queue 3 vector 3 > [ 2032.105177] blk_mq_rdma_map_queues: set->mq_map[4] queue 4 vector 4 > [ 2032.111689] blk_mq_rdma_map_queues: set->mq_map[5] queue 5 vector 5 > [ 2032.118208] blk_mq_rdma_map_queues: set->mq_map[6] queue 6 vector 6 > [ 2032.124728] blk_mq_rdma_map_queues: set->mq_map[7] queue 7 vector 7 > [ 2032.131246] blk_mq_rdma_map_queues: set->mq_map[8] queue 15 vector 15 > [ 2032.137938] blk_mq_rdma_map_queues: set->mq_map[9] queue 15 vector 15 > [ 2032.144629] blk_mq_rdma_map_queues: set->mq_map[10] queue 15 vector 15 > [ 2032.151401] blk_mq_rdma_map_queues: set->mq_map[11] queue 15 vector 15 > [ 2032.158172] blk_mq_rdma_map_queues: set->mq_map[12] queue 15 vector 15 > [ 2032.164940] blk_mq_rdma_map_queues: set->mq_map[13] queue 15 vector 15 > [ 2032.171709] blk_mq_rdma_map_queues: set->mq_map[14] queue 15 vector 15 > [ 2032.178477] blk_mq_rdma_map_queues: set->mq_map[15] queue 15 vector 15 > [ 2032.187409] nvme nvme0: Connect command failed, error wo/DNR bit: -16402 > [ 2032.194376] nvme nvme0: failed to connect queue: 9 ret=-18 queue 9 is not mapped (overlap). please try the bellow: --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/block/blk-mq-rdma.c b/block/blk-mq-rdma.c index 996167f..a91d611 100644 --- a/block/blk-mq-rdma.c +++ b/block/blk-mq-rdma.c @@ -34,14 +34,55 @@ int blk_mq_rdma_map_queues(struct blk_mq_tag_set *set, { const struct cpumask *mask; unsigned int queue, cpu; + bool mapped; + /* reset all CPUs mapping */ + for_each_possible_cpu(cpu) + set->mq_map[cpu] = UINT_MAX; + + /* Try to map the queues according to affinity */ for (queue = 0; queue < set->nr_hw_queues; queue++) { mask = ib_get_vector_affinity(dev, first_vec + queue); if (!mask) goto fallback; - for_each_cpu(cpu, mask) - set->mq_map[cpu] = queue; + for_each_cpu(cpu, mask) { + if (set->mq_map[cpu] == UINT_MAX) { + set->mq_map[cpu] = queue; + /* Initialy each queue mapped to 1 cpu */ + break; + } + } + } + + /* Map the unmapped queues in a naive way */ + for (queue = 0; queue < set->nr_hw_queues; queue++) { + mapped = false; + for_each_possible_cpu(cpu) { + if (set->mq_map[cpu] == queue) { + mapped = true; + break; + } + } + if (!mapped) { + for_each_possible_cpu(cpu) { + if (set->mq_map[cpu] == UINT_MAX) { + set->mq_map[cpu] = queue; + mapped = true; + break; + } + } + } + /* This case should never happen */ + if (WARN_ON_ONCE(!mapped)) + goto fallback; + } + + /* set all the rest of the CPUs */ + queue = 0; + for_each_possible_cpu(cpu) { + if (set->mq_map[cpu] == UINT_MAX) + set->mq_map[cpu] = queue++ % set->nr_hw_queues; } return 0;