From patchwork Tue Jan 12 12:16:42 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Or Gerlitz X-Patchwork-Id: 8017501 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 374619F6FA for ; Tue, 12 Jan 2016 13:50:34 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 5661B203E3 for ; Tue, 12 Jan 2016 13:50:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 28186203DC for ; Tue, 12 Jan 2016 13:50:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752394AbcALNub (ORCPT ); Tue, 12 Jan 2016 08:50:31 -0500 Received: from mail-am1on0068.outbound.protection.outlook.com ([157.56.112.68]:32948 "EHLO emea01-am1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752249AbcALNu3 (ORCPT ); Tue, 12 Jan 2016 08:50:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:To:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=nh1Kn7VuNSVbSPWQkaldEfSyBJr3HUyg4PSMZhfBWQ0=; b=CzXb0SxbCMc0GjGGqkD4QuEi/l5oyAl1+frJEW1i9Wu28iMyF7tYRCoWP7HfpqWf6m2zBOgalPe1ATab/TrKgAajMGN/tDXZzt4VB+lYK6JmJPv5vsCeSGFfB0OVkUOQp5P4TrZNqjIJ+HyPUgrw5+dKcTggViM5i9RUtsD47jw= Received: from HE1PR05CA0123.eurprd05.prod.outlook.com (10.169.113.177) by DB5PR05MB1223.eurprd05.prod.outlook.com (10.161.244.28) with Microsoft SMTP Server (TLS) id 15.1.361.13; Tue, 12 Jan 2016 12:17:36 +0000 Received: from AM1FFO11FD038.protection.gbl (2a01:111:f400:7e00::182) by HE1PR05CA0123.outlook.office365.com (2a01:111:e400:7a1a::49) with Microsoft SMTP Server (TLS) id 15.1.365.19 via Frontend Transport; Tue, 12 Jan 2016 12:17:35 +0000 Authentication-Results: spf=pass (sender IP is 193.47.165.134) smtp.mailfrom=mellanox.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=pass action=none header.from=mellanox.com; Received-SPF: Pass (protection.outlook.com: domain of mellanox.com designates 193.47.165.134 as permitted sender) receiver=protection.outlook.com; client-ip=193.47.165.134; helo=mtlcas13.mtl.com; Received: from mtlcas13.mtl.com (193.47.165.134) by AM1FFO11FD038.mail.protection.outlook.com (10.174.64.227) with Microsoft SMTP Server (TLS) id 15.1.355.15 via Frontend Transport; Tue, 12 Jan 2016 12:17:33 +0000 Received: from MTLCAS13.mtl.com (10.0.8.78) by mtlcas13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38; Tue, 12 Jan 2016 14:17:32 +0200 Received: from MTLCAS01.mtl.com (10.0.8.71) by MTLCAS13.mtl.com (10.0.8.78) with Microsoft SMTP Server (TLS) id 15.0.775.38 via Frontend Transport; Tue, 12 Jan 2016 14:17:32 +0200 Received: from [10.12.129.83] (10.12.129.83) by MTLCAS01.mtl.com (10.0.8.71) with Microsoft SMTP Server (TLS) id 14.3.123.3; Tue, 12 Jan 2016 14:16:43 +0200 Subject: Re: [PATCH] IB/mlx5: Reduce mlx5_ib_wq cacheline bouncing To: Sagi Grimberg References: <1452594732-9573-1-git-send-email-sagig@mellanox.com> CC: , Matan Barak , "Leon Romanovsky" From: Or Gerlitz Message-ID: <5694EEAA.3050600@mellanox.com> Date: Tue, 12 Jan 2016 14:16:42 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 In-Reply-To: <1452594732-9573-1-git-send-email-sagig@mellanox.com> X-Originating-IP: [10.12.129.83] X-EOPAttributedMessage: 0 X-Microsoft-Exchange-Diagnostics: 1; AM1FFO11FD038; 1:7SPAz1/W5UeAHeY4lc1QOVhfNTqnj31ObM3CDnXFYlAgjHefFxoigftQMbPeHtDAcGuWZHKlAaD4ABTmnAEh2PC7N2Xfpoi4juPn8k45dudV0M17cOKF/lOn1+qvBn7zNwRKJFQ0lWyJsu3g6Tx4K8b+BGD/6/ZMViQrLeeGNCpFZHtwZ19i2KqQICePwRYbsoMmez5PrTaT+oYdHFpaDvS7/4NniYotVbo0vmQI7MHB/A+s1rrs+oZIkGlaDmHEAaQEb0HP5C5Euv5Xw7gALXDqL6xhc+RsuXLA8z4/ecaBlr4cTQc3L9g/6G4e+VmawIgTP05MUEWASP5KcIuC3l3O/C1bjgEA3hjytSIOBhu9jKU8Yz2TNubTj8mwajwZ5vYt/eqYI6wvbH/XbF3qCg== X-Forefront-Antispam-Report: CIP:193.47.165.134; CTRY:IL; IPV:NLI; EFV:NLI; SFV:NSPM; SFS:(10009020)(6009001)(2980300002)(438002)(209900001)(24454002)(377454003)(199003)(189002)(479174004)(1220700001)(450100001)(4326007)(65806001)(19580395003)(83506001)(59896002)(47776003)(50986999)(87266999)(54356999)(15395725005)(189998001)(107886002)(110136002)(65816999)(76176999)(4001350100001)(80316001)(33656002)(50466002)(86362001)(65956001)(23746002)(106466001)(97736004)(87936001)(77096005)(36756003)(15975445007)(5008740100001)(64126003)(4001430100002)(92566002)(6806005)(230700001)(2906002)(1096002)(5004730100002)(586003)(3846002)(6116002)(11100500001)(2950100001)(4001450100002)(3940600001)(6606295002); DIR:OUT; SFP:1101; SCL:1; SRVR:DB5PR05MB1223; H:mtlcas13.mtl.com; FPR:; SPF:Pass; PTR:ErrorRetry; A:1; MX:1; LANG:en; X-Microsoft-Exchange-Diagnostics: 1; DB5PR05MB1223; 2:QS3e/5jNvaVQ4fJaWnwTglsmlcdHqPSuUGdfwnQj1RPIEyQs+s/RKOCMWeIMEdEATAcjSn6gOaRT93Q5J8LPmJ/pcvQ0c8IbIyPvWag0RfvNd5NaENbeDccs1lpQ14oBO61Gih8i1OnJEulEoBHwyw==; 3:ONk4J+h86C4/Gyk+l6CGSWzDgbjhvZGgEZ0ihlub9D7xnZh+lgLdNwhWwYQ7nrJbKcl/FUHjp/rAJsXh0ZyGq+WIYcRRp96daNexZxUnHDFK0c+eoEmfwPQfbv/L5BIHy1fGWJEva01tmh6AU8w96q/fnG0JXa5mYjcwutLSagDQylAgmYe/8BVQMtZRwtNFS+OgctLyRHJUb89s9YouKv5L8us7BUHxUjsTJdU8VYnBdxnIXihIX0HfwLm/N2hxnXkWhvPWdrlt4qpDFbPhMA==; 25:ibvFsOHSi+t6mP24MPYsPdQiEAb4F8w5mAnEm4MwK9eXTieUUGfM50oKZ/WtlredMPsJf1jtmt/DNl5568Zxm4RBzkVMjDBytuKXBkwr1yVforWplGZ+YbxSOrWuYKZI8hqfqYBDzuiRdJmrWf1CDD+aRpgmJu4Fl6XABZXOTAj/cz34ZWkgZFKQIjIRkv5jFwCMrJ8FR74ypUsWEp/WWeTuH9tH5xauyr/AQ3VjT34p00J9LBq5Uw4b3h0d/qW+ X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(8251501001); SRVR:DB5PR05MB1223; X-MS-Office365-Filtering-Correlation-Id: adbb5000-0fd9-40a7-802b-08d31b4a59b2 X-Microsoft-Exchange-Diagnostics: 1; DB5PR05MB1223; 20:SSYEQVm1DlK/iNNkwQikj0WrsI9p7+6KWvlBDsLIEn+1H3/26VNzJ8J0LZH4ENeN5SrexvYAXlxrtol0W3uvn2SjzSmWmsjCCe3t4twRwkZJvDDivKj0vTN6Qb8BfGb3KL1CjOFSPb49D3iMC1AeDqOFeQ3pbG2plpol1kJF77hkmilRmcBoVnJpHjrSdLWrHJs4E/1fqLGiPlNEXIBgEjOnf0/rMuN+7ysFUW+YFOHO0ShAj8cWWqz1Moncr72BmTNR/j77g39/V4Ww/fzepMiqeZCAlSnL8DtIj84ScKh/3/ySx6MdEhbMUMNqXono2rrEdfR9szzjjUiNYFswKDrpmWLwikWE8h8OimbTgWLvKMsvhoC21lnrHr6WDJ6nfBlaMY0INX+kluzHJCJn2HxsfNc/kmcDs6Osgtdy0z5BiA0QDTdFibSq86u8DO/+mCPCTzGLV2cIgA2rr+ObXuDOgYk5ZX7ub8+RlCeiiaXIR+mUvV0bNDNTxc/Pmbiy; 4:Be4wTLuH1eDnaimlWVDDNsFwYaxW1HGrt8R7BIbQJAmZpl+7dLHmxP68urCwTcsDABDTJpcVzw7qGmBKcYKBX2pObJ8IXGpVTsMI9CrwVFeeLxTesoMfq7iVTpoD8Nh9oZhCzrgZpXVeyTNRBD9BzmP8UOq1+yyAPYZTJ20F6v0TpLCbahbGhDHw3ZY+LckgP5vJTbAic3sVywneLXpWbIs154up/69I+0Bfy+dzb3z7ufQ7ruP1AmtBUw3Y7DVwhtpW2+Od22BJbMxo9NjX0bA+uKWcC/WmA9/XVr9Jg8Un4PtSnUS8o+oLM4L+QHeU5hBbtWDWATy5jfp+9gariL7h1T9l5d7kMYfMy7/He7zIcxrwsbjm3Hjha7FVoFCcHE2QdN0HLQBFm5TClyJ7GCj5TmH03LE8bzSZ9X 1HQdLVrCa5WZt4B9qGOH+QYK4f X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(601004)(2401047)(13018025)(5005006)(13015025)(13017025)(520078)(8121501046)(10201501046)(3002001); SRVR:DB5PR05MB1223; BCL:0; PCL:0; RULEID:; SRVR:DB5PR05MB1223; X-Forefront-PRVS: 081904387B X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1; DB5PR05MB1223; 23:djBdHi5I4F6JIt5g/powz8tE9C6vwFHylqYos?= =?Windows-1252?Q?GCox4PgbYrS1BUi6zMP1HROzTI6N4qvUs6MciFYa8QOCynWYk1SBUAOY?= =?Windows-1252?Q?qEKQV8EPUoNfGA/yMhNAaiFmOcldKKT5uNxSxzje8GALKB+g/xGA1qLf?= =?Windows-1252?Q?YKAYUUia/1psvcy+rBmRuscdVbSq5sUCQNxPJlt0q8CWMkWuQvRY/j2x?= =?Windows-1252?Q?DW5DOLu4/gyC0kJO+ba1CFOxcHekERRJpcd4OdDbaDPtYFs3mTmsN8m4?= =?Windows-1252?Q?v1PE3bCmvY+YCib/R+QGnkc7NRvmy6i8qHwig7dEEd4nT9etko3kPmY7?= =?Windows-1252?Q?neu2WwNVT2MYpo6gx1MA7ODemYc8mUuzn5pFgCc4BxxG53P7/sBt0F16?= =?Windows-1252?Q?74C3zv2tVfr3nqVTYH5KJxcuZZyLRT8SkqN2QcrmSSNXFe4gyLHUjBmO?= =?Windows-1252?Q?BM/2DTwsqzXM0Zsq9+CkTiD4kcOW8pmLk0EyLWMcB/dmWhpKfqHUDqiz?= =?Windows-1252?Q?Osvh2nQghm+jQUFvmbesMHZO3tmPzOlx6MuSOh9LKz4qpDg7sx6kJ/e1?= =?Windows-1252?Q?KlNdTUjlTphBDwZzy4WgBrJX3dQ/ZDQfpylBO324mXVsNnLQKC152Olt?= =?Windows-1252?Q?0tCK7GI/tzu9ioMiyr5Sp3BJuf/P8m61Bn2oxCHV8pX0CanX2juXvM56?= =?Windows-1252?Q?MuZ7LxCRNTgEjGtIPJdaMMKJeFi8oOWq8zKjQQmHLSVsH5xiBU1/gAdk?= =?Windows-1252?Q?cnAm2qenaJ++kbkJ1Vk+Pu2N7lMkidL4wj8Cks1NUSoIZ6X5vJsmMyGd?= =?Windows-1252?Q?cqg1ArjU/hqkAWkyFnRYq/w2u4PsOGT3E1bVSz2lLAfqxRBjUlw8rJ7O?= =?Windows-1252?Q?nZRrkJAm2Q1akb+6QCXZnJ5PGsc02qfGzeIoyCzOgQ5gPl2jFQF5k7qS?= =?Windows-1252?Q?So+B/b+qUYlDKvbMQfetQuJH0Dzh2MEexLpUT5TsFoyPZ8NJuMwKhfd2?= =?Windows-1252?Q?MxSNR3ifafDUFJeILtvEdi862mD5TeaR0zFjMO8Ulpzudj7pKHc/tdOo?= =?Windows-1252?Q?l1aWMW5upjW6Hwzg8dwNJrr0u69gPCAuctttzlDRIJknsUcDkfOJ9E+m?= =?Windows-1252?Q?Ic1sCgB9QpCyzTg5nqrrYzeQbLu/qbjZNrrryGUQB1uFhduDR8C+UxZG?= =?Windows-1252?Q?rRZT1sqvfQvoKy9Qo8bHOYBWrzBUsCb5wKfZvhykKqY3OdEZzR9DegqT?= =?Windows-1252?Q?IHxcAXfwQWfj+vqAp3GqfcJ6SFPrGf1BllPq8BHaEIw4kaZ12N0z1WOr?= =?Windows-1252?Q?pv4SIYWjv7zs8U7b7MFFOpLVn35EG3uG/0HYGf19w9smvRIUBYlxxgoO?= =?Windows-1252?Q?nglfpJdCKPG6QcajT8HTzM5PyfSELey4oPDoFg6qcLsiVDBdGBMansS3?= =?Windows-1252?Q?h8e4HBS60tV5Zq6AdUxpYro66kzscwRyzUd22On8w/Hh4Wrq7yl0jG2E?= =?Windows-1252?Q?z6rBX5TtBXG0k1oMi6+32S6KN7HY78TxRrO4uQTVUSq7TLDg0EUCV/RB?= =?Windows-1252?Q?JlayzJiJnoKTF4=3D?= X-Microsoft-Exchange-Diagnostics: 1; DB5PR05MB1223; 5:bNoXdjH5tyF/1wXU+AFD4+eVlUiqPppmiwlNGTDbsYa7IfrMzQBVZhcAhLCL6V239C37LJOWU+piSgq9EBrRVpLqD8KneAXDAk+meIX4ZC+3qK3PoRNqTy/LsPXx/2/zdqfd8NAvrzOvmMX/W7vV+w==; 24:Q+EqazBKE/PZTjSq0KBPW7daD2AJ2pSWvbtHkwxZqp6pjmA2bK10R6M+XP2ZG/ESC85f+MhRScb4DC+zINNhyM8TevCB/6z4W6UWFEoen6I= SpamDiagnosticOutput: 1:23 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jan 2016 12:17:33.3120 (UTC) X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b; Ip=[193.47.165.134]; Helo=[mtlcas13.mtl.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB5PR05MB1223 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On 1/12/2016 12:32 PM, Sagi Grimberg wrote: > mlx5 keeps a lot of internal accounting for wr processing. > mlx5_ib_wq consists of multiple arrays: > struct mlx5_ib_wq { > u64 *wrid; > u32 *wr_data; > struct wr_list *w_list; > unsigned *wqe_head; > ... > } > > Each time we access each of these arrays, even for a single index > we fetch a cacheline. Reduce cacheline bounces by fitting these members > in a cacheline aligned struct (swr_ctx) and allocate an array. Accessing > this array will fetch all of these members in a single shot. > > Since the receive queue needs only the wrid we use a nameless union > where in the rwr_ctx we only have wrid member. Have some performance numbers before/after this patch to support the proposed change? Also, I have asked you the same question re the iser remote invalidation series [1], this datais needed there too. Or. [1] http://marc.info/?l=linux-rdma&m=145188949807652&w=2 > + u64 wrid; > +}__packed; > + --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -129,11 +129,24 @@ struct wr_list { u16 next; }; +/* Please don't let this exceed a single cacheline */ +struct swr_ctx { + u64 wrid; + u32 wr_data; + struct wr_list w_list; + u32 wqe_head; + u8 rsvd[12]; +}__packed; what the role of the reserved field, is that for alignment purposes? if yes, maybe better namewould be "align" Nit, (same) checkpatch error here and below ERROR: space required after that close brace '}' #111: FILE: drivers/infiniband/hw/mlx5/mlx5_ib.h:139: +}__packed; > + > +struct rwr_ctx {