From patchwork Tue Aug 15 15:17:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 13353901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA669C0015E for ; Tue, 15 Aug 2023 15:17:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EBB518D000F; Tue, 15 Aug 2023 11:17:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E451F8D0001; Tue, 15 Aug 2023 11:17:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE4D28D000F; Tue, 15 Aug 2023 11:17:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B9F948D0001 for ; Tue, 15 Aug 2023 11:17:45 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5B5E2160D80 for ; Tue, 15 Aug 2023 15:17:45 +0000 (UTC) X-FDA: 81126693690.03.D9EF0C2 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf11.hostedemail.com (Postfix) with ESMTP id 9A74840010 for ; Tue, 15 Aug 2023 15:17:43 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf11.hostedemail.com: domain of hawk@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=hawk@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692112663; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references; bh=M/mKrj+ee5WgvYQsQvWU+roSEz8/50MqLvk/Dk2bKrU=; b=lofzgsjJ9QkVyD9b/Ls33U4VZSA+Wagc7jhg7ZmKZtNWzwtZEe1e9y+3Es+ezzMv2B91rP BG3pnjyZ7Oy3HjtfTTg3JH06+kPw3sUfnlBMPmXOQAZdWHp3eYnuzNGLlj26CjwCEtLv1y vRyMLY7h2/F/aPmnqPrwmevQiLkHSSM= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf11.hostedemail.com: domain of hawk@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=hawk@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692112663; a=rsa-sha256; cv=none; b=jHNNW1YUDzOy68XUxTXQFpIA2VrFVGW5aQg0a5E5EjTs9LOnuOy2LQAsBwgfvkVrKPBlbF X3VF8N/xtvzEkR644gwgrq648Zsaf2Danj+BFMbSXzInGCcdme+jM84fvrUmNKBu+TfYKs UaUhLgQTod0nPK0pQKVrIlXYpfEbDtc= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 881F4631EA; Tue, 15 Aug 2023 15:17:42 +0000 (UTC) Received: from [192.168.42.3] (194-45-78-10.static.kviknet.net [194.45.78.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp.kernel.org (Postfix) with ESMTPSA id 06313C433C8; Tue, 15 Aug 2023 15:17:38 +0000 (UTC) Subject: [PATCH net] net: use SLAB_NO_MERGE for kmem_cache skbuff_head_cache From: Jesper Dangaard Brouer To: netdev@vger.kernel.org, vbabka@suse.cz Cc: Jesper Dangaard Brouer , Eric Dumazet , "David S. Miller" , Jakub Kicinski , Paolo Abeni , linux-mm@kvack.org, Andrew Morton , Mel Gorman , Christoph Lameter , roman.gushchin@linux.dev, dsterba@suse.com Date: Tue, 15 Aug 2023 17:17:36 +0200 Message-ID: <169211265663.1491038.8580163757548985946.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 9A74840010 X-Stat-Signature: eqth37t9gsgpoaah86sfmbzpc4ke8bkn X-HE-Tag: 1692112663-905554 X-HE-Meta: U2FsdGVkX1+SOSTNcIGoqdZYXAnpBApbIy/O2lYHFIIdwrKsaxsPslLrKkQZ8peEPXhVPnQMz6fAnSNxdtf33KWSCeFE3gBlJiLWA4NsypxZ/uuaJNfLtZt1AkzrcjNezd6sxTutf7WhpB0LYpZ8K1yvE4+t34VQwTCpDJDYEwSCOiJBFdl4qnfM6AazCKxNOeT77GGWOr+ucC/sNMt/6qOcnxnGYaUqpoSUES/4k6I9K2BeiiqtfrFODuN2WqjtaoClm8bWzYK9VtukuzGVIo+YHEoGpmvVSiwRWPuMY5AomQxaQ3vsTnvBeKFXtSieJGRpTqB+QfjSxIds0XIwV7TGByIUo8B29QvyHfNwkTduezaRf8sIzwjFvmunBQNrWMkPz9bIJu8E/fxox07g9U2Ro9aPP83Nj+mJ+9KJtEq4aFEtl9wx5Ono5RT8iIguSpU0D8Qdzx6yj7TV72tqCEJs/eSyMbiI+WNVdveqwq0koalvmT1rEf9e1Ni3bCRwiLXgHbyvYNQY3o7YR698lNp4B97ELn6DuE6CwBventpgzS8dUGYNizS3YUzvGgLK/9u0iFnJcELucfRQlUHJ+MZBPtg5NdF2fRWTzDk897u28S6bqebondeYliGRT+fyaiYVMNCO5+nK8JI7d6yFdM4XzgHdJAUOyvcZbCu+jSl1RcEUhN7mirnyk6ktHI5llL58EEtwqjl/M7Gf2fHazcMJ7XApN/2SENFWivWgiw1jUh5YFRc+wBkkbPgcLQw5V1t8HLI3+mr3po/4okywKVWijCiDsOSSFnAj1l8UuFzgIGxnTVt1exonB0Dnba205fbw6Xj3AC3qr02G/W0J/MTIx/63nE+kzwoSCd0Ol53TvSLaaDXB82bTZ7uW1y/TK2PPSiayOM6BN+7y2yOCGSYBI6E3WDmLNzCBxHfZtcYgWJLIC3/CCkHGxeq2NkiJu8fh3+s0+fEYNLm8gzM 0BwVA5r9 Uo9zj75uqUMT79ZVv0J2A8TKwLppVI+6o/a1U7uYdcMIL65yb8rKB4K17QeiMbbe9ak7N3DmZunN3kqrBxhD3LlXVN9h1gRrZzS/x8QgGb3baFIIKXYClH5S7NtqfTxylEZ3hdsyYBvDNQxWXTfI5xCXP/UXzlOaLbyvfqphtN3I0TuY1faXr9qFExwV5bqirhGmezAvjRvKUng0/7+uXT0M04qa+kqg/SXfdJNNdiSJp3fU2HaIQjpEQ8glY5Jk7s76FtAElMthfJ/rfqoniemio/KiGa4LJhtUdSx5b4i8kF7ehSVCErC0n95B/zBgc7J3U4MWYKz1Sl8xdKl83ro5f0WLiKuUAyn1o7iuvyxDEXVbQnWsmfvV508vJ0mCwo/sq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since v6.5-rc1 MM-tree is merged and contains a new flag SLAB_NO_MERGE in commit d0bf7d5759c1 ("mm/slab: introduce kmem_cache flag SLAB_NO_MERGE") now is the time to use this flag for networking as proposed earlier see link. The SKB (sk_buff) kmem_cache slab is critical for network performance. Network stack uses kmem_cache_{alloc,free}_bulk APIs to gain performance by amortising the alloc/free cost. For the bulk API to perform efficiently the slub fragmentation need to be low. Especially for the SLUB allocator, the efficiency of bulk free API depend on objects belonging to the same slab (page). When running different network performance microbenchmarks, I started to notice that performance was reduced (slightly) when machines had longer uptimes. I believe the cause was 'skbuff_head_cache' got aliased/merged into the general slub for 256 bytes sized objects (with my kernel config, without CONFIG_HARDENED_USERCOPY). For SKB kmem_cache network stack have other various reasons for not merging, but it varies depending on kernel config (e.g. CONFIG_HARDENED_USERCOPY). We want to explicitly set SLAB_NO_MERGE for this kmem_cache to get most out of kmem_cache_{alloc,free}_bulk APIs. When CONFIG_SLUB_TINY is configured the bulk APIs are essentially disabled. Thus, for this case drop the SLAB_NO_MERGE flag. Link: https://lore.kernel.org/all/167396280045.539803.7540459812377220500.stgit@firesoul/ Signed-off-by: Jesper Dangaard Brouer Acked-by: Vlastimil Babka --- net/core/skbuff.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a298992060e6..92aee3e0376a 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -4750,12 +4750,23 @@ static void skb_extensions_init(void) static void skb_extensions_init(void) {} #endif +/* The SKB kmem_cache slab is critical for network performance. Never + * merge/alias the slab with similar sized objects. This avoids fragmentation + * that hurts performance of kmem_cache_{alloc,free}_bulk APIs. + */ +#ifndef CONFIG_SLUB_TINY +#define FLAG_SKB_NO_MERGE SLAB_NO_MERGE +#else /* CONFIG_SLUB_TINY - simple loop in kmem_cache_alloc_bulk */ +#define FLAG_SKB_NO_MERGE 0 +#endif + void __init skb_init(void) { skbuff_cache = kmem_cache_create_usercopy("skbuff_head_cache", sizeof(struct sk_buff), 0, - SLAB_HWCACHE_ALIGN|SLAB_PANIC, + SLAB_HWCACHE_ALIGN|SLAB_PANIC| + FLAG_SKB_NO_MERGE, offsetof(struct sk_buff, cb), sizeof_field(struct sk_buff, cb), NULL);