From patchwork Thu Feb 2 18:57:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13126675 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2A41C05027 for ; Thu, 2 Feb 2023 18:58:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233089AbjBBS6J (ORCPT ); Thu, 2 Feb 2023 13:58:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233015AbjBBS6I (ORCPT ); Thu, 2 Feb 2023 13:58:08 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5EB9E12F19 for ; Thu, 2 Feb 2023 10:58:05 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5065604854eso29049577b3.16 for ; Thu, 02 Feb 2023 10:58:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zo5Vq3MAn/Li5H6n621bkQpdnHht3zfcLn68T0vVpDQ=; b=sH7nonY4kCQVPsu3hze+evYr81/o0DoOzq7D7UIKYjAQ3/7jaGWMHgz+SJInipMDpt Ee37HL0FCjpVidoaCCKCTha06BMZb44+z0J2tVMlOegyQG+ANDapaU8oXeEkzBnCeo6r z9TbGOdU6wKZud9yq0Pz8uKVkcJc/kdAu4Fq9LX/GDymXLPb+jkcIQOp8j03/YExv1J1 isKUlL2cYzj8KbW3PmE9rfj6qe9+c8qNMHaAWmXOGMvRj5GqGgHtRpYM9ibdEyIL2Gqg FTslEk+4vn6R8sRbFZFY48fDv/yR3Np38fZbSwL49SdDUIUW4r5yf9lcKatYCfcKAKe7 7ZyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zo5Vq3MAn/Li5H6n621bkQpdnHht3zfcLn68T0vVpDQ=; b=2Pn/INH7jgS8lY6MTuoh6BalMNGk0Mpbd2wGTSMGd5BgD2tyHnQqsdIRZeVpHrO7SR jQA1n6hwBQr4KZhE6xtelf4TTzEfH+1OGhLHR37C57nwxKHqa8jmSaxXsRtQ7c4h4Od2 CWmE0EhDZjdzx6VwTzYIB7WvyoT3L4dCzGZ/afI9r1znb1xMy5SFoy5pZM4LqTbCW2xH E2ZwjJ30GApuAUGsymmtIbF+RZvzPCZgEoadzReAgodYRJTJDNJgM84FrPnBEQOQJldJ aHvexK1WQqMcAajd7mIW0q8Yr6Rjr7Kzf8mbhrEOkO7tO+aV35I4ZB13Xgb9fQtqyMRO zJhw== X-Gm-Message-State: AO0yUKXUv8WwExEMP594FjtvgsTQqNaIZ50WaAmBQEBMbMiorR0YelJX 0cDlIHZd87RtUwfXecdDPJEZfoLcseHSFg== X-Google-Smtp-Source: AK7set9bMmBizAHUloRVsCbSu0u8g7qVikmv1d9SgF1YbrWIiEjWfPtvaH4PmxHIAQjhOijLDlikE/33zP75Kw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a25:d093:0:b0:80b:a1ca:21fc with SMTP id h141-20020a25d093000000b0080ba1ca21fcmr910513ybg.403.1675364284685; Thu, 02 Feb 2023 10:58:04 -0800 (PST) Date: Thu, 2 Feb 2023 18:57:58 +0000 In-Reply-To: <20230202185801.4179599-1-edumazet@google.com> Mime-Version: 1.0 References: <20230202185801.4179599-1-edumazet@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230202185801.4179599-2-edumazet@google.com> Subject: [PATCH net-next 1/4] net: add SKB_HEAD_ALIGN() helper From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Alexander Duyck , Soheil Hassas Yeganeh , Eric Dumazet Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org We have many places using this expression: SKB_DATA_ALIGN(sizeof(struct skb_shared_info)) Use of SKB_HEAD_ALIGN() will allow to clean them. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- include/linux/skbuff.h | 8 ++++++++ net/core/skbuff.c | 18 ++++++------------ 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 5ba12185f43e311e37c9045763c3ee0efc274f2a..f2141b7e3940cee060e8443dbaa147b843eb43a0 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -255,6 +255,14 @@ #define SKB_DATA_ALIGN(X) ALIGN(X, SMP_CACHE_BYTES) #define SKB_WITH_OVERHEAD(X) \ ((X) - SKB_DATA_ALIGN(sizeof(struct skb_shared_info))) + +/* For X bytes available in skb->head, what is the minimal + * allocation needed, knowing struct skb_shared_info needs + * to be aligned. + */ +#define SKB_HEAD_ALIGN(X) (SKB_DATA_ALIGN(X) + \ + SKB_DATA_ALIGN(sizeof(struct skb_shared_info))) + #define SKB_MAX_ORDER(X, ORDER) \ SKB_WITH_OVERHEAD((PAGE_SIZE << (ORDER)) - (X)) #define SKB_MAX_HEAD(X) (SKB_MAX_ORDER((X), 0)) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index bb79b4cb89db344d23609f93b2bcca5103f1e92d..b73de8fb0756c02cf9ba4b7e90854c9c17728463 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -558,8 +558,7 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, * aligned memory blocks, unless SLUB/SLAB debug is enabled. * Both skb->head and skb_shared_info are cache line aligned. */ - size = SKB_DATA_ALIGN(size); - size += SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); + size = SKB_HEAD_ALIGN(size); osize = kmalloc_size_roundup(size); data = kmalloc_reserve(osize, gfp_mask, node, &pfmemalloc); if (unlikely(!data)) @@ -632,8 +631,7 @@ struct sk_buff *__netdev_alloc_skb(struct net_device *dev, unsigned int len, goto skb_success; } - len += SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); - len = SKB_DATA_ALIGN(len); + len = SKB_HEAD_ALIGN(len); if (sk_memalloc_socks()) gfp_mask |= __GFP_MEMALLOC; @@ -732,8 +730,7 @@ struct sk_buff *__napi_alloc_skb(struct napi_struct *napi, unsigned int len, data = page_frag_alloc_1k(&nc->page_small, gfp_mask); pfmemalloc = NAPI_SMALL_PAGE_PFMEMALLOC(nc->page_small); } else { - len += SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); - len = SKB_DATA_ALIGN(len); + len = SKB_HEAD_ALIGN(len); data = page_frag_alloc(&nc->page, len, gfp_mask); pfmemalloc = nc->page.pfmemalloc; @@ -1936,8 +1933,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail, if (skb_pfmemalloc(skb)) gfp_mask |= __GFP_MEMALLOC; - size = SKB_DATA_ALIGN(size); - size += SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); + size = SKB_HEAD_ALIGN(size); size = kmalloc_size_roundup(size); data = kmalloc_reserve(size, gfp_mask, NUMA_NO_NODE, NULL); if (!data) @@ -6288,8 +6284,7 @@ static int pskb_carve_inside_header(struct sk_buff *skb, const u32 off, if (skb_pfmemalloc(skb)) gfp_mask |= __GFP_MEMALLOC; - size = SKB_DATA_ALIGN(size); - size += SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); + size = SKB_HEAD_ALIGN(size); size = kmalloc_size_roundup(size); data = kmalloc_reserve(size, gfp_mask, NUMA_NO_NODE, NULL); if (!data) @@ -6407,8 +6402,7 @@ static int pskb_carve_inside_nonlinear(struct sk_buff *skb, const u32 off, if (skb_pfmemalloc(skb)) gfp_mask |= __GFP_MEMALLOC; - size = SKB_DATA_ALIGN(size); - size += SKB_DATA_ALIGN(sizeof(struct skb_shared_info)); + size = SKB_HEAD_ALIGN(size); size = kmalloc_size_roundup(size); data = kmalloc_reserve(size, gfp_mask, NUMA_NO_NODE, NULL); if (!data) From patchwork Thu Feb 2 18:57:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13126677 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03EA2C61DA4 for ; Thu, 2 Feb 2023 18:58:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233092AbjBBS6S (ORCPT ); Thu, 2 Feb 2023 13:58:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232776AbjBBS6L (ORCPT ); Thu, 2 Feb 2023 13:58:11 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 248A62A989 for ; Thu, 2 Feb 2023 10:58:10 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-4ff7c6679f2so29087837b3.12 for ; Thu, 02 Feb 2023 10:58:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rkepuJ2zoHdPzygKI1n27HnD325lkrHrRjgMMEPWapE=; b=q/W+3bFKPT3g+bygipMtMesVcowRidKkCMo5PWZzy1KTK4SBsocVdsYasyrnjJX8CE mPsjUzd9mJ26OjvCuAgFZoM9gzCYxEjfR/wWK83W/iKnk3jwFE2aWY2yCPrjF+0e8CM0 DapjsXiSQxbMRzQ0OnHUEIBlIrEVVFNhHtcJrIDofdcJ8Tv6oBjlLJv5s26G1wJnjCnA 8XA2ZypG8H7LdZgKqMpwEtL8RjeW1JKVgR4uEx2uaywkciF59eBFw28msJen6FZ2JuVV UIOS/xXHoW/4HhT5J6flJy4S0CkDjmigdEFQZWeRyiJTHBJF/a04/fwbieFEWwM3nF8x x0AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rkepuJ2zoHdPzygKI1n27HnD325lkrHrRjgMMEPWapE=; b=kpyWICn2HTtBcu3qU7aeUSqoyT5nJOoJQDq89kXHdlfPPY/BgICvUfyz3q7e8Msp04 ptvfiKAQE26W3Hk0i4f1w6nzBdgjeWPmfF4bzdvf/Ec4CKnewzOcgDMPtwCjAnPbjdpu 2EsN1uZ1UuQgOgIjoGTcoauAxfC6n0SJ7VIz+EHwuvzQSFgHtFxjy8nET7QhNs1cDedA aScT++zOjRGhw8ZIob9iEEnfNZQnzC6I9nWYxlz6/oxU/GZpd9/oOy9Yms9IbemUcqcF RO/8fHS1CFyqVDPhH7o9Tbw5wcI69e7YCYirGZRme4G8+9IZSMK+6IJYK0JE7Is655H5 skhw== X-Gm-Message-State: AO0yUKUh9oQPXoCt+lkct4toF/4gluqFykCBEqSfRkQBdEQaqAMOxm/E op5DlafNUsoC0gJM5puoIMia/jXyLlJz6Q== X-Google-Smtp-Source: AK7set/X7P6HQRE4XpVbshE9a9X3A6wWuTvOL1gVxnlutuE5os7JSE4BPap+IMWP/Eklq2w9HzfT7Hwa/M+jcQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a25:9191:0:b0:855:fafc:7419 with SMTP id w17-20020a259191000000b00855fafc7419mr2ybl.0.1675364286187; Thu, 02 Feb 2023 10:58:06 -0800 (PST) Date: Thu, 2 Feb 2023 18:57:59 +0000 In-Reply-To: <20230202185801.4179599-1-edumazet@google.com> Mime-Version: 1.0 References: <20230202185801.4179599-1-edumazet@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230202185801.4179599-3-edumazet@google.com> Subject: [PATCH net-next 2/4] net: remove osize variable in __alloc_skb() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Alexander Duyck , Soheil Hassas Yeganeh , Eric Dumazet Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org This is a cleanup patch, to prepare following change. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- net/core/skbuff.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index b73de8fb0756c02cf9ba4b7e90854c9c17728463..a82df5289208d69716e60c5c1f201ec3ca50a258 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -533,7 +533,6 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, { struct kmem_cache *cache; struct sk_buff *skb; - unsigned int osize; bool pfmemalloc; u8 *data; @@ -559,16 +558,15 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, * Both skb->head and skb_shared_info are cache line aligned. */ size = SKB_HEAD_ALIGN(size); - osize = kmalloc_size_roundup(size); - data = kmalloc_reserve(osize, gfp_mask, node, &pfmemalloc); + size = kmalloc_size_roundup(size); + data = kmalloc_reserve(size, gfp_mask, node, &pfmemalloc); if (unlikely(!data)) goto nodata; /* kmalloc_size_roundup() might give us more room than requested. * Put skb_shared_info exactly at the end of allocated zone, * to allow max possible filling before reallocation. */ - size = SKB_WITH_OVERHEAD(osize); - prefetchw(data + size); + prefetchw(data + SKB_WITH_OVERHEAD(size)); /* * Only clear those fields we need to clear, not those that we will @@ -576,7 +574,7 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, * the tail pointer in struct sk_buff! */ memset(skb, 0, offsetof(struct sk_buff, tail)); - __build_skb_around(skb, data, osize); + __build_skb_around(skb, data, size); skb->pfmemalloc = pfmemalloc; if (flags & SKB_ALLOC_FCLONE) { From patchwork Thu Feb 2 18:58:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13126678 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FC12C636D3 for ; Thu, 2 Feb 2023 18:58:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233097AbjBBS6T (ORCPT ); Thu, 2 Feb 2023 13:58:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233094AbjBBS6P (ORCPT ); Thu, 2 Feb 2023 13:58:15 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF7303B3DB for ; Thu, 2 Feb 2023 10:58:11 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id y125-20020a25c883000000b0086349255277so890489ybf.8 for ; Thu, 02 Feb 2023 10:58:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8VHO+GQDTTptIyIkKxxaBfs2xV8urt7m1x/p/R1+WTA=; b=RonGC3uTj31BJmEL67HUYezC0Szr6RjQJeUnd/RXlc80yxMLhfIjCVg/PHm39v6xQl woaemFmMqRG9S50Fwdw2Mw4jwK/yy6OS13ChlAa0R8XhaPw0/L2OJ2TPRljPFL/FyG7K il+50cLywUl40ZoHcQrV27uf0wiSmSnuQhL75uuYRkqrWfuOIA5yKf394ycK8mu69xUI eO4JHbgyYtgBfDxzNeA++07abNK2Nd3SHagPM4cpiwNPvkjSucTgP0YW092uya3k0dsD P8BjBvchdoUBBPY9DKkHKZselObaxl3EUTG1oX407ghuy2je19In7FnS5sd0Des71Z/d 3xWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8VHO+GQDTTptIyIkKxxaBfs2xV8urt7m1x/p/R1+WTA=; b=Tda6vtAJ8s3oTaSdL2EyBR2RUCcmrqxjf5WDafvmOHSB+XzICckeMn6y5kpSiGg1ga YayL1+5f1kTyRfhSPGDpq3yiWsWlmMbUtD9YTVxFbQZHj1VxOzQcCpzS6aj3GCp7KYL1 0OoXAQlDsWXXkQhQxM8ZeUCtthC36YEJsnc6lkf04sWuLbxvxKi2iXuaZxSAkuB/X6Oq sqY4/Fxltk4c/r9fHuH3ozz+PZGZEIN/SQfT9IAg1wZWkhhQeAJc/SSVKVCubtupebjc JETv+cn5tZkOsBD+Mlcz4mmHqwpgLSgXmv1E9I2ZaSGk3e/1c5TCmniIlIzSYPHGIJ5P sSOQ== X-Gm-Message-State: AO0yUKV+raP9OlUJ8QxJojJVQr8MXlWzvlKbQ45PH2hS8xDxo4wX7t7u tEh0C3lvKoDad3W+oUCUjBkAiFH9tV17AA== X-Google-Smtp-Source: AK7set9Oe6D6H9UCmqllq96MoMt2cq/s2i13RuqBA9bplSI3vAt1St4+KJTtqZImpe6D8IweRATB6jqGQ2OBZg== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a5b:889:0:b0:763:8e38:3fb3 with SMTP id e9-20020a5b0889000000b007638e383fb3mr471273ybq.547.1675364291125; Thu, 02 Feb 2023 10:58:11 -0800 (PST) Date: Thu, 2 Feb 2023 18:58:00 +0000 In-Reply-To: <20230202185801.4179599-1-edumazet@google.com> Mime-Version: 1.0 References: <20230202185801.4179599-1-edumazet@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230202185801.4179599-4-edumazet@google.com> Subject: [PATCH net-next 3/4] net: factorize code in kmalloc_reserve() From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Alexander Duyck , Soheil Hassas Yeganeh , Eric Dumazet Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org All kmalloc_reserve() callers have to make the same computation, we can factorize them, to prepare following patch in the series. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- net/core/skbuff.c | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a82df5289208d69716e60c5c1f201ec3ca50a258..ae0b2aa1f01e8060cc4fe69137e9bd98e44280cc 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -478,17 +478,20 @@ EXPORT_SYMBOL(napi_build_skb); * may be used. Otherwise, the packet data may be discarded until enough * memory is free */ -static void *kmalloc_reserve(size_t size, gfp_t flags, int node, +static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node, bool *pfmemalloc) { - void *obj; bool ret_pfmemalloc = false; + unsigned int obj_size; + void *obj; + obj_size = SKB_HEAD_ALIGN(*size); + *size = obj_size = kmalloc_size_roundup(obj_size); /* * Try a regular allocation, when that fails and we're not entitled * to the reserves, fail. */ - obj = kmalloc_node_track_caller(size, + obj = kmalloc_node_track_caller(obj_size, flags | __GFP_NOMEMALLOC | __GFP_NOWARN, node); if (obj || !(gfp_pfmemalloc_allowed(flags))) @@ -496,7 +499,7 @@ static void *kmalloc_reserve(size_t size, gfp_t flags, int node, /* Try again but now we are using pfmemalloc reserves */ ret_pfmemalloc = true; - obj = kmalloc_node_track_caller(size, flags, node); + obj = kmalloc_node_track_caller(obj_size, flags, node); out: if (pfmemalloc) @@ -557,9 +560,7 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, * aligned memory blocks, unless SLUB/SLAB debug is enabled. * Both skb->head and skb_shared_info are cache line aligned. */ - size = SKB_HEAD_ALIGN(size); - size = kmalloc_size_roundup(size); - data = kmalloc_reserve(size, gfp_mask, node, &pfmemalloc); + data = kmalloc_reserve(&size, gfp_mask, node, &pfmemalloc); if (unlikely(!data)) goto nodata; /* kmalloc_size_roundup() might give us more room than requested. @@ -1931,9 +1932,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail, if (skb_pfmemalloc(skb)) gfp_mask |= __GFP_MEMALLOC; - size = SKB_HEAD_ALIGN(size); - size = kmalloc_size_roundup(size); - data = kmalloc_reserve(size, gfp_mask, NUMA_NO_NODE, NULL); + data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL); if (!data) goto nodata; size = SKB_WITH_OVERHEAD(size); @@ -6282,9 +6281,7 @@ static int pskb_carve_inside_header(struct sk_buff *skb, const u32 off, if (skb_pfmemalloc(skb)) gfp_mask |= __GFP_MEMALLOC; - size = SKB_HEAD_ALIGN(size); - size = kmalloc_size_roundup(size); - data = kmalloc_reserve(size, gfp_mask, NUMA_NO_NODE, NULL); + data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL); if (!data) return -ENOMEM; size = SKB_WITH_OVERHEAD(size); @@ -6400,9 +6397,7 @@ static int pskb_carve_inside_nonlinear(struct sk_buff *skb, const u32 off, if (skb_pfmemalloc(skb)) gfp_mask |= __GFP_MEMALLOC; - size = SKB_HEAD_ALIGN(size); - size = kmalloc_size_roundup(size); - data = kmalloc_reserve(size, gfp_mask, NUMA_NO_NODE, NULL); + data = kmalloc_reserve(&size, gfp_mask, NUMA_NO_NODE, NULL); if (!data) return -ENOMEM; size = SKB_WITH_OVERHEAD(size); From patchwork Thu Feb 2 18:58:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13126679 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B2E8C05027 for ; Thu, 2 Feb 2023 18:58:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233102AbjBBS6U (ORCPT ); Thu, 2 Feb 2023 13:58:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233103AbjBBS6Q (ORCPT ); Thu, 2 Feb 2023 13:58:16 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45FBD367C2 for ; Thu, 2 Feb 2023 10:58:13 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id y14-20020a2586ce000000b0086167203873so1217604ybm.13 for ; Thu, 02 Feb 2023 10:58:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YN98cNrpXDkdIzhcVIFV9D+C1oGPSiwZEV1Jck03X/k=; b=LHtAj0g2cRL2d50o/VRlQ5X5Vsew/dV9XGyFAGjvZO2n3BY8ehVB9ep3eDyIXS8vHR c+yqhOXg2APkL9cgvGTwvxrwTXqH6yxw6qOOcISw3QBTKCNHURc8XNeGxwQG0wiIKmyC saZje2uIyZ60dxCfP2TRmIYYWjTNeb8ggkCThsR8DZ/Y9yp6a3T6VJbt3ESp3zvaapN/ Rb6ZVgdA7ejtvfDOS6NILGONmSTCidxv8cqflqn5yXLzj4Ez6WqC3xjMLpn7s/sbiwlg w5LBLK+Sy2L4/tlbhJCye0o34B/AF1PLhCU3xGcZvjz6HQP9u/5uJFpvbmkmzntSYRUK nXrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YN98cNrpXDkdIzhcVIFV9D+C1oGPSiwZEV1Jck03X/k=; b=SwrFuzF0Vm/ncWSN2T/3LNSY6qZZppWxVYpBUKnVfdrWfQ4o8vHeMZXD7RG/lZsn5m 9jFYqWd3MvwtKn6e5gaZkh04ub6iNdRtqv7L+fdnoQxTlMTg42cXpYcj8aju+PEXAHvU aTrO1xxMCbgYAbI3M78ZSgWpMaMxE24f57Lf3Vp/HDZ1QUMHsiF2UdEpcbH47wWGhe+O PZrysN55mwiQPgaclw9eKR/JAGuvt2HGsfwGW792usBbklKCx8LAgp2CgIad7TxFX74x KyruvtplxMx+m8fHAmzqurCaj6DehIfTF8C8TDmma1BbDEJHHu+uYaVZWKiCmyA76Kfk qCpA== X-Gm-Message-State: AO0yUKW8ChKfgC5of3JWkAWUwEofWqcCBwQFUWnr41WuczvNhTDUYSDe mrg3Y8yUuZdMhSkZgh22+GzrEcsvEwQlzA== X-Google-Smtp-Source: AK7set9kLcju52ic8X2PGOCqTOvyETaXxHwKQkg72i7xemfCNN31rlNpBX6+k8jAe+F3mTyxNXelmWVhB1mTCw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a81:1d83:0:b0:524:896a:5a2a with SMTP id d125-20020a811d83000000b00524896a5a2amr1ywd.9.1675364292746; Thu, 02 Feb 2023 10:58:12 -0800 (PST) Date: Thu, 2 Feb 2023 18:58:01 +0000 In-Reply-To: <20230202185801.4179599-1-edumazet@google.com> Mime-Version: 1.0 References: <20230202185801.4179599-1-edumazet@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230202185801.4179599-5-edumazet@google.com> Subject: [PATCH net-next 4/4] net: add dedicated kmem_cache for typical/small skb->head From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, eric.dumazet@gmail.com, Alexander Duyck , Soheil Hassas Yeganeh , Eric Dumazet Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Recent removal of ksize() in alloc_skb() increased performance because we no longer read the associated struct page. We have an equivalent cost at kfree_skb() time. kfree(skb->head) has to access a struct page, often cold in cpu caches to get the owning struct kmem_cache. Considering that many allocations are small, we can have our own kmem_cache to avoid the cache line miss. This also saves memory because these small heads are no longer padded to 1024 bytes. CONFIG_SLUB=y $ grep skbuff_small_head /proc/slabinfo skbuff_small_head 2907 2907 640 51 8 : tunables 0 0 0 : slabdata 57 57 0 CONFIG_SLAB=y $ grep skbuff_small_head /proc/slabinfo skbuff_small_head 607 624 640 6 1 : tunables 54 27 8 : slabdata 104 104 5 Note: after Kees Cook patches and this one, we might be able to revert commit dbae2b062824 ("net: skb: introduce and use a single page frag cache") because GRO_MAX_HEAD is also small. Signed-off-by: Eric Dumazet Cc: Paolo Abeni Acked-by: Soheil Hassas Yeganeh --- net/core/skbuff.c | 52 ++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 47 insertions(+), 5 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index ae0b2aa1f01e8060cc4fe69137e9bd98e44280cc..3e540b4924701cc57b6fbd1b668bab3b652ee94c 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -89,6 +89,19 @@ static struct kmem_cache *skbuff_fclone_cache __ro_after_init; #ifdef CONFIG_SKB_EXTENSIONS static struct kmem_cache *skbuff_ext_cache __ro_after_init; #endif +static struct kmem_cache *skb_small_head_cache __ro_after_init; + +#define SKB_SMALL_HEAD_SIZE SKB_HEAD_ALIGN(MAX_TCP_HEADER) + +/* We want SKB_SMALL_HEAD_CACHE_SIZE to not be a power of two. */ +#define SKB_SMALL_HEAD_CACHE_SIZE \ + (is_power_of_2(SKB_SMALL_HEAD_SIZE) ? \ + (SKB_SMALL_HEAD_SIZE + L1_CACHE_BYTES) : \ + SKB_SMALL_HEAD_SIZE) + +#define SKB_SMALL_HEAD_HEADROOM \ + SKB_WITH_OVERHEAD(SKB_SMALL_HEAD_CACHE_SIZE) + int sysctl_max_skb_frags __read_mostly = MAX_SKB_FRAGS; EXPORT_SYMBOL(sysctl_max_skb_frags); @@ -486,6 +499,21 @@ static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node, void *obj; obj_size = SKB_HEAD_ALIGN(*size); + if (obj_size <= SKB_SMALL_HEAD_CACHE_SIZE && + !(flags & KMALLOC_NOT_NORMAL_BITS)) { + + /* skb_small_head_cache has non power of two size, + * likely forcing SLUB to use order-3 pages. + * We deliberately attempt a NOMEMALLOC allocation only. + */ + obj = kmem_cache_alloc_node(skb_small_head_cache, + flags | __GFP_NOMEMALLOC | __GFP_NOWARN, + node); + if (obj) { + *size = SKB_SMALL_HEAD_CACHE_SIZE; + goto out; + } + } *size = obj_size = kmalloc_size_roundup(obj_size); /* * Try a regular allocation, when that fails and we're not entitled @@ -805,6 +833,14 @@ static bool skb_pp_recycle(struct sk_buff *skb, void *data) return page_pool_return_skb_page(virt_to_page(data)); } +static void skb_kfree_head(void *head, unsigned int end_offset) +{ + if (end_offset == SKB_SMALL_HEAD_HEADROOM) + kmem_cache_free(skb_small_head_cache, head); + else + kfree(head); +} + static void skb_free_head(struct sk_buff *skb) { unsigned char *head = skb->head; @@ -814,7 +850,7 @@ static void skb_free_head(struct sk_buff *skb) return; skb_free_frag(head); } else { - kfree(head); + skb_kfree_head(head, skb_end_offset(skb)); } } @@ -1995,7 +2031,7 @@ int pskb_expand_head(struct sk_buff *skb, int nhead, int ntail, return 0; nofrags: - kfree(data); + skb_kfree_head(data, size); nodata: return -ENOMEM; } @@ -4633,6 +4669,12 @@ void __init skb_init(void) 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC, NULL); + skb_small_head_cache = kmem_cache_create("skbuff_small_head", + SKB_SMALL_HEAD_CACHE_SIZE, + 0, + SLAB_HWCACHE_ALIGN | SLAB_PANIC, + NULL); + skb_extensions_init(); } @@ -6297,7 +6339,7 @@ static int pskb_carve_inside_header(struct sk_buff *skb, const u32 off, if (skb_cloned(skb)) { /* drop the old head gracefully */ if (skb_orphan_frags(skb, gfp_mask)) { - kfree(data); + skb_kfree_head(data, size); return -ENOMEM; } for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) @@ -6405,7 +6447,7 @@ static int pskb_carve_inside_nonlinear(struct sk_buff *skb, const u32 off, memcpy((struct skb_shared_info *)(data + size), skb_shinfo(skb), offsetof(struct skb_shared_info, frags[0])); if (skb_orphan_frags(skb, gfp_mask)) { - kfree(data); + skb_kfree_head(data, size); return -ENOMEM; } shinfo = (struct skb_shared_info *)(data + size); @@ -6441,7 +6483,7 @@ static int pskb_carve_inside_nonlinear(struct sk_buff *skb, const u32 off, /* skb_frag_unref() is not needed here as shinfo->nr_frags = 0. */ if (skb_has_frag_list(skb)) kfree_skb_list(skb_shinfo(skb)->frag_list); - kfree(data); + skb_kfree_head(data, size); return -ENOMEM; } skb_release_data(skb, SKB_CONSUMED);