From patchwork Tue Mar 5 16:03:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582614 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9DA05128372 for ; Tue, 5 Mar 2024 16:04:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654660; cv=none; b=GxUJbgrq+xQy+hf+JnzU0avbHQ3P1/rrHmk9zT042A50h0AOgQJaKpxtVcTwjn++cnczl0f4Bc+GwklBhTVLUVn5jpTYrrvg8Efbpo5g/FQiCJiE5sLzpTPC83Wr519thQwdG2co+m6w8DX5zNPtOdzdSxwO3REpDEIgeuLulF4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654660; c=relaxed/simple; bh=lkRFVLV27FDOA478lzcOz2mJAgCuUYQ+GWyEJJp1w5Q=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=C4CrEG5b6wVRYqfUJxTUzOSicfPSyuy4I/4QRlTQUmW6yQUHFSlLw1Osj04fG890Pgi935kbg2G67QKf/lPwdkhEtIKq8s/FWnbiHvgMKVkS10hA6xNMS9Q8+njZPnVgf8rNe1uqhfr1P+FfZ/J8HyB61ru//97lOaTiFrJel20= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=F5LnMqcp; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="F5LnMqcp" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc743cc50a6so7590445276.2 for ; Tue, 05 Mar 2024 08:04:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654657; x=1710259457; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UbLX2AYPg1cLWgj5PFHrnbQLL+wLioeOEdysz4St+Ks=; b=F5LnMqcpktpfzY+BIYJNstApYineFGZm0DontP2AIGC4QHDDOWnJhh0bvJGdEc02RN XhzNfZlGSf+WQXawB4rjb0v6JsRMynMzP7gdEK4ogQBP/Kkwdmf2Vv6NXB4/RktHK9z7 tWS8z9VWsrkSL7IV7yYTGbxmBkfI04t+5PObIRqdI+G51H1AP0nAzG8dDo42IlxCQzNr OskHns8r9UVdwYLKN6B0dpwLmuOjG2YQtGDUQfvsyA8xz++v7Tl3K/K1NbAGviQvWkF3 cvKyeK7rNg3EOs3bNAS1m2/HEEo7Pr9UHD1WTkJuQrzROBd+/tcX88JZLMATl8BMNLNW UY9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654657; x=1710259457; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UbLX2AYPg1cLWgj5PFHrnbQLL+wLioeOEdysz4St+Ks=; b=cqNcpTAiRYwOtEX/1VBhOrL4HxQ8nw2+JcDpFxdDKiodYUDZdkvuSwJbwgwewbTOXC 8iPstVj+oiiVgwV2sw2jxybW+tzO+JGRjsxgZTHHyJI8NBApx/ByZTgPRni0kACuVNVa 8zg5qwv/02EUYoEVvwdQr+/J9kUzFQ0mGusgtRl2BeQNDDr3YFb7BRK/MpIrt8LpPM55 k2t0oWi4uhVgo8xo+zS7mUqG1xPfB+93u3tyYEz4LKR/pZ4LRzbh4AM6kJJ/N4RExfMc dfxxHBQ2ckxXX5y++RlbLcphRRlqRDp0iyyydqyPnUzoQgtG1MbawqXx5TKC9HHWkQa1 DdEQ== X-Gm-Message-State: AOJu0Yw4CGmoDuNBLGFDQ8ZkzWmGBCWy0sYWsL8LmY6agOxeNf8FZr6d th8w2nsf5KAymq2iYUWvEN6zKF50HhL88ZOwTLzBfwkUrPBy+Ho0QYmsOTVUDdUAelzY/+IrI2w FHGs7DYYKyw== X-Google-Smtp-Source: AGHT+IHjczfvIsRGPOszux8vj7Q9MLZaHiDt0o9/P9+pxVhYhKf0WlSd+S+AA3ZF2HnSYVwOpfi4hQlcVABSbQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1143:b0:dc7:6cfa:dc59 with SMTP id p3-20020a056902114300b00dc76cfadc59mr475366ybu.4.1709654657706; Tue, 05 Mar 2024 08:04:17 -0800 (PST) Date: Tue, 5 Mar 2024 16:03:56 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-2-edumazet@google.com> Subject: [PATCH net-next 01/18] net: introduce struct net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Instead of spreading networking critical fields all over the places, add a custom net_hotdata structure so that we can precisely control its layout. In this first patch, move : - gro_normal_batch used in rx (GRO stack) - offload_base used in rx and tx (GRO and TSO stacks) Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 1 - include/net/gro.h | 5 ++--- include/net/hotdata.h | 15 +++++++++++++++ net/core/Makefile | 1 + net/core/gro.c | 15 ++++++--------- net/core/gso.c | 4 ++-- net/core/hotdata.c | 9 +++++++++ net/core/sysctl_net_core.c | 3 ++- 8 files changed, 37 insertions(+), 16 deletions(-) create mode 100644 include/net/hotdata.h create mode 100644 net/core/hotdata.c diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index c41019f3417948d09ae9a50b57b856be1dc8ae42..15ce809e0541078bff7a48b8d7cb2cf2c1ac8a93 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -4790,7 +4790,6 @@ void dev_get_tstats64(struct net_device *dev, struct rtnl_link_stats64 *s); extern int netdev_max_backlog; extern int dev_rx_weight; extern int dev_tx_weight; -extern int gro_normal_batch; enum { NESTED_SYNC_IMM_BIT, diff --git a/include/net/gro.h b/include/net/gro.h index 2b58671a65492bf3f9dabf1e7a2d985cee007e11..d6fc8fbd37302338fc09ab01fead899002c5833f 100644 --- a/include/net/gro.h +++ b/include/net/gro.h @@ -9,6 +9,7 @@ #include #include #include +#include struct napi_gro_cb { union { @@ -446,7 +447,7 @@ static inline void gro_normal_one(struct napi_struct *napi, struct sk_buff *skb, { list_add_tail(&skb->list, &napi->rx_list); napi->rx_count += segs; - if (napi->rx_count >= READ_ONCE(gro_normal_batch)) + if (napi->rx_count >= READ_ONCE(net_hotdata.gro_normal_batch)) gro_normal_list(napi); } @@ -493,6 +494,4 @@ static inline void inet6_get_iif_sdif(const struct sk_buff *skb, int *iif, int * #endif } -extern struct list_head offload_base; - #endif /* _NET_IPV6_GRO_H */ diff --git a/include/net/hotdata.h b/include/net/hotdata.h new file mode 100644 index 0000000000000000000000000000000000000000..6ed32e4e34aa3bdc6e860f5a8a6cab69c36c7fad --- /dev/null +++ b/include/net/hotdata.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _NET_HOTDATA_H +#define _NET_HOTDATA_H + +#include + +/* Read mostly data used in network fast paths. */ +struct net_hotdata { + struct list_head offload_base; + int gro_normal_batch; +}; + +extern struct net_hotdata net_hotdata; + +#endif /* _NET_HOTDATA_H */ diff --git a/net/core/Makefile b/net/core/Makefile index 821aec06abf1460d3504de4b6b66a328bba748d8..6e6548011fae570e345717e43eb3c1a6133571c7 100644 --- a/net/core/Makefile +++ b/net/core/Makefile @@ -18,6 +18,7 @@ obj-y += dev.o dev_addr_lists.o dst.o netevent.o \ obj-$(CONFIG_NETDEV_ADDR_LIST_TEST) += dev_addr_lists_test.o obj-y += net-sysfs.o +obj-y += hotdata.o obj-$(CONFIG_PAGE_POOL) += page_pool.o page_pool_user.o obj-$(CONFIG_PROC_FS) += net-procfs.o obj-$(CONFIG_NET_PKTGEN) += pktgen.o diff --git a/net/core/gro.c b/net/core/gro.c index 6a0edbd826a17573b51c5f71e20ff0c09364fc21..ee30d4f0c03876e78795397d1c495881a2c9e80f 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -10,9 +10,6 @@ #define GRO_MAX_HEAD (MAX_HEADER + 128) static DEFINE_SPINLOCK(offload_lock); -struct list_head offload_base __read_mostly = LIST_HEAD_INIT(offload_base); -/* Maximum number of GRO_NORMAL skbs to batch up for list-RX */ -int gro_normal_batch __read_mostly = 8; /** * dev_add_offload - register offload handlers @@ -31,7 +28,7 @@ void dev_add_offload(struct packet_offload *po) struct packet_offload *elem; spin_lock(&offload_lock); - list_for_each_entry(elem, &offload_base, list) { + list_for_each_entry(elem, &net_hotdata.offload_base, list) { if (po->priority < elem->priority) break; } @@ -55,7 +52,7 @@ EXPORT_SYMBOL(dev_add_offload); */ static void __dev_remove_offload(struct packet_offload *po) { - struct list_head *head = &offload_base; + struct list_head *head = &net_hotdata.offload_base; struct packet_offload *po1; spin_lock(&offload_lock); @@ -235,9 +232,9 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb) static void napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb) { + struct list_head *head = &net_hotdata.offload_base; struct packet_offload *ptype; __be16 type = skb->protocol; - struct list_head *head = &offload_base; int err = -ENOENT; BUILD_BUG_ON(sizeof(struct napi_gro_cb) > sizeof(skb->cb)); @@ -444,7 +441,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff { u32 bucket = skb_get_hash_raw(skb) & (GRO_HASH_BUCKETS - 1); struct gro_list *gro_list = &napi->gro_hash[bucket]; - struct list_head *head = &offload_base; + struct list_head *head = &net_hotdata.offload_base; struct packet_offload *ptype; __be16 type = skb->protocol; struct sk_buff *pp = NULL; @@ -550,7 +547,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff struct packet_offload *gro_find_receive_by_type(__be16 type) { - struct list_head *offload_head = &offload_base; + struct list_head *offload_head = &net_hotdata.offload_base; struct packet_offload *ptype; list_for_each_entry_rcu(ptype, offload_head, list) { @@ -564,7 +561,7 @@ EXPORT_SYMBOL(gro_find_receive_by_type); struct packet_offload *gro_find_complete_by_type(__be16 type) { - struct list_head *offload_head = &offload_base; + struct list_head *offload_head = &net_hotdata.offload_base; struct packet_offload *ptype; list_for_each_entry_rcu(ptype, offload_head, list) { diff --git a/net/core/gso.c b/net/core/gso.c index 9e1803bfc9c6cac2fe7054661f8995909a6c28d9..bcd156372f4df080f83cc45fc96df1789125a8ae 100644 --- a/net/core/gso.c +++ b/net/core/gso.c @@ -17,7 +17,7 @@ struct sk_buff *skb_eth_gso_segment(struct sk_buff *skb, struct packet_offload *ptype; rcu_read_lock(); - list_for_each_entry_rcu(ptype, &offload_base, list) { + list_for_each_entry_rcu(ptype, &net_hotdata.offload_base, list) { if (ptype->type == type && ptype->callbacks.gso_segment) { segs = ptype->callbacks.gso_segment(skb, features); break; @@ -48,7 +48,7 @@ struct sk_buff *skb_mac_gso_segment(struct sk_buff *skb, __skb_pull(skb, vlan_depth); rcu_read_lock(); - list_for_each_entry_rcu(ptype, &offload_base, list) { + list_for_each_entry_rcu(ptype, &net_hotdata.offload_base, list) { if (ptype->type == type && ptype->callbacks.gso_segment) { segs = ptype->callbacks.gso_segment(skb, features); break; diff --git a/net/core/hotdata.c b/net/core/hotdata.c new file mode 100644 index 0000000000000000000000000000000000000000..abb8ad19d59acc0d7d6e1b06f4506afa42bde44b --- /dev/null +++ b/net/core/hotdata.c @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include +#include +#include + +struct net_hotdata net_hotdata __cacheline_aligned = { + .offload_base = LIST_HEAD_INIT(net_hotdata.offload_base), + .gro_normal_batch = 8, +}; diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 986f15e5d6c41250c8b9099fc1d2883112e77ffb..0eb1242eabbe0d3ea58886b1db409c9d991ac672 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -23,6 +23,7 @@ #include #include #include +#include #include "dev.h" @@ -632,7 +633,7 @@ static struct ctl_table net_core_table[] = { }, { .procname = "gro_normal_batch", - .data = &gro_normal_batch, + .data = &net_hotdata.gro_normal_batch, .maxlen = sizeof(unsigned int), .mode = 0644, .proc_handler = proc_dointvec_minmax, From patchwork Tue Mar 5 16:03:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582615 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 353A48613F for ; Tue, 5 Mar 2024 16:04:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654661; cv=none; b=dIJ6NzW7rl+TM80CNiu7hshn1gRWZ13OQ+yK2lhj3BeuWer/KGJNN9Jb4fqL0RWya1lrrE934/lG0lHB7BrQ1g09derJE2+0drKviJqJSCWFXy4ScBzj2BmlgiExzoj4KS2iA1AFzXB1OXbtk37DpwGK5CXsshcR0080R84kmKU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654661; c=relaxed/simple; bh=qp1qgt2zDEDBM+0n/pxRpAEVbnIJipRGuXxeSRTPy/g=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UxDmtgKgmGTPj6Ln7U+4QmpE0h1SrtSNwTrg0KrPSyL6Jncqljh+Kvk01WFQc+7NLKIaezArBx8RZ9Up+n992VFHjTfvaX2RDw6ZiEUf+gNjEUJ8XrXQZ96wJEbz4zuXUKWCs9N459Igk+M8G7fZXsbAtPYomU0JahAuTdIXtt4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=dbXyHV46; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="dbXyHV46" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dced704f17cso9808761276.1 for ; Tue, 05 Mar 2024 08:04:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654659; x=1710259459; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BlPfYhfDEF30sgDH3HFcx4bwCeATqFumLekjJ9n7FH0=; b=dbXyHV46uzA61i6eEC120OHEh/Y0n7QXb0JBdcU3brwzKw4JMpm9wiUoHB2rFGZVHm 2eZzEVpKgQ8CJLGGoUBiVtqoF46CGSC3kwJQRps3bNFjgARlyVn40OpRQvYoJneAxjko Biqba0fFyuHWOy21LAFiXIuvaQbVTs1Hr3z+ZBsAcmo1vETYPZp39yFmilrK/nD6j5e0 z8dnFAQn3hQrPMPpcDgVmf6c359h8fDRg23NTjnKgmFe5OKe1ICF75gwP/P8oyKCmOAG MZ8S0mQErJcCwgYmU7/z0og4GAXMuoyL0+ZgZwameWh+LZVmBQEp3tNlGSfTQBcBsBKs vJRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654659; x=1710259459; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BlPfYhfDEF30sgDH3HFcx4bwCeATqFumLekjJ9n7FH0=; b=iud7WmI8QtFVxg9TT9bazoWAdjzCxo6mo4bspuBurUvSoJgenaf/aYH2mZVVs+Oc54 bmKjYhXjLIzmgA2P+dGiaBFVgepqICzm5PGj9kwRahWkGdUrpxhGW48Hfilfy+2PjaMR oxXHc4HC1PeFlzD83eM15sMg0V7msj0cV5t+MmPLrmJAgs4jCo+GtEeAk5EmJk5N8hAC yv4rJZyuc4URw1CJ6GXTbPzqMlZhxgMNAHo6IyrtMoporDN8/RrGWOaqhQrUaSewYPdI yFtZPAwjh2Z04uyZvuidevDrv5/hyuj4nERTWTxEgBDqldWK40AUOSY0fvz90wWCO+X9 AtvQ== X-Gm-Message-State: AOJu0Yye2aR7G5eNqoH5FKxTxPMyLNOUWcIPvyQs8VTr5NaGEbuVEqYW 2sW/W01piAKVkYjrknWcYBLLoVUyuPvKm8/BVcBhmafSxx3zvowzecIrHk2ijVcrJMWSH0dfgln TYYthgk8btA== X-Google-Smtp-Source: AGHT+IGKDieeyo3h8ch4c0hBIwR9S8G86/YrRoZF1TPDL4z3Joh4864uVlawC3FeDD7mzsIWFfg1AyQm8RNQmw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1081:b0:dcc:41ad:fb3b with SMTP id v1-20020a056902108100b00dcc41adfb3bmr475061ybu.10.1709654659198; Tue, 05 Mar 2024 08:04:19 -0800 (PST) Date: Tue, 5 Mar 2024 16:03:57 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-3-edumazet@google.com> Subject: [PATCH net-next 02/18] net: move netdev_budget and netdev_budget to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org netdev_budget and netdev_budget are used in rx path (net_rx_action()) Move them into net_hotdata for better cache locality. Signed-off-by: Eric Dumazet --- include/net/hotdata.h | 2 ++ net/core/dev.c | 7 ++----- net/core/dev.h | 2 -- net/core/hotdata.c | 6 ++++++ net/core/sysctl_net_core.c | 4 ++-- 5 files changed, 12 insertions(+), 9 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 6ed32e4e34aa3bdc6e860f5a8a6cab69c36c7fad..72170223385ebe65cce42f762b3686c072291d36 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -8,6 +8,8 @@ struct net_hotdata { struct list_head offload_base; int gro_normal_batch; + int netdev_budget; + int netdev_budget_usecs; }; extern struct net_hotdata net_hotdata; diff --git a/net/core/dev.c b/net/core/dev.c index fe054cbd41e92cbca87f1c0640c6ebe4fb6b2d86..0102a1810e7b148f465b87886b743e3d12c0e578 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4410,9 +4410,6 @@ EXPORT_SYMBOL(netdev_max_backlog); int netdev_tstamp_prequeue __read_mostly = 1; unsigned int sysctl_skb_defer_max __read_mostly = 64; -int netdev_budget __read_mostly = 300; -/* Must be at least 2 jiffes to guarantee 1 jiffy timeout */ -unsigned int __read_mostly netdev_budget_usecs = 2 * USEC_PER_SEC / HZ; int weight_p __read_mostly = 64; /* old backlog weight */ int dev_weight_rx_bias __read_mostly = 1; /* bias for backlog weight */ int dev_weight_tx_bias __read_mostly = 1; /* bias for output_queue quota */ @@ -6790,8 +6787,8 @@ static __latent_entropy void net_rx_action(struct softirq_action *h) { struct softnet_data *sd = this_cpu_ptr(&softnet_data); unsigned long time_limit = jiffies + - usecs_to_jiffies(READ_ONCE(netdev_budget_usecs)); - int budget = READ_ONCE(netdev_budget); + usecs_to_jiffies(READ_ONCE(net_hotdata.netdev_budget_usecs)); + int budget = READ_ONCE(net_hotdata.netdev_budget); LIST_HEAD(list); LIST_HEAD(repoll); diff --git a/net/core/dev.h b/net/core/dev.h index 45892267848d7a35a09aea95f04cfd9b72204d3c..9a6170530850c78508f9234ec82b174f4bf5a4a3 100644 --- a/net/core/dev.h +++ b/net/core/dev.h @@ -38,8 +38,6 @@ int dev_addr_init(struct net_device *dev); void dev_addr_check(struct net_device *dev); /* sysctls not referred to from outside net/core/ */ -extern int netdev_budget; -extern unsigned int netdev_budget_usecs; extern unsigned int sysctl_skb_defer_max; extern int netdev_tstamp_prequeue; extern int netdev_unregister_timeout_secs; diff --git a/net/core/hotdata.c b/net/core/hotdata.c index abb8ad19d59acc0d7d6e1b06f4506afa42bde44b..907d69120397dfb8d5a901912b72580fe256c762 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -1,9 +1,15 @@ // SPDX-License-Identifier: GPL-2.0-or-later #include #include +#include #include + struct net_hotdata net_hotdata __cacheline_aligned = { .offload_base = LIST_HEAD_INIT(net_hotdata.offload_base), .gro_normal_batch = 8, + + .netdev_budget = 300, + /* Must be at least 2 jiffes to guarantee 1 jiffy timeout */ + .netdev_budget_usecs = 2 * USEC_PER_SEC / HZ, }; diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 0eb1242eabbe0d3ea58886b1db409c9d991ac672..a9c2d798b219506da75a5d0a30d490ff4011d668 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -577,7 +577,7 @@ static struct ctl_table net_core_table[] = { #endif { .procname = "netdev_budget", - .data = &netdev_budget, + .data = &net_hotdata.netdev_budget, .maxlen = sizeof(int), .mode = 0644, .proc_handler = proc_dointvec @@ -600,7 +600,7 @@ static struct ctl_table net_core_table[] = { }, { .procname = "netdev_budget_usecs", - .data = &netdev_budget_usecs, + .data = &net_hotdata.netdev_budget_usecs, .maxlen = sizeof(unsigned int), .mode = 0644, .proc_handler = proc_dointvec_minmax, From patchwork Tue Mar 5 16:03:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582616 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EC08128804 for ; Tue, 5 Mar 2024 16:04:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654663; cv=none; b=Wy1lOKwOnqFm57QDNElzazf0rKNL67NmiUGNaddVEemJs1Cbsy88bIbQzbA+ojewELGELemqDuF28Z6t1TXNDYIV7E8NT64jv8oZ3u1ZKMVOy1JFQytiKDNb0qWe9AjJHg8OXbpb9dzoCenIqceSeF/HU0NwQ4tddv1hq6A4BDE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654663; c=relaxed/simple; bh=JzNyBUgM5kqm0qh/j6abNXmcE/WZtRpjVnkJZqx2Vvg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=H3ZrD7CpagVCTGB4giWZLR2XlH7CoxIp5eAzjUH8iuBj/gEhJs1W7OmjObanXBBBVjMA2NXS5ZoLkZcIzoBc+GCSTmTTwKGdNiBceIbKi3VdpXODvuQgFBdTr6JX0Sagdi6vxMunxz/5UxGrBsUC8/iXsN6sGSBoMLAV81M5Gr8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EhWhZbCa; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EhWhZbCa" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc64e0fc7c8so8781220276.2 for ; Tue, 05 Mar 2024 08:04:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654660; x=1710259460; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TfYSoeGWUA6oA9i9yl8PIu9Ha00NoLjunGBNWRckx+M=; b=EhWhZbCa7GlDJm/espnif/X1ImOGth4LpWUU8/ZE8lr8cGC8Dvi+0EpWminsHwxLVN gTS7+DHElJU5xHDGMPzl6RFRRhB5yj5LCs/EcrPy5YM5+Lxz52WSythO5uPFJVw3lFYt IVQQwWGBkFS5j2oBVY8sSZgYoSoGWZnwH3EvpkqIiqUzKCwBYPPDUgQXFjPi2rwtvLbG YKweF0cHYiMipSFvBTI0e7YUh5qf3vHHGaI3+3ElDHAsGUXsVjH354y0yv5wIMArbEvK 32oluEPlXg99X8s6g4i7HOkG9zNBodZLnq0EKGmVCRDsrgNR1g1nQqB13HBEjwUwvGrj Z2/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654660; x=1710259460; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TfYSoeGWUA6oA9i9yl8PIu9Ha00NoLjunGBNWRckx+M=; b=Jme70hKu68I7O6m3r0gfRsbo2YufTWU2FkWEX8YFsGgarMlvVkIcnRLtRF73qPGAiN HWXC0P7vj1g1OWmBpHSlBu1XLJCveRVTRKpgDUyEv2P1OSDjiwGj8YFEsdt6FE+0lrAf Hp07wJK8ez7YOpal9iNprwPSfMJF7LefXEnRrRqfCEWtggjFnobO4ykohyqEdo5X/Umo EO0qzBB65uEyIWtEkbhvMFBzKsx9A5FMxd/I5VE+BVWY/sweAWrAlzhqAdX+VTDOHcUK hR3lHdyi2tkb8zr21OMoh7sBAOozii6OvIutc7HZiNuOO7sh9OylGU7gJ0XyFBXpnxz0 zNSg== X-Gm-Message-State: AOJu0YwGAzGlU3mXseQhnVTCB5UddiegfYQTsI5+jYkKgK7Mz30e6zAs 04s33HqXqkJEpijsB/mL7J0033aLRXiWyRJRZoVlUux7CaStvRMW1O3yEfA9QoEcub/+9SAWgPN 2QcHaay4R1g== X-Google-Smtp-Source: AGHT+IHvyW25y+EGIWbprAvVgm5OeELPoXUHCIi3TtfNeSV+acnJfl4seredc4JFVV7R3RlNDR42ke9cx1QudQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:102e:b0:dcd:c091:e86 with SMTP id x14-20020a056902102e00b00dcdc0910e86mr523862ybt.13.1709654660663; Tue, 05 Mar 2024 08:04:20 -0800 (PST) Date: Tue, 5 Mar 2024 16:03:58 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-4-edumazet@google.com> Subject: [PATCH net-next 03/18] net: move netdev_tstamp_prequeue into net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org netdev_tstamp_prequeue is used in rx path. Move it to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet --- include/net/hotdata.h | 1 + net/core/dev.c | 10 +++++----- net/core/dev.h | 1 - net/core/hotdata.c | 2 ++ net/core/sysctl_net_core.c | 2 +- 5 files changed, 9 insertions(+), 7 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 72170223385ebe65cce42f762b3686c072291d36..149e56528537d8ed3365e46d6dc96e39c73a733a 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -10,6 +10,7 @@ struct net_hotdata { int gro_normal_batch; int netdev_budget; int netdev_budget_usecs; + int tstamp_prequeue; }; extern struct net_hotdata net_hotdata; diff --git a/net/core/dev.c b/net/core/dev.c index 0102a1810e7b148f465b87886b743e3d12c0e578..53ebdb55e8b7c0a6522eb3fdbb7bdd00948eb9a5 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4408,7 +4408,6 @@ EXPORT_SYMBOL(__dev_direct_xmit); int netdev_max_backlog __read_mostly = 1000; EXPORT_SYMBOL(netdev_max_backlog); -int netdev_tstamp_prequeue __read_mostly = 1; unsigned int sysctl_skb_defer_max __read_mostly = 64; int weight_p __read_mostly = 64; /* old backlog weight */ int dev_weight_rx_bias __read_mostly = 1; /* bias for backlog weight */ @@ -5052,7 +5051,7 @@ static int netif_rx_internal(struct sk_buff *skb) { int ret; - net_timestamp_check(READ_ONCE(netdev_tstamp_prequeue), skb); + net_timestamp_check(READ_ONCE(net_hotdata.tstamp_prequeue), skb); trace_netif_rx(skb); @@ -5344,7 +5343,7 @@ static int __netif_receive_skb_core(struct sk_buff **pskb, bool pfmemalloc, int ret = NET_RX_DROP; __be16 type; - net_timestamp_check(!READ_ONCE(netdev_tstamp_prequeue), skb); + net_timestamp_check(!READ_ONCE(net_hotdata.tstamp_prequeue), skb); trace_netif_receive_skb(skb); @@ -5728,7 +5727,7 @@ static int netif_receive_skb_internal(struct sk_buff *skb) { int ret; - net_timestamp_check(READ_ONCE(netdev_tstamp_prequeue), skb); + net_timestamp_check(READ_ONCE(net_hotdata.tstamp_prequeue), skb); if (skb_defer_rx_timestamp(skb)) return NET_RX_SUCCESS; @@ -5758,7 +5757,8 @@ void netif_receive_skb_list_internal(struct list_head *head) INIT_LIST_HEAD(&sublist); list_for_each_entry_safe(skb, next, head, list) { - net_timestamp_check(READ_ONCE(netdev_tstamp_prequeue), skb); + net_timestamp_check(READ_ONCE(net_hotdata.tstamp_prequeue), + skb); skb_list_del_init(skb); if (!skb_defer_rx_timestamp(skb)) list_add_tail(&skb->list, &sublist); diff --git a/net/core/dev.h b/net/core/dev.h index 9a6170530850c78508f9234ec82b174f4bf5a4a3..2bcaf8eee50c179db2ca59880521b9be6ecd45c8 100644 --- a/net/core/dev.h +++ b/net/core/dev.h @@ -39,7 +39,6 @@ void dev_addr_check(struct net_device *dev); /* sysctls not referred to from outside net/core/ */ extern unsigned int sysctl_skb_defer_max; -extern int netdev_tstamp_prequeue; extern int netdev_unregister_timeout_secs; extern int weight_p; extern int dev_weight_rx_bias; diff --git a/net/core/hotdata.c b/net/core/hotdata.c index 907d69120397dfb8d5a901912b72580fe256c762..087c4c84987df07f11a87112a778a5cac608a654 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -12,4 +12,6 @@ struct net_hotdata net_hotdata __cacheline_aligned = { .netdev_budget = 300, /* Must be at least 2 jiffes to guarantee 1 jiffy timeout */ .netdev_budget_usecs = 2 * USEC_PER_SEC / HZ, + + .tstamp_prequeue = 1, }; diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index a9c2d798b219506da75a5d0a30d490ff4011d668..bddd07da099886f2747f2ac4ba39a482b0f4231d 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -499,7 +499,7 @@ static struct ctl_table net_core_table[] = { #endif { .procname = "netdev_tstamp_prequeue", - .data = &netdev_tstamp_prequeue, + .data = &net_hotdata.tstamp_prequeue, .maxlen = sizeof(int), .mode = 0644, .proc_handler = proc_dointvec From patchwork Tue Mar 5 16:03:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582617 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA898128816 for ; Tue, 5 Mar 2024 16:04:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654664; cv=none; b=grjBzsSOcMJEDRDP70O5M1FbnWUId1ipU6v2HvrJy9ysxsCc4xJg6xEyo3KP3y2v47Sm6tlm2LhLdmzODT/mxzFVxtfX0AzEsuMQBSh2pqzZZuCrZ3NTPIDDUs8wGdaJ8czXNHc4b4/7J4OHE2DSXHyRj/eOVcDghVH7xBj/xlI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654664; c=relaxed/simple; bh=Q8ZqJk7HUmRJS8hfnpURnkWgcpiWkE0RQFvtDSawFQQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=aICIVh+ocuoZihMGEvnb7by8HmIJqZj+8vYNnyaFFObBil4WPnbQIplOo9VXFlu031mTj7oIA5pBmMx6SW9TmcZP/QNmE9L8GV3IGWrY6Fm2oBGg9+XeFH+ogcfMOgIXuxYL3D7ago6LZySMhjCDU6r3TdyjTRkBHWc6SoDiciE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ef0a/L4H; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ef0a/L4H" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-609a8fc232bso33028637b3.0 for ; Tue, 05 Mar 2024 08:04:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654662; x=1710259462; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nPp+FKZCe5WrKAC/L6DMRtNnUj6zHw/P99nkgEHXMdE=; b=ef0a/L4HePAVJzctE0nZ5UmbHpgXCwFmrD8M4FjXk2Yx2yJ8wBZDEcibqoP1wFaDZP 2fEcPDcP5GULokHtFPvad852SfqQxxo6/lDcpXEDphAQetUV6cpKQcCJYCIjqPVEXdrL Xwn3VQl4ujy5pCMlAJ8pLq9MLJvcOj766CXKJtf+vS6eLEJ94PEXvUZVQikZZJxeJkyn WiH/pnLbMpv4GnIP0pXddFA6WJbZl7yc810v9xvfQHvIHNh9n0nfJDqLMAmf9aqU200E j5v1Oyhe2bFf/pucBdrwEBVt9iE+pYS6TkNwcpvbbAu68OyFIBNOdmiM7EowWF+D8OdY 3oIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654662; x=1710259462; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nPp+FKZCe5WrKAC/L6DMRtNnUj6zHw/P99nkgEHXMdE=; b=dDB/1K+SepLHAso8k2WRFdDIhDyJ7oFJdX22dD+Dvy60PfmpTS04pZJaNg+vyaobzp Ysef86cBbxfkJCoVKmm612Le90b8F+8aliBH53POOPtAbkBE8u4CrLs4Ij8eoymF6mFv XHRVPbLFX1Bwjw9HJqUyZIQ6SZF9Lh4NpT1WC867t9kOX0n1cSVHjQwG4+OpmiZqf0R/ d/Yb5VmdpQ/HLdX8UbHr4Wco0/5Ah19BZVM8yohnS/aMetMwDNa5Nuo/yonGb/JPpTQD /e9QFACadePEYWM76Z/7RmGdFv3Kzp5TqxoZoLMZ0QM3dAu0kW6anHkNa3WP/OFErEWl MFBQ== X-Gm-Message-State: AOJu0YzdK0eO6jqfHSp6lMihSxQS0eCFH/1b8r4Pj8HWZr95FkngBhid KwypsbbCHuPgMOTdRYzMeNy9zH00G536HbmraZNEph6bQN5S7dSEsyhP9o8D5Ss85jiDYEev1uv +wpqUwukMhw== X-Google-Smtp-Source: AGHT+IG0VfqF5jYZoKSQY52ujme8HXdeOlHLHU4kOgSMfl7+fh/azEQQyg+z84+eb5EDl8LLTLT6eDzHyV+kyw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:154e:b0:dc7:3189:4e75 with SMTP id r14-20020a056902154e00b00dc731894e75mr415653ybu.3.1709654661987; Tue, 05 Mar 2024 08:04:21 -0800 (PST) Date: Tue, 5 Mar 2024 16:03:59 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-5-edumazet@google.com> Subject: [PATCH net-next 04/18] net: move ptype_all into net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org ptype_all is used in rx/tx fast paths. Move it to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 1 - include/net/hotdata.h | 1 + net/core/dev.c | 16 +++++++--------- net/core/hotdata.c | 1 + net/core/net-procfs.c | 7 ++++--- 5 files changed, 13 insertions(+), 13 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 15ce809e0541078bff7a48b8d7cb2cf2c1ac8a93..e2e7d5a7ef34de165cd293eb71800e1e6b450432 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -5300,7 +5300,6 @@ static inline const char *netdev_reg_state(const struct net_device *dev) #define PTYPE_HASH_SIZE (16) #define PTYPE_HASH_MASK (PTYPE_HASH_SIZE - 1) -extern struct list_head ptype_all __read_mostly; extern struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly; extern struct net_device *blackhole_netdev; diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 149e56528537d8ed3365e46d6dc96e39c73a733a..d462cb8f16bad459b439c566274c01a0f18a95d0 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -7,6 +7,7 @@ /* Read mostly data used in network fast paths. */ struct net_hotdata { struct list_head offload_base; + struct list_head ptype_all; int gro_normal_batch; int netdev_budget; int netdev_budget_usecs; diff --git a/net/core/dev.c b/net/core/dev.c index 53ebdb55e8b7c0a6522eb3fdbb7bdd00948eb9a5..119b7004a8e51ea6785c60e558988d369eec8935 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -161,7 +161,6 @@ static DEFINE_SPINLOCK(ptype_lock); struct list_head ptype_base[PTYPE_HASH_SIZE] __read_mostly; -struct list_head ptype_all __read_mostly; /* Taps */ static int netif_rx_internal(struct sk_buff *skb); static int call_netdevice_notifiers_extack(unsigned long val, @@ -540,7 +539,7 @@ static inline void netdev_set_addr_lockdep_class(struct net_device *dev) static inline struct list_head *ptype_head(const struct packet_type *pt) { if (pt->type == htons(ETH_P_ALL)) - return pt->dev ? &pt->dev->ptype_all : &ptype_all; + return pt->dev ? &pt->dev->ptype_all : &net_hotdata.ptype_all; else return pt->dev ? &pt->dev->ptype_specific : &ptype_base[ntohs(pt->type) & PTYPE_HASH_MASK]; @@ -2226,7 +2225,8 @@ static inline bool skb_loop_sk(struct packet_type *ptype, struct sk_buff *skb) */ bool dev_nit_active(struct net_device *dev) { - return !list_empty(&ptype_all) || !list_empty(&dev->ptype_all); + return !list_empty(&net_hotdata.ptype_all) || + !list_empty(&dev->ptype_all); } EXPORT_SYMBOL_GPL(dev_nit_active); @@ -2237,10 +2237,9 @@ EXPORT_SYMBOL_GPL(dev_nit_active); void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev) { - struct packet_type *ptype; + struct list_head *ptype_list = &net_hotdata.ptype_all; + struct packet_type *ptype, *pt_prev = NULL; struct sk_buff *skb2 = NULL; - struct packet_type *pt_prev = NULL; - struct list_head *ptype_list = &ptype_all; rcu_read_lock(); again: @@ -2286,7 +2285,7 @@ void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev) pt_prev = ptype; } - if (ptype_list == &ptype_all) { + if (ptype_list == &net_hotdata.ptype_all) { ptype_list = &dev->ptype_all; goto again; } @@ -5387,7 +5386,7 @@ static int __netif_receive_skb_core(struct sk_buff **pskb, bool pfmemalloc, if (pfmemalloc) goto skip_taps; - list_for_each_entry_rcu(ptype, &ptype_all, list) { + list_for_each_entry_rcu(ptype, &net_hotdata.ptype_all, list) { if (pt_prev) ret = deliver_skb(skb, pt_prev, orig_dev); pt_prev = ptype; @@ -11771,7 +11770,6 @@ static int __init net_dev_init(void) if (netdev_kobject_init()) goto out; - INIT_LIST_HEAD(&ptype_all); for (i = 0; i < PTYPE_HASH_SIZE; i++) INIT_LIST_HEAD(&ptype_base[i]); diff --git a/net/core/hotdata.c b/net/core/hotdata.c index 087c4c84987df07f11a87112a778a5cac608a654..29fcfe89fd9a697120f826dbe2fd36a1617581a1 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -7,6 +7,7 @@ struct net_hotdata net_hotdata __cacheline_aligned = { .offload_base = LIST_HEAD_INIT(net_hotdata.offload_base), + .ptype_all = LIST_HEAD_INIT(net_hotdata.ptype_all), .gro_normal_batch = 8, .netdev_budget = 300, diff --git a/net/core/net-procfs.c b/net/core/net-procfs.c index 2e4e96d30ee1a7a51e49587378aab47aed1290da..a97eceb84e61ec347ad132ff0f22c8ce82f12e90 100644 --- a/net/core/net-procfs.c +++ b/net/core/net-procfs.c @@ -3,6 +3,7 @@ #include #include #include +#include #include "dev.h" @@ -183,7 +184,7 @@ static void *ptype_get_idx(struct seq_file *seq, loff_t pos) } } - list_for_each_entry_rcu(pt, &ptype_all, list) { + list_for_each_entry_rcu(pt, &net_hotdata.ptype_all, list) { if (i == pos) return pt; ++i; @@ -231,13 +232,13 @@ static void *ptype_seq_next(struct seq_file *seq, void *v, loff_t *pos) } } - nxt = ptype_all.next; + nxt = net_hotdata.ptype_all.next; goto ptype_all; } if (pt->type == htons(ETH_P_ALL)) { ptype_all: - if (nxt != &ptype_all) + if (nxt != &net_hotdata.ptype_all) goto found; hash = 0; nxt = ptype_base[0].next; From patchwork Tue Mar 5 16:04:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582618 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5829A128830 for ; Tue, 5 Mar 2024 16:04:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654666; cv=none; b=gaoSGxCWyJlgbjpgwAbpB63UYJHoIyU3pc0y4lCcPw9lr45UVvh5F4k1wZCFgS05BDX0eoR/D62xciKW+gJAjYIUy1X9zHhzeU3gKA6eQ9Wdwul6q4xXFs3PJD3SgdExxjhHq75QSDxABYmTfWh8TbKjxqhP3RFyTW/Gv1T91GI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654666; c=relaxed/simple; bh=0yw0C0XAQ5Askvz5OGBEXGb8dMuNgFf9izDpTevmaDA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=M2v2FRzn4XcU4S7SLHp9+m1ctaogDYTEDKuEPhecfi4eqbKCjqCMfs8/o72Ix0Q6/+qR3GRDXXMg/f4tx437jl2F3DcgOCn/Z8v1L5qcs5MmrZkJ+b5XSe7QGVwgi2jDnX4ehJrJSzo/kem5TA3ezbv3fUEmR8ALpMgm7G7miiA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=o09aTMVI; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="o09aTMVI" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dced704f17cso9808852276.1 for ; Tue, 05 Mar 2024 08:04:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654663; x=1710259463; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rI8hlkbcYR6A41wLQsMFj8e6JRiA/d3SlGACllYlHvw=; b=o09aTMVIYkxxDCmcQRo4G9rqMO5DFEz1NT82gkpZ0rWHO4pMVT64RJoaT0b5blCkZM UqBOQ8H5YfpkMUerpDLDWTEUB+GQuzigzlOreyjIvnWFwUYI+GO1HgKKSe9p4dBoLvuM e4x1a2dKayOHnPTCBWEmBZM/zT3sJVBt/JqBa6YaQJT/UIHEgCFZjODUMRopXARGYIIR i4leSzf+grGPgejoEzGDNY2ppKwmsq/yCuN/UucDyL1IN5HVjGn/CZafWVAr99hEVV8T lBlYvHZih7HyCG8ol83eerRhRQYuxg4j8VlILOlrA8f2KPvFen20X6MRGDu92y5Ek/dI c0Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654663; x=1710259463; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rI8hlkbcYR6A41wLQsMFj8e6JRiA/d3SlGACllYlHvw=; b=tUk9bZRNy1h5HBgReHc7IKj5cjf+QaKXt//4WdRNyB/8o1uA/Ks8wT3opcupLU0KM1 RmvO3RgxJND1eUBCaeHg61oVJWeXmchsNxbgWrYKi+7U7oNsYkisGfuRhd1++3M4MW+X xaik2kPO6OwB/JJQN5yH4wGjfeHxiVcgbJvXeAUsQYjQIQvLvck/KgePm6RcygiUjbIB RahhwmPJSjpc6djcrNCR5VlmLD6iiD84H2LGRXcC/2JPX4zC7tsqYpU113DYSY67kQFt 7GZtUFW3+azROEYs85vlBbgvf6eY0azBYn2sIl/YKRU1Qmlj/XPc+EhRM+ln2ozrj569 jXtA== X-Gm-Message-State: AOJu0YxltOvhyiD0/Uag0G6aJPswjM6drueTaVwExZ0H2vSUmHXPiSdw 9DoSnAiIMgkYa0xMD/nzN+MTG92KekSY8fMnHyoIPd7RNp+IfFb39Xe9VYOAqSphre/gCo0MqFt VEHmtJdmVyw== X-Google-Smtp-Source: AGHT+IEQ6CSpyvi526a/E6/437rLfO+gQ8YqMPl0yg9tYAlQpPoG/gWA/JLqAQjreKevsghwp4HVBtOGic7T1w== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a25:f307:0:b0:dc7:4af0:8c6c with SMTP id c7-20020a25f307000000b00dc74af08c6cmr473643ybs.6.1709654663380; Tue, 05 Mar 2024 08:04:23 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:00 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-6-edumazet@google.com> Subject: [PATCH net-next 05/18] net: move netdev_max_backlog to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org netdev_max_backlog is used in rx fat path. Move it to net_hodata for better cache locality. Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 1 - include/net/hotdata.h | 1 + net/core/dev.c | 8 +++----- net/core/gro_cells.c | 3 ++- net/core/hotdata.c | 2 ++ net/core/sysctl_net_core.c | 2 +- net/xfrm/espintcp.c | 4 +++- net/xfrm/xfrm_input.c | 3 ++- 8 files changed, 14 insertions(+), 10 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index e2e7d5a7ef34de165cd293eb71800e1e6b450432..044d6f5b2ace3e2decd4296e01c8d3e200c6c7dc 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -4787,7 +4787,6 @@ void dev_fetch_sw_netstats(struct rtnl_link_stats64 *s, const struct pcpu_sw_netstats __percpu *netstats); void dev_get_tstats64(struct net_device *dev, struct rtnl_link_stats64 *s); -extern int netdev_max_backlog; extern int dev_rx_weight; extern int dev_tx_weight; diff --git a/include/net/hotdata.h b/include/net/hotdata.h index d462cb8f16bad459b439c566274c01a0f18a95d0..dc50b200a94b6b935cd79d8e0406a61209fdc68e 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -12,6 +12,7 @@ struct net_hotdata { int netdev_budget; int netdev_budget_usecs; int tstamp_prequeue; + int max_backlog; }; extern struct net_hotdata net_hotdata; diff --git a/net/core/dev.c b/net/core/dev.c index 119b7004a8e51ea6785c60e558988d369eec8935..1b112c4db983c2d7cd280bc8c2ebc621ea3c6145 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4404,9 +4404,6 @@ EXPORT_SYMBOL(__dev_direct_xmit); * Receiver routines *************************************************************************/ -int netdev_max_backlog __read_mostly = 1000; -EXPORT_SYMBOL(netdev_max_backlog); - unsigned int sysctl_skb_defer_max __read_mostly = 64; int weight_p __read_mostly = 64; /* old backlog weight */ int dev_weight_rx_bias __read_mostly = 1; /* bias for backlog weight */ @@ -4713,7 +4710,7 @@ static bool skb_flow_limit(struct sk_buff *skb, unsigned int qlen) struct softnet_data *sd; unsigned int old_flow, new_flow; - if (qlen < (READ_ONCE(netdev_max_backlog) >> 1)) + if (qlen < (READ_ONCE(net_hotdata.max_backlog) >> 1)) return false; sd = this_cpu_ptr(&softnet_data); @@ -4761,7 +4758,8 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, if (!netif_running(skb->dev)) goto drop; qlen = skb_queue_len(&sd->input_pkt_queue); - if (qlen <= READ_ONCE(netdev_max_backlog) && !skb_flow_limit(skb, qlen)) { + if (qlen <= READ_ONCE(net_hotdata.max_backlog) && + !skb_flow_limit(skb, qlen)) { if (qlen) { enqueue: __skb_queue_tail(&sd->input_pkt_queue, skb); diff --git a/net/core/gro_cells.c b/net/core/gro_cells.c index ed5ec5de47f670753924bd0c72db1e3ceb9b9e7a..ff8e5b64bf6b76451a69e3eae132b593c60ee204 100644 --- a/net/core/gro_cells.c +++ b/net/core/gro_cells.c @@ -3,6 +3,7 @@ #include #include #include +#include struct gro_cell { struct sk_buff_head napi_skbs; @@ -26,7 +27,7 @@ int gro_cells_receive(struct gro_cells *gcells, struct sk_buff *skb) cell = this_cpu_ptr(gcells->cells); - if (skb_queue_len(&cell->napi_skbs) > READ_ONCE(netdev_max_backlog)) { + if (skb_queue_len(&cell->napi_skbs) > READ_ONCE(net_hotdata.max_backlog)) { drop: dev_core_stats_rx_dropped_inc(dev); kfree_skb(skb); diff --git a/net/core/hotdata.c b/net/core/hotdata.c index 29fcfe89fd9a697120f826dbe2fd36a1617581a1..35ed5a83ecc7ebda513fe4fafc596e053f0252c5 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -15,4 +15,6 @@ struct net_hotdata net_hotdata __cacheline_aligned = { .netdev_budget_usecs = 2 * USEC_PER_SEC / HZ, .tstamp_prequeue = 1, + .max_backlog = 1000, }; +EXPORT_SYMBOL(net_hotdata); diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index bddd07da099886f2747f2ac4ba39a482b0f4231d..8eaeeb289914258f90cf940e906d5c6be0cc0cd6 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -440,7 +440,7 @@ static struct ctl_table net_core_table[] = { }, { .procname = "netdev_max_backlog", - .data = &netdev_max_backlog, + .data = &net_hotdata.max_backlog, .maxlen = sizeof(int), .mode = 0644, .proc_handler = proc_dointvec diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c index d3b3f9e720b3b6c2a4ea89df4257a564100b2c4b..fe82e2d073006e5ab1b03868c851147c0422d26d 100644 --- a/net/xfrm/espintcp.c +++ b/net/xfrm/espintcp.c @@ -10,6 +10,7 @@ #if IS_ENABLED(CONFIG_IPV6) #include #endif +#include static void handle_nonesp(struct espintcp_ctx *ctx, struct sk_buff *skb, struct sock *sk) @@ -169,7 +170,8 @@ int espintcp_queue_out(struct sock *sk, struct sk_buff *skb) { struct espintcp_ctx *ctx = espintcp_getctx(sk); - if (skb_queue_len(&ctx->out_queue) >= READ_ONCE(netdev_max_backlog)) + if (skb_queue_len(&ctx->out_queue) >= + READ_ONCE(net_hotdata.max_backlog)) return -ENOBUFS; __skb_queue_tail(&ctx->out_queue, skb); diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c index bd4ce21d76d7551d8f0a4a11f4b75705a7f634c5..161f535c8b9495b01f6d9689e14c40e5c0885968 100644 --- a/net/xfrm/xfrm_input.c +++ b/net/xfrm/xfrm_input.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "xfrm_inout.h" @@ -764,7 +765,7 @@ int xfrm_trans_queue_net(struct net *net, struct sk_buff *skb, trans = this_cpu_ptr(&xfrm_trans_tasklet); - if (skb_queue_len(&trans->queue) >= READ_ONCE(netdev_max_backlog)) + if (skb_queue_len(&trans->queue) >= READ_ONCE(net_hotdata.max_backlog)) return -ENOBUFS; BUILD_BUG_ON(sizeof(struct xfrm_trans_cb) > sizeof(skb->cb)); From patchwork Tue Mar 5 16:04:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582619 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C907128833 for ; Tue, 5 Mar 2024 16:04:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654667; cv=none; b=c/z+ydY8KdTXrDH9Ac+Cp0JOBDeSaCwA2t3AMgMEtJZZWhEFSPhzxnER1bWVu/pJW8PIZRh5c9eU2UwSt5ZXQ4CeT4NQ43mam1IG6uPC/PuiGenJr5xpmo5vtAJrP61yeEV66bmuXd9rc7X3Gjizz0HcntrYKcc3oKfz9/EUstQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654667; c=relaxed/simple; bh=ClIk7kQtxWQZHnPBckWAKi71GT8HcpiekxkfprrzHdg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=KT1c3/QtLOjBsa7PZbKDDQQw9BLBFuMzfv20OMhiqX0VrNt7Afh+Ols9gkhQfjeALgUv6qrn9maq7DO7sDNhZJHbsdkoCeoL4WkbeQ4CPY0Cl8OPjb/7HUvUNzP4oI5q2blaP+xiegM8aEDKD8/VjYk0Fo+f2QlNSJfx0CNc1BQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=VFEsv4XY; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="VFEsv4XY" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dcc0bcf9256so7580992276.3 for ; Tue, 05 Mar 2024 08:04:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654665; x=1710259465; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=OnrJVah67Ylh005tHnrwHvl3p9vempg6J3tNuVCiAdo=; b=VFEsv4XY1/D92+AiKEVANGY1YYfF40qHBlpqHSUJawDtJ9kPXkUNOvnyqWiKPH682E AVq1RbGX2ATmL9noxwQaRAvTB5iBEsXFNL3AskZZ7sr3SzuNL92FnF3u+a8WvvKmwql+ 3aP1+lLlCGBCdNO4RgRnNB0bg0EwCQa9nTO8Y5Z5QQ5gX98+VTxntT5EivSZc00QML30 /H4q6YtBtFcra/S3Ayol7YIlv3JYX16GH6IRrldXMz74gfQoV//yNHEXzueZpPud+xAz 04vtct0qhhG+FDM1agxY8OLj9mDwJO3zuD6QrJSbi7lqajVM6+yPxUiPxylayVB6OXSe a5tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654665; x=1710259465; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=OnrJVah67Ylh005tHnrwHvl3p9vempg6J3tNuVCiAdo=; b=nS4UOSVs63lQpLgwRXz0YeMIqcVQZAwaG5IdUNPnxZzvU0U9Fo8F9BzczsnPmyCxiz cFcltmnKRjOru8WE9raILcmxcibX87T+rSU8QryTLh/wpwbnxW7tqeiZsrOZ4ibHDps+ G5AeXlC9XpVo1AwIdIEPBv6dwHQ3JesTHWCChYOG8Gh/41oIgqQ6sBXk60u6Y3bz7mHn Prsgk+b0za9BWDrbHUGlEgTEzwc04M/fNBx2NNa8boE6T70cuS22FLYH9YYm4OfRwhSX S3fRlkoYZ/nfGF6QK4LCBRb0qTbGhGslTysI6LEK8QnRrKx8AwLwPTAWcTOVkW/WO9EQ 26oA== X-Gm-Message-State: AOJu0YzraCAHgz/z6Xz5szxi3YFXFNn20tZjAqQjbtI6mDariwn4PPwq vcfVzIKXF7yMUAKOnBaMDr2c44RvgA0cQQWnosyvLJvPWnskHtQ8gqt0/dzwY/W34TNxMIOTkeo 5HTH6jXGUhQ== X-Google-Smtp-Source: AGHT+IGZ39/TBmeRRiAYM1QUFg9NoSkMD1E3edAmIIN5oACkYKJBss/EEXyNVcLnfHJi1IJxv1sGf/GXoL5u1Q== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1009:b0:dbe:387d:a8ef with SMTP id w9-20020a056902100900b00dbe387da8efmr416752ybt.1.1709654664767; Tue, 05 Mar 2024 08:04:24 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:01 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-7-edumazet@google.com> Subject: [PATCH net-next 06/18] net: move ip_packet_offload and ipv6_packet_offload to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org These structures are used in GRO and GSO paths. Signed-off-by: Eric Dumazet --- include/net/hotdata.h | 7 +++++++ net/ipv4/af_inet.c | 18 +++++++++--------- net/ipv6/ip6_offload.c | 18 +++++++++--------- 3 files changed, 25 insertions(+), 18 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index dc50b200a94b6b935cd79d8e0406a61209fdc68e..ec752d234c5ed4b9c110c9e61c143fe8fc27089e 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -3,9 +3,16 @@ #define _NET_HOTDATA_H #include +#include /* Read mostly data used in network fast paths. */ struct net_hotdata { +#if IS_ENABLED(CONFIG_INET) + struct packet_offload ip_packet_offload; +#endif +#if IS_ENABLED(CONFIG_IPV6) + struct packet_offload ipv6_packet_offload; +#endif struct list_head offload_base; struct list_head ptype_all; int gro_normal_batch; diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index 5daebdcbca326aa1fc042e1e1ff1e82a18bd283d..08dda6955562ea6b89e02b8299b03ab52b342f27 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -1904,14 +1904,6 @@ static int ipv4_proc_init(void); * IP protocol layer initialiser */ -static struct packet_offload ip_packet_offload __read_mostly = { - .type = cpu_to_be16(ETH_P_IP), - .callbacks = { - .gso_segment = inet_gso_segment, - .gro_receive = inet_gro_receive, - .gro_complete = inet_gro_complete, - }, -}; static const struct net_offload ipip_offload = { .callbacks = { @@ -1938,7 +1930,15 @@ static int __init ipv4_offload_init(void) if (ipip_offload_init() < 0) pr_crit("%s: Cannot add IPIP protocol offload\n", __func__); - dev_add_offload(&ip_packet_offload); + net_hotdata.ip_packet_offload = (struct packet_offload) { + .type = cpu_to_be16(ETH_P_IP), + .callbacks = { + .gso_segment = inet_gso_segment, + .gro_receive = inet_gro_receive, + .gro_complete = inet_gro_complete, + }, + }; + dev_add_offload(&net_hotdata.ip_packet_offload); return 0; } diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c index cca64c7809bee9a0360cbfab6a645d3f8d2ffea3..b41e35af69ea2835aa47d6ca01d9b109d4092462 100644 --- a/net/ipv6/ip6_offload.c +++ b/net/ipv6/ip6_offload.c @@ -419,14 +419,6 @@ static int ip4ip6_gro_complete(struct sk_buff *skb, int nhoff) return inet_gro_complete(skb, nhoff); } -static struct packet_offload ipv6_packet_offload __read_mostly = { - .type = cpu_to_be16(ETH_P_IPV6), - .callbacks = { - .gso_segment = ipv6_gso_segment, - .gro_receive = ipv6_gro_receive, - .gro_complete = ipv6_gro_complete, - }, -}; static struct sk_buff *sit_gso_segment(struct sk_buff *skb, netdev_features_t features) @@ -486,7 +478,15 @@ static int __init ipv6_offload_init(void) if (ipv6_exthdrs_offload_init() < 0) pr_crit("%s: Cannot add EXTHDRS protocol offload\n", __func__); - dev_add_offload(&ipv6_packet_offload); + net_hotdata.ipv6_packet_offload = (struct packet_offload) { + .type = cpu_to_be16(ETH_P_IPV6), + .callbacks = { + .gso_segment = ipv6_gso_segment, + .gro_receive = ipv6_gro_receive, + .gro_complete = ipv6_gro_complete, + }, + }; + dev_add_offload(&net_hotdata.ipv6_packet_offload); inet_add_offload(&sit_offload, IPPROTO_IPV6); inet6_add_offload(&ip6ip6_offload, IPPROTO_IPV6); From patchwork Tue Mar 5 16:04:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582620 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E38ED128385 for ; Tue, 5 Mar 2024 16:04:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654668; cv=none; b=rO98WTnBEAl7tkVIEB/DrYK2gDFW1Mk0rqIWBGzUkSA3uSW2k+c0dNH6hzaGgJ+oBfgyDchd0e0VyuZfplwEGHAIMl0Fyk+sMX1AuWnX1GShZHLnFutsF+OdDmNejvJ+03QJaUldFVWjy/3CnnfpcH8s70AcjlNqxEEMBIAhLHc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654668; c=relaxed/simple; bh=1TqMyKP5wpIwmazXaRPJbac3u4QVLvg8OenG8Op8gLw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=r/6tu0fbleL/HAkB2lHtCfbqgpAT3UagkTx5KyYp+3rK/CW+boiO6kvyKSfO0epdGTwee9p/bZYL0JgGVpQzpdJnAArEdmS4+VGYJgfy/Lde9GFz8ulgFHgSgtjvsXkn/lQ8nE9sfTfIMcsGitX9jnOn/r1cJjseMJ6XmfG4V/A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MIs46S5v; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MIs46S5v" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b26783b4so7154389276.0 for ; Tue, 05 Mar 2024 08:04:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654666; x=1710259466; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EsoBf0bFuUIa9NPM7+ThGlhD8PUNm3GCIAPGt/l5R2g=; b=MIs46S5vLWm4zB7AVw3Ge91n6CRtPE7fYMrBMBmWE5W/wK0tPQhWPLj/zz4rCDBjxL ERSxmDXT+Pt92uXlOWdU/BxS+QQh56xvcz+bSVj6dV1nkoc93cVPs4goRrhJ3Mje9ABn afKYFTJjbm8tx8eE2+i27g5E0K+mLhuBztFKXHLrzWVIBVw41993s9+UoBz9qG6rf6hu Q+3gsBSdHaVCkiyRgrA9G6gfDl7fzVtkr/AQbwfPE3BDYfzVngCmFc2CUZgHeUy2Q3RJ /21xe0ZBfbd2AEjlugVcOZJuEd51Fs/AjAT2img+KuBciFxPFbxnoC7/JhJ5USNAraF/ D9+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654666; x=1710259466; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EsoBf0bFuUIa9NPM7+ThGlhD8PUNm3GCIAPGt/l5R2g=; b=fWDPqpYCQdK1m6VIGeVh+k7nZVIQksZQZ/jIEqqAu6Zl2GeBh0tfCUaR+T60ANCRq6 gJPxCtgUmsZqb1Da4NQYQCUCcZWFhhv5ajzvpizTdvSDfXXag8B7eG/ELbI/bsKk4G5N UZgOcK5fyRbsK+y5td9kIw/fNzXp9QUz0ZglALfMP6h7H1UTdNOhGSd+Cs5FH9bfT4ul ZoVHzORajIXIKTFv8sd8G/l1DigPd1dd3uAzTAruT8uoe0LdkKbPjqBqWZbnThDfXWdl AOBULUT5sn4uk//Pt7cR6Qsotd2KiG/0D8SodQlOtSBOWtJJLeD9phmWZ7dL4My4jewY mQDg== X-Gm-Message-State: AOJu0Yz3oc2BwyYaBpiaFXQO2wT6+rBGpWZoavzgEl1LJz4qY2Jrpfrp iRwBSdAc2iMO5KFZ/qpp1elUCUXqyGYLfNen+assZE+czQWk3Ky8pDhZYK5KpXwWa2u91O0uPxb ZUzVRH1L2/A== X-Google-Smtp-Source: AGHT+IEV9qcXtUrAahx863zMFezlkGq4pLsUazJp6xCIaJC6ovEapmfICwve/gGwzrllzUfw5t+Ef1hzp3jbwg== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1009:b0:dbe:387d:a8ef with SMTP id w9-20020a056902100900b00dbe387da8efmr416755ybt.1.1709654666018; Tue, 05 Mar 2024 08:04:26 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:02 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-8-edumazet@google.com> Subject: [PATCH net-next 07/18] net: move tcpv4_offload and tcpv6_offload to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org These are used in TCP fast paths. Move them into net_hotdata for better cache locality. Signed-off-by: Eric Dumazet --- include/net/hotdata.h | 3 +++ net/ipv4/tcp_offload.c | 17 ++++++++--------- net/ipv6/tcpv6_offload.c | 16 ++++++++-------- 3 files changed, 19 insertions(+), 17 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index ec752d234c5ed4b9c110c9e61c143fe8fc27089e..a4a8df3bc0dea1b4c9589bd70f7ac457ebc5b634 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -4,14 +4,17 @@ #include #include +#include /* Read mostly data used in network fast paths. */ struct net_hotdata { #if IS_ENABLED(CONFIG_INET) struct packet_offload ip_packet_offload; + struct net_offload tcpv4_offload; #endif #if IS_ENABLED(CONFIG_IPV6) struct packet_offload ipv6_packet_offload; + struct net_offload tcpv6_offload; #endif struct list_head offload_base; struct list_head ptype_all; diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index b955ab3b236d965a38054efa004fe12f03074c70..ebe4722bb0204433936e69724879779141288789 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -345,15 +345,14 @@ INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) return 0; } -static const struct net_offload tcpv4_offload = { - .callbacks = { - .gso_segment = tcp4_gso_segment, - .gro_receive = tcp4_gro_receive, - .gro_complete = tcp4_gro_complete, - }, -}; - int __init tcpv4_offload_init(void) { - return inet_add_offload(&tcpv4_offload, IPPROTO_TCP); + net_hotdata.tcpv4_offload = (struct net_offload) { + .callbacks = { + .gso_segment = tcp4_gso_segment, + .gro_receive = tcp4_gro_receive, + .gro_complete = tcp4_gro_complete, + }, + }; + return inet_add_offload(&net_hotdata.tcpv4_offload, IPPROTO_TCP); } diff --git a/net/ipv6/tcpv6_offload.c b/net/ipv6/tcpv6_offload.c index bf0c957e4b5eaaabc0ac3a7e55c7de6608cec156..4b07d1e6c952957419924c83c5d3ac0e9d0b2565 100644 --- a/net/ipv6/tcpv6_offload.c +++ b/net/ipv6/tcpv6_offload.c @@ -66,15 +66,15 @@ static struct sk_buff *tcp6_gso_segment(struct sk_buff *skb, return tcp_gso_segment(skb, features); } -static const struct net_offload tcpv6_offload = { - .callbacks = { - .gso_segment = tcp6_gso_segment, - .gro_receive = tcp6_gro_receive, - .gro_complete = tcp6_gro_complete, - }, -}; int __init tcpv6_offload_init(void) { - return inet6_add_offload(&tcpv6_offload, IPPROTO_TCP); + net_hotdata.tcpv6_offload = (struct net_offload) { + .callbacks = { + .gso_segment = tcp6_gso_segment, + .gro_receive = tcp6_gro_receive, + .gro_complete = tcp6_gro_complete, + }, + }; + return inet6_add_offload(&net_hotdata.tcpv6_offload, IPPROTO_TCP); } From patchwork Tue Mar 5 16:04:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582621 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4005D1292FD for ; Tue, 5 Mar 2024 16:04:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654669; cv=none; b=VojtT9LUX/1DQ3pGrST9ScJaU6+56Bp1j8ZNEkAJz+Ey/bOfMwphFuL11KktL0J43jHAyWtGYaEBnJmiVCN0AP7g2ic00RtXJxh9L1Nj1idZQWNCL0Q7A0MDE+UAXLLflye7K92Ks5vx95daiMZABErVCzRD1F2E6XI1Z/TNnR4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654669; c=relaxed/simple; bh=snMuZN97+kOSp/1k7e6f4mv9qsuzm6DWJUPob6gN3yU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=SfBw4XmYVPeCHTjpiTTmQYtEXVc+IaqT37Wj8rvucgY356sKLFBgUNy2pyGDxygMmg1w5gCFlRjHzJMydHCBSKtRM+8I58JON6c2Ly2qQQTuE/Ch8uCOJehJMkuiCM2xKMzbPYnzKVOhPXpQNr6T4rIR5zZ4F10NsGLeWIWstn4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=wufV5YEC; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="wufV5YEC" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc746178515so8506459276.2 for ; Tue, 05 Mar 2024 08:04:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654667; x=1710259467; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kFWf2CBTF8+VR6S687evZT9Hnp4r7DP5+F9I/vsP0fY=; b=wufV5YECY7pG+kW6TLarBd31bNs9cAPi47+J16v9ffFL0SlQaRjFqUrRsFcLpyflQj 9sVfaBFpyTt5h09gOGAGglgn2ekH6zD0w9W2dkAs1BIm5dpIoge6l0OACWjht/A8tVYf Dy8FdA0WQ87t6FX+cxu9IjpF/3ZfbMDCrSlk6pOwjkvYA2s31TvIe+ZxiJM7ZxU+HW5l zcDtRYkz848lQSpDWYzAPApZ9mQ6B5kNe9xMvGk4dPeq7WPHRx7Wk1cnQO6w9tCFqWh/ krbTV7NmZ8P1wp+hEAcQTQ7B3+AYBTQOyXusbD3kNNDDUboHX7oYwAwCeBnQayQh1o86 eo5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654667; x=1710259467; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kFWf2CBTF8+VR6S687evZT9Hnp4r7DP5+F9I/vsP0fY=; b=pmFRA0SRz5Sru5SeVkO+dbXoEnN+znRio9rhwvwX1SBhyKMTcwymlQvZfB8lkrnvmG e6ttZxkFvxYIBEsfcgApS7y6By27V+JPR8iZkXWTjX8ru1FAt+K+IRNMoQA3PXZwHhew OaocfmF6z5QwM81A3f1gIjYmP8FSC9roxr2hoyAP7dGJ+k0uLwCQhfpjeYxAVIGHyi51 Q3Y/2TQBqzstowK/j8C5SLOQj+n93ADp7t9DgCp3kmVuFzu73Vncn/2yALs2n3ZxGXOq asxAPnXWy/1+phad0pHAVQ7Eb2Y5fptvEIhNbeDlpo9eOZFG84AA3UXUwTLBAqwYvOdz HpfQ== X-Gm-Message-State: AOJu0Yxslye3iVnDfw3CFRdI6ru+aV4W/TiyOhhYD0+JkHRl0Ma8wHyX bu3qEkKfzq6whCy+wTaSdVHV1E/GbZB5KkYOGILnefAJv/GMDCnNPq7wrxo7rkl2t5YFjlPsbad 2WV5XCQK0sw== X-Google-Smtp-Source: AGHT+IFR/J4nQPkY/BmaUZQbMg+GvDpovKC3txgtx9O2ofXDBlT0aug27UDE1RoQPsxE9HwEgWaP2bPImT+T5Q== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:110a:b0:dc7:49a9:6666 with SMTP id o10-20020a056902110a00b00dc749a96666mr3121810ybu.3.1709654667534; Tue, 05 Mar 2024 08:04:27 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:03 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-9-edumazet@google.com> Subject: [PATCH net-next 08/18] net: move dev_tx_weight to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org dev_tx_weight is used in tx fast path. Move it to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 1 - include/net/hotdata.h | 1 + net/core/dev.c | 1 - net/core/hotdata.c | 1 + net/core/sysctl_net_core.c | 2 +- net/sched/sch_generic.c | 3 ++- 6 files changed, 5 insertions(+), 4 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 044d6f5b2ace3e2decd4296e01c8d3e200c6c7dc..c2a735edc44be95fbd4bbd1e234d883582bfde10 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -4788,7 +4788,6 @@ void dev_fetch_sw_netstats(struct rtnl_link_stats64 *s, void dev_get_tstats64(struct net_device *dev, struct rtnl_link_stats64 *s); extern int dev_rx_weight; -extern int dev_tx_weight; enum { NESTED_SYNC_IMM_BIT, diff --git a/include/net/hotdata.h b/include/net/hotdata.h index a4a8df3bc0dea1b4c9589bd70f7ac457ebc5b634..2b0eb6b7f1f2c9b1273b07e06ba0b5c12a2934bf 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -23,6 +23,7 @@ struct net_hotdata { int netdev_budget_usecs; int tstamp_prequeue; int max_backlog; + int dev_tx_weight; }; extern struct net_hotdata net_hotdata; diff --git a/net/core/dev.c b/net/core/dev.c index 1b112c4db983c2d7cd280bc8c2ebc621ea3c6145..3f8d451f42fb9ba621f1ea19e784c7f0be0bf2d8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4409,7 +4409,6 @@ int weight_p __read_mostly = 64; /* old backlog weight */ int dev_weight_rx_bias __read_mostly = 1; /* bias for backlog weight */ int dev_weight_tx_bias __read_mostly = 1; /* bias for output_queue quota */ int dev_rx_weight __read_mostly = 64; -int dev_tx_weight __read_mostly = 64; /* Called with irq disabled */ static inline void ____napi_schedule(struct softnet_data *sd, diff --git a/net/core/hotdata.c b/net/core/hotdata.c index 35ed5a83ecc7ebda513fe4fafc596e053f0252c5..ec8c3b48e8fea57491c5870055cffb44c779db44 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -16,5 +16,6 @@ struct net_hotdata net_hotdata __cacheline_aligned = { .tstamp_prequeue = 1, .max_backlog = 1000, + .dev_tx_weight = 64, }; EXPORT_SYMBOL(net_hotdata); diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 8eaeeb289914258f90cf940e906d5c6be0cc0cd6..a30016a8660e09db89b3153e4103c185a800a2ef 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -302,7 +302,7 @@ static int proc_do_dev_weight(struct ctl_table *table, int write, if (!ret && write) { weight = READ_ONCE(weight_p); WRITE_ONCE(dev_rx_weight, weight * dev_weight_rx_bias); - WRITE_ONCE(dev_tx_weight, weight * dev_weight_tx_bias); + WRITE_ONCE(net_hotdata.dev_tx_weight, weight * dev_weight_tx_bias); } mutex_unlock(&dev_weight_mutex); diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c index 9b3e9262040b6ef6516752c558c8997bf4054123..ff5336493777507242320d7e9214c637663f0734 100644 --- a/net/sched/sch_generic.c +++ b/net/sched/sch_generic.c @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -409,7 +410,7 @@ static inline bool qdisc_restart(struct Qdisc *q, int *packets) void __qdisc_run(struct Qdisc *q) { - int quota = READ_ONCE(dev_tx_weight); + int quota = READ_ONCE(net_hotdata.dev_tx_weight); int packets; while (qdisc_restart(q, &packets)) { From patchwork Tue Mar 5 16:04:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582622 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1A54129A87 for ; Tue, 5 Mar 2024 16:04:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654671; cv=none; b=huGW6wKIDEu2wR3coxcACDXAqu2m9zRw32SbLTSstXPafNIXI/FCPZ+07yVhjmncGK0yOVebhbCH9DaYmc5i0GAsq7p3tWCIMMWuOuu9xYvw7JphpxNk+x7zNGmCEF9wkPWVEAP03DikKjQ3BFhF0tZ2p62XZm7bUGHKaWoY3mY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654671; c=relaxed/simple; bh=2galitw1vnNWkoOIoi3x8QmdLxZubnHTMRQfqPrTm2w=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=euewfLZQabftj0jGsTRLDD3rh0DrhLf6Qp8LngjgGWqjLmkkJWtnd4sUTsiHPZDLhkz6nc8VbcmiR1TJesRC8Mw08JHXLyW7kU0leeNloPDqYH5jcTHjlp/OiPQTg+Ebg9aHzrJTZtJqxFsWRGyPFb4aTFDtQ3F78zMW5OOXkH8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BFgjDblu; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BFgjDblu" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc6b26783b4so7154448276.0 for ; Tue, 05 Mar 2024 08:04:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654669; x=1710259469; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bZWA9akZOOSAULCC8k5FbB+CJKWtlh5EikBpmDhk9u8=; b=BFgjDbluvB4NnQGyhMQ4RhLfgX2kjmYBSv76wlYlwLo9aJus7fmbrY/4Utscc/JJKh Puw4BlfbHKdBBeGq1egV9wmaF85HWiImnjOPZjbTVzcSU6SruaVsAoh+EgpD2Ecycsn9 biGn8b2HUcMgQdtTCyel8staGQRtDEFLjPS+TMGi/Et9ivj7N/SAcr8v2N80T7q2n0iH qHHPXIiNeQn94ppLsjwGNSlobYl9rjYOsazWtUwzwUnCjw6JK6a8WLK9Sn838N5f3b03 Qcr47vVNvnmloYq+wXjbEPCrREeqDnthqa1xhg6uPzxxVSnc5LGuYOic0ImXjU1XXVvw NM0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654669; x=1710259469; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bZWA9akZOOSAULCC8k5FbB+CJKWtlh5EikBpmDhk9u8=; b=TSgIAtiaitavdI6Q+UsScF/NysSrBhs6Bsjz+9VlT/M1hUcbmWpiUC0VljM4cfQ85y lQLSAFrH5AE3RtAWFSR0Ghcbq2Ltyc9RWBNLdSSdJvna5DXegvhtrBQwQusx0AqjcEuG 1giIJ9BsEQHNmOJI3qiY6GYWFqiSbnC+YWDlxWJB+YTTI1/4xDk5Ot1yI5ydUcYP6rfg Z9PWu56Ryju9uI0DwR/pMqZPHXt2ICRwHWwR4kT7J0ad+6g2OOnpjOlTWXlzvSY5+grw zLR8YcmoeNWC7LmRxAKlR2WKkBG94v42aVOjXu/QpS0PbSUxJc4V6sNLAA+zzDLkqFcK iF7Q== X-Gm-Message-State: AOJu0YwYzNSF+rM0eX9Un0lzq3JML9w39BTEuHNaJClxmfF+/9nLN3zi iD+05a1NIsErJ+5Yf6fSVA1IaWNo6MrRYhBJ/h9kQqACDU1nG4ecD2/mkrqLMyCzC2F4DNSNffs aalI/vWCZqQ== X-Google-Smtp-Source: AGHT+IH7l/+ToW6rIS8SW7Neu9VBv9FdYCcJyXa5bwVuuTtW/R4T/QNNYFoXlTH/wzAJG1b/Zz7vSL/0NfHZXw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:70d:b0:dc6:c94e:fb85 with SMTP id k13-20020a056902070d00b00dc6c94efb85mr417157ybt.2.1709654668968; Tue, 05 Mar 2024 08:04:28 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:04 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-10-edumazet@google.com> Subject: [PATCH net-next 09/18] net: move dev_rx_weight to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org dev_rx_weight is read from process_backlog(). Move it to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet --- include/linux/netdevice.h | 2 -- include/net/hotdata.h | 1 + net/core/dev.c | 3 +-- net/core/hotdata.c | 1 + net/core/sysctl_net_core.c | 2 +- 5 files changed, 4 insertions(+), 5 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index c2a735edc44be95fbd4bbd1e234d883582bfde10..d3d4d1052ecb0bc704e9bb40fffbea85cfb6ab03 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -4787,8 +4787,6 @@ void dev_fetch_sw_netstats(struct rtnl_link_stats64 *s, const struct pcpu_sw_netstats __percpu *netstats); void dev_get_tstats64(struct net_device *dev, struct rtnl_link_stats64 *s); -extern int dev_rx_weight; - enum { NESTED_SYNC_IMM_BIT, NESTED_SYNC_TODO_BIT, diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 2b0eb6b7f1f2c9b1273b07e06ba0b5c12a2934bf..f45085eddbc997c2a9686b5b4de9da4cc8c4ede8 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -24,6 +24,7 @@ struct net_hotdata { int tstamp_prequeue; int max_backlog; int dev_tx_weight; + int dev_rx_weight; }; extern struct net_hotdata net_hotdata; diff --git a/net/core/dev.c b/net/core/dev.c index 3f8d451f42fb9ba621f1ea19e784c7f0be0bf2d8..26676dbd6c8594371a4d30e0dd95d72342e226ca 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4408,7 +4408,6 @@ unsigned int sysctl_skb_defer_max __read_mostly = 64; int weight_p __read_mostly = 64; /* old backlog weight */ int dev_weight_rx_bias __read_mostly = 1; /* bias for backlog weight */ int dev_weight_tx_bias __read_mostly = 1; /* bias for output_queue quota */ -int dev_rx_weight __read_mostly = 64; /* Called with irq disabled */ static inline void ____napi_schedule(struct softnet_data *sd, @@ -5978,7 +5977,7 @@ static int process_backlog(struct napi_struct *napi, int quota) net_rps_action_and_irq_enable(sd); } - napi->weight = READ_ONCE(dev_rx_weight); + napi->weight = READ_ONCE(net_hotdata.dev_rx_weight); while (again) { struct sk_buff *skb; diff --git a/net/core/hotdata.c b/net/core/hotdata.c index ec8c3b48e8fea57491c5870055cffb44c779db44..c8a7a451c18a383d091e413a510d84d163473f2f 100644 --- a/net/core/hotdata.c +++ b/net/core/hotdata.c @@ -17,5 +17,6 @@ struct net_hotdata net_hotdata __cacheline_aligned = { .tstamp_prequeue = 1, .max_backlog = 1000, .dev_tx_weight = 64, + .dev_rx_weight = 64, }; EXPORT_SYMBOL(net_hotdata); diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index a30016a8660e09db89b3153e4103c185a800a2ef..8a4c698dad9c97636ca9cebfad925d4220e98f2a 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -301,7 +301,7 @@ static int proc_do_dev_weight(struct ctl_table *table, int write, ret = proc_dointvec(table, write, buffer, lenp, ppos); if (!ret && write) { weight = READ_ONCE(weight_p); - WRITE_ONCE(dev_rx_weight, weight * dev_weight_rx_bias); + WRITE_ONCE(net_hotdata.dev_rx_weight, weight * dev_weight_rx_bias); WRITE_ONCE(net_hotdata.dev_tx_weight, weight * dev_weight_tx_bias); } mutex_unlock(&dev_weight_mutex); From patchwork Tue Mar 5 16:04:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582625 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3B0C12A154 for ; Tue, 5 Mar 2024 16:04:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654685; cv=none; b=S4YRwhLUd5NwhotePtDnpwv1Meew2rd9fECZHL4tBxEikpI4zPT8d3j6qPOztp2ftd34ZkbqxWIvY4RqDP4w5+TtvYXschVucxd9/PnrnVPntoZszES7aUmEvNZeyk7Hbs6FiWhGsWazZ3vpPNdhNRtp2tpAke+EMSNHiN4kYos= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654685; c=relaxed/simple; bh=qvJR7RDeGnkkEji13PDzCTJbcfAMtHrb5DFW3Ufc/cc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=uXp/TPGXkrU3hdsLelM5Es5AjCwxQvdL0HhKa9UCJu7bmpKJp+kto/pL0gOB1yrrSaEAPkAgDCzLDWV/pToBDTC4oNoFgnx8hyjlP9KvZ5qMd23Alkztd6iamxT/4MaEM9MMzCQE4U2Top5KkpSdNQcNVvamfS/q/zbVTtlqbXw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Chrmcago; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Chrmcago" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60971264c48so82452687b3.0 for ; Tue, 05 Mar 2024 08:04:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654670; x=1710259470; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AXuzjaWpuPGGuxoMls62ICUjw0KDzzH1deWBD6dbTUc=; b=ChrmcagoWrEf9/yxnsC5SLOIl3QbxwmKOYQNESH72p0AIOiNALdXCrhOMc8GMKb0GO Xko+1TFH70HVsgGpjH14kK7p7j7wn449QSsi2icMq8+Ms8w+Ic9bMqflAmM8YBr9ece9 wggnpRsXDY4NAhqzoLgt+ErMgtQIoAknQpBqawHZsvhrs7qMokpECf+XPkxVYw4OCyBM evQK7E9aZrOHPga5a0A5n+n1OcPYqZriRewyfpdSTQM5IqLEjIhZn+7xPB8JuP3aevNj 48F4F10Jy0yx3IWxtGSV79EeI7BshD8YjX4QQVVsD1FWSxaSqlpkxfUSxid8qOlDsjqy mGBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654670; x=1710259470; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AXuzjaWpuPGGuxoMls62ICUjw0KDzzH1deWBD6dbTUc=; b=hIPm797zpqwBsgGhIw0nbMuUQpoXqriVqbvPN0ya8jCB8fEoPJUA6xn+MWufvHJaZ4 2/TwY+DP7Cs+pWRZeM+OiDW7ny+Wt+cjRXIzoP3LNmFs3vPuHB+sQ+hnxQumzb8SWiyp Afe5N/m/KdjaYosPnJDzrheU9g577B+EQJDi6J3414f27W9JQ8uqfMY99Ugbi+lIdJ5g yj+uYcZUCUcorn3wTyBmFHbc2unKc3YR+VJmXLLY4ZDWX2FH/9jLHE0aROmUhrtuHl95 tfTC0DquJ5rq4ekSyRtt3lpZII9PP5dEkoAbQEYDJnbMsIGlFLVcYmzZvSyr3I7CwUtN MtFA== X-Gm-Message-State: AOJu0Yx+PsCchs1g+sUPmWj/RW99vy8mHcfhVgB9Tgek5K+KIrCnlxkV ldn9eTzWnagM2vH4R3vI1aGqK7DD+eZXY/2la3Wvn14QgMUiE5lGMMspXzuRbta++waAVEX5raf GgKRS+rbTjA== X-Google-Smtp-Source: AGHT+IGtrR9iNBnuqRs7p2je6kZyXq3Xvu8AiLbG9RFaPXwLaGjy2u/ewh5fjjfHa8an5/TQf9QLnvA7eG4UFw== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1821:b0:dc7:9218:df3b with SMTP id cf33-20020a056902182100b00dc79218df3bmr519963ybb.10.1709654670769; Tue, 05 Mar 2024 08:04:30 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:05 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-11-edumazet@google.com> Subject: [PATCH net-next 10/18] net: move skbuff_cache(s) to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org skbuff_cache, skbuff_fclone_cache and skb_small_head_cache are used in rx/tx fast paths. Move them to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet --- include/linux/skbuff.h | 1 - include/net/hotdata.h | 3 +++ kernel/bpf/cpumap.c | 4 +++- net/bpf/test_run.c | 4 +++- net/core/skbuff.c | 44 +++++++++++++++++++----------------------- net/core/xdp.c | 5 +++-- 6 files changed, 32 insertions(+), 29 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 3013355b63f5f29acfcb0331bfd1c3308aba034d..d0508f90bed50ca66850b29383033e11985dad34 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1271,7 +1271,6 @@ static inline void consume_skb(struct sk_buff *skb) void __consume_stateless_skb(struct sk_buff *skb); void __kfree_skb(struct sk_buff *skb); -extern struct kmem_cache *skbuff_cache; void kfree_skb_partial(struct sk_buff *skb, bool head_stolen); bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, diff --git a/include/net/hotdata.h b/include/net/hotdata.h index f45085eddbc997c2a9686b5b4de9da4cc8c4ede8..728aee1cf07c8f0d85873d912248a99e148f84b1 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -18,6 +18,9 @@ struct net_hotdata { #endif struct list_head offload_base; struct list_head ptype_all; + struct kmem_cache *skbuff_cache; + struct kmem_cache *skbuff_fclone_cache; + struct kmem_cache *skb_small_head_cache; int gro_normal_batch; int netdev_budget; int netdev_budget_usecs; diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c index 8a0bb80fe48a344964e4029fec5e895ee512babf..33f4246cdf0de8232bfbba6a4300f158db3ca9d5 100644 --- a/kernel/bpf/cpumap.c +++ b/kernel/bpf/cpumap.c @@ -24,6 +24,7 @@ #include #include #include +#include #include #include @@ -326,7 +327,8 @@ static int cpu_map_kthread_run(void *data) /* Support running another XDP prog on this CPU */ nframes = cpu_map_bpf_prog_run(rcpu, frames, xdp_n, &stats, &list); if (nframes) { - m = kmem_cache_alloc_bulk(skbuff_cache, gfp, nframes, skbs); + m = kmem_cache_alloc_bulk(net_hotdata.skbuff_cache, + gfp, nframes, skbs); if (unlikely(m == 0)) { for (i = 0; i < nframes; i++) skbs[i] = NULL; /* effect: xdp_return_frame */ diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 5535f9adc6589d028138026ebff24286098dc46d..61efeadaff8db0529a62d074f441e2a7c35eaa9e 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -254,7 +255,8 @@ static int xdp_recv_frames(struct xdp_frame **frames, int nframes, int i, n; LIST_HEAD(list); - n = kmem_cache_alloc_bulk(skbuff_cache, gfp, nframes, (void **)skbs); + n = kmem_cache_alloc_bulk(net_hotdata.skbuff_cache, gfp, nframes, + (void **)skbs); if (unlikely(n == 0)) { for (i = 0; i < nframes; i++) xdp_return_frame(frames[i]); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 43d7fc150acc9263760162c3f5778fa0a646bcc4..766219011aeaf5782df9d624696d273ef6c1577c 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -69,6 +69,7 @@ #include #include #include +#include #include #include #include @@ -88,15 +89,10 @@ #include "dev.h" #include "sock_destructor.h" -struct kmem_cache *skbuff_cache __ro_after_init; -static struct kmem_cache *skbuff_fclone_cache __ro_after_init; #ifdef CONFIG_SKB_EXTENSIONS static struct kmem_cache *skbuff_ext_cache __ro_after_init; #endif - -static struct kmem_cache *skb_small_head_cache __ro_after_init; - #define SKB_SMALL_HEAD_SIZE SKB_HEAD_ALIGN(MAX_TCP_HEADER) /* We want SKB_SMALL_HEAD_CACHE_SIZE to not be a power of two. @@ -349,7 +345,7 @@ static struct sk_buff *napi_skb_cache_get(void) struct sk_buff *skb; if (unlikely(!nc->skb_count)) { - nc->skb_count = kmem_cache_alloc_bulk(skbuff_cache, + nc->skb_count = kmem_cache_alloc_bulk(net_hotdata.skbuff_cache, GFP_ATOMIC, NAPI_SKB_CACHE_BULK, nc->skb_cache); @@ -358,7 +354,7 @@ static struct sk_buff *napi_skb_cache_get(void) } skb = nc->skb_cache[--nc->skb_count]; - kasan_mempool_unpoison_object(skb, kmem_cache_size(skbuff_cache)); + kasan_mempool_unpoison_object(skb, kmem_cache_size(net_hotdata.skbuff_cache)); return skb; } @@ -416,7 +412,7 @@ struct sk_buff *slab_build_skb(void *data) struct sk_buff *skb; unsigned int size; - skb = kmem_cache_alloc(skbuff_cache, GFP_ATOMIC); + skb = kmem_cache_alloc(net_hotdata.skbuff_cache, GFP_ATOMIC); if (unlikely(!skb)) return NULL; @@ -467,7 +463,7 @@ struct sk_buff *__build_skb(void *data, unsigned int frag_size) { struct sk_buff *skb; - skb = kmem_cache_alloc(skbuff_cache, GFP_ATOMIC); + skb = kmem_cache_alloc(net_hotdata.skbuff_cache, GFP_ATOMIC); if (unlikely(!skb)) return NULL; @@ -578,7 +574,7 @@ static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node, obj_size = SKB_HEAD_ALIGN(*size); if (obj_size <= SKB_SMALL_HEAD_CACHE_SIZE && !(flags & KMALLOC_NOT_NORMAL_BITS)) { - obj = kmem_cache_alloc_node(skb_small_head_cache, + obj = kmem_cache_alloc_node(net_hotdata.skb_small_head_cache, flags | __GFP_NOMEMALLOC | __GFP_NOWARN, node); *size = SKB_SMALL_HEAD_CACHE_SIZE; @@ -586,7 +582,7 @@ static void *kmalloc_reserve(unsigned int *size, gfp_t flags, int node, goto out; /* Try again but now we are using pfmemalloc reserves */ ret_pfmemalloc = true; - obj = kmem_cache_alloc_node(skb_small_head_cache, flags, node); + obj = kmem_cache_alloc_node(net_hotdata.skb_small_head_cache, flags, node); goto out; } @@ -649,7 +645,7 @@ struct sk_buff *__alloc_skb(unsigned int size, gfp_t gfp_mask, u8 *data; cache = (flags & SKB_ALLOC_FCLONE) - ? skbuff_fclone_cache : skbuff_cache; + ? net_hotdata.skbuff_fclone_cache : net_hotdata.skbuff_cache; if (sk_memalloc_socks() && (flags & SKB_ALLOC_RX)) gfp_mask |= __GFP_MEMALLOC; @@ -1095,7 +1091,7 @@ static int skb_pp_frag_ref(struct sk_buff *skb) static void skb_kfree_head(void *head, unsigned int end_offset) { if (end_offset == SKB_SMALL_HEAD_HEADROOM) - kmem_cache_free(skb_small_head_cache, head); + kmem_cache_free(net_hotdata.skb_small_head_cache, head); else kfree(head); } @@ -1162,7 +1158,7 @@ static void kfree_skbmem(struct sk_buff *skb) switch (skb->fclone) { case SKB_FCLONE_UNAVAILABLE: - kmem_cache_free(skbuff_cache, skb); + kmem_cache_free(net_hotdata.skbuff_cache, skb); return; case SKB_FCLONE_ORIG: @@ -1183,7 +1179,7 @@ static void kfree_skbmem(struct sk_buff *skb) if (!refcount_dec_and_test(&fclones->fclone_ref)) return; fastpath: - kmem_cache_free(skbuff_fclone_cache, fclones); + kmem_cache_free(net_hotdata.skbuff_fclone_cache, fclones); } void skb_release_head_state(struct sk_buff *skb) @@ -1280,7 +1276,7 @@ static void kfree_skb_add_bulk(struct sk_buff *skb, sa->skb_array[sa->skb_count++] = skb; if (unlikely(sa->skb_count == KFREE_SKB_BULK_SIZE)) { - kmem_cache_free_bulk(skbuff_cache, KFREE_SKB_BULK_SIZE, + kmem_cache_free_bulk(net_hotdata.skbuff_cache, KFREE_SKB_BULK_SIZE, sa->skb_array); sa->skb_count = 0; } @@ -1305,7 +1301,7 @@ kfree_skb_list_reason(struct sk_buff *segs, enum skb_drop_reason reason) } if (sa.skb_count) - kmem_cache_free_bulk(skbuff_cache, sa.skb_count, sa.skb_array); + kmem_cache_free_bulk(net_hotdata.skbuff_cache, sa.skb_count, sa.skb_array); } EXPORT_SYMBOL(kfree_skb_list_reason); @@ -1467,9 +1463,9 @@ static void napi_skb_cache_put(struct sk_buff *skb) if (unlikely(nc->skb_count == NAPI_SKB_CACHE_SIZE)) { for (i = NAPI_SKB_CACHE_HALF; i < NAPI_SKB_CACHE_SIZE; i++) kasan_mempool_unpoison_object(nc->skb_cache[i], - kmem_cache_size(skbuff_cache)); + kmem_cache_size(net_hotdata.skbuff_cache)); - kmem_cache_free_bulk(skbuff_cache, NAPI_SKB_CACHE_HALF, + kmem_cache_free_bulk(net_hotdata.skbuff_cache, NAPI_SKB_CACHE_HALF, nc->skb_cache + NAPI_SKB_CACHE_HALF); nc->skb_count = NAPI_SKB_CACHE_HALF; } @@ -2066,7 +2062,7 @@ struct sk_buff *skb_clone(struct sk_buff *skb, gfp_t gfp_mask) if (skb_pfmemalloc(skb)) gfp_mask |= __GFP_MEMALLOC; - n = kmem_cache_alloc(skbuff_cache, gfp_mask); + n = kmem_cache_alloc(net_hotdata.skbuff_cache, gfp_mask); if (!n) return NULL; @@ -5005,7 +5001,7 @@ static void skb_extensions_init(void) {} void __init skb_init(void) { - skbuff_cache = kmem_cache_create_usercopy("skbuff_head_cache", + net_hotdata.skbuff_cache = kmem_cache_create_usercopy("skbuff_head_cache", sizeof(struct sk_buff), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC| @@ -5013,7 +5009,7 @@ void __init skb_init(void) offsetof(struct sk_buff, cb), sizeof_field(struct sk_buff, cb), NULL); - skbuff_fclone_cache = kmem_cache_create("skbuff_fclone_cache", + net_hotdata.skbuff_fclone_cache = kmem_cache_create("skbuff_fclone_cache", sizeof(struct sk_buff_fclones), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC, @@ -5022,7 +5018,7 @@ void __init skb_init(void) * struct skb_shared_info is located at the end of skb->head, * and should not be copied to/from user. */ - skb_small_head_cache = kmem_cache_create_usercopy("skbuff_small_head", + net_hotdata.skb_small_head_cache = kmem_cache_create_usercopy("skbuff_small_head", SKB_SMALL_HEAD_CACHE_SIZE, 0, SLAB_HWCACHE_ALIGN | SLAB_PANIC, @@ -5895,7 +5891,7 @@ void kfree_skb_partial(struct sk_buff *skb, bool head_stolen) { if (head_stolen) { skb_release_head_state(skb); - kmem_cache_free(skbuff_cache, skb); + kmem_cache_free(net_hotdata.skbuff_cache, skb); } else { __kfree_skb(skb); } diff --git a/net/core/xdp.c b/net/core/xdp.c index 0e3709a29175baf3ee90fdc9cc2d97d861da11a3..41693154e426f78656cc18fc97ca7e82e648270b 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -16,6 +16,7 @@ #include #include +#include #include #include /* struct xdp_mem_allocator */ #include @@ -589,7 +590,7 @@ EXPORT_SYMBOL_GPL(xdp_warn); int xdp_alloc_skb_bulk(void **skbs, int n_skb, gfp_t gfp) { - n_skb = kmem_cache_alloc_bulk(skbuff_cache, gfp, n_skb, skbs); + n_skb = kmem_cache_alloc_bulk(net_hotdata.skbuff_cache, gfp, n_skb, skbs); if (unlikely(!n_skb)) return -ENOMEM; @@ -658,7 +659,7 @@ struct sk_buff *xdp_build_skb_from_frame(struct xdp_frame *xdpf, { struct sk_buff *skb; - skb = kmem_cache_alloc(skbuff_cache, GFP_ATOMIC); + skb = kmem_cache_alloc(net_hotdata.skbuff_cache, GFP_ATOMIC); if (unlikely(!skb)) return NULL; From patchwork Tue Mar 5 16:04:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582624 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 191A512A16B for ; Tue, 5 Mar 2024 16:04:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654686; cv=none; b=JAsdZY26qsrfRgPIkhTHUC5wREidGzRXoHbauctbJcqSXUiimQZpBXG5VIeaJ0RBcTG8fls9ZNDC7j6ZMgYoGKTauiGMVzGKXzuvTWCe/Lpp7x2RtgA7j8tT0b6VwMslsoWHITDdIR7Q0rKsg2wATI6kIjHQZkx3/tYaXB7HbEA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654686; c=relaxed/simple; bh=DYBb33Z/UVRcDZcDfSjIuQmUv3x7A0LmtslLfcKZ4z8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XTybyGk6N5B3UNF866200lyw8cySoEZHDPlB2ncGWn2BPN0uETadnguovrdjFq7bt6vk+leJVZCHopWjtbmBF0n3uUylowMbfPvTVUj69Fj5840XY7Ytvub/x0rtTqtqzaZ00sqrL3V//tvVTp3yL8T0trUBlxFWyxR2n0kmO0k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EfWNw97Y; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EfWNw97Y" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-d9a541b720aso10674395276.0 for ; Tue, 05 Mar 2024 08:04:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654672; x=1710259472; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ml+T6Ku/RpBJ6TPqtwbz51dgmo/A7GQ9j0i5zvMTCTc=; b=EfWNw97Y8GzqyiKaEvHN2Bz4IdIqqJdw9/hsS1PV2iyhlp5fGu8W+ZCzm78cCTHN5r hj9QgmqtAUq7eObH+lKduEjTtO0bX23ubK7zjNhOvusR9VM1VNBc4z7zFLROSPkzSH5i n1dRJpJkIwu77OwOroU38ZgEzpeXzU0zTpbRd4kk7B3pUC4Hu4LFO2VYM0oQKOEttydD PtHQallGPoLQoUL1Hg/7eUC6MHsT+AG+YmK6LRQqWDCqW/QS3DCUzVlmWo7gy4+a4+2i LSi/d/VT/y48i9r3gK617WfpPkS7b2j3vPSyxuv0uUGJN4G6284Op73QI9gt2E6lu17F Bmww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654672; x=1710259472; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ml+T6Ku/RpBJ6TPqtwbz51dgmo/A7GQ9j0i5zvMTCTc=; b=Yjnoxt98Oa0x/IWWk2+G4XWd85Kkrk63esu7akemnagktWxMdIN0qWkTFf3SgumlE7 L480FcMPBqJAUFkhrtbJ3L9j+NrDMj10YTtgzePwgd6zNzMXSMr5kI0TbuyafGOWi5n8 B+yCtJJFhSSyPFB42lwvI8HUm8MjTxn6W/A4Xq7be2SYpAhSX1zPwYTImS/7a/sdOm4B 8njdGPjReP7zXElrARRgKQTQjWdhwfwAYoMigQxMd2lnaN8wO7KgQM/yMgYN86uAMkFW +j9qqvF2iS5r3sXLJnXxLzDMjVaojaZWHXZEs+MkAsjzWb3dvSgj8x4tRdEl8/KiyZqZ FI+w== X-Gm-Message-State: AOJu0Yy0GNFwdKLYz3XQqZyClPW2clQ0OoWzyFY3mmcHTXVULlqnv2eE aSiATQqxVkUxxDvAa3JbSb/EXkgp06RLzIOoEuVzkocMgRnb3yf3vxObEaNA64NED3EmDp1QqaC CnMcGtP/VKA== X-Google-Smtp-Source: AGHT+IEhDi5G8bUSHeErNh+sj84jAoNG7jvaaXga2zkZ02i8ofaTj+I/DZTU5ouyWHqdRNjLNXhbvF0eRqc88A== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1004:b0:dc7:42:ecd with SMTP id w4-20020a056902100400b00dc700420ecdmr3125522ybt.6.1709654672270; Tue, 05 Mar 2024 08:04:32 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:06 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-12-edumazet@google.com> Subject: [PATCH net-next 11/18] udp: move udpv4_offload and udpv6_offload to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org These structures are used in GRO and GSO paths. Move them to net_hodata for better cache locality. Signed-off-by: Eric Dumazet --- include/net/hotdata.h | 2 ++ net/ipv4/udp_offload.c | 17 ++++++++--------- net/ipv6/udp_offload.c | 21 ++++++++++----------- 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 728aee1cf07c8f0d85873d912248a99e148f84b1..086114e4d3bcc9184304b024ed93c9024888fcf1 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -11,10 +11,12 @@ struct net_hotdata { #if IS_ENABLED(CONFIG_INET) struct packet_offload ip_packet_offload; struct net_offload tcpv4_offload; + struct net_offload udpv4_offload; #endif #if IS_ENABLED(CONFIG_IPV6) struct packet_offload ipv6_packet_offload; struct net_offload tcpv6_offload; + struct net_offload udpv6_offload; #endif struct list_head offload_base; struct list_head ptype_all; diff --git a/net/ipv4/udp_offload.c b/net/ipv4/udp_offload.c index 6c95d28d0c4a7e56d587a986113b3711f8de964c..b9880743765c6c24c28bea095f16f0cf091664ce 100644 --- a/net/ipv4/udp_offload.c +++ b/net/ipv4/udp_offload.c @@ -737,15 +737,14 @@ INDIRECT_CALLABLE_SCOPE int udp4_gro_complete(struct sk_buff *skb, int nhoff) return udp_gro_complete(skb, nhoff, udp4_lib_lookup_skb); } -static const struct net_offload udpv4_offload = { - .callbacks = { - .gso_segment = udp4_ufo_fragment, - .gro_receive = udp4_gro_receive, - .gro_complete = udp4_gro_complete, - }, -}; - int __init udpv4_offload_init(void) { - return inet_add_offload(&udpv4_offload, IPPROTO_UDP); + net_hotdata.udpv4_offload = (struct net_offload) { + .callbacks = { + .gso_segment = udp4_ufo_fragment, + .gro_receive = udp4_gro_receive, + .gro_complete = udp4_gro_complete, + }, + }; + return inet_add_offload(&net_hotdata.udpv4_offload, IPPROTO_UDP); } diff --git a/net/ipv6/udp_offload.c b/net/ipv6/udp_offload.c index 6b95ba241ebe2af7e5f2760d8a9c1d78f08579c5..312bcaeea96fb78ac488124cf7795aa834392c64 100644 --- a/net/ipv6/udp_offload.c +++ b/net/ipv6/udp_offload.c @@ -192,20 +192,19 @@ INDIRECT_CALLABLE_SCOPE int udp6_gro_complete(struct sk_buff *skb, int nhoff) return udp_gro_complete(skb, nhoff, udp6_lib_lookup_skb); } -static const struct net_offload udpv6_offload = { - .callbacks = { - .gso_segment = udp6_ufo_fragment, - .gro_receive = udp6_gro_receive, - .gro_complete = udp6_gro_complete, - }, -}; - -int udpv6_offload_init(void) +int __init udpv6_offload_init(void) { - return inet6_add_offload(&udpv6_offload, IPPROTO_UDP); + net_hotdata.udpv6_offload = (struct net_offload) { + .callbacks = { + .gso_segment = udp6_ufo_fragment, + .gro_receive = udp6_gro_receive, + .gro_complete = udp6_gro_complete, + }, + }; + return inet6_add_offload(&net_hotdata.udpv6_offload, IPPROTO_UDP); } int udpv6_offload_exit(void) { - return inet6_del_offload(&udpv6_offload, IPPROTO_UDP); + return inet6_del_offload(&net_hotdata.udpv6_offload, IPPROTO_UDP); } From patchwork Tue Mar 5 16:04:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582623 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E305012A16C for ; Tue, 5 Mar 2024 16:04:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654690; cv=none; b=USE0P5Qv9sAiEfpIQKs7KGDdrbzp75n94eqsTilN6rMtX8iJPrfEUd2MbMCULeT7956Ociw7k+Fetx5lGPMqKnI7DXvOH4tv4QD7CslscYuJNg5Iiz+YdZ1+nAe5FYukDIDFtv87a7uvxEDHN/CpGuCDnPPYvRoAIHj0FS7HzRU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654690; c=relaxed/simple; bh=r+JAa2ZCZuzUlZ5+j7doFlqH2VLOA5hH0qAehVbTY/I=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=smiyx4MuuaA1u8AwP+T2v+ooGvLKJ2IsaR+iv4+8vSknLQGCJpnJVEChNjlVjF381PAkmGlSGwDfDToie778dl0cSMSgfB7X7KwZUKyRm0oqu9luDx6U3ZOJOATVg1L7eZ0qJsed/URfk4PZYdCsTI+lR21hS5UuyzRdeVfk2ew= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=RuOjrpGA; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="RuOjrpGA" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6b269686aso8633254276.1 for ; Tue, 05 Mar 2024 08:04:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654674; x=1710259474; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BP/NiSx0xOGEgpLnrjFAiBUOlZjPqz/JsAge1tk5TKI=; b=RuOjrpGAj+G3ERObc3q/T2cjELp0hDqeYVTcicsRAQmnQwEOo7ncu5ueuAhMh9dgd3 R2IGa9ZIjd0lS2zZC9QmRJxuzlmfG2JxCCAkonN2Gc9CSho6cMVAV/vNmfSI3zYjVt00 daMPrBq8q+ZmZJIlCmbYDRkA4Evy9sIzoc2JrC+Fy6NNypLMPi1Br6O9+F9RARQ0br5u MoA9NOy4NWA0JcghzVnOvZGmn23z9auzVxAAw/73cpMrMEINK7jnt4ZJP0VhpHEjDnly 80mzhlnAudyanuTJhylmatukG/npnUbXq7ihNaYmq5e9OV9ga4t7rya4fRI1RBPMCccz 9VoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654674; x=1710259474; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BP/NiSx0xOGEgpLnrjFAiBUOlZjPqz/JsAge1tk5TKI=; b=Tz+0uNra08Wf9OJ8P5mFYSeHtzQxU0JJwtPzyPyFmw2/CDrimHycqfD0qy3vhGwFE5 vxyI/g/1Unb3zaTVOhmUiJ5wZHJRzRY4gDr6uSKhguCixvEO7u/2LgYrvcKXRMeK22Np Gncx0/mHhB+YVfkRayIP+MR4RODxu9E6aaCHZia8Sv9lqUj8+waQxfr8TVVL7ITTNSTK wPZkTvd6yZW6fX8xnOhZz97tygKeV+ecRHKEMj5tuWeVlBY0JVw9a2ymqLEwtWpkoiRJ IM9U8Wp+MzQmAROl86QNDGq0Xb4PLogYPP0ZK4EpcGojXKamnH6w+TmulcxaA077pU8W kjmQ== X-Gm-Message-State: AOJu0YwG6s4QTnVS42KR4NW+T2AtpOwBGx3mJHxyo4eIoluCxaJ+EtSo H15vwB6XIL0BAQ7GYybINW61yuvSvyMPBjc7ajIXWv6DicGFiLkF1rXSkGCcKPm+3L20qnTQvYm bMamLHrt4Pg== X-Google-Smtp-Source: AGHT+IE3AbpGzdM17tI9UVSEIDUauX16zYWFGTSlS7ldhfccc8dKfWKhtqDDt6eGB8SCf/QUl2twx2epeiL7DQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:124b:b0:dc6:dfd9:d423 with SMTP id t11-20020a056902124b00b00dc6dfd9d423mr526001ybu.3.1709654673966; Tue, 05 Mar 2024 08:04:33 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:07 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-13-edumazet@google.com> Subject: [PATCH net-next 12/18] ipv6: move tcpv6_protocol and udpv6_protocol to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org These structures are read in rx path, move them to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet --- include/net/hotdata.h | 2 ++ net/ipv6/tcp_ipv6.c | 17 +++++++++-------- net/ipv6/udp.c | 16 ++++++++-------- 3 files changed, 19 insertions(+), 16 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 086114e4d3bcc9184304b024ed93c9024888fcf1..97617acb75e1f2141fe7170d93c06f9813c725a3 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -16,7 +16,9 @@ struct net_hotdata { #if IS_ENABLED(CONFIG_IPV6) struct packet_offload ipv6_packet_offload; struct net_offload tcpv6_offload; + struct inet6_protocol tcpv6_protocol; struct net_offload udpv6_offload; + struct inet6_protocol udpv6_protocol; #endif struct list_head offload_base; struct list_head ptype_all; diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index f677f0fa51968d00c3571d55ae7850742387f2d1..3f4cba49e9ee6520987993dcea082e6065b4688b 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -58,6 +58,7 @@ #include #include #include +#include #include #include @@ -2367,11 +2368,6 @@ struct proto tcpv6_prot = { }; EXPORT_SYMBOL_GPL(tcpv6_prot); -static const struct inet6_protocol tcpv6_protocol = { - .handler = tcp_v6_rcv, - .err_handler = tcp_v6_err, - .flags = INET6_PROTO_NOPOLICY|INET6_PROTO_FINAL, -}; static struct inet_protosw tcpv6_protosw = { .type = SOCK_STREAM, @@ -2408,7 +2404,12 @@ int __init tcpv6_init(void) { int ret; - ret = inet6_add_protocol(&tcpv6_protocol, IPPROTO_TCP); + net_hotdata.tcpv6_protocol = (struct inet6_protocol) { + .handler = tcp_v6_rcv, + .err_handler = tcp_v6_err, + .flags = INET6_PROTO_NOPOLICY | INET6_PROTO_FINAL, + }; + ret = inet6_add_protocol(&net_hotdata.tcpv6_protocol, IPPROTO_TCP); if (ret) goto out; @@ -2433,7 +2434,7 @@ int __init tcpv6_init(void) out_tcpv6_protosw: inet6_unregister_protosw(&tcpv6_protosw); out_tcpv6_protocol: - inet6_del_protocol(&tcpv6_protocol, IPPROTO_TCP); + inet6_del_protocol(&net_hotdata.tcpv6_protocol, IPPROTO_TCP); goto out; } @@ -2441,5 +2442,5 @@ void tcpv6_exit(void) { unregister_pernet_subsys(&tcpv6_net_ops); inet6_unregister_protosw(&tcpv6_protosw); - inet6_del_protocol(&tcpv6_protocol, IPPROTO_TCP); + inet6_del_protocol(&net_hotdata.tcpv6_protocol, IPPROTO_TCP); } diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 3f2249b4cd5f6a594dd9768e29f20f0d9a57faed..97d86909aabb6588d0bba901f6df1f23a4f2e561 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -1702,11 +1702,6 @@ int udpv6_getsockopt(struct sock *sk, int level, int optname, return ipv6_getsockopt(sk, level, optname, optval, optlen); } -static const struct inet6_protocol udpv6_protocol = { - .handler = udpv6_rcv, - .err_handler = udpv6_err, - .flags = INET6_PROTO_NOPOLICY|INET6_PROTO_FINAL, -}; /* ------------------------------------------------------------------------ */ #ifdef CONFIG_PROC_FS @@ -1803,7 +1798,12 @@ int __init udpv6_init(void) { int ret; - ret = inet6_add_protocol(&udpv6_protocol, IPPROTO_UDP); + net_hotdata.udpv6_protocol = (struct inet6_protocol) { + .handler = udpv6_rcv, + .err_handler = udpv6_err, + .flags = INET6_PROTO_NOPOLICY | INET6_PROTO_FINAL, + }; + ret = inet6_add_protocol(&net_hotdata.udpv6_protocol, IPPROTO_UDP); if (ret) goto out; @@ -1814,12 +1814,12 @@ int __init udpv6_init(void) return ret; out_udpv6_protocol: - inet6_del_protocol(&udpv6_protocol, IPPROTO_UDP); + inet6_del_protocol(&net_hotdata.udpv6_protocol, IPPROTO_UDP); goto out; } void udpv6_exit(void) { inet6_unregister_protosw(&udpv6_protosw); - inet6_del_protocol(&udpv6_protocol, IPPROTO_UDP); + inet6_del_protocol(&net_hotdata.udpv6_protocol, IPPROTO_UDP); } From patchwork Tue Mar 5 16:04:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582627 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1FD04128801 for ; Tue, 5 Mar 2024 16:04:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654691; cv=none; b=XdHACwmDlrPLZaQmKrWS9sEwMpV1MOLNY0GmiqVplBaAxCW7wQcBzlWkbpHI4Fi1Lfi+3/xhdwOdpMDGn5LQOF+cyi0YsXUowtKJBmhqHrLd+KG9ETPLaBk+B7QbRzvRFHmwf+lRqlyXyIk6DTYP7SOKkYRzvMlOKKB/bLieueM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654691; c=relaxed/simple; bh=gvy+5YrH62b/erUFJ2UXKwPrjN6eRVv04Yn7bkAJCdE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pd2S1ezb+b5UVZHjUMEUlKWTy2I4bOLXvaon37p2yo0QDhpVFszRBhsO0Qulo9Z+3x4WH9O4dVojq9r2hcsK9uAqEyE2HiJazZVeb1XSET9aqc2037qHhS3RNZ2Puga1BjuWWyDzgLX6mtDpNzSPQWsqs24j/It/PgTHZ6LDcuY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=SLrvxvWI; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="SLrvxvWI" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dced704f17cso9809142276.1 for ; Tue, 05 Mar 2024 08:04:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654675; x=1710259475; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uPkfSDEDaD+ot1oycTy547baCEwC3QtbZqcPHtahwi0=; b=SLrvxvWIi1doFSzEpeZWC7Z9BAZ6q7E8SQRdFoqKsfRWD//L8u14lrq7llC4uTAzFg oh9c+Oy2eI/grlfyU0MR2E3aydXtalDZDQL37q2rYHroTTJa6UQLvUCuw7g2wz7i2b2h XdrKE7gbxEWMS7Ij8FSLuZSxwbLMULPDRGoSTXUhQZOYaB1x5KHRVg2+ln3cQSIa6O8d +UJZkHisqTLzaKXdSX2GQ+g/BvTV7BwkbQEmYr5VfSrmkI3FHKOlvFV0f5NTHxIPiEdP 8XnyZmTKzMQj9YsFLywhXg0mt9m0g73eKfgkfO2prQbmL6BAWmxDtdcIjbgdujLoujd+ 6osw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654675; x=1710259475; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uPkfSDEDaD+ot1oycTy547baCEwC3QtbZqcPHtahwi0=; b=XV5lF1aU4g5dOuEpO126MxOU8T/Zpiy/rytEd4wQQfcA3arA0/qGAVEPVonjQ388Pm Jrme3b4ForaPUf29xnOqxI9/on9xRWVorrQM2aKMQIe7/ZPBlFFdGUBc7LZ/QS5jYmCp ix9BjhQfFbPw/E396CGsj5wIWxBAPgra9PUbviJgF5mItSghn5YDvkt/mjjF0XCNC8hZ XpJe+7uEROYjFyLlcXuW+51xnUbojmcvqXyzdQeoougAn4t+z8buHfdy/bJwo8WBwqEZ GEnMEl7utiqi9rHVejahnWIoXf32SZxfrWLGfTdsu2bmRMOoo2hIgVnGqljuQoBqMJZc zwaQ== X-Gm-Message-State: AOJu0Yze3kBhF+SEK3/qglLxFrrIkU7JxyUYPMfROLGxw3Q4h6b4cf0J 1oHG0QHokcveoBj9sC0rp+7yeJn2m44hVDP/n8gD+faw1IEbn7ZUN01bPGtbSlHNrMMts6Oukeg qdTMYLmUOlw== X-Google-Smtp-Source: AGHT+IEDdaYTVAMn+9+o9ytS/uLNGtu5wGyDYhiNG1kONQkuPsByYt6TD8z4jfHVOwAm9UGpCcoQvb26418UAA== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1081:b0:dcc:41ad:fb3b with SMTP id v1-20020a056902108100b00dcc41adfb3bmr475163ybu.10.1709654675207; Tue, 05 Mar 2024 08:04:35 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:08 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-14-edumazet@google.com> Subject: [PATCH net-next 13/18] inet: move tcp_protocol and udp_protocol to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org These structures are read in rx path, move them to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet --- include/net/hotdata.h | 2 ++ net/ipv4/af_inet.c | 30 +++++++++++++++--------------- 2 files changed, 17 insertions(+), 15 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 97617acb75e1f2141fe7170d93c06f9813c725a3..4d1cb3c29d4edfbc18cf56c370a1e04e5fcb1cbd 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -11,7 +11,9 @@ struct net_hotdata { #if IS_ENABLED(CONFIG_INET) struct packet_offload ip_packet_offload; struct net_offload tcpv4_offload; + struct net_protocol tcp_protocol; struct net_offload udpv4_offload; + struct net_protocol udp_protocol; #endif #if IS_ENABLED(CONFIG_IPV6) struct packet_offload ipv6_packet_offload; diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index 08dda6955562ea6b89e02b8299b03ab52b342f27..6f1cfd176e7b84f23d8a5e505bf8e13b2b755f06 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -1751,19 +1751,6 @@ static const struct net_protocol igmp_protocol = { }; #endif -static const struct net_protocol tcp_protocol = { - .handler = tcp_v4_rcv, - .err_handler = tcp_v4_err, - .no_policy = 1, - .icmp_strict_tag_validation = 1, -}; - -static const struct net_protocol udp_protocol = { - .handler = udp_rcv, - .err_handler = udp_err, - .no_policy = 1, -}; - static const struct net_protocol icmp_protocol = { .handler = icmp_rcv, .err_handler = icmp_err, @@ -1992,9 +1979,22 @@ static int __init inet_init(void) if (inet_add_protocol(&icmp_protocol, IPPROTO_ICMP) < 0) pr_crit("%s: Cannot add ICMP protocol\n", __func__); - if (inet_add_protocol(&udp_protocol, IPPROTO_UDP) < 0) + + net_hotdata.udp_protocol = (struct net_protocol) { + .handler = udp_rcv, + .err_handler = udp_err, + .no_policy = 1, + }; + if (inet_add_protocol(&net_hotdata.udp_protocol, IPPROTO_UDP) < 0) pr_crit("%s: Cannot add UDP protocol\n", __func__); - if (inet_add_protocol(&tcp_protocol, IPPROTO_TCP) < 0) + + net_hotdata.tcp_protocol = (struct net_protocol) { + .handler = tcp_v4_rcv, + .err_handler = tcp_v4_err, + .no_policy = 1, + .icmp_strict_tag_validation = 1, + }; + if (inet_add_protocol(&net_hotdata.tcp_protocol, IPPROTO_TCP) < 0) pr_crit("%s: Cannot add TCP protocol\n", __func__); #ifdef CONFIG_IP_MULTICAST if (inet_add_protocol(&igmp_protocol, IPPROTO_IGMP) < 0) From patchwork Tue Mar 5 16:04:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582626 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qk1-f202.google.com (mail-qk1-f202.google.com [209.85.222.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 12528128805 for ; Tue, 5 Mar 2024 16:04:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654693; cv=none; b=YmON66WLhs7+BaQpKUfQvLeMmrA9xPDwU2HMnaCMzez5UcJKEJ9wEDGPuDZ/QNeYDkr2r0GcfhnMb604AScFxBgLqF+L8XtZmrAZKw/ZaeOXPBhXuKVWhV06jleJhhASQ2OApV8TmM6tWBHCwtpxg6D4Qe8knJ43R4odGPFSLdY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654693; c=relaxed/simple; bh=e0+PLrT73nEBv75eicC4a+8lpbZ9f7PeNh5w4yzgsKo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=PJFC1Vo08Xu1eqcYtk1D/MJWB8sEOrydNxt2JhUTYlPJrKSVDQ4SG9F+qCzIESfp0zdpu9Ce5iZ7ipC4BkmTqQDoTNmKOxmKuWFj+6UpoLACVQ2dvv2gFbGRRsHL4Q5Hg+S+SwMHM+qShHbShO/vlbbVvFzhsP+UdPeac9DxqMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=X1xlMR8b; arc=none smtp.client-ip=209.85.222.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="X1xlMR8b" Received: by mail-qk1-f202.google.com with SMTP id af79cd13be357-78825e47824so201372685a.1 for ; Tue, 05 Mar 2024 08:04:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654677; x=1710259477; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gZ2xzFYq3fEPMpr/MUwgp+HG/Vy0UXwzCdvLWEnfl/o=; b=X1xlMR8b8EsV4A9PS7+F4GVrVkY8SO8hf8PeXzTC0NK6jFKFh2fPqRYqPlrtQm4ieM OKD+85MI8ym/9rEg/ZR4WqDgeoDrRgunKrgiH23tWLUOlyQANU+s7a79RNP49Ip4Y8OX +zUGF0mOdI86A2ldoVBMDoCSAq0xdY06AphIqYBAZPIGPJ1hk3OwvEYGIc4pdyNayR1W ggo7gQj4iLisGVp+vl8pg7hFdfIJW63tMxEoGaLlzis6v/1StpSQvDkqsK5tWgKo2LSj n12lbL3JrFQMWZ+03IzQL9BkoPBYBSB1U3xdVz8q87HR62Zo6+3X0edmbT6oKYljiFvK T3lA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654677; x=1710259477; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gZ2xzFYq3fEPMpr/MUwgp+HG/Vy0UXwzCdvLWEnfl/o=; b=DPEWlQBi9gLj/1ci8kwxe0bnO0BpGmIZPfq1W/ZMEoBSGOqhTVorjj2qSFL3scc2Yz yTIeA7D9zD+LoNxcGP9SR3rVdmzizo1BXwSbPjgpJxDPqyXb5HRAOCa7suEKn6hYiBDk P5maXaQ40O5oAHjvll0TMeJhCFSgqXMZztbFGZVVN+80xyWAiLl6CsppIWe3cYbN3pSP FNQZl6bfnmycxvq4dl7ZF7GHlBMiNu5C71Lr6vnFOL2aLBMcB3dVui3g/9sriPqqOm2r 1hUGW6ayfCdpBMOJqYV4podHGy/fuUPM0lNhNdQ8Gl/mb78srHWcmNHDrLMJSMJasU6G n/lg== X-Gm-Message-State: AOJu0YzmIScF1BVPW4poFrCRziEh/OYnDCooT9egOrEPbRo+LT6guPzt hWtZV1W+PuNp3n+Ime1rXrriNCEsWiEy401C4BGy+GruDCP/ST0DeFrNGggW/4AJjkJVEAEoInS 1uwc8fLLcNg== X-Google-Smtp-Source: AGHT+IGxJI2wAtvikhj+WpuERa04vwzy+AKZ+7ZxJrtW2zxWF2lTVgBoTQA2b5+vzKcHQKBHzEAVsfgOXFd1ew== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:620a:1a04:b0:788:13c3:dad7 with SMTP id bk4-20020a05620a1a0400b0078813c3dad7mr148662qkb.8.1709654677177; Tue, 05 Mar 2024 08:04:37 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:09 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-15-edumazet@google.com> Subject: [PATCH net-next 14/18] inet: move inet_ehash_secret and udp_ehash_secret into net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org "struct net_protocol" has a 32bit hole in 32bit arches. Use it to store the 32bit secret used by UDP and TCP, to increase cache locality in rx path. Signed-off-by: Eric Dumazet --- include/net/hotdata.h | 3 +++ include/net/protocol.h | 1 + net/ipv4/inet_hashtables.c | 3 +-- net/ipv4/udp.c | 2 -- 4 files changed, 5 insertions(+), 4 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 4d1cb3c29d4edfbc18cf56c370a1e04e5fcb1cbd..4dd86be99116ff83f2524461a006565b2ade2241 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -36,6 +36,9 @@ struct net_hotdata { int dev_rx_weight; }; +#define inet_ehash_secret net_hotdata.tcp_protocol.secret +#define udp_ehash_secret net_hotdata.udp_protocol.secret + extern struct net_hotdata net_hotdata; #endif /* _NET_HOTDATA_H */ diff --git a/include/net/protocol.h b/include/net/protocol.h index 6aef8cb11cc8c409e5f7a2519f5e747be584c8d5..3ff26e66735cec98b08eadb1c3f129e011923cb0 100644 --- a/include/net/protocol.h +++ b/include/net/protocol.h @@ -46,6 +46,7 @@ struct net_protocol { * socket lookup? */ icmp_strict_tag_validation:1; + u32 secret; }; #if IS_ENABLED(CONFIG_IPV6) diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c index 308ff34002ea6b5e0620004f65ffd833087afbc1..7498af3201647fd937bf8177f04c200bea178a79 100644 --- a/net/ipv4/inet_hashtables.c +++ b/net/ipv4/inet_hashtables.c @@ -24,6 +24,7 @@ #include #endif #include +#include #include #include #include @@ -32,8 +33,6 @@ u32 inet_ehashfn(const struct net *net, const __be32 laddr, const __u16 lport, const __be32 faddr, const __be16 fport) { - static u32 inet_ehash_secret __read_mostly; - net_get_random_once(&inet_ehash_secret, sizeof(inet_ehash_secret)); return __inet_ehashfn(laddr, lport, faddr, fport, diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index a8acea17b4e5344d022ae8f8eb674d1a36f8035a..2beabf5b2d8628f1fed69a0212c57bd3cd638483 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -411,8 +411,6 @@ INDIRECT_CALLABLE_SCOPE u32 udp_ehashfn(const struct net *net, const __be32 laddr, const __u16 lport, const __be32 faddr, const __be16 fport) { - static u32 udp_ehash_secret __read_mostly; - net_get_random_once(&udp_ehash_secret, sizeof(udp_ehash_secret)); return __inet_ehashfn(laddr, lport, faddr, fport, From patchwork Tue Mar 5 16:04:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582628 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7C3F1292C6 for ; Tue, 5 Mar 2024 16:04:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654695; cv=none; b=Kfn9VdNNVGB4Q4gJFFyVZhRdhKQ/zV07uwJHypAkZtvtr6q9YDQRXqxfV52x/qBTnG/a3kvhIsoxr7TWbT9PHU3raXUyZ3NubZn/IwkYCsNr7IXoy63y6YICQVPfiBuEhlzoRFow58yPVQavtOimE2xIr/Vm3QT8mMX1JixbpI0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654695; c=relaxed/simple; bh=8lQlopQP1vQVZnm4MX7x/WGwOt74+X00EmL1jeuL3Wg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Q9PzC6LFTTEUCLih6kvh6GbH9SwcdUqqGFKGsPW0/7A6cHt7VESDMMomqawPMA2FnLwYxZo8hKkflQEOfohp1ZLtkNOMkpnNb77IXBI7ipLFtxrJcjG0j2F6ITvyctNRSNQnlOHaJdr3j5Z28PxAXu5GlTdUaiXCO75aMNMylWQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=v3MLJfAC; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="v3MLJfAC" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dbf618042daso8836124276.0 for ; Tue, 05 Mar 2024 08:04:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654679; x=1710259479; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=J812lKlAoiPYdEqSHKAyJuB8QWUBUqNQavlJ1/QHB4Q=; b=v3MLJfACrONnDBM5zRMAqVi/7RDe32TLAfRSpGhY48AEmZnzSfQtafKa8wJ+G83upz Ijb2F4wkOlX0OiRggauj6OpuJdSUcAqcZE4EuqDVIb2ZKlxUuYd5ebuHSt/E9Dcq0MuV rSxvw0oG6U/j1vDnihiihKpu/8hlN+bG0jR8Z8mFVPCpPBXLJ3FHm4EzxuQKlmx2sR8v rx1vaFmrbQYnKYkqZ68p55AJV/PqnAtUqM/S1IHMtzH+mP0rW6MsJOwsMbT86qAKapMl Sk15i46ZVZ/92BgrwHUJGtNHpNbx4IJ3QCsxg4oJmUsgqzVjj23bUVnDdvEQvHc8/BJd INMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654679; x=1710259479; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=J812lKlAoiPYdEqSHKAyJuB8QWUBUqNQavlJ1/QHB4Q=; b=EpUrRbpABl6r13NTzVe/tsc2m8yoyEHDlpf0RttGwVEaI5J0tkYESMeG4KehHFCALf plw6slW6au8lIbtc6PONk3imV0OlAUWEpi+XluZCFoXfYSSeCALp8LYFRsQmJzrQHcU6 2nozBxQZQBrSXRDwMdKyfod5ZIxMVP29a70ZzwLwZevpY+UvOUBKo6G4ztKJQlw2GI9q O2h5iI/liB90Qx+FecipQBxRobTSlPSHf+NRzk1Zpl8s849TzWMwYpkzAs4VjMyrGsEs mWDfx1hmmu4amD3VRKfSCkQoWVsEAgAUrC56XYd+swr3Nxq1DAyfQaC/C/drvMWQo4+m VcMA== X-Gm-Message-State: AOJu0YyLKN2a2Ge3yuAFx6ywnivvF9UMoU8DiWA2qXAdM+csoJWOfWTb 3a7fVe/0DVuTzqZuQCiiV6SwB5N/OKyOW/4rjinE8YTtE2Sl2ykDqySRfZ0+DPQvyWK3g7gSQ0m dDRbFBtigPg== X-Google-Smtp-Source: AGHT+IGV/1LaaidQF4LXBK9bopigujJbsqID9K2WmoimL4rj9Kl+QVYDfqvb3N9Agufc7nICMEpLMpq1lZQTzQ== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1507:b0:dc7:59d9:7b46 with SMTP id q7-20020a056902150700b00dc759d97b46mr476959ybu.3.1709654679032; Tue, 05 Mar 2024 08:04:39 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:10 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-16-edumazet@google.com> Subject: [PATCH net-next 15/18] ipv6: move inet6_ehash_secret and udp6_ehash_secret into net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org "struct inet6_protocol" has a 32bit hole in 32bit arches. Use it to store the 32bit secret used by UDP and TCP, to increase cache locality in rx path. Signed-off-by: Eric Dumazet --- include/net/hotdata.h | 2 ++ include/net/protocol.h | 1 + net/ipv6/inet6_hashtables.c | 2 +- net/ipv6/udp.c | 1 - 4 files changed, 4 insertions(+), 2 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 4dd86be99116ff83f2524461a006565b2ade2241..e4dac2f859efd421b975a61360536af949046d0e 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -38,6 +38,8 @@ struct net_hotdata { #define inet_ehash_secret net_hotdata.tcp_protocol.secret #define udp_ehash_secret net_hotdata.udp_protocol.secret +#define inet6_ehash_secret net_hotdata.tcpv6_protocol.secret +#define udp6_ehash_secret net_hotdata.udpv6_protocol.secret extern struct net_hotdata net_hotdata; diff --git a/include/net/protocol.h b/include/net/protocol.h index 3ff26e66735cec98b08eadb1c3f129e011923cb0..213649d2ab098edc4eb9f44a5403540887c2b8ef 100644 --- a/include/net/protocol.h +++ b/include/net/protocol.h @@ -60,6 +60,7 @@ struct inet6_protocol { __be32 info); unsigned int flags; /* INET6_PROTO_xxx */ + u32 secret; }; #define INET6_PROTO_NOPOLICY 0x1 diff --git a/net/ipv6/inet6_hashtables.c b/net/ipv6/inet6_hashtables.c index b0e8d278e8a9b794d0001efdc0f43716f9a34f8f..0fee97f3166cf8326b3b714ac6bde48ca5188cec 100644 --- a/net/ipv6/inet6_hashtables.c +++ b/net/ipv6/inet6_hashtables.c @@ -14,6 +14,7 @@ #include #include +#include #include #include #include @@ -25,7 +26,6 @@ u32 inet6_ehashfn(const struct net *net, const struct in6_addr *laddr, const u16 lport, const struct in6_addr *faddr, const __be16 fport) { - static u32 inet6_ehash_secret __read_mostly; static u32 ipv6_hash_secret __read_mostly; u32 lhash, fhash; diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 97d86909aabb6588d0bba901f6df1f23a4f2e561..1e1c67a51675e8534a953a6d4d63211388d95ca9 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -79,7 +79,6 @@ u32 udp6_ehashfn(const struct net *net, const struct in6_addr *faddr, const __be16 fport) { - static u32 udp6_ehash_secret __read_mostly; static u32 udp_ipv6_hash_secret __read_mostly; u32 lhash, fhash; From patchwork Tue Mar 5 16:04:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582629 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83D5C12880D for ; Tue, 5 Mar 2024 16:04:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654697; cv=none; b=bXVIMkuezhMFdJlP42QxW1FMaiX7EcgFj2vH1Lea5BNBwDUTQ+TCuhNhmQxNuY6Y16pGb8byQJmqIgf368OUga8/SfF8Yu4AFsSc+M3ZEl17hz7evtB9NoxTs4kyucxioYt3+vjdapRJ9w0tjFXhBc+L6IJ8MzXIEXRsJJeezQ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654697; c=relaxed/simple; bh=wuJAK4u5Z43xMfjz8Ab/ZSWgWF3a+56F0FV9pa0Q9lU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ExhyjqCMBNWwcP80BwQatnuz3TB7HZEeg9X/vE52LGDWkS5O8WCjIt9nMm/f9YLd8Tb2ZBtEwoPig7tiwpLQ2xHTquzVCZSu7UJJcsYDsRV0+2ejaU0/SA9MRy9gVk12DjbFZ/d1n6PGeDF5QdeuPuuvOCaLObhyfjBK19hqcho= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XwRTFptb; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XwRTFptb" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60998466af4so51198667b3.3 for ; Tue, 05 Mar 2024 08:04:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654680; x=1710259480; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=a1vkV6Sni2i06OF9GaK5ZpbS2xl9biWoyHnzpUvKBKg=; b=XwRTFptbx0KFhVvpDcibO2Ek3aCjdC80ttvIjlrSujbMRzJAvx656VsuleHDQgHhx1 Vsp0rZniPefKaFWExnx201c0cwX1VPpJYZ8GIG1vjqf1ioiWIM08QZqUzqLHaLKZQo+2 xNTyFQvTcfL8fNXOXNz9dVEUxpJY2bA87LJ/Y03Dv0IX8nMYhA9zG1XWIYhxZCRUtrvZ r+CG9saWTj+dW4a4CK10KopMj/iSLCDlfC6672lE4mRWo6Nl1m8IPVetDb5/P+g7pkOJ XwNkA5KdJ1jySOZeFMvLMpvyNWmq2FRyHT4nQH/YrpgORRCgoPNcpyFLSuReQ+iHKEvV 1PPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654680; x=1710259480; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=a1vkV6Sni2i06OF9GaK5ZpbS2xl9biWoyHnzpUvKBKg=; b=rfSt+hXMbde/kOAKmM4Yr+ajoNEwDEtI4KtGPSC3S3ZWY+pcdwaUYLPM7f+vp0IQc5 YnCjPss6dmzTE/G7xKFO5kR2V1zRNFq6WStlzhy99g/8FNNb0SbgV6xYs/UT0ZKGpHC1 VNlEeaMXezvM8y9yWsdznmx+jGHd04PQrb5KR+6YVAm/gNKkOEmjY/FjJHwtloyoB7QJ 8keuCQNpfWjZc1kO/BjlV0LHqtpyI+zEA+4178xDmWek7L18XuDrvp1K4n6c+UJajbkt 2OfguJXXEYAZmuC9rm75trirYCPSUY7CPWsKd93FlGTge5APvgBeDRNYHEx6i8X63cSR GJfw== X-Gm-Message-State: AOJu0YxD3FBemOxGbGls+xRRHc2fzrXNJAtyEyyewZHKQOYeN3d1TsT2 cW2ATRM+5NH8LP/ZM6UFEaBXuLRHzFa9y+5kZ62HeUxbRktSHuFn+x+N4CSXe2SaLnnPV1qCkFt hvab64ELGBg== X-Google-Smtp-Source: AGHT+IERW3RLs7pAvrtK7+8wenp15EOf0eYUuFkXQS/5Va7zov6earse9wZ2K8Bqz+MSmoFd135jNea/Uk3/DA== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:690c:d21:b0:608:1b39:246b with SMTP id cn33-20020a05690c0d2100b006081b39246bmr3810484ywb.3.1709654680656; Tue, 05 Mar 2024 08:04:40 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:11 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-17-edumazet@google.com> Subject: [PATCH net-next 16/18] ipv6: move tcp_ipv6_hash_secret and udp_ipv6_hash_secret to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Use a 32bit hole in "struct net_offload" to store the remaining 32bit secrets used by TCPv6 and UDPv6. Signed-off-by: Eric Dumazet --- include/net/hotdata.h | 2 ++ include/net/protocol.h | 1 + net/ipv6/inet6_hashtables.c | 6 ++---- net/ipv6/udp.c | 2 -- 4 files changed, 5 insertions(+), 6 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index e4dac2f859efd421b975a61360536af949046d0e..0a0a9106b40030f56b04c1e48083c13498ce0939 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -39,7 +39,9 @@ struct net_hotdata { #define inet_ehash_secret net_hotdata.tcp_protocol.secret #define udp_ehash_secret net_hotdata.udp_protocol.secret #define inet6_ehash_secret net_hotdata.tcpv6_protocol.secret +#define tcp_ipv6_hash_secret net_hotdata.tcpv6_offload.secret #define udp6_ehash_secret net_hotdata.udpv6_protocol.secret +#define udp_ipv6_hash_secret net_hotdata.udpv6_offload.secret extern struct net_hotdata net_hotdata; diff --git a/include/net/protocol.h b/include/net/protocol.h index 213649d2ab098edc4eb9f44a5403540887c2b8ef..b2499f88f8f8199de28555d0cbd6b4523325205b 100644 --- a/include/net/protocol.h +++ b/include/net/protocol.h @@ -70,6 +70,7 @@ struct inet6_protocol { struct net_offload { struct offload_callbacks callbacks; unsigned int flags; /* Flags used by IPv6 for now */ + u32 secret; }; /* This should be set for any extension header which is compatible with GSO. */ #define INET6_PROTO_GSO_EXTHDR 0x1 diff --git a/net/ipv6/inet6_hashtables.c b/net/ipv6/inet6_hashtables.c index 0fee97f3166cf8326b3b714ac6bde48ca5188cec..2e81383b663b71b95719a295fd9629f1193e4225 100644 --- a/net/ipv6/inet6_hashtables.c +++ b/net/ipv6/inet6_hashtables.c @@ -26,15 +26,13 @@ u32 inet6_ehashfn(const struct net *net, const struct in6_addr *laddr, const u16 lport, const struct in6_addr *faddr, const __be16 fport) { - static u32 ipv6_hash_secret __read_mostly; - u32 lhash, fhash; net_get_random_once(&inet6_ehash_secret, sizeof(inet6_ehash_secret)); - net_get_random_once(&ipv6_hash_secret, sizeof(ipv6_hash_secret)); + net_get_random_once(&tcp_ipv6_hash_secret, sizeof(tcp_ipv6_hash_secret)); lhash = (__force u32)laddr->s6_addr32[3]; - fhash = __ipv6_addr_jhash(faddr, ipv6_hash_secret); + fhash = __ipv6_addr_jhash(faddr, tcp_ipv6_hash_secret); return __inet6_ehashfn(lhash, lport, fhash, fport, inet6_ehash_secret + net_hash_mix(net)); diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 1e1c67a51675e8534a953a6d4d63211388d95ca9..80ad8f436b179d7279cbbd5108c3494a9773c0d8 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -79,8 +79,6 @@ u32 udp6_ehashfn(const struct net *net, const struct in6_addr *faddr, const __be16 fport) { - static u32 udp_ipv6_hash_secret __read_mostly; - u32 lhash, fhash; net_get_random_once(&udp6_ehash_secret, From patchwork Tue Mar 5 16:04:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582630 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60917128807 for ; Tue, 5 Mar 2024 16:04:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654699; cv=none; b=f3VJ7M9ia51XzcJkJvoTXfmvRKSrY1w/pcQEL/0Nc3fbIbWXRSd/iNkmRQiQ4Tb9eDK4mevlS67V0K/SOc6lKFyUnP9Np7iiU6BxV0EHYv8wvbWpH7DVHapkw3SEuDmDBcDD7gK+dfCc0gy0OHmqTcjphxeNExtK5WUr5/XGISc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654699; c=relaxed/simple; bh=+RpDZ4umHT16KgkibZ6MLC6oD8ZUCZVpsTThjwhrJS4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HIhkyTWDW85dFuaFwCqPHgQ+kKyJ8rLWOWEhBPfDcfee7K9uOaef0zPzUWL/pV3zrg9gk5DT0ZT7P67YXfWF+7kDDC/UB8+i/c7oUYbWx4sAwJHEO/Lkz2Zm0nQynVh4B8KPODJIznC27mJErH382kgLWLFlzY+LBftVVXJguto= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=E1PLC2Pt; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="E1PLC2Pt" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60998ed9df5so53891917b3.3 for ; Tue, 05 Mar 2024 08:04:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654682; x=1710259482; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+XW2iTaBEBFjs9ewuQar/cXIujcUM7/EhsgXAWDpU3Q=; b=E1PLC2PtTsNYljRXBcZWQ7Rq6H6iKIZxGD8nI+4yDtxmO3NoHRW+BthvpwSgxGvqDY LGHzBy8GhndTElPEFDnN2779S4We0C8eBmwLBbR7ig1Ks9Y/e3lnMkguUMQ0p9tqQ60Q 4Tq4DAt7ECFHldt9jmGqv7ixe67D5xcMwG3UpPwRqLBTo7ipPZmPpX/5xkLLu6PlaIl6 U9bpcB5dC2xXH9NFCoWND9ccOunhkUUghlnFEu7BK4IV1ydt2ARYQvn26+RvDZaCVAif oMO+EE0j8B/HfRpixElvdt9ORM0/p54Gyi2pV5TcQ3XVqGaOld9ZO4KsDzE2QpoEtCFN PCtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654682; x=1710259482; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+XW2iTaBEBFjs9ewuQar/cXIujcUM7/EhsgXAWDpU3Q=; b=YDD9t1WDccVWkezkI8RtCQAtFj7htcTARKuI+nW9zlSj/Nnsy8104+Ax1y9jXd5Tox 8zAxXBBYOZqj9ZK8419nzYB3q1t2yxxBjCJTdaZEZtHkhS+plTw9IdhHhqVtIHKbDsdP 0+jK04Y8j7sULH66pfpwj6XYFTc/F3xOhhCW34Vg78XHddigy+N+gag2O3NDEjIXZY59 B7AYUue5vdSh6/99XzyswDuW7q+41JbU9ul8hykaykvLHdQ9APBe8oVgtGVivmDOZSnI 5UTzMvwJ4kjTVI48gsBFHupkLRvXMj/OeU2PTYBuMWcxOxcOso4b7ozImcFH/o7szEJt uLAQ== X-Gm-Message-State: AOJu0YwIKPr0kggy8GbEGYeUC8LWIphHrem0h3iX9Fn+7narhwVbFxUb Gdh1dd9dWwmabITta+bnuQFq3e3aZC6DexsXRf/2URK69kyNOEFy5O22P4EfmCylyDKZlggXF1x 1ul5A/S+NNA== X-Google-Smtp-Source: AGHT+IEYBRYeWySyG8pUIpAZ2LXSGW7xA5jA+AqbRnhM+9qiwRXlWG/aYFQ/a4h0yIsXFfm27zpCNYAC1nLPpg== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:72c:b0:dbe:a0c2:df25 with SMTP id l12-20020a056902072c00b00dbea0c2df25mr437273ybt.8.1709654682475; Tue, 05 Mar 2024 08:04:42 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:12 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-18-edumazet@google.com> Subject: [PATCH net-next 17/18] net: introduce include/net/rps.h From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org Move RPS related structures and helpers from include/linux/netdevice.h and include/net/sock.h to a new include file. Signed-off-by: Eric Dumazet --- drivers/net/ethernet/intel/ice/ice_arfs.c | 1 + .../net/ethernet/mellanox/mlx4/en_netdev.c | 1 + .../net/ethernet/mellanox/mlx5/core/en_arfs.c | 1 + drivers/net/ethernet/sfc/rx_common.c | 1 + drivers/net/ethernet/sfc/siena/rx_common.c | 1 + drivers/net/tun.c | 1 + include/linux/netdevice.h | 82 ----------- include/net/rps.h | 127 ++++++++++++++++++ include/net/sock.h | 35 ----- net/core/dev.c | 1 + net/core/net-sysfs.c | 1 + net/core/sysctl_net_core.c | 1 + net/ipv4/af_inet.c | 1 + net/ipv4/tcp.c | 1 + net/ipv6/af_inet6.c | 1 + net/sctp/socket.c | 1 + 16 files changed, 140 insertions(+), 117 deletions(-) create mode 100644 include/net/rps.h diff --git a/drivers/net/ethernet/intel/ice/ice_arfs.c b/drivers/net/ethernet/intel/ice/ice_arfs.c index cca0e753f38ff92be24f8e8fc8963e2bf2416cfa..7cee365cc7d167865511c8b0dab08520814d4ec1 100644 --- a/drivers/net/ethernet/intel/ice/ice_arfs.c +++ b/drivers/net/ethernet/intel/ice/ice_arfs.c @@ -2,6 +2,7 @@ /* Copyright (C) 2018-2020, Intel Corporation. */ #include "ice.h" +#include /** * ice_is_arfs_active - helper to check is aRFS is active diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c index d7da62cda821f70af530f95d23b63afd64423219..5d3fde63b273922f52f2466c7b339f3c9908e705 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c index e66f486faafe1a6b0cfc75f0f11b2e957b040842..c7f542d0b8f08c635a6fad868a364a8f5f91ba8c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c @@ -34,6 +34,7 @@ #include #include #include +#include #include "en.h" #define ARFS_HASH_SHIFT BITS_PER_BYTE diff --git a/drivers/net/ethernet/sfc/rx_common.c b/drivers/net/ethernet/sfc/rx_common.c index fac227d372db4c832a52a62144ff9c6c89995335..dcd901eccfc8f1f9c68ee391de548a2c1602ab94 100644 --- a/drivers/net/ethernet/sfc/rx_common.c +++ b/drivers/net/ethernet/sfc/rx_common.c @@ -11,6 +11,7 @@ #include "net_driver.h" #include #include +#include #include "efx.h" #include "nic.h" #include "rx_common.h" diff --git a/drivers/net/ethernet/sfc/siena/rx_common.c b/drivers/net/ethernet/sfc/siena/rx_common.c index 4579f43484c3675963de1ad8c7753a200c680fe1..219fb358a646d399dbd3c66cc34039f70bbd7341 100644 --- a/drivers/net/ethernet/sfc/siena/rx_common.c +++ b/drivers/net/ethernet/sfc/siena/rx_common.c @@ -11,6 +11,7 @@ #include "net_driver.h" #include #include +#include #include "efx.h" #include "nic.h" #include "rx_common.h" diff --git a/drivers/net/tun.c b/drivers/net/tun.c index be37235af55d2d282b162adcd9d0e599cd5e6b3e..c1be5cec0f350d61c604d5d2776e047aa48c90b8 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -78,6 +78,7 @@ #include #include #include +#include #include #include diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index d3d4d1052ecb0bc704e9bb40fffbea85cfb6ab03..aaacf86df56ab480485b34b19fecebfd00c34c59 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -227,12 +227,6 @@ struct net_device_core_stats { #include #include -#ifdef CONFIG_RPS -#include -extern struct static_key_false rps_needed; -extern struct static_key_false rfs_needed; -#endif - struct neighbour; struct neigh_parms; struct sk_buff; @@ -732,86 +726,10 @@ static inline void netdev_queue_numa_node_write(struct netdev_queue *q, int node #endif } -#ifdef CONFIG_RPS -/* - * This structure holds an RPS map which can be of variable length. The - * map is an array of CPUs. - */ -struct rps_map { - unsigned int len; - struct rcu_head rcu; - u16 cpus[]; -}; -#define RPS_MAP_SIZE(_num) (sizeof(struct rps_map) + ((_num) * sizeof(u16))) - -/* - * The rps_dev_flow structure contains the mapping of a flow to a CPU, the - * tail pointer for that CPU's input queue at the time of last enqueue, and - * a hardware filter index. - */ -struct rps_dev_flow { - u16 cpu; - u16 filter; - unsigned int last_qtail; -}; -#define RPS_NO_FILTER 0xffff - -/* - * The rps_dev_flow_table structure contains a table of flow mappings. - */ -struct rps_dev_flow_table { - unsigned int mask; - struct rcu_head rcu; - struct rps_dev_flow flows[]; -}; -#define RPS_DEV_FLOW_TABLE_SIZE(_num) (sizeof(struct rps_dev_flow_table) + \ - ((_num) * sizeof(struct rps_dev_flow))) - -/* - * The rps_sock_flow_table contains mappings of flows to the last CPU - * on which they were processed by the application (set in recvmsg). - * Each entry is a 32bit value. Upper part is the high-order bits - * of flow hash, lower part is CPU number. - * rps_cpu_mask is used to partition the space, depending on number of - * possible CPUs : rps_cpu_mask = roundup_pow_of_two(nr_cpu_ids) - 1 - * For example, if 64 CPUs are possible, rps_cpu_mask = 0x3f, - * meaning we use 32-6=26 bits for the hash. - */ -struct rps_sock_flow_table { - u32 mask; - - u32 ents[] ____cacheline_aligned_in_smp; -}; -#define RPS_SOCK_FLOW_TABLE_SIZE(_num) (offsetof(struct rps_sock_flow_table, ents[_num])) - -#define RPS_NO_CPU 0xffff - -extern u32 rps_cpu_mask; -extern struct rps_sock_flow_table __rcu *rps_sock_flow_table; - -static inline void rps_record_sock_flow(struct rps_sock_flow_table *table, - u32 hash) -{ - if (table && hash) { - unsigned int index = hash & table->mask; - u32 val = hash & ~rps_cpu_mask; - - /* We only give a hint, preemption can change CPU under us */ - val |= raw_smp_processor_id(); - - /* The following WRITE_ONCE() is paired with the READ_ONCE() - * here, and another one in get_rps_cpu(). - */ - if (READ_ONCE(table->ents[index]) != val) - WRITE_ONCE(table->ents[index], val); - } -} - #ifdef CONFIG_RFS_ACCEL bool rps_may_expire_flow(struct net_device *dev, u16 rxq_index, u32 flow_id, u16 filter_id); #endif -#endif /* CONFIG_RPS */ /* XPS map type and offset of the xps map within net_device->xps_maps[]. */ enum xps_map_type { diff --git a/include/net/rps.h b/include/net/rps.h new file mode 100644 index 0000000000000000000000000000000000000000..6081d817d2458b7b34036d87fbdef3fa6dc914ea --- /dev/null +++ b/include/net/rps.h @@ -0,0 +1,127 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _NET_RPS_H +#define _NET_RPS_H + +#include +#include +#include + +#ifdef CONFIG_RPS + +extern struct static_key_false rps_needed; +extern struct static_key_false rfs_needed; + +/* + * This structure holds an RPS map which can be of variable length. The + * map is an array of CPUs. + */ +struct rps_map { + unsigned int len; + struct rcu_head rcu; + u16 cpus[]; +}; +#define RPS_MAP_SIZE(_num) (sizeof(struct rps_map) + ((_num) * sizeof(u16))) + +/* + * The rps_dev_flow structure contains the mapping of a flow to a CPU, the + * tail pointer for that CPU's input queue at the time of last enqueue, and + * a hardware filter index. + */ +struct rps_dev_flow { + u16 cpu; + u16 filter; + unsigned int last_qtail; +}; +#define RPS_NO_FILTER 0xffff + +/* + * The rps_dev_flow_table structure contains a table of flow mappings. + */ +struct rps_dev_flow_table { + unsigned int mask; + struct rcu_head rcu; + struct rps_dev_flow flows[]; +}; +#define RPS_DEV_FLOW_TABLE_SIZE(_num) (sizeof(struct rps_dev_flow_table) + \ + ((_num) * sizeof(struct rps_dev_flow))) + +/* + * The rps_sock_flow_table contains mappings of flows to the last CPU + * on which they were processed by the application (set in recvmsg). + * Each entry is a 32bit value. Upper part is the high-order bits + * of flow hash, lower part is CPU number. + * rps_cpu_mask is used to partition the space, depending on number of + * possible CPUs : rps_cpu_mask = roundup_pow_of_two(nr_cpu_ids) - 1 + * For example, if 64 CPUs are possible, rps_cpu_mask = 0x3f, + * meaning we use 32-6=26 bits for the hash. + */ +struct rps_sock_flow_table { + u32 mask; + + u32 ents[] ____cacheline_aligned_in_smp; +}; +#define RPS_SOCK_FLOW_TABLE_SIZE(_num) (offsetof(struct rps_sock_flow_table, ents[_num])) + +#define RPS_NO_CPU 0xffff + +extern u32 rps_cpu_mask; +extern struct rps_sock_flow_table __rcu *rps_sock_flow_table; + +static inline void rps_record_sock_flow(struct rps_sock_flow_table *table, + u32 hash) +{ + unsigned int index = hash & table->mask; + u32 val = hash & ~rps_cpu_mask; + + /* We only give a hint, preemption can change CPU under us */ + val |= raw_smp_processor_id(); + + /* The following WRITE_ONCE() is paired with the READ_ONCE() + * here, and another one in get_rps_cpu(). + */ + if (READ_ONCE(table->ents[index]) != val) + WRITE_ONCE(table->ents[index], val); +} + +#endif /* CONFIG_RPS */ + +static inline void sock_rps_record_flow_hash(__u32 hash) +{ +#ifdef CONFIG_RPS + struct rps_sock_flow_table *sock_flow_table; + + if (!hash) + return; + rcu_read_lock(); + sock_flow_table = rcu_dereference(rps_sock_flow_table); + if (sock_flow_table) + rps_record_sock_flow(sock_flow_table, hash); + rcu_read_unlock(); +#endif +} + +static inline void sock_rps_record_flow(const struct sock *sk) +{ +#ifdef CONFIG_RPS + if (static_branch_unlikely(&rfs_needed)) { + /* Reading sk->sk_rxhash might incur an expensive cache line + * miss. + * + * TCP_ESTABLISHED does cover almost all states where RFS + * might be useful, and is cheaper [1] than testing : + * IPv4: inet_sk(sk)->inet_daddr + * IPv6: ipv6_addr_any(&sk->sk_v6_daddr) + * OR an additional socket flag + * [1] : sk_state and sk_prot are in the same cache line. + */ + if (sk->sk_state == TCP_ESTABLISHED) { + /* This READ_ONCE() is paired with the WRITE_ONCE() + * from sock_rps_save_rxhash() and sock_rps_reset_rxhash(). + */ + sock_rps_record_flow_hash(READ_ONCE(sk->sk_rxhash)); + } + } +#endif +} + +#endif /* _NET_RPS_H */ diff --git a/include/net/sock.h b/include/net/sock.h index 09a0cde8bf52286167f3b9e9b32f632f3d1b5487..b5e00702acc1f037df7eb8ad085d00e0b18079a8 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1117,41 +1117,6 @@ static inline void sk_incoming_cpu_update(struct sock *sk) WRITE_ONCE(sk->sk_incoming_cpu, cpu); } -static inline void sock_rps_record_flow_hash(__u32 hash) -{ -#ifdef CONFIG_RPS - struct rps_sock_flow_table *sock_flow_table; - - rcu_read_lock(); - sock_flow_table = rcu_dereference(rps_sock_flow_table); - rps_record_sock_flow(sock_flow_table, hash); - rcu_read_unlock(); -#endif -} - -static inline void sock_rps_record_flow(const struct sock *sk) -{ -#ifdef CONFIG_RPS - if (static_branch_unlikely(&rfs_needed)) { - /* Reading sk->sk_rxhash might incur an expensive cache line - * miss. - * - * TCP_ESTABLISHED does cover almost all states where RFS - * might be useful, and is cheaper [1] than testing : - * IPv4: inet_sk(sk)->inet_daddr - * IPv6: ipv6_addr_any(&sk->sk_v6_daddr) - * OR an additional socket flag - * [1] : sk_state and sk_prot are in the same cache line. - */ - if (sk->sk_state == TCP_ESTABLISHED) { - /* This READ_ONCE() is paired with the WRITE_ONCE() - * from sock_rps_save_rxhash() and sock_rps_reset_rxhash(). - */ - sock_rps_record_flow_hash(READ_ONCE(sk->sk_rxhash)); - } - } -#endif -} static inline void sock_rps_save_rxhash(struct sock *sk, const struct sk_buff *skb) diff --git a/net/core/dev.c b/net/core/dev.c index 26676dbd6c8594371a4d30e0dd95d72342e226ca..e9f24a31ae121f713e6ef5a530a65218bbb457e8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -155,6 +155,7 @@ #include #include #include +#include #include "dev.h" #include "net-sysfs.h" diff --git a/net/core/net-sysfs.c b/net/core/net-sysfs.c index af238026ac3c6d81267bf0b53a7b240ee9ba32b1..5560083774b1ad4e2612637990dfa3f80f750c83 100644 --- a/net/core/net-sysfs.c +++ b/net/core/net-sysfs.c @@ -24,6 +24,7 @@ #include #include #include +#include #include "dev.h" #include "net-sysfs.h" diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 8a4c698dad9c97636ca9cebfad925d4220e98f2a..4b93e27404e83a5b3afaa23ebd18cf55b1fdc6e8 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -24,6 +24,7 @@ #include #include #include +#include #include "dev.h" diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index 6f1cfd176e7b84f23d8a5e505bf8e13b2b755f06..55bd72997b31063b7baad350fdcff40e938aecb8 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -119,6 +119,7 @@ #endif #include #include +#include #include diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 7e1b848398d04f2da2a91c3af97b1e2e3895b8ee..c5b83875411aee6037d0afd060eb9d3e16f8b6b0 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -279,6 +279,7 @@ #include #include #include +#include /* Track pending CMSGs. */ enum { diff --git a/net/ipv6/af_inet6.c b/net/ipv6/af_inet6.c index b90d46533cdcc1ffb61ca483e6f67ab358ede55c..8041dc181bd42e5e1af2d9e9fe6af50057e5b58f 100644 --- a/net/ipv6/af_inet6.c +++ b/net/ipv6/af_inet6.c @@ -64,6 +64,7 @@ #include #include #include +#include #include #include diff --git a/net/sctp/socket.c b/net/sctp/socket.c index 6b9fcdb0952a0fe599ae5d1d1cc6fa9557a3a3bc..c67679a41044fc8e801d175b235249f2c8b99dc0 100644 --- a/net/sctp/socket.c +++ b/net/sctp/socket.c @@ -67,6 +67,7 @@ #include #include #include +#include /* Forward declarations for internal helper functions. */ static bool sctp_writeable(const struct sock *sk); From patchwork Tue Mar 5 16:04:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 13582631 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B2A312AAC0 for ; Tue, 5 Mar 2024 16:04:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654700; cv=none; b=ol/hU7f/jTj98/uJHGf4KGq+GWET+3/lORXL6oOJImUq7qolyfe9hG5XF7IrzwtAcl1+H4JuUENnLxTHg5mUXX6uhrYZk2r7LlBI4K+Uc17T+KxqDGcUJtdMNlPlc+61axd9J7l+7Tcvzyt7KcUYdAzf+f7QqZopBnLyueT3OLs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709654700; c=relaxed/simple; bh=4afPVHI5thGgU7g6ZCN0kgXLob+BBsFTw9blvYx9pHw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kzHcMSrZXyu6vVnD4k/UHfv/yAkO1iRUGHgmcZPh69eYIg40nsLh26sE7ZSerRmIyTDE6Ggpm9XtdyEV53b6et/sSZK/0FUNedZNJjrrYTiSokzzcWV+6Rb6+R8xoDlLBlzrM54ZC+2WPb7oUNWIBNgTiTer7ZRHLXfHwxCodTs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bMlpgen9; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--edumazet.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bMlpgen9" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc647f65573so10204708276.2 for ; Tue, 05 Mar 2024 08:04:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709654684; x=1710259484; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rFQRLyiXLEhWnxUI5NjDOfXwmcvwbMhEFic/5d2k5A8=; b=bMlpgen9mnNxZ2cEewmKY2Qq5vU+zh/A2lppgnvW18bF6kGN0MvmFm8ntUd98Ohmat BVG9EJ5lhTzHdmUCXYeOrU0PxyOT09pStH8bha8W2CuyBZlF5vm/CfQkWZA3DB3vxbpI R5nH8FJ4vsLQpvdDCOC3aHj9nabMsNr1Ycm1UZaTnVyZ2+WbvBdY0WFvG/KoVXA8L4fc 4yNSeI4RP4TDTvHWjheh6Ux4aR8dcTg0HY+7C91/l4VOHGs1xVdDZDAozOKk+RjoMdlV KURNtHr7Oz+C57XoKD+Wn/+z39q6TS89ZSroMQKnpByFJVl3i4DXDzIY6Lds6Ma8DbhU +lqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654684; x=1710259484; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rFQRLyiXLEhWnxUI5NjDOfXwmcvwbMhEFic/5d2k5A8=; b=s+fv05tAMrIN7JECihxUVvrk5GUWTA2tqrOLBMBAQbrw8wDvgVDCFNC+PgVRAehhoA iwDuE0+qi1a4tkYdFDlJElSlsB1I4uhwySfpp0nChJqBfizdlzbLvhU1VNoq3hJtg8Mt NYgp4iEu9uHcLGjrzTwrWHQHISRWxw8wV8TrlK551ZPh1bLFjK/4mH6dJoZ5GX5cRth2 ZyKHFizIU5ywvibm8uATAhJkNuHCMRsMYRQ7DefC3LUWMjTD9ob+AgPpV2dsIfDobc1A 6YlDAdhaiXQC6jRAqWBLwkfQ596elIVfi4JiDtBvtq18rPj7Su4iM6xyUIkQkTFzkNhk Cydw== X-Gm-Message-State: AOJu0YxJKEznPhE1NGlMkIBuEgN6PxdYR8ws0SKDwMZS9GJRsu6DaU2y n7Ec+MJ4GEKkTgwfex5xhRwJCtk6cap0HQTWIiK3bTyZVci0X0xmSGTE4jc5HBiG+sZqyJ+MQvV Jyhw9PWY7rw== X-Google-Smtp-Source: AGHT+IHce2ux2BZZB57E5FwCGXXKXy93aS+WYrt2VCfgS1Vh8DQI3PM6KmFsu5Lla102GXz7lu8Ct/Yjco6X1w== X-Received: from edumazet1.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:395a]) (user=edumazet job=sendgmr) by 2002:a05:6902:1081:b0:dc7:8e30:e2e3 with SMTP id v1-20020a056902108100b00dc78e30e2e3mr3178799ybu.2.1709654684142; Tue, 05 Mar 2024 08:04:44 -0800 (PST) Date: Tue, 5 Mar 2024 16:04:13 +0000 In-Reply-To: <20240305160413.2231423-1-edumazet@google.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240305160413.2231423-1-edumazet@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240305160413.2231423-19-edumazet@google.com> Subject: [PATCH net-next 18/18] net: move rps_sock_flow_table to net_hotdata From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev@vger.kernel.org, David Ahern , Willem de Bruijn , Soheil Hassas Yeganeh , Neal Cardwell , eric.dumazet@gmail.com, Eric Dumazet X-Patchwork-Delegate: kuba@kernel.org rps_sock_flow_table and rps_cpu_mask are used in fast path. Move them to net_hotdata for better cache locality. Signed-off-by: Eric Dumazet --- include/net/hotdata.h | 4 ++++ include/net/rps.h | 8 +++----- net/core/dev.c | 12 +++--------- net/core/sysctl_net_core.c | 9 ++++++--- 4 files changed, 16 insertions(+), 17 deletions(-) diff --git a/include/net/hotdata.h b/include/net/hotdata.h index 0a0a9106b40030f56b04c1e48083c13498ce0939..7bb6e46aec8f19deff42112041feb47724cdd538 100644 --- a/include/net/hotdata.h +++ b/include/net/hotdata.h @@ -27,6 +27,10 @@ struct net_hotdata { struct kmem_cache *skbuff_cache; struct kmem_cache *skbuff_fclone_cache; struct kmem_cache *skb_small_head_cache; +#ifdef CONFIG_RPS + struct rps_sock_flow_table __rcu *rps_sock_flow_table; + u32 rps_cpu_mask; +#endif int gro_normal_batch; int netdev_budget; int netdev_budget_usecs; diff --git a/include/net/rps.h b/include/net/rps.h index 6081d817d2458b7b34036d87fbdef3fa6dc914ea..7660243e905b92651a41292e04caf72c5f12f26e 100644 --- a/include/net/rps.h +++ b/include/net/rps.h @@ -5,6 +5,7 @@ #include #include #include +#include #ifdef CONFIG_RPS @@ -64,14 +65,11 @@ struct rps_sock_flow_table { #define RPS_NO_CPU 0xffff -extern u32 rps_cpu_mask; -extern struct rps_sock_flow_table __rcu *rps_sock_flow_table; - static inline void rps_record_sock_flow(struct rps_sock_flow_table *table, u32 hash) { unsigned int index = hash & table->mask; - u32 val = hash & ~rps_cpu_mask; + u32 val = hash & ~net_hotdata.rps_cpu_mask; /* We only give a hint, preemption can change CPU under us */ val |= raw_smp_processor_id(); @@ -93,7 +91,7 @@ static inline void sock_rps_record_flow_hash(__u32 hash) if (!hash) return; rcu_read_lock(); - sock_flow_table = rcu_dereference(rps_sock_flow_table); + sock_flow_table = rcu_dereference(net_hotdata.rps_sock_flow_table); if (sock_flow_table) rps_record_sock_flow(sock_flow_table, hash); rcu_read_unlock(); diff --git a/net/core/dev.c b/net/core/dev.c index e9f24a31ae121f713e6ef5a530a65218bbb457e8..9f54d8e61ec0f4a4ad1824e333f89ad08c5ec431 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4450,12 +4450,6 @@ static inline void ____napi_schedule(struct softnet_data *sd, #ifdef CONFIG_RPS -/* One global table that all flow-based protocols share. */ -struct rps_sock_flow_table __rcu *rps_sock_flow_table __read_mostly; -EXPORT_SYMBOL(rps_sock_flow_table); -u32 rps_cpu_mask __read_mostly; -EXPORT_SYMBOL(rps_cpu_mask); - struct static_key_false rps_needed __read_mostly; EXPORT_SYMBOL(rps_needed); struct static_key_false rfs_needed __read_mostly; @@ -4547,7 +4541,7 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, if (!hash) goto done; - sock_flow_table = rcu_dereference(rps_sock_flow_table); + sock_flow_table = rcu_dereference(net_hotdata.rps_sock_flow_table); if (flow_table && sock_flow_table) { struct rps_dev_flow *rflow; u32 next_cpu; @@ -4557,10 +4551,10 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, * This READ_ONCE() pairs with WRITE_ONCE() from rps_record_sock_flow(). */ ident = READ_ONCE(sock_flow_table->ents[hash & sock_flow_table->mask]); - if ((ident ^ hash) & ~rps_cpu_mask) + if ((ident ^ hash) & ~net_hotdata.rps_cpu_mask) goto try_rps; - next_cpu = ident & rps_cpu_mask; + next_cpu = ident & net_hotdata.rps_cpu_mask; /* OK, now we know there is a match, * we can look at the local (per receive queue) flow table diff --git a/net/core/sysctl_net_core.c b/net/core/sysctl_net_core.c index 4b93e27404e83a5b3afaa23ebd18cf55b1fdc6e8..6973dda3abda63e0924efa4b6b7026786e8bfb4f 100644 --- a/net/core/sysctl_net_core.c +++ b/net/core/sysctl_net_core.c @@ -140,7 +140,8 @@ static int rps_sock_flow_sysctl(struct ctl_table *table, int write, mutex_lock(&sock_flow_mutex); - orig_sock_table = rcu_dereference_protected(rps_sock_flow_table, + orig_sock_table = rcu_dereference_protected( + net_hotdata.rps_sock_flow_table, lockdep_is_held(&sock_flow_mutex)); size = orig_size = orig_sock_table ? orig_sock_table->mask + 1 : 0; @@ -161,7 +162,8 @@ static int rps_sock_flow_sysctl(struct ctl_table *table, int write, mutex_unlock(&sock_flow_mutex); return -ENOMEM; } - rps_cpu_mask = roundup_pow_of_two(nr_cpu_ids) - 1; + net_hotdata.rps_cpu_mask = + roundup_pow_of_two(nr_cpu_ids) - 1; sock_table->mask = size - 1; } else sock_table = orig_sock_table; @@ -172,7 +174,8 @@ static int rps_sock_flow_sysctl(struct ctl_table *table, int write, sock_table = NULL; if (sock_table != orig_sock_table) { - rcu_assign_pointer(rps_sock_flow_table, sock_table); + rcu_assign_pointer(net_hotdata.rps_sock_flow_table, + sock_table); if (sock_table) { static_branch_inc(&rps_needed); static_branch_inc(&rfs_needed);