From patchwork Thu Jun 9 06:34:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12874851 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01952C43334 for ; Thu, 9 Jun 2022 06:34:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234804AbiFIGeU (ORCPT ); Thu, 9 Jun 2022 02:34:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229754AbiFIGeT (ORCPT ); Thu, 9 Jun 2022 02:34:19 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F11AF186FD for ; Wed, 8 Jun 2022 23:34:18 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id o6-20020a17090a0a0600b001e2c6566046so25881100pjo.0 for ; Wed, 08 Jun 2022 23:34:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0abhAtFaXlCnrdg1se2pezovH988jn6vXslYDVO4DCQ=; b=LozJ88UFfUpkyR6WO4meNvzURoaajlF8NtbtbQhNzgiI0JGy5tZ5oezDz/HZjym23X IAciUJFN/GuLFLXwEX+AFHCYxILqZhfSx+eVlpE3ABje9GO09NIZ+CgQRW9W6jXFuwSL VPoXXt1zAvnNwPriuzxNQzhR4Qq14xEkFDD8AwzK/fhXS2HQNpkuygMZNf8JTNj6Zc4O a0YtbJ8WCBex/JFqA8cVPMOAFHG4eb5rTC2NKUAi2ZY6T2zQoiFxElV3JsJ47v1Rwe64 8rN57MGpjxcUO4N7ZJUbYVW5g3fXUfdD5M3j3yLuiOBperU/VY1En7lAiXT9sDf1fU64 5mfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0abhAtFaXlCnrdg1se2pezovH988jn6vXslYDVO4DCQ=; b=5xeaWN/rpoh+ymb86bGSdbo4zNUn0AD6V1jzlfYGzthOPq344O8EG0EO8l6PCnJwse WEYSZB2JqKU6FK/zlwVgKVoPglMd+9ngxGP/LXci59W+WmY7hdQUh0gE+FvBOKNlZuVd te1sUeibC85AHFDlLL915eKs7MSZI5UbTqt3+Aq4jweWjKmBWEAHh55CtniUDWjivX05 7jqWUS/EHaa1kCuUzpOn9qT6bybWcLPSbEacj0uKmSjqBo8BKL7s+BjBVT8yY7iUfA8R +I1y2OzgIRy33UxObgbns9wd2FBUBEEWgt+Lx83syCFVi2diwyMnJZBuDNBWHhXamd4a qoYA== X-Gm-Message-State: AOAM5300EWKU0Kt6dyyCBfhc3SRRIvhqoYH6GvM/9dDjzz9wjY+wsxBO gdpKiWTfB38+wzurHJUt4Ik= X-Google-Smtp-Source: ABdhPJyM0H2GjPUUYYBzTNwmKfhlhMAQfBAoCvWrFQoeemFwRYJ9l+uoj5+6z1AQYQXfoXJjM0miAg== X-Received: by 2002:a17:903:2287:b0:164:95f:b4d6 with SMTP id b7-20020a170903228700b00164095fb4d6mr37713685plh.140.1654756458553; Wed, 08 Jun 2022 23:34:18 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:f579:a14f:f943:9d9a]) by smtp.gmail.com with ESMTPSA id 199-20020a6215d0000000b0051b9c0af43dsm16340050pfv.155.2022.06.08.23.34.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jun 2022 23:34:18 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev , Soheil Hassas Yeganeh , Wei Wang , Shakeel Butt , Neal Cardwell , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 1/7] Revert "net: set SK_MEM_QUANTUM to 4096" Date: Wed, 8 Jun 2022 23:34:06 -0700 Message-Id: <20220609063412.2205738-2-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220609063412.2205738-1-eric.dumazet@gmail.com> References: <20220609063412.2205738-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet This reverts commit bd68a2a854ad5a85f0c8d0a9c8048ca3f6391efb. This change broke memcg on arches with PAGE_SIZE != 4096 Later, commit 2bb2f5fb21b04 ("net: add new socket option SO_RESERVE_MEM") also assumed PAGE_SIZE==SK_MEM_QUANTUM Following patches in the series will greatly reduce the over allocations problem. Signed-off-by: Eric Dumazet Reviewed-by: Shakeel Butt --- include/net/sock.h | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 657873e2d90fbb2254144aa2405a0551a789edbb..5c5265269899091a7bb8f14766085a463a476403 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1532,25 +1532,15 @@ int __sk_mem_schedule(struct sock *sk, int size, int kind); void __sk_mem_reduce_allocated(struct sock *sk, int amount); void __sk_mem_reclaim(struct sock *sk, int amount); -/* We used to have PAGE_SIZE here, but systems with 64KB pages - * do not necessarily have 16x time more memory than 4KB ones. - */ -#define SK_MEM_QUANTUM 4096 +#define SK_MEM_QUANTUM ((int)PAGE_SIZE) #define SK_MEM_QUANTUM_SHIFT ilog2(SK_MEM_QUANTUM) #define SK_MEM_SEND 0 #define SK_MEM_RECV 1 -/* sysctl_mem values are in pages, we convert them in SK_MEM_QUANTUM units */ +/* sysctl_mem values are in pages */ static inline long sk_prot_mem_limits(const struct sock *sk, int index) { - long val = sk->sk_prot->sysctl_mem[index]; - -#if PAGE_SIZE > SK_MEM_QUANTUM - val <<= PAGE_SHIFT - SK_MEM_QUANTUM_SHIFT; -#elif PAGE_SIZE < SK_MEM_QUANTUM - val >>= SK_MEM_QUANTUM_SHIFT - PAGE_SHIFT; -#endif - return val; + return sk->sk_prot->sysctl_mem[index]; } static inline int sk_mem_pages(int amt) From patchwork Thu Jun 9 06:34:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12874853 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98231C43334 for ; Thu, 9 Jun 2022 06:34:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238514AbiFIGeY (ORCPT ); Thu, 9 Jun 2022 02:34:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50158 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237314AbiFIGeW (ORCPT ); Thu, 9 Jun 2022 02:34:22 -0400 Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02B7B2FFD8 for ; Wed, 8 Jun 2022 23:34:21 -0700 (PDT) Received: by mail-pl1-x629.google.com with SMTP id n18so19517483plg.5 for ; Wed, 08 Jun 2022 23:34:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fhWyeWhT9LwHFgXSPn+7wxyv2FqOKN3ir1NGekf3NRw=; b=NUTnoKQ+aPiz36k+CZatMIILvI682jMO7gn9wDKmA86cHASrxzKdlhLlEqgrMYxmV2 oEFeWrwVq9XbDetboYZ5AddXxCrRkOjdzXdpKHEgBLZNVNrJSA75zZFcuXqsEzIDLSGR BifB0Wd2oRjK33+I0OEaA5J8jmXEUoABTLcTJKTsSqBURVEtatEUB8xioyqL+jNf44SK In2RtMvxnpy+O0qURApQu1yZtJ7yfaQNFpP8nE/9m4ivAUm7RS3qDw3ZPZA5ja1bb/Al hzTKmzyQ1/iTpSR5fvjADLwGTQvS5UFpxPshgYKZy/SxIzf9tODWs0lRt79YBZS1ZNWH Y8YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fhWyeWhT9LwHFgXSPn+7wxyv2FqOKN3ir1NGekf3NRw=; b=3JeXzkCom7vF4+EK1QhepJg6hzkPiibYCOVpLvflbGpL+z6apuoFzi00r0gAhoV6W+ 2DNPB5rSlfBh5p0Fvx4NXFzWkIeKw1PT+XDOW7LaPDrdi1cvEkoI15S//4sKtFSCnWwA GgCTUtM66NfoSA+xcwf0c4HexP3PGZTCKcFvsHlKakopFjLfmLhZGhg79nlVPb5e4U3S T62TEfIZofOhgGIoHXZq8CaSAuw1l/U9qKwUjqf7kTl8k4CqF1DYlS1eDZ9U5+im8bYf Q0TLvt+8U2pDBjhHDlTyyUYD1OUAKlKmnNxEZyox6V3vAz8ziYQUsQ0e4dfDXSQ+j1ZG gLDA== X-Gm-Message-State: AOAM531gRpS1FA0JmSQr4ekyQGed+C01BDqjEj4HxBY3JOnfuELbn8WR +MeTt8QL3BHvH7JCHi3tPSQ= X-Google-Smtp-Source: ABdhPJwEi24tlEUZQnD/tmOv02ysZhDPLkftZrASzabAHxG4CTDxc/wxf2jsLAKusXDM5MqEvhWlLA== X-Received: by 2002:a17:902:ecc9:b0:163:f779:f97a with SMTP id a9-20020a170902ecc900b00163f779f97amr37455965plh.167.1654756460417; Wed, 08 Jun 2022 23:34:20 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:f579:a14f:f943:9d9a]) by smtp.gmail.com with ESMTPSA id 199-20020a6215d0000000b0051b9c0af43dsm16340050pfv.155.2022.06.08.23.34.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jun 2022 23:34:20 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev , Soheil Hassas Yeganeh , Wei Wang , Shakeel Butt , Neal Cardwell , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 2/7] net: remove SK_MEM_QUANTUM and SK_MEM_QUANTUM_SHIFT Date: Wed, 8 Jun 2022 23:34:07 -0700 Message-Id: <20220609063412.2205738-3-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220609063412.2205738-1-eric.dumazet@gmail.com> References: <20220609063412.2205738-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Due to memcg interface, SK_MEM_QUANTUM is effectively PAGE_SIZE. This might change in the future, but it seems better to avoid the confusion. Signed-off-by: Eric Dumazet Reviewed-by: Shakeel Butt --- include/net/sock.h | 8 +++----- net/core/sock.c | 16 ++++++++-------- net/ipv4/tcp.c | 4 ++-- net/ipv4/tcp_input.c | 2 +- net/ipv4/tcp_output.c | 2 +- net/ipv4/udp.c | 10 +++++----- net/mptcp/protocol.c | 8 ++++---- net/sctp/protocol.c | 4 ++-- 8 files changed, 26 insertions(+), 28 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 5c5265269899091a7bb8f14766085a463a476403..298897bbfb3a3ea6ba88f76bc486ae636e2b1cfd 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1532,8 +1532,6 @@ int __sk_mem_schedule(struct sock *sk, int size, int kind); void __sk_mem_reduce_allocated(struct sock *sk, int amount); void __sk_mem_reclaim(struct sock *sk, int amount); -#define SK_MEM_QUANTUM ((int)PAGE_SIZE) -#define SK_MEM_QUANTUM_SHIFT ilog2(SK_MEM_QUANTUM) #define SK_MEM_SEND 0 #define SK_MEM_RECV 1 @@ -1545,7 +1543,7 @@ static inline long sk_prot_mem_limits(const struct sock *sk, int index) static inline int sk_mem_pages(int amt) { - return (amt + SK_MEM_QUANTUM - 1) >> SK_MEM_QUANTUM_SHIFT; + return (amt + PAGE_SIZE - 1) >> PAGE_SHIFT; } static inline bool sk_has_account(struct sock *sk) @@ -1594,7 +1592,7 @@ static inline void sk_mem_reclaim(struct sock *sk) reclaimable = sk->sk_forward_alloc - sk_unused_reserved_mem(sk); - if (reclaimable >= SK_MEM_QUANTUM) + if (reclaimable >= (int)PAGE_SIZE) __sk_mem_reclaim(sk, reclaimable); } @@ -1613,7 +1611,7 @@ static inline void sk_mem_reclaim_partial(struct sock *sk) reclaimable = sk->sk_forward_alloc - sk_unused_reserved_mem(sk); - if (reclaimable > SK_MEM_QUANTUM) + if (reclaimable > (int)PAGE_SIZE) __sk_mem_reclaim(sk, reclaimable - 1); } diff --git a/net/core/sock.c b/net/core/sock.c index 2ff40dd0a7a652029cca1743109286b50c2a17f3..6b786e836c7f5fc74307f050d4f32b4b554eb53b 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -991,7 +991,7 @@ EXPORT_SYMBOL(sock_set_mark); static void sock_release_reserved_memory(struct sock *sk, int bytes) { /* Round down bytes to multiple of pages */ - bytes &= ~(SK_MEM_QUANTUM - 1); + bytes = round_down(bytes, PAGE_SIZE); WARN_ON(bytes > sk->sk_reserved_mem); sk->sk_reserved_mem -= bytes; @@ -1028,9 +1028,9 @@ static int sock_reserve_memory(struct sock *sk, int bytes) mem_cgroup_uncharge_skmem(sk->sk_memcg, pages); return -ENOMEM; } - sk->sk_forward_alloc += pages << SK_MEM_QUANTUM_SHIFT; + sk->sk_forward_alloc += pages << PAGE_SHIFT; - sk->sk_reserved_mem += pages << SK_MEM_QUANTUM_SHIFT; + sk->sk_reserved_mem += pages << PAGE_SHIFT; return 0; } @@ -3003,10 +3003,10 @@ int __sk_mem_schedule(struct sock *sk, int size, int kind) { int ret, amt = sk_mem_pages(size); - sk->sk_forward_alloc += amt << SK_MEM_QUANTUM_SHIFT; + sk->sk_forward_alloc += amt << PAGE_SHIFT; ret = __sk_mem_raise_allocated(sk, size, amt, kind); if (!ret) - sk->sk_forward_alloc -= amt << SK_MEM_QUANTUM_SHIFT; + sk->sk_forward_alloc -= amt << PAGE_SHIFT; return ret; } EXPORT_SYMBOL(__sk_mem_schedule); @@ -3034,12 +3034,12 @@ EXPORT_SYMBOL(__sk_mem_reduce_allocated); /** * __sk_mem_reclaim - reclaim sk_forward_alloc and memory_allocated * @sk: socket - * @amount: number of bytes (rounded down to a SK_MEM_QUANTUM multiple) + * @amount: number of bytes (rounded down to a PAGE_SIZE multiple) */ void __sk_mem_reclaim(struct sock *sk, int amount) { - amount >>= SK_MEM_QUANTUM_SHIFT; - sk->sk_forward_alloc -= amount << SK_MEM_QUANTUM_SHIFT; + amount >>= PAGE_SHIFT; + sk->sk_forward_alloc -= amount << PAGE_SHIFT; __sk_mem_reduce_allocated(sk, amount); } EXPORT_SYMBOL(__sk_mem_reclaim); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 9984d23a7f3e1353d2e1fc9053d98c77268c577e..9e696758a4c213f22919483dcd6740b10ee3294b 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -4661,11 +4661,11 @@ void __init tcp_init(void) max_wshare = min(4UL*1024*1024, limit); max_rshare = min(6UL*1024*1024, limit); - init_net.ipv4.sysctl_tcp_wmem[0] = SK_MEM_QUANTUM; + init_net.ipv4.sysctl_tcp_wmem[0] = PAGE_SIZE; init_net.ipv4.sysctl_tcp_wmem[1] = 16*1024; init_net.ipv4.sysctl_tcp_wmem[2] = max(64*1024, max_wshare); - init_net.ipv4.sysctl_tcp_rmem[0] = SK_MEM_QUANTUM; + init_net.ipv4.sysctl_tcp_rmem[0] = PAGE_SIZE; init_net.ipv4.sysctl_tcp_rmem[1] = 131072; init_net.ipv4.sysctl_tcp_rmem[2] = max(131072, max_rshare); diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 2e2a9ece9af27372e6b653d685a89a2c71ba05d1..3fb117022558a408a664ea7c8fe2303296247ead 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -5287,7 +5287,7 @@ static void tcp_collapse_ofo_queue(struct sock *sk) before(TCP_SKB_CB(skb)->end_seq, start)) { /* Do not attempt collapsing tiny skbs */ if (range_truesize != head->truesize || - end - start >= SKB_WITH_OVERHEAD(SK_MEM_QUANTUM)) { + end - start >= SKB_WITH_OVERHEAD(PAGE_SIZE)) { tcp_collapse(sk, NULL, &tp->out_of_order_queue, head, skb, start, end); } else { diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 1c054431e358328fe3849f5a45aaa88308a1e1c8..8ab98e1aca6797a51eaaf8886680d2001a616948 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -3367,7 +3367,7 @@ void sk_forced_mem_schedule(struct sock *sk, int size) if (size <= sk->sk_forward_alloc) return; amt = sk_mem_pages(size); - sk->sk_forward_alloc += amt * SK_MEM_QUANTUM; + sk->sk_forward_alloc += amt << PAGE_SHIFT; sk_memory_allocated_add(sk, amt); if (mem_cgroup_sockets_enabled && sk->sk_memcg) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index aa9f2ec3dc4681f767e8be9d580096ba8b439327..bbc9970fa2e947ce8fdd08763033b6b5912af042 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -1461,11 +1461,11 @@ static void udp_rmem_release(struct sock *sk, int size, int partial, sk->sk_forward_alloc += size; - amt = (sk->sk_forward_alloc - partial) & ~(SK_MEM_QUANTUM - 1); + amt = (sk->sk_forward_alloc - partial) & ~(PAGE_SIZE - 1); sk->sk_forward_alloc -= amt; if (amt) - __sk_mem_reduce_allocated(sk, amt >> SK_MEM_QUANTUM_SHIFT); + __sk_mem_reduce_allocated(sk, amt >> PAGE_SHIFT); atomic_sub(size, &sk->sk_rmem_alloc); @@ -1558,7 +1558,7 @@ int __udp_enqueue_schedule_skb(struct sock *sk, struct sk_buff *skb) spin_lock(&list->lock); if (size >= sk->sk_forward_alloc) { amt = sk_mem_pages(size); - delta = amt << SK_MEM_QUANTUM_SHIFT; + delta = amt << PAGE_SHIFT; if (!__sk_mem_raise_allocated(sk, delta, amt, SK_MEM_RECV)) { err = -ENOBUFS; spin_unlock(&list->lock); @@ -3263,8 +3263,8 @@ EXPORT_SYMBOL(udp_flow_hashrnd); static void __udp_sysctl_init(struct net *net) { - net->ipv4.sysctl_udp_rmem_min = SK_MEM_QUANTUM; - net->ipv4.sysctl_udp_wmem_min = SK_MEM_QUANTUM; + net->ipv4.sysctl_udp_rmem_min = PAGE_SIZE; + net->ipv4.sysctl_udp_wmem_min = PAGE_SIZE; #ifdef CONFIG_NET_L3_MASTER_DEV net->ipv4.sysctl_udp_l3mdev_accept = 0; diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 17e13396024ad8807ce00a28ab1d86c23a582e32..080a630d6902caa2022fda1c6b3edb65e4e74a8c 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -167,8 +167,8 @@ static bool mptcp_ooo_try_coalesce(struct mptcp_sock *msk, struct sk_buff *to, static void __mptcp_rmem_reclaim(struct sock *sk, int amount) { - amount >>= SK_MEM_QUANTUM_SHIFT; - mptcp_sk(sk)->rmem_fwd_alloc -= amount << SK_MEM_QUANTUM_SHIFT; + amount >>= PAGE_SHIFT; + mptcp_sk(sk)->rmem_fwd_alloc -= amount << PAGE_SHIFT; __sk_mem_reduce_allocated(sk, amount); } @@ -327,7 +327,7 @@ static bool mptcp_rmem_schedule(struct sock *sk, struct sock *ssk, int size) return true; amt = sk_mem_pages(size); - amount = amt << SK_MEM_QUANTUM_SHIFT; + amount = amt << PAGE_SHIFT; msk->rmem_fwd_alloc += amount; if (!__sk_mem_raise_allocated(sk, size, amt, SK_MEM_RECV)) { if (ssk->sk_forward_alloc < amount) { @@ -972,7 +972,7 @@ static void __mptcp_mem_reclaim_partial(struct sock *sk) lockdep_assert_held_once(&sk->sk_lock.slock); - if (reclaimable > SK_MEM_QUANTUM) + if (reclaimable > (int)PAGE_SIZE) __mptcp_rmem_reclaim(sk, reclaimable - 1); sk_mem_reclaim_partial(sk); diff --git a/net/sctp/protocol.c b/net/sctp/protocol.c index 35928fefae3327f97688f0857de63bc17e3429d6..fa500ea3a1f1bb779e264ee999ac4b7252e716ee 100644 --- a/net/sctp/protocol.c +++ b/net/sctp/protocol.c @@ -1523,11 +1523,11 @@ static __init int sctp_init(void) limit = (sysctl_sctp_mem[1]) << (PAGE_SHIFT - 7); max_share = min(4UL*1024*1024, limit); - sysctl_sctp_rmem[0] = SK_MEM_QUANTUM; /* give each asoc 1 page min */ + sysctl_sctp_rmem[0] = PAGE_SIZE; /* give each asoc 1 page min */ sysctl_sctp_rmem[1] = 1500 * SKB_TRUESIZE(1); sysctl_sctp_rmem[2] = max(sysctl_sctp_rmem[1], max_share); - sysctl_sctp_wmem[0] = SK_MEM_QUANTUM; + sysctl_sctp_wmem[0] = PAGE_SIZE; sysctl_sctp_wmem[1] = 16*1024; sysctl_sctp_wmem[2] = max(64*1024, max_share); From patchwork Thu Jun 9 06:34:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12874854 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C530FC433EF for ; Thu, 9 Jun 2022 06:34:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238687AbiFIGe0 (ORCPT ); Thu, 9 Jun 2022 02:34:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238404AbiFIGeY (ORCPT ); Thu, 9 Jun 2022 02:34:24 -0400 Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA6B8326EE for ; Wed, 8 Jun 2022 23:34:22 -0700 (PDT) Received: by mail-pl1-x631.google.com with SMTP id h1so19489408plf.11 for ; Wed, 08 Jun 2022 23:34:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gDZCiATz4+88NA7QeL5ZEdLf1h8/4cV4veaCgP5CCF0=; b=FPckFk8k0LL06aVygqxmc0roO1XGokTuRUu7YWenlPNaDs8ZwT3x70eivmN2YDBujP J2IhNWlj6T5ilZf+spbpkXztc6iws+5g5Rdgi0JVwJBFz9znP62lojexlQxmsikR6lsT A6KUjivlgbkfAMi6eM2yawsW39WhMJSwF1EbtXBM9zTy1k4mF818EZaJB+Icm0YxnNEj TM69FebxlVGDrhmxclBGc7vxLsl1VJ6OjjRc1UtihkPUYHIU7KNifxu1eeVr8hYmc6+n QUy4aHBijhyA/iKi3QrLVjYsTg8ACRYLS1zI+ZDNcq7bQLyd4sXoBPcTUNhubBmcTBAT uTgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gDZCiATz4+88NA7QeL5ZEdLf1h8/4cV4veaCgP5CCF0=; b=FHjBOfWJuE6W030E9gUBzNKM7p6QlJAPt1O6yDh4H6R6bn8XduqlLtHh63E9ay6ToO D9trGyo3Nh1Dqtv1nQPAHT664/lwqcWnbsEfsHWCCIUTQyc7k1znjaYaxz7RSryqSTdr qsks2i1K3ArTIs7TaZ3OsUcETnog609QprRnJD/GwUnlUmLeXYKOfUs0+ZvY1k7zdQ6g Jr2h5sUdV8gPEXzcXoSiyQmV3vpNjmtivmAz8hyhGMq4Jh3X4lSGNFESwsOzKq2P/xr+ 1//tVig/2Je1N3yxb4x2872GOjrKtjlASwsfTGp5p75mfDFCs9rd3OC9NmdNGyOv9Rj6 PhDg== X-Gm-Message-State: AOAM530y5kk1jK2AuQt9ML3G5a9+zMViWx7qHfQE+zqY1TzKSW7sw1aj MfGBMAEtxwLbQJyO9XbcK5A= X-Google-Smtp-Source: ABdhPJyUBVE10o4y7h/V6mVF+OIGgN9M/uF5UQD5kvX1uUtzpYqOBDP8oY52knezhrH/ATU7uhSq9w== X-Received: by 2002:a17:902:d551:b0:168:93b6:a94a with SMTP id z17-20020a170902d55100b0016893b6a94amr7410806plf.149.1654756462367; Wed, 08 Jun 2022 23:34:22 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:f579:a14f:f943:9d9a]) by smtp.gmail.com with ESMTPSA id 199-20020a6215d0000000b0051b9c0af43dsm16340050pfv.155.2022.06.08.23.34.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jun 2022 23:34:22 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev , Soheil Hassas Yeganeh , Wei Wang , Shakeel Butt , Neal Cardwell , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 3/7] net: add per_cpu_fw_alloc field to struct proto Date: Wed, 8 Jun 2022 23:34:08 -0700 Message-Id: <20220609063412.2205738-4-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220609063412.2205738-1-eric.dumazet@gmail.com> References: <20220609063412.2205738-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Each protocol having a ->memory_allocated pointer gets a corresponding per-cpu reserve, that following patches will use. Instead of having reserved bytes per socket, we want to have per-cpu reserves. Signed-off-by: Eric Dumazet Reviewed-by: Shakeel Butt --- include/net/sock.h | 1 + include/net/tcp.h | 2 ++ include/net/udp.h | 1 + net/core/sock.c | 4 ++++ net/decnet/af_decnet.c | 4 ++++ net/ipv4/tcp.c | 2 ++ net/ipv4/tcp_ipv4.c | 3 +++ net/ipv4/udp.c | 4 ++++ net/ipv4/udplite.c | 3 +++ net/ipv6/tcp_ipv6.c | 3 +++ net/ipv6/udp.c | 3 +++ net/ipv6/udplite.c | 3 +++ net/mptcp/protocol.c | 3 +++ net/sctp/socket.c | 7 +++++++ 14 files changed, 43 insertions(+) diff --git a/include/net/sock.h b/include/net/sock.h index 298897bbfb3a3ea6ba88f76bc486ae636e2b1cfd..825f8cbf791f02d798f17dd4f7a2659cebb0e98a 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1254,6 +1254,7 @@ struct proto { void (*enter_memory_pressure)(struct sock *sk); void (*leave_memory_pressure)(struct sock *sk); atomic_long_t *memory_allocated; /* Current allocated memory. */ + int __percpu *per_cpu_fw_alloc; struct percpu_counter *sockets_allocated; /* Current number of sockets. */ /* diff --git a/include/net/tcp.h b/include/net/tcp.h index 1e99f5c61f8499c121a9fc50643db559a6021a38..4794cae4577e4c64ce2664ed734ae90bbc531782 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -253,6 +253,8 @@ extern long sysctl_tcp_mem[3]; #define TCP_RACK_NO_DUPTHRESH 0x4 /* Do not use DUPACK threshold in RACK */ extern atomic_long_t tcp_memory_allocated; +DECLARE_PER_CPU(int, tcp_memory_per_cpu_fw_alloc); + extern struct percpu_counter tcp_sockets_allocated; extern unsigned long tcp_memory_pressure; diff --git a/include/net/udp.h b/include/net/udp.h index b83a003305667d1c1cd1bac00580fec9164958b0..b60eea2e3fae2f3f2d2acfdffdb7f442bcef4478 100644 --- a/include/net/udp.h +++ b/include/net/udp.h @@ -95,6 +95,7 @@ static inline struct udp_hslot *udp_hashslot2(struct udp_table *table, extern struct proto udp_prot; extern atomic_long_t udp_memory_allocated; +DECLARE_PER_CPU(int, udp_memory_per_cpu_fw_alloc); /* sysctl variables for udp */ extern long sysctl_udp_mem[3]; diff --git a/net/core/sock.c b/net/core/sock.c index 6b786e836c7f5fc74307f050d4f32b4b554eb53b..3bb406167da93b7526ff85787b89fa65e44dce8b 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -3798,6 +3798,10 @@ int proto_register(struct proto *prot, int alloc_slab) pr_err("%s: missing sysctl_mem\n", prot->name); return -EINVAL; } + if (prot->memory_allocated && !prot->per_cpu_fw_alloc) { + pr_err("%s: missing per_cpu_fw_alloc\n", prot->name); + return -EINVAL; + } if (alloc_slab) { prot->slab = kmem_cache_create_usercopy(prot->name, prot->obj_size, 0, diff --git a/net/decnet/af_decnet.c b/net/decnet/af_decnet.c index dc92a67baea39484cd4c93913d3eae8ac4463538..aa4f43f52499b55309784de0a2b5a3e434e9a19c 100644 --- a/net/decnet/af_decnet.c +++ b/net/decnet/af_decnet.c @@ -149,6 +149,7 @@ static DEFINE_RWLOCK(dn_hash_lock); static struct hlist_head dn_sk_hash[DN_SK_HASH_SIZE]; static struct hlist_head dn_wild_sk; static atomic_long_t decnet_memory_allocated; +static DEFINE_PER_CPU(int, decnet_memory_per_cpu_fw_alloc); static int __dn_setsockopt(struct socket *sock, int level, int optname, sockptr_t optval, unsigned int optlen, int flags); @@ -454,7 +455,10 @@ static struct proto dn_proto = { .owner = THIS_MODULE, .enter_memory_pressure = dn_enter_memory_pressure, .memory_pressure = &dn_memory_pressure, + .memory_allocated = &decnet_memory_allocated, + .per_cpu_fw_alloc = &decnet_memory_per_cpu_fw_alloc, + .sysctl_mem = sysctl_decnet_mem, .sysctl_wmem = sysctl_decnet_wmem, .sysctl_rmem = sysctl_decnet_rmem, diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 9e696758a4c213f22919483dcd6740b10ee3294b..e6bdf8e2c09a156c45eae9490419b93b35f6e191 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -294,6 +294,8 @@ EXPORT_SYMBOL(sysctl_tcp_mem); atomic_long_t tcp_memory_allocated ____cacheline_aligned_in_smp; /* Current allocated memory. */ EXPORT_SYMBOL(tcp_memory_allocated); +DEFINE_PER_CPU(int, tcp_memory_per_cpu_fw_alloc); +EXPORT_PER_CPU_SYMBOL_GPL(tcp_memory_per_cpu_fw_alloc); #if IS_ENABLED(CONFIG_SMC) DEFINE_STATIC_KEY_FALSE(tcp_have_smc); diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index fe8f23b95d32ca4a35d05166d471327bc608fa91..fda811a5251f2d76ac24a036e6b4f4e7d7d96d6f 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -3045,7 +3045,10 @@ struct proto tcp_prot = { .stream_memory_free = tcp_stream_memory_free, .sockets_allocated = &tcp_sockets_allocated, .orphan_count = &tcp_orphan_count, + .memory_allocated = &tcp_memory_allocated, + .per_cpu_fw_alloc = &tcp_memory_per_cpu_fw_alloc, + .memory_pressure = &tcp_memory_pressure, .sysctl_mem = sysctl_tcp_mem, .sysctl_wmem_offset = offsetof(struct net, ipv4.sysctl_tcp_wmem), diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index bbc9970fa2e947ce8fdd08763033b6b5912af042..6172b4750a888bf792650a59f8ce0cd97d781fad 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -125,6 +125,8 @@ EXPORT_SYMBOL(sysctl_udp_mem); atomic_long_t udp_memory_allocated ____cacheline_aligned_in_smp; EXPORT_SYMBOL(udp_memory_allocated); +DEFINE_PER_CPU(int, udp_memory_per_cpu_fw_alloc); +EXPORT_PER_CPU_SYMBOL_GPL(udp_memory_per_cpu_fw_alloc); #define MAX_UDP_PORTS 65536 #define PORTS_PER_CHAIN (MAX_UDP_PORTS / UDP_HTABLE_SIZE_MIN) @@ -2946,6 +2948,8 @@ struct proto udp_prot = { .psock_update_sk_prot = udp_bpf_update_proto, #endif .memory_allocated = &udp_memory_allocated, + .per_cpu_fw_alloc = &udp_memory_per_cpu_fw_alloc, + .sysctl_mem = sysctl_udp_mem, .sysctl_wmem_offset = offsetof(struct net, ipv4.sysctl_udp_wmem_min), .sysctl_rmem_offset = offsetof(struct net, ipv4.sysctl_udp_rmem_min), diff --git a/net/ipv4/udplite.c b/net/ipv4/udplite.c index cd1cd68adeec88e4cfcc75c3f32b6243229df670..6e08a76ae1e7e13905fa13ea12e075b94308a8ff 100644 --- a/net/ipv4/udplite.c +++ b/net/ipv4/udplite.c @@ -51,7 +51,10 @@ struct proto udplite_prot = { .unhash = udp_lib_unhash, .rehash = udp_v4_rehash, .get_port = udp_v4_get_port, + .memory_allocated = &udp_memory_allocated, + .per_cpu_fw_alloc = &udp_memory_per_cpu_fw_alloc, + .sysctl_mem = sysctl_udp_mem, .obj_size = sizeof(struct udp_sock), .h.udp_table = &udplite_table, diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index f37dd4aa91c6bae1df92a1e4edca362c50cd97b9..c72448ba6dc9c9e8762f2ff84e55fd5d50987c49 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -2159,7 +2159,10 @@ struct proto tcpv6_prot = { .leave_memory_pressure = tcp_leave_memory_pressure, .stream_memory_free = tcp_stream_memory_free, .sockets_allocated = &tcp_sockets_allocated, + .memory_allocated = &tcp_memory_allocated, + .per_cpu_fw_alloc = &tcp_memory_per_cpu_fw_alloc, + .memory_pressure = &tcp_memory_pressure, .orphan_count = &tcp_orphan_count, .sysctl_mem = sysctl_tcp_mem, diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 55afd7f39c0450ff442a0499b7f8e42bf1a613bc..be074f07073a532d47b83497a2b6808f4271d43e 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -1740,7 +1740,10 @@ struct proto udpv6_prot = { #ifdef CONFIG_BPF_SYSCALL .psock_update_sk_prot = udp_bpf_update_proto, #endif + .memory_allocated = &udp_memory_allocated, + .per_cpu_fw_alloc = &udp_memory_per_cpu_fw_alloc, + .sysctl_mem = sysctl_udp_mem, .sysctl_wmem_offset = offsetof(struct net, ipv4.sysctl_udp_wmem_min), .sysctl_rmem_offset = offsetof(struct net, ipv4.sysctl_udp_rmem_min), diff --git a/net/ipv6/udplite.c b/net/ipv6/udplite.c index fbb700d3f437ee73824b369486010e152a659abb..b707258562597ebddf5e0d75e6415b6f967cca33 100644 --- a/net/ipv6/udplite.c +++ b/net/ipv6/udplite.c @@ -48,7 +48,10 @@ struct proto udplitev6_prot = { .unhash = udp_lib_unhash, .rehash = udp_v6_rehash, .get_port = udp_v6_get_port, + .memory_allocated = &udp_memory_allocated, + .per_cpu_fw_alloc = &udp_memory_per_cpu_fw_alloc, + .sysctl_mem = sysctl_udp_mem, .obj_size = sizeof(struct udp6_sock), .h.udp_table = &udplite_table, diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 080a630d6902caa2022fda1c6b3edb65e4e74a8c..9563124ac8af9f9f45b16f23d5d77f17dd83d0c5 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -3437,7 +3437,10 @@ static struct proto mptcp_prot = { .get_port = mptcp_get_port, .forward_alloc_get = mptcp_forward_alloc_get, .sockets_allocated = &mptcp_sockets_allocated, + .memory_allocated = &tcp_memory_allocated, + .per_cpu_fw_alloc = &tcp_memory_per_cpu_fw_alloc, + .memory_pressure = &tcp_memory_pressure, .sysctl_wmem_offset = offsetof(struct net, ipv4.sysctl_tcp_wmem), .sysctl_rmem_offset = offsetof(struct net, ipv4.sysctl_tcp_rmem), diff --git a/net/sctp/socket.c b/net/sctp/socket.c index 6d37d2dfb3da87fd509c21121d743f44bf6ee00c..05174acd981a8c304300b7a39b366d758d6fbafa 100644 --- a/net/sctp/socket.c +++ b/net/sctp/socket.c @@ -93,6 +93,7 @@ static int sctp_sock_migrate(struct sock *oldsk, struct sock *newsk, static unsigned long sctp_memory_pressure; static atomic_long_t sctp_memory_allocated; +static DEFINE_PER_CPU(int, sctp_memory_per_cpu_fw_alloc); struct percpu_counter sctp_sockets_allocated; static void sctp_enter_memory_pressure(struct sock *sk) @@ -9657,7 +9658,10 @@ struct proto sctp_prot = { .sysctl_wmem = sysctl_sctp_wmem, .memory_pressure = &sctp_memory_pressure, .enter_memory_pressure = sctp_enter_memory_pressure, + .memory_allocated = &sctp_memory_allocated, + .per_cpu_fw_alloc = &sctp_memory_per_cpu_fw_alloc, + .sockets_allocated = &sctp_sockets_allocated, }; @@ -9700,7 +9704,10 @@ struct proto sctpv6_prot = { .sysctl_wmem = sysctl_sctp_wmem, .memory_pressure = &sctp_memory_pressure, .enter_memory_pressure = sctp_enter_memory_pressure, + .memory_allocated = &sctp_memory_allocated, + .per_cpu_fw_alloc = &sctp_memory_per_cpu_fw_alloc, + .sockets_allocated = &sctp_sockets_allocated, }; #endif /* IS_ENABLED(CONFIG_IPV6) */ From patchwork Thu Jun 9 06:34:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12874855 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8585AC433EF for ; Thu, 9 Jun 2022 06:34:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239051AbiFIGef (ORCPT ); Thu, 9 Jun 2022 02:34:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229754AbiFIGeZ (ORCPT ); Thu, 9 Jun 2022 02:34:25 -0400 Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54FFF2FFD8 for ; Wed, 8 Jun 2022 23:34:24 -0700 (PDT) Received: by mail-pj1-x1034.google.com with SMTP id w13-20020a17090a780d00b001e8961b355dso8656156pjk.5 for ; Wed, 08 Jun 2022 23:34:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=66EdC7/JGy2H2xFnuQqp5vgFdI8PL+EsBB1vb5TBCso=; b=JT93uofFNR9zTsGDR9FhhpalfY0RofLtsLsn6RrilTSkhVQHCiHbEP34gNggdQHsuK JaBETh8IfrFjmPbqT9EhxaUbw3CjAHrbRuRAsd3igsHm3CxZh2s6vY9tJMjesz6EYcNb 3a0AbVuQH05nxYHa4+eZyxqVXto491dVnOckd3wI4tGvI1DODQT9HSCY7gDkzf9kgc5Q aG8LLDNPrFjFV+YhUsmrQTMVIP7qOyB5zMsgqYC+2852c8LHtJp/DBlYMiR5rsKBo74u Yw4YThg97dqk+gezXWd+aza+M6WHdFSbNmySHwXR7SkUIiSaCXNNKChaT2U8hmzykkyk LWkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=66EdC7/JGy2H2xFnuQqp5vgFdI8PL+EsBB1vb5TBCso=; b=b62mUgpsZJ0R/QHtNDO/lAxyolCUVVzOnohyJnDS49SW8Skv5Kz3V6MwywLZSsNPZF S0fmko0jAyJoJg7xcHc+fOy07/NHVsAgNIbQ5mqAC3NNhgx3f0p1mUyXl+7odMjHzySH Gi2x6CbmG89+fzEnpsZXbshqO0JN7s9CEz76nnTN86JSPXGk1U/+pbnIU4+lcejUbkWj qc3O8yKx0fNML5L6+fR7b3oCLPObBak1Nv5rOsfx4rIwlk8Er3s+hSyMpwpYMmJdpFpk U/jwAoL5xHLC8/0K9R5GE6LtObpUVwSuBe2csCAceMqNsYYQJG5qOyClVXNJUeX7xfXv WPvQ== X-Gm-Message-State: AOAM530GJBqsiuoIhIprK1mV8KASnOGNrh9DKDXEXCThW/5sHIHjepdH iBDqm2jauaMx9nvojbWq0UEWhujhl1A= X-Google-Smtp-Source: ABdhPJxdwp+Q7Jbo95DhOyiX+Wo34gKz1fYIeyfsTxcN95WW76tn0wBM0RG36j3DSP95Ws1TRE3GFQ== X-Received: by 2002:a17:90b:1a8d:b0:1e8:a809:af4b with SMTP id ng13-20020a17090b1a8d00b001e8a809af4bmr1931721pjb.76.1654756463872; Wed, 08 Jun 2022 23:34:23 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:f579:a14f:f943:9d9a]) by smtp.gmail.com with ESMTPSA id 199-20020a6215d0000000b0051b9c0af43dsm16340050pfv.155.2022.06.08.23.34.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jun 2022 23:34:23 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev , Soheil Hassas Yeganeh , Wei Wang , Shakeel Butt , Neal Cardwell , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 4/7] net: implement per-cpu reserves for memory_allocated Date: Wed, 8 Jun 2022 23:34:09 -0700 Message-Id: <20220609063412.2205738-5-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220609063412.2205738-1-eric.dumazet@gmail.com> References: <20220609063412.2205738-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet We plan keeping sk->sk_forward_alloc as small as possible in future patches. This means we are going to call sk_memory_allocated_add() and sk_memory_allocated_sub() more often. Implement a per-cpu cache of +1/-1 MB, to reduce number of changes to sk->sk_prot->memory_allocated, which would otherwise be cause of false sharing. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh Reviewed-by: Shakeel Butt --- include/net/sock.h | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 825f8cbf791f02d798f17dd4f7a2659cebb0e98a..59040fee74e7de8d63fbf719f46e172906c134bb 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1397,22 +1397,48 @@ static inline bool sk_under_memory_pressure(const struct sock *sk) return !!*sk->sk_prot->memory_pressure; } +static inline long +proto_memory_allocated(const struct proto *prot) +{ + return max(0L, atomic_long_read(prot->memory_allocated)); +} + static inline long sk_memory_allocated(const struct sock *sk) { - return atomic_long_read(sk->sk_prot->memory_allocated); + return proto_memory_allocated(sk->sk_prot); } +/* 1 MB per cpu, in page units */ +#define SK_MEMORY_PCPU_RESERVE (1 << (20 - PAGE_SHIFT)) + static inline long sk_memory_allocated_add(struct sock *sk, int amt) { - return atomic_long_add_return(amt, sk->sk_prot->memory_allocated); + int local_reserve; + + preempt_disable(); + local_reserve = __this_cpu_add_return(*sk->sk_prot->per_cpu_fw_alloc, amt); + if (local_reserve >= SK_MEMORY_PCPU_RESERVE) { + __this_cpu_sub(*sk->sk_prot->per_cpu_fw_alloc, local_reserve); + atomic_long_add(local_reserve, sk->sk_prot->memory_allocated); + } + preempt_enable(); + return sk_memory_allocated(sk); } static inline void sk_memory_allocated_sub(struct sock *sk, int amt) { - atomic_long_sub(amt, sk->sk_prot->memory_allocated); + int local_reserve; + + preempt_disable(); + local_reserve = __this_cpu_sub_return(*sk->sk_prot->per_cpu_fw_alloc, amt); + if (local_reserve <= -SK_MEMORY_PCPU_RESERVE) { + __this_cpu_sub(*sk->sk_prot->per_cpu_fw_alloc, local_reserve); + atomic_long_add(local_reserve, sk->sk_prot->memory_allocated); + } + preempt_enable(); } #define SK_ALLOC_PERCPU_COUNTER_BATCH 16 @@ -1441,12 +1467,6 @@ proto_sockets_allocated_sum_positive(struct proto *prot) return percpu_counter_sum_positive(prot->sockets_allocated); } -static inline long -proto_memory_allocated(struct proto *prot) -{ - return atomic_long_read(prot->memory_allocated); -} - static inline bool proto_memory_pressure(struct proto *prot) { From patchwork Thu Jun 9 06:34:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12874856 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C432C433EF for ; Thu, 9 Jun 2022 06:34:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239001AbiFIGei (ORCPT ); Thu, 9 Jun 2022 02:34:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50496 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238970AbiFIGe1 (ORCPT ); Thu, 9 Jun 2022 02:34:27 -0400 Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E955C369D3 for ; Wed, 8 Jun 2022 23:34:25 -0700 (PDT) Received: by mail-pg1-x535.google.com with SMTP id e66so20959798pgc.8 for ; Wed, 08 Jun 2022 23:34:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l94o/1eJJb3mEn0w56d5zkZI41cFtm3Fy1loLlVyw3k=; b=fDJiMy6WyYd1usVQpjedtmNc/cAmLBkC85c0aOZLKpK6Z3y3oBvESQ462UUB4iwSMx ucD7TV9oIcPb93CbeYIazzTWPfcYRN8G4hGYpT3uEReY2JJibCvhz2LeN8dAobzkh1Kd cbUmYrjXQFkAFP9d4q1TOTvQpLJ24ebOuCg+F/3qI7WIbGmdHMN1bMMHlcxheJv1nxu8 GdD4/xSqfPPq9MiJFcCwGJklA/fr2T2tLqMC7t6UOOjyXfwpt1evxtC6a3AkT6kh9mid CVWNB3xM67i6h2RX9JWMIRk9iTCfnp/0L7/hfIXmgC2FMLYC+jwfb6zfgODQk50UKosC UM6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l94o/1eJJb3mEn0w56d5zkZI41cFtm3Fy1loLlVyw3k=; b=UH1d+DQsuByF8aedumYk+G4C7aS+R95zoxhpF4vnYdW6evhT50V9ZmHORxCppuSZSC lYlw7EGG3EEQ+C9cDutXFVQBpA2SIT0OATaIC74s6+iRHp1JqZon/Fsi6bdcJxcrN7Om 9xsTfbWmG+PdKiCLxHGlYmw5HJuHY0F33zIzqI4DJQiREUgSEQeMMzipM8OEQddLfGBm lzNgNAvGwwAA5NWDc7artL7rt/2MY5nZ6vBsukSovbAXaSHg5MM6OXGKrC1PZ5aCnmlR JwaH9N2PZahSO1QscDCIo/xy0mnfTYfUtEMQcDNqovktoYM0cp/6j+WMMA4CxmOmaU2W Vokg== X-Gm-Message-State: AOAM531EyGhPi8JKBlKbhF/ssjsrZnAKCg67L19EYrRgEepBp9J6ftkd CDPDhCjVV3WCq/DlXPfPOVmrU2hR14U= X-Google-Smtp-Source: ABdhPJw3Cp83RP4AOWRMrrD4CfvWfqwy59DpORe4O1n+c2PKK7dLIcYh24NN41x3N1iVPqIJ/sNPXg== X-Received: by 2002:a65:6459:0:b0:3fc:700a:ce88 with SMTP id s25-20020a656459000000b003fc700ace88mr33882596pgv.143.1654756465338; Wed, 08 Jun 2022 23:34:25 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:f579:a14f:f943:9d9a]) by smtp.gmail.com with ESMTPSA id 199-20020a6215d0000000b0051b9c0af43dsm16340050pfv.155.2022.06.08.23.34.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jun 2022 23:34:25 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev , Soheil Hassas Yeganeh , Wei Wang , Shakeel Butt , Neal Cardwell , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 5/7] net: fix sk_wmem_schedule() and sk_rmem_schedule() errors Date: Wed, 8 Jun 2022 23:34:10 -0700 Message-Id: <20220609063412.2205738-6-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220609063412.2205738-1-eric.dumazet@gmail.com> References: <20220609063412.2205738-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet If sk->sk_forward_alloc is 150000, and we need to schedule 150001 bytes, we want to allocate 1 byte more (rounded up to one page), instead of 150001 :/ Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2") Signed-off-by: Eric Dumazet Reviewed-by: Shakeel Butt --- include/net/sock.h | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 59040fee74e7de8d63fbf719f46e172906c134bb..cf288f7e9019106dfb466be707d34dacf33b339c 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1575,19 +1575,23 @@ static inline bool sk_has_account(struct sock *sk) static inline bool sk_wmem_schedule(struct sock *sk, int size) { + int delta; + if (!sk_has_account(sk)) return true; - return size <= sk->sk_forward_alloc || - __sk_mem_schedule(sk, size, SK_MEM_SEND); + delta = size - sk->sk_forward_alloc; + return delta <= 0 || __sk_mem_schedule(sk, delta, SK_MEM_SEND); } static inline bool sk_rmem_schedule(struct sock *sk, struct sk_buff *skb, int size) { + int delta; + if (!sk_has_account(sk)) return true; - return size <= sk->sk_forward_alloc || - __sk_mem_schedule(sk, size, SK_MEM_RECV) || + delta = size - sk->sk_forward_alloc; + return delta <= 0 || __sk_mem_schedule(sk, delta, SK_MEM_RECV) || skb_pfmemalloc(skb); } From patchwork Thu Jun 9 06:34:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12874857 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA909C43334 for ; Thu, 9 Jun 2022 06:34:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239026AbiFIGel (ORCPT ); Thu, 9 Jun 2022 02:34:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239019AbiFIGef (ORCPT ); Thu, 9 Jun 2022 02:34:35 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C66EB39B8A for ; Wed, 8 Jun 2022 23:34:27 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id v11-20020a17090a4ecb00b001e2c5b837ccso25829741pjl.3 for ; Wed, 08 Jun 2022 23:34:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=33tdRE/1/Zw7WdIazFpVj3yQuE9jkTZYCHdzZhOXJcs=; b=fkkK83E5ugE6pPmDPxjKafb9ijLJryzgUiIKaZgIqGApmwRGToEeMy9X1tYqiBY4Sj z/1F4gYJOFO8hv341bVaFKKPYFXfxRAjHXTHn2XAxn2yO+aEZrn3F/C3uqpYNFec8keB nkrSDyab2qQmjtxJCpiILZvGnPqj/YIu9dF99ge4B/xlYcgZQnP4mSicsugolDCFiBLq zHEqaO9SbM2o6VsBmreLTY90gBvUcjaXNF+BDgmwFzFpiFTzljAhUuntxTpojfW6I64O xhcFCe2Rqxr228yoWwRDTwBum7dazuoYwOOXFBcS9a5ikD67N7IfxuYOCBu/tnrrzWMt sawA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=33tdRE/1/Zw7WdIazFpVj3yQuE9jkTZYCHdzZhOXJcs=; b=0Mihhu4G/6k1W5bLhDfrtIKjwaNRNgUM38N+S4/Ab4g171eTiASFtiyBNPt5dT5YaH 8ZZlZ6uGzYoVZzKUgMvs4lHU5ZVwH2MX6WPPqLn2L4J2HVpRusuNcXw6uKpkQerNDZN2 jntT6noNpW6LWyT8kj892q28I3igW5T2fK4TOiVuYdiIoP4tyi2LGJgSAk+4aJ3wMeRl ywRzaNmvYxjPruTkmn0GGxS3xzc8RXFF3V4iQGnxKyyKsPJPeHoi0OeVOaMeR+M/rVBm ILh3AraLHGdlC5aG/W8P2r+WwwpYG7jamCm1tQhi7OY/nCLiLiU3GkjTyGyk+Gom89N1 C7JA== X-Gm-Message-State: AOAM532/p3pIp73a86pHh5Zrko1e2nyCQzjrnARhyimAMSRMcuIqniU1 zCCWMbpZ5SlpV39VsoJX8yM= X-Google-Smtp-Source: ABdhPJwoWgOyubbQ+wx20Ua4W85jICqQg1g9XtUElDd7E75vlgAPWp8TMNXS4qx4GAYjl0dZ1H6hyQ== X-Received: by 2002:a17:902:ecca:b0:166:3e34:4d01 with SMTP id a10-20020a170902ecca00b001663e344d01mr36813281plh.97.1654756467165; Wed, 08 Jun 2022 23:34:27 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:f579:a14f:f943:9d9a]) by smtp.gmail.com with ESMTPSA id 199-20020a6215d0000000b0051b9c0af43dsm16340050pfv.155.2022.06.08.23.34.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jun 2022 23:34:26 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev , Soheil Hassas Yeganeh , Wei Wang , Shakeel Butt , Neal Cardwell , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 6/7] net: keep sk->sk_forward_alloc as small as possible Date: Wed, 8 Jun 2022 23:34:11 -0700 Message-Id: <20220609063412.2205738-7-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220609063412.2205738-1-eric.dumazet@gmail.com> References: <20220609063412.2205738-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Currently, tcp_memory_allocated can hit tcp_mem[] limits quite fast. Each TCP socket can forward allocate up to 2 MB of memory, even after flow became less active. 10,000 sockets can have reserved 20 GB of memory, and we have no shrinker in place to reclaim that. Instead of trying to reclaim the extra allocations in some places, just keep sk->sk_forward_alloc values as small as possible. This should not impact performance too much now we have per-cpu reserves: Changes to tcp_memory_allocated should not be too frequent. For sockets not using SO_RESERVE_MEM: - idle sockets (no packets in tx/rx queues) have zero forward alloc. - non idle sockets have a forward alloc smaller than one page. Note: - Removal of SK_RECLAIM_CHUNK and SK_RECLAIM_THRESHOLD is left to MPTCP maintainers as a follow up. Signed-off-by: Eric Dumazet Reviewed-by: Shakeel Butt --- include/net/sock.h | 29 ++--------------------------- net/core/datagram.c | 3 --- net/ipv4/tcp.c | 7 ------- net/ipv4/tcp_input.c | 4 ---- net/ipv4/tcp_timer.c | 19 ++++--------------- net/iucv/af_iucv.c | 2 -- net/mptcp/protocol.c | 2 +- net/sctp/sm_statefuns.c | 2 -- net/sctp/socket.c | 5 ----- net/sctp/stream_interleave.c | 2 -- net/sctp/ulpqueue.c | 4 ---- 11 files changed, 7 insertions(+), 72 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index cf288f7e9019106dfb466be707d34dacf33b339c..0063e8410a4e3ed91aef9cf34eb1127f7ce33b93 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1627,19 +1627,6 @@ static inline void sk_mem_reclaim_final(struct sock *sk) sk_mem_reclaim(sk); } -static inline void sk_mem_reclaim_partial(struct sock *sk) -{ - int reclaimable; - - if (!sk_has_account(sk)) - return; - - reclaimable = sk->sk_forward_alloc - sk_unused_reserved_mem(sk); - - if (reclaimable > (int)PAGE_SIZE) - __sk_mem_reclaim(sk, reclaimable - 1); -} - static inline void sk_mem_charge(struct sock *sk, int size) { if (!sk_has_account(sk)) @@ -1647,29 +1634,17 @@ static inline void sk_mem_charge(struct sock *sk, int size) sk->sk_forward_alloc -= size; } -/* the following macros control memory reclaiming in sk_mem_uncharge() +/* the following macros control memory reclaiming in mptcp_rmem_uncharge() */ #define SK_RECLAIM_THRESHOLD (1 << 21) #define SK_RECLAIM_CHUNK (1 << 20) static inline void sk_mem_uncharge(struct sock *sk, int size) { - int reclaimable; - if (!sk_has_account(sk)) return; sk->sk_forward_alloc += size; - reclaimable = sk->sk_forward_alloc - sk_unused_reserved_mem(sk); - - /* Avoid a possible overflow. - * TCP send queues can make this happen, if sk_mem_reclaim() - * is not called and more than 2 GBytes are released at once. - * - * If we reach 2 MBytes, reclaim 1 MBytes right now, there is - * no need to hold that much forward allocation anyway. - */ - if (unlikely(reclaimable >= SK_RECLAIM_THRESHOLD)) - __sk_mem_reclaim(sk, SK_RECLAIM_CHUNK); + sk_mem_reclaim(sk); } /* diff --git a/net/core/datagram.c b/net/core/datagram.c index 50f4faeea76cc9b7ac48f7814c3badf1251421a5..35791f86bd1a0b96352dab5657ebb730db4ada2c 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -320,7 +320,6 @@ EXPORT_SYMBOL(skb_recv_datagram); void skb_free_datagram(struct sock *sk, struct sk_buff *skb) { consume_skb(skb); - sk_mem_reclaim_partial(sk); } EXPORT_SYMBOL(skb_free_datagram); @@ -336,7 +335,6 @@ void __skb_free_datagram_locked(struct sock *sk, struct sk_buff *skb, int len) slow = lock_sock_fast(sk); sk_peek_offset_bwd(sk, len); skb_orphan(skb); - sk_mem_reclaim_partial(sk); unlock_sock_fast(sk, slow); /* skb is now orphaned, can be freed outside of locked section */ @@ -396,7 +394,6 @@ int skb_kill_datagram(struct sock *sk, struct sk_buff *skb, unsigned int flags) NULL); kfree_skb(skb); - sk_mem_reclaim_partial(sk); return err; } EXPORT_SYMBOL(skb_kill_datagram); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index e6bdf8e2c09a156c45eae9490419b93b35f6e191..14ebb4ec4a51f3c55501aa53423ce897599e8637 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -858,9 +858,6 @@ struct sk_buff *tcp_stream_alloc_skb(struct sock *sk, int size, gfp_t gfp, { struct sk_buff *skb; - if (unlikely(tcp_under_memory_pressure(sk))) - sk_mem_reclaim_partial(sk); - skb = alloc_skb_fclone(size + MAX_TCP_HEADER, gfp); if (likely(skb)) { bool mem_scheduled; @@ -2764,8 +2761,6 @@ void __tcp_close(struct sock *sk, long timeout) __kfree_skb(skb); } - sk_mem_reclaim(sk); - /* If socket has been already reset (e.g. in tcp_reset()) - kill it. */ if (sk->sk_state == TCP_CLOSE) goto adjudge_to_death; @@ -2873,7 +2868,6 @@ void __tcp_close(struct sock *sk, long timeout) } } if (sk->sk_state != TCP_CLOSE) { - sk_mem_reclaim(sk); if (tcp_check_oom(sk, 0)) { tcp_set_state(sk, TCP_CLOSE); tcp_send_active_reset(sk, GFP_ATOMIC); @@ -2951,7 +2945,6 @@ void tcp_write_queue_purge(struct sock *sk) } tcp_rtx_queue_purge(sk); INIT_LIST_HEAD(&tcp_sk(sk)->tsorted_sent_queue); - sk_mem_reclaim(sk); tcp_clear_all_retrans_hints(tcp_sk(sk)); tcp_sk(sk)->packets_out = 0; inet_csk(sk)->icsk_backoff = 0; diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 3fb117022558a408a664ea7c8fe2303296247ead..fdc7beb81b684560ed9e66b24464d870fdd6650e 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -805,7 +805,6 @@ static void tcp_event_data_recv(struct sock *sk, struct sk_buff *skb) * restart window, so that we send ACKs quickly. */ tcp_incr_quickack(sk, TCP_MAX_QUICKACKS); - sk_mem_reclaim(sk); } } icsk->icsk_ack.lrcvtime = now; @@ -4390,7 +4389,6 @@ void tcp_fin(struct sock *sk) skb_rbtree_purge(&tp->out_of_order_queue); if (tcp_is_sack(tp)) tcp_sack_reset(&tp->rx_opt); - sk_mem_reclaim(sk); if (!sock_flag(sk, SOCK_DEAD)) { sk->sk_state_change(sk); @@ -5336,7 +5334,6 @@ static bool tcp_prune_ofo_queue(struct sock *sk) tcp_drop_reason(sk, rb_to_skb(node), SKB_DROP_REASON_TCP_OFO_QUEUE_PRUNE); if (!prev || goal <= 0) { - sk_mem_reclaim(sk); if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf && !tcp_under_memory_pressure(sk)) break; @@ -5383,7 +5380,6 @@ static int tcp_prune_queue(struct sock *sk) skb_peek(&sk->sk_receive_queue), NULL, tp->copied_seq, tp->rcv_nxt); - sk_mem_reclaim(sk); if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf) return 0; diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c index 20cf4a98c69d85d07c884d7bc8316191ff962bd8..2208755e8efc6e6847459aa222cd576aadc0f387 100644 --- a/net/ipv4/tcp_timer.c +++ b/net/ipv4/tcp_timer.c @@ -290,15 +290,13 @@ void tcp_delack_timer_handler(struct sock *sk) { struct inet_connection_sock *icsk = inet_csk(sk); - sk_mem_reclaim_partial(sk); - if (((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN)) || !(icsk->icsk_ack.pending & ICSK_ACK_TIMER)) - goto out; + return; if (time_after(icsk->icsk_ack.timeout, jiffies)) { sk_reset_timer(sk, &icsk->icsk_delack_timer, icsk->icsk_ack.timeout); - goto out; + return; } icsk->icsk_ack.pending &= ~ICSK_ACK_TIMER; @@ -317,10 +315,6 @@ void tcp_delack_timer_handler(struct sock *sk) tcp_send_ack(sk); __NET_INC_STATS(sock_net(sk), LINUX_MIB_DELAYEDACKS); } - -out: - if (tcp_under_memory_pressure(sk)) - sk_mem_reclaim(sk); } @@ -600,11 +594,11 @@ void tcp_write_timer_handler(struct sock *sk) if (((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN)) || !icsk->icsk_pending) - goto out; + return; if (time_after(icsk->icsk_timeout, jiffies)) { sk_reset_timer(sk, &icsk->icsk_retransmit_timer, icsk->icsk_timeout); - goto out; + return; } tcp_mstamp_refresh(tcp_sk(sk)); @@ -626,9 +620,6 @@ void tcp_write_timer_handler(struct sock *sk) tcp_probe_timer(sk); break; } - -out: - sk_mem_reclaim(sk); } static void tcp_write_timer(struct timer_list *t) @@ -743,8 +734,6 @@ static void tcp_keepalive_timer (struct timer_list *t) elapsed = keepalive_time_when(tp) - elapsed; } - sk_mem_reclaim(sk); - resched: inet_csk_reset_keepalive_timer (sk, elapsed); goto out; diff --git a/net/iucv/af_iucv.c b/net/iucv/af_iucv.c index a0385ddbffcfc775b74768d5a889fcf7cbec1ee8..498a0c35b7bb20c9eae5aa8e5fee78170c08e831 100644 --- a/net/iucv/af_iucv.c +++ b/net/iucv/af_iucv.c @@ -278,8 +278,6 @@ static void iucv_sock_destruct(struct sock *sk) skb_queue_purge(&sk->sk_receive_queue); skb_queue_purge(&sk->sk_error_queue); - sk_mem_reclaim(sk); - if (!sock_flag(sk, SOCK_DEAD)) { pr_err("Attempt to release alive iucv socket %p\n", sk); return; diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 9563124ac8af9f9f45b16f23d5d77f17dd83d0c5..e0fb9f96c45c81e70f8f72dd1df7c49ff3150370 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -975,7 +975,7 @@ static void __mptcp_mem_reclaim_partial(struct sock *sk) if (reclaimable > (int)PAGE_SIZE) __mptcp_rmem_reclaim(sk, reclaimable - 1); - sk_mem_reclaim_partial(sk); + sk_mem_reclaim(sk); } static void mptcp_mem_reclaim_partial(struct sock *sk) diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c index 52edee1322fc36b0400569f8e6de2133adf6cd74..f6ee7f4040c14ef087ab8faca4552a2f9c505e9a 100644 --- a/net/sctp/sm_statefuns.c +++ b/net/sctp/sm_statefuns.c @@ -6590,8 +6590,6 @@ static int sctp_eat_data(const struct sctp_association *asoc, pr_debug("%s: under pressure, reneging for tsn:%u\n", __func__, tsn); deliver = SCTP_CMD_RENEGE; - } else { - sk_mem_reclaim(sk); } } diff --git a/net/sctp/socket.c b/net/sctp/socket.c index 05174acd981a8c304300b7a39b366d758d6fbafa..171f1a35d205225cdec9dbc6d0a07f446ce92996 100644 --- a/net/sctp/socket.c +++ b/net/sctp/socket.c @@ -1824,9 +1824,6 @@ static int sctp_sendmsg_to_asoc(struct sctp_association *asoc, if (sctp_wspace(asoc) < (int)msg_len) sctp_prsctp_prune(asoc, sinfo, msg_len - sctp_wspace(asoc)); - if (sk_under_memory_pressure(sk)) - sk_mem_reclaim(sk); - if (sctp_wspace(asoc) <= 0 || !sk_wmem_schedule(sk, msg_len)) { timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); err = sctp_wait_for_sndbuf(asoc, &timeo, msg_len); @@ -9195,8 +9192,6 @@ static int sctp_wait_for_sndbuf(struct sctp_association *asoc, long *timeo_p, goto do_error; if (signal_pending(current)) goto do_interrupted; - if (sk_under_memory_pressure(sk)) - sk_mem_reclaim(sk); if ((int)msg_len <= sctp_wspace(asoc) && sk_wmem_schedule(sk, msg_len)) break; diff --git a/net/sctp/stream_interleave.c b/net/sctp/stream_interleave.c index 6b13f737ebf2e437114ea1fae6157aab7b1a6fb8..bb22b71df7a34202748faf93e8b4a4a08ab312ca 100644 --- a/net/sctp/stream_interleave.c +++ b/net/sctp/stream_interleave.c @@ -979,8 +979,6 @@ static void sctp_renege_events(struct sctp_ulpq *ulpq, struct sctp_chunk *chunk, if (freed >= needed && sctp_ulpevent_idata(ulpq, chunk, gfp) <= 0) sctp_intl_start_pd(ulpq, gfp); - - sk_mem_reclaim(asoc->base.sk); } static void sctp_intl_stream_abort_pd(struct sctp_ulpq *ulpq, __u16 sid, diff --git a/net/sctp/ulpqueue.c b/net/sctp/ulpqueue.c index 407fed46931b9f1a7f07f3d1657b24eb2a39a147..0a8510a0c5e69789f8911cacd59aeb9c4b5ae20d 100644 --- a/net/sctp/ulpqueue.c +++ b/net/sctp/ulpqueue.c @@ -1100,12 +1100,8 @@ void sctp_ulpq_renege(struct sctp_ulpq *ulpq, struct sctp_chunk *chunk, else if (retval == 1) sctp_ulpq_reasm_drain(ulpq); } - - sk_mem_reclaim(asoc->base.sk); } - - /* Notify the application if an association is aborted and in * partial delivery mode. Send up any pending received messages. */ From patchwork Thu Jun 9 06:34:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12874858 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29FA4C43334 for ; Thu, 9 Jun 2022 06:34:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239128AbiFIGew (ORCPT ); Thu, 9 Jun 2022 02:34:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51244 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239034AbiFIGef (ORCPT ); Thu, 9 Jun 2022 02:34:35 -0400 Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 64D5C3AA42 for ; Wed, 8 Jun 2022 23:34:29 -0700 (PDT) Received: by mail-pf1-x42f.google.com with SMTP id g205so20252424pfb.11 for ; Wed, 08 Jun 2022 23:34:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yXHe682qxb99LehCpjB+F9buG2tUtQQhdUB4QoqRzoo=; b=FsLxodBjjbi6jYx8bETl7Wi6ja+9KCLgujuxRa+DGWnIO8bUJ477SqdRV/aUNyKIwj M0bQdTnWuKx/uR5Jk5W0pG2xPFlsVXBPv/eeAjA/Ost+GqiTKIwMWEGx1yiWzncBmSnS W50AqnuKyZkhSEUl9cxVL4ENiuQFbdaO8TV2YL+G6tW3diJW69EOGVj06WTdrZAenh+g yQlOkxwPnFcm+SXofd6RM1D2HwCQ79ARXVxwUNFam1MeAJ+RE+DFPQIKxG0d+lk7h54O pn+zNlhcH5e/c6CxFC/YYZBK5HIDjF84y1ezh/0/w65PP5e4NT6wmDIelREcdq8bLy3e AOeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yXHe682qxb99LehCpjB+F9buG2tUtQQhdUB4QoqRzoo=; b=yAFKeCrzsTKWmwnGbKEOWEWJ1EZRDQowIH2FM29hrF2UlnATAWX1jIRDVCRaFQTFXn THMlG8c4GnyFD7GW0GZDcjicD8mV0HedZepU/Hcc84dm53HFVjPtv7jFOHKRayzhQ4St KEYYbrPiq/I37wrDR678r45npnN31KptuWPm96u5e2UMN34ha5nQJ+fYHRRf0B/PSuYV heV+c1GEIESaBQGqnVL87odx7YxELgPMC3WWgEBnQufAuJDAVhYD/NqMHhf9ERae67Hu 8ZxP4xRoEwpZR9ZEQnKF8ml+yDx+luoZjOJqhsCv0jXWIro/rs7RvjjJVaVFMOAsUah8 UZTw== X-Gm-Message-State: AOAM531tnonfrXFSKpdQrUjDOylpbdHKUywLREPBXvItlSAhUkeqmBZE UzXQj7Qu/c8sv3yhYcchiQg= X-Google-Smtp-Source: ABdhPJyW+DKbWB18MJlsg2SQf9yigWTiX4+LCHGmQKi66LAdwPXZHc0XFRn/N3O8fF5PoLKqpQwSYg== X-Received: by 2002:a63:c00c:0:b0:3fc:dc35:6192 with SMTP id h12-20020a63c00c000000b003fcdc356192mr32444806pgg.7.1654756468910; Wed, 08 Jun 2022 23:34:28 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:f579:a14f:f943:9d9a]) by smtp.gmail.com with ESMTPSA id 199-20020a6215d0000000b0051b9c0af43dsm16340050pfv.155.2022.06.08.23.34.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 Jun 2022 23:34:28 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski , Paolo Abeni Cc: netdev , Soheil Hassas Yeganeh , Wei Wang , Shakeel Butt , Neal Cardwell , Eric Dumazet , Eric Dumazet Subject: [PATCH net-next 7/7] net: unexport __sk_mem_{raise|reduce}_allocated Date: Wed, 8 Jun 2022 23:34:12 -0700 Message-Id: <20220609063412.2205738-8-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog In-Reply-To: <20220609063412.2205738-1-eric.dumazet@gmail.com> References: <20220609063412.2205738-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet These two helpers are only used from core networking. Signed-off-by: Eric Dumazet Reviewed-by: Shakeel Butt --- net/core/sock.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/net/core/sock.c b/net/core/sock.c index 3bb406167da93b7526ff85787b89fa65e44dce8b..c911013bbdf7670ac55a559580e68f3d57c20200 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2987,7 +2987,6 @@ int __sk_mem_raise_allocated(struct sock *sk, int size, int amt, int kind) return 0; } -EXPORT_SYMBOL(__sk_mem_raise_allocated); /** * __sk_mem_schedule - increase sk_forward_alloc and memory_allocated @@ -3029,7 +3028,6 @@ void __sk_mem_reduce_allocated(struct sock *sk, int amount) (sk_memory_allocated(sk) < sk_prot_mem_limits(sk, 0))) sk_leave_memory_pressure(sk); } -EXPORT_SYMBOL(__sk_mem_reduce_allocated); /** * __sk_mem_reclaim - reclaim sk_forward_alloc and memory_allocated