From patchwork Mon Oct 25 16:48:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12582363 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D9F5C433EF for ; Mon, 25 Oct 2021 16:48:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2FA5C60EBB for ; Mon, 25 Oct 2021 16:48:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233257AbhJYQuz (ORCPT ); Mon, 25 Oct 2021 12:50:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234104AbhJYQuw (ORCPT ); Mon, 25 Oct 2021 12:50:52 -0400 Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75784C061220 for ; Mon, 25 Oct 2021 09:48:30 -0700 (PDT) Received: by mail-pf1-x430.google.com with SMTP id a26so474502pfr.11 for ; Mon, 25 Oct 2021 09:48:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yvoFbHItcgmslFM0wRmjTR9AgHVL2RK29yZdl/SUDVI=; b=R3grmXXwTy+pul1i48HY4pqUZj/2+StvPob/eYswt7B2nUPL8snThwJEAYiH8FpDIi Hbo3vXJsPN5nWnA94cevhvmkxh+VhfpFy9kS7yYb6CVE4WHqFz3mgC2oMxrARn7besQH b40viDIAcy2OUTGh5dvV3ou+N/F5dsGTwSX6mJKjPm6X6SXm0S80O6k9IMUYCB9P2/rl RKkOFNQyEEeii0iQfm1fsbb59EuhGx2e1MO09qCB+MDz/AQRoD2r1L706duuGvNaFZaI 9R3GLSdnJaZ76ldJeN7kH57Lz3NnxeGtZLH1EjQjnZSBvqREj9sQ+43cNCzd1ZqIo1Yw bAVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yvoFbHItcgmslFM0wRmjTR9AgHVL2RK29yZdl/SUDVI=; b=qPB+4wtOfc+btN42rmRf4EnjOOWUhHuExro8R6SWNG4sa9uV73Y9gA1bMsCXH14FWe 3DLQGdai9Q842y2zrFUX4lSJlS3xEBDVgMos4FIjZxZ4Kul0ebwalo9hdNw9WGbI9m+0 duafL+KndIPqJBEoIq6I69CJZFCZggwqWKLdpTIVn8HxfS//e+btzYRe90SdlJJ5+3FV KVFBKfwDUb9RvTJdiRrE/G8qtzskkHPbYYZ0v0vfg4jGyuDfHdaut4UVHpVyFKDB9rEX 0YN75yU5D1TBfsRZzdlayDFM1SgtBBIt9yIroQMwu0WaMx+jguQIA9Mq8bPbZSG3ANP2 Ao+g== X-Gm-Message-State: AOAM530KUl3MQ7cgbC22tuY0gEUPL+mxtl4siB643U9yvE5L6jdpsnAI 90YycdWBSOlc6n+JZAqeB84= X-Google-Smtp-Source: ABdhPJygYzLQaYR1I06Oe1yIVeMGp33pacXe8/rZt/MjL5co+sYoPnY59cRF7+/OWBs2irnEJR264A== X-Received: by 2002:a62:7c0b:0:b0:47b:df8d:816 with SMTP id x11-20020a627c0b000000b0047bdf8d0816mr14458376pfc.11.1635180509941; Mon, 25 Oct 2021 09:48:29 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:b7cd:daa3:a144:1652]) by smtp.gmail.com with ESMTPSA id b3sm17052582pfm.54.2021.10.25.09.48.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Oct 2021 09:48:29 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell Subject: [PATCH v2 net-next 01/10] tcp: move inet->rx_dst_ifindex to sk->sk_rx_dst_ifindex Date: Mon, 25 Oct 2021 09:48:16 -0700 Message-Id: <20211025164825.259415-2-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog In-Reply-To: <20211025164825.259415-1-eric.dumazet@gmail.com> References: <20211025164825.259415-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Increase cache locality by moving rx_dst_ifindex next to sk->sk_rx_dst This is part of an effort to reduce cache line misses in TCP fast path. This removes one cache line miss in early demux. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- include/net/inet_sock.h | 3 +-- include/net/sock.h | 3 +++ net/ipv4/tcp_ipv4.c | 6 +++--- net/ipv6/tcp_ipv6.c | 6 +++--- 4 files changed, 10 insertions(+), 8 deletions(-) diff --git a/include/net/inet_sock.h b/include/net/inet_sock.h index 89163ef8cf4be2aaf99d09806749911a121a56e0..9e1111f5915bd03b6ec5e2e4a74ea0079ede8263 100644 --- a/include/net/inet_sock.h +++ b/include/net/inet_sock.h @@ -207,11 +207,10 @@ struct inet_sock { __be32 inet_saddr; __s16 uc_ttl; __u16 cmsg_flags; + struct ip_options_rcu __rcu *inet_opt; __be16 inet_sport; __u16 inet_id; - struct ip_options_rcu __rcu *inet_opt; - int rx_dst_ifindex; __u8 tos; __u8 min_ttl; __u8 mc_ttl; diff --git a/include/net/sock.h b/include/net/sock.h index 596ba85611bc786affed2bf2b18e455b015f3774..0bfb3f138bdab01bd97498e1126d111743000c8c 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -259,6 +259,7 @@ struct bpf_local_storage; * @sk_rcvbuf: size of receive buffer in bytes * @sk_wq: sock wait queue and async head * @sk_rx_dst: receive input route used by early demux + * @sk_rx_dst_ifindex: ifindex for @sk_rx_dst * @sk_dst_cache: destination cache * @sk_dst_pending_confirm: need to confirm neighbour * @sk_policy: flow policy @@ -430,6 +431,8 @@ struct sock { struct xfrm_policy __rcu *sk_policy[2]; #endif struct dst_entry *sk_rx_dst; + int sk_rx_dst_ifindex; + struct dst_entry __rcu *sk_dst_cache; atomic_t sk_omem_alloc; int sk_sndbuf; diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 5e6d8cb82ce5d8897ec33bf9113d94c42b55bb34..2bdc32c1afb65bb123a27444d9f6e4d01a188074 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -1703,7 +1703,7 @@ int tcp_v4_do_rcv(struct sock *sk, struct sk_buff *skb) sock_rps_save_rxhash(sk, skb); sk_mark_napi_id(sk, skb); if (dst) { - if (inet_sk(sk)->rx_dst_ifindex != skb->skb_iif || + if (sk->sk_rx_dst_ifindex != skb->skb_iif || !INDIRECT_CALL_1(dst->ops->check, ipv4_dst_check, dst, 0)) { dst_release(dst); @@ -1788,7 +1788,7 @@ int tcp_v4_early_demux(struct sk_buff *skb) if (dst) dst = dst_check(dst, 0); if (dst && - inet_sk(sk)->rx_dst_ifindex == skb->skb_iif) + sk->sk_rx_dst_ifindex == skb->skb_iif) skb_dst_set_noref(skb, dst); } } @@ -2195,7 +2195,7 @@ void inet_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb) if (dst && dst_hold_safe(dst)) { sk->sk_rx_dst = dst; - inet_sk(sk)->rx_dst_ifindex = skb->skb_iif; + sk->sk_rx_dst_ifindex = skb->skb_iif; } } EXPORT_SYMBOL(inet_sk_rx_dst_set); diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 1ef27cd7dbff713a22835fc13b57e2eca529f87e..3e8669b6d636c2971f2afc0abf0f2ef51ca6e2d4 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -108,7 +108,7 @@ static void inet6_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb) const struct rt6_info *rt = (const struct rt6_info *)dst; sk->sk_rx_dst = dst; - inet_sk(sk)->rx_dst_ifindex = skb->skb_iif; + sk->sk_rx_dst_ifindex = skb->skb_iif; tcp_inet6_sk(sk)->rx_dst_cookie = rt6_get_cookie(rt); } } @@ -1509,7 +1509,7 @@ static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb) sock_rps_save_rxhash(sk, skb); sk_mark_napi_id(sk, skb); if (dst) { - if (inet_sk(sk)->rx_dst_ifindex != skb->skb_iif || + if (sk->sk_rx_dst_ifindex != skb->skb_iif || INDIRECT_CALL_1(dst->ops->check, ip6_dst_check, dst, np->rx_dst_cookie) == NULL) { dst_release(dst); @@ -1874,7 +1874,7 @@ INDIRECT_CALLABLE_SCOPE void tcp_v6_early_demux(struct sk_buff *skb) if (dst) dst = dst_check(dst, tcp_inet6_sk(sk)->rx_dst_cookie); if (dst && - inet_sk(sk)->rx_dst_ifindex == skb->skb_iif) + sk->sk_rx_dst_ifindex == skb->skb_iif) skb_dst_set_noref(skb, dst); } } From patchwork Mon Oct 25 16:48:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12582365 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2E19C433FE for ; Mon, 25 Oct 2021 16:48:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B811960EB9 for ; Mon, 25 Oct 2021 16:48:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234085AbhJYQu4 (ORCPT ); Mon, 25 Oct 2021 12:50:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234100AbhJYQuy (ORCPT ); Mon, 25 Oct 2021 12:50:54 -0400 Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCF03C061745 for ; Mon, 25 Oct 2021 09:48:31 -0700 (PDT) Received: by mail-pl1-x62a.google.com with SMTP id t21so8345359plr.6 for ; Mon, 25 Oct 2021 09:48:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JjKcZRwf7Vlbxnn6gpdZiVEu3+XDIiSkdFofSruGkuo=; b=bfG7JqsYhMzVUblDYT+noDyg4q0IoOW1EIKVF9tfPGSRFTCEJNFqJZ5GmMNzFjEMEY mcYLHmrQvxYet9NkCaHx2NUvnSPDPdwvylf5sHtx9S1uUwPutG4zRU5XWjIEwO6cSgfL PtoJV+BQieJsHxrqFOvbXNPEdDE5CB7U+GDEOKYhTEBo9NKgyL3p4p+4Y8ItmdayEmOA Vx0xYVUcgWp2oXTSPm54IrLgoAoaUzk8zhzzayhedfDzf4UI3nqKoSD+PtBmDhL+nqse DLhk+cKiDybT7b1b03BW2oP0tVrZ2wIBRj6QdR9f+Gf2ITaePwhEUTgKorl2k7s7Stkh EbWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JjKcZRwf7Vlbxnn6gpdZiVEu3+XDIiSkdFofSruGkuo=; b=cCgFLJccyiKr4UeiMuIbxZ+0Z24uyAJyspKYDlFXh3FmBIOCcBKUAjNQIvvXYJH8uv LOECgg9FLEm5B9eX830HWjt9Wsl7BkeztvpV7TfFQtmOk8D59b0Tz/qMmY+e9IVYJOaa 4dq5GSz/MIyQBX74PLKJqutyASECEAUi/nx+rdG2l68XxWgZxcqlrQMSxLsZ+CX/I3If N3/JrQwtqHr+hGmnaZ2VLQCP+y6813TnXldExa/VNPxc9K/ctUY74al3m8xXsUea+jN9 mAa8nmaxH6caXAD1WYQIWWkoVzHhaUEx2/7wFxDCNg0Bki63ByNm58W38bg/Bf7ubTb2 fOVg== X-Gm-Message-State: AOAM533E7N0CrIglMHkoZQxL++4whJK2MdYReaRc2VFHDISfoIoF6ay6 XFVddWy4lIlcALFoBrgfgyI= X-Google-Smtp-Source: ABdhPJzOGSj0nYtuArg0Jf50Swxf70SA+/gx5SgiRU93j9ADkBX6XBewk8/qNMIs69+UCnOJiJfV4Q== X-Received: by 2002:a17:902:dac5:b0:140:3b56:c7e4 with SMTP id q5-20020a170902dac500b001403b56c7e4mr12823798plx.78.1635180511237; Mon, 25 Oct 2021 09:48:31 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:b7cd:daa3:a144:1652]) by smtp.gmail.com with ESMTPSA id b3sm17052582pfm.54.2021.10.25.09.48.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Oct 2021 09:48:30 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell Subject: [PATCH v2 net-next 02/10] ipv6: move inet6_sk(sk)->rx_dst_cookie to sk->sk_rx_dst_cookie Date: Mon, 25 Oct 2021 09:48:17 -0700 Message-Id: <20211025164825.259415-3-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog In-Reply-To: <20211025164825.259415-1-eric.dumazet@gmail.com> References: <20211025164825.259415-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Increase cache locality by moving rx_dst_coookie next to sk->sk_rx_dst This removes one or two cache line misses in IPv6 early demux (TCP/UDP) Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- include/linux/ipv6.h | 1 - include/net/sock.h | 2 ++ net/ipv6/tcp_ipv6.c | 6 +++--- net/ipv6/udp.c | 4 ++-- 4 files changed, 7 insertions(+), 6 deletions(-) diff --git a/include/linux/ipv6.h b/include/linux/ipv6.h index ef4a69865737cee82a72c35f3421a535b607c7a6..c383630d3f0658908eac65c030daf97b0a0d0c7c 100644 --- a/include/linux/ipv6.h +++ b/include/linux/ipv6.h @@ -282,7 +282,6 @@ struct ipv6_pinfo { __be32 rcv_flowinfo; __u32 dst_cookie; - __u32 rx_dst_cookie; struct ipv6_mc_socklist __rcu *ipv6_mc_list; struct ipv6_ac_socklist *ipv6_ac_list; diff --git a/include/net/sock.h b/include/net/sock.h index 0bfb3f138bdab01bd97498e1126d111743000c8c..99c4194cb61add848e3a35db0f952c4193f5ea1f 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -260,6 +260,7 @@ struct bpf_local_storage; * @sk_wq: sock wait queue and async head * @sk_rx_dst: receive input route used by early demux * @sk_rx_dst_ifindex: ifindex for @sk_rx_dst + * @sk_rx_dst_cookie: cookie for @sk_rx_dst * @sk_dst_cache: destination cache * @sk_dst_pending_confirm: need to confirm neighbour * @sk_policy: flow policy @@ -432,6 +433,7 @@ struct sock { #endif struct dst_entry *sk_rx_dst; int sk_rx_dst_ifindex; + u32 sk_rx_dst_cookie; struct dst_entry __rcu *sk_dst_cache; atomic_t sk_omem_alloc; diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 3e8669b6d636c2971f2afc0abf0f2ef51ca6e2d4..50d9578e945bd2965247a46bc6d8b1adeb21d2f4 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -109,7 +109,7 @@ static void inet6_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb) sk->sk_rx_dst = dst; sk->sk_rx_dst_ifindex = skb->skb_iif; - tcp_inet6_sk(sk)->rx_dst_cookie = rt6_get_cookie(rt); + sk->sk_rx_dst_cookie = rt6_get_cookie(rt); } } @@ -1511,7 +1511,7 @@ static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb) if (dst) { if (sk->sk_rx_dst_ifindex != skb->skb_iif || INDIRECT_CALL_1(dst->ops->check, ip6_dst_check, - dst, np->rx_dst_cookie) == NULL) { + dst, sk->sk_rx_dst_cookie) == NULL) { dst_release(dst); sk->sk_rx_dst = NULL; } @@ -1872,7 +1872,7 @@ INDIRECT_CALLABLE_SCOPE void tcp_v6_early_demux(struct sk_buff *skb) struct dst_entry *dst = READ_ONCE(sk->sk_rx_dst); if (dst) - dst = dst_check(dst, tcp_inet6_sk(sk)->rx_dst_cookie); + dst = dst_check(dst, sk->sk_rx_dst_cookie); if (dst && sk->sk_rx_dst_ifindex == skb->skb_iif) skb_dst_set_noref(skb, dst); diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 8d785232b4796b7cafe14a35dedcbb0aaa2c37c2..14a94cddcf0bcf63d8351c66b94a08770694a9c8 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -884,7 +884,7 @@ static void udp6_sk_rx_dst_set(struct sock *sk, struct dst_entry *dst) if (udp_sk_rx_dst_set(sk, dst)) { const struct rt6_info *rt = (const struct rt6_info *)dst; - inet6_sk(sk)->rx_dst_cookie = rt6_get_cookie(rt); + sk->sk_rx_dst_cookie = rt6_get_cookie(rt); } } @@ -1073,7 +1073,7 @@ INDIRECT_CALLABLE_SCOPE void udp_v6_early_demux(struct sk_buff *skb) dst = READ_ONCE(sk->sk_rx_dst); if (dst) - dst = dst_check(dst, inet6_sk(sk)->rx_dst_cookie); + dst = dst_check(dst, sk->sk_rx_dst_cookie); if (dst) { /* set noref for now. * any place which wants to hold dst has to call From patchwork Mon Oct 25 16:48:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12582367 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BF79C4332F for ; Mon, 25 Oct 2021 16:48:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 83CF860F22 for ; Mon, 25 Oct 2021 16:48:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234093AbhJYQu4 (ORCPT ); Mon, 25 Oct 2021 12:50:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44650 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234071AbhJYQuz (ORCPT ); Mon, 25 Oct 2021 12:50:55 -0400 Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D052BC061348 for ; Mon, 25 Oct 2021 09:48:32 -0700 (PDT) Received: by mail-pg1-x533.google.com with SMTP id c4so11504960pgv.11 for ; Mon, 25 Oct 2021 09:48:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sYXfm/4e8postm1njOaXmHy+JuD2lpfYTQAHOwn6i88=; b=Ab28K0ihWm8FsFYwesYLdcCxJzuZrZ8GlKPQ/hwZ3Mtrrg9ptqMOynD4b/xaYsD47e OLOF4cJo/OeIeK/dM9g4QsmP5A388T7D5THpV3HwbEl5KIBoFznXrVS6CnIzcTrEuasA D+lkmxSXqZMtsHcvXq9DdalPZponHGo0ABeUAG44c8bOG7+mj3367WMHtcrwfJGj7WLh q0AANbM4QXCQyruW3fpvMXcSVhprcoZODYv9P1eX453PQh2a9vO6BALjgQCPcx33dVZ6 p2NBX0rD9hHlvBIACZvKJMj0TM3wl03wHebW5pPn4xyxGZ3pYScGgvb/O4qZvaw2OoBw GlaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sYXfm/4e8postm1njOaXmHy+JuD2lpfYTQAHOwn6i88=; b=A5Ss+lWpQqtzn/BVDZclXGXbdikhCTcBHD9dfD0x92L+E3TSe+CwIIULEhFBzk4MX8 2nsD1CvFBkmzsM0oHMU9zGWN7Zv9CuE3jzzT6Z223OqA0ZMzoi+fy0lZI7IoeD2at/CW dc7jyfsLtyPUN9WOaGps6oWNO2p52k4KI1YzrNpEdud8E8Sz3xTJOekI2VsgSs87byfo p5AcezjfKXUphnwM6txJs6WzdmuKORB42MDxMMeVrFMkC9EPOVrNodxix72dreKGwGu4 Z+740LuIZQ0JXZ5HLVq1Fbk2P8cjFqWaTequo9Ca9AD3SD1Y9YnkN4dnLLq3gmycP1sj efhw== X-Gm-Message-State: AOAM531wFimjNO8xe3DAmmqlsh+8rq8DYNL8Ska5K8bh9McfQg1LoFuq YxeEPPZSZoReTxdpw9l1VgI= X-Google-Smtp-Source: ABdhPJzBrTx25uZ+ZbA/bbW+MEsLU81GuBzOyERRqx/Iiu5cBGZzAh58P5ztD03QkbzAtFcFQVxwpg== X-Received: by 2002:a63:f306:: with SMTP id l6mr14405955pgh.72.1635180512470; Mon, 25 Oct 2021 09:48:32 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:b7cd:daa3:a144:1652]) by smtp.gmail.com with ESMTPSA id b3sm17052582pfm.54.2021.10.25.09.48.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Oct 2021 09:48:32 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell Subject: [PATCH v2 net-next 03/10] net: avoid dirtying sk->sk_napi_id Date: Mon, 25 Oct 2021 09:48:18 -0700 Message-Id: <20211025164825.259415-4-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog In-Reply-To: <20211025164825.259415-1-eric.dumazet@gmail.com> References: <20211025164825.259415-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet sk_napi_id is located in a cache line that can be kept read mostly. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- include/net/busy_poll.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/net/busy_poll.h b/include/net/busy_poll.h index 40296ed976a9778ceb239b99ad783cb99b8b92ef..4202c609bb0b09345c0f1c5105adf409a3a89f74 100644 --- a/include/net/busy_poll.h +++ b/include/net/busy_poll.h @@ -130,7 +130,8 @@ static inline void skb_mark_napi_id(struct sk_buff *skb, static inline void sk_mark_napi_id(struct sock *sk, const struct sk_buff *skb) { #ifdef CONFIG_NET_RX_BUSY_POLL - WRITE_ONCE(sk->sk_napi_id, skb->napi_id); + if (unlikely(READ_ONCE(sk->sk_napi_id) != skb->napi_id)) + WRITE_ONCE(sk->sk_napi_id, skb->napi_id); #endif sk_rx_queue_set(sk, skb); } From patchwork Mon Oct 25 16:48:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12582369 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24B72C433F5 for ; Mon, 25 Oct 2021 16:48:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0FD7660EBB for ; Mon, 25 Oct 2021 16:48:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234100AbhJYQu5 (ORCPT ); Mon, 25 Oct 2021 12:50:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44660 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234086AbhJYQu4 (ORCPT ); Mon, 25 Oct 2021 12:50:56 -0400 Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1DF75C061745 for ; Mon, 25 Oct 2021 09:48:34 -0700 (PDT) Received: by mail-pg1-x52d.google.com with SMTP id r28so7183162pga.0 for ; Mon, 25 Oct 2021 09:48:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=653wW3nxLwDc1GqFAR6Umq7l0o7whuAkrBQPp700DBU=; b=mnN25bki8goIr12D5uRIkW40mhYYrMTCu97J5Gei1yE4fCGG/zlzIIlazOWMjoXlUx FfCXWP5lqYRm3tnJ7B8Ig5SMqtGUjIukxNj+wW8kZObNNCexuYrqYuGA2LwPTPj2apL0 MmARvWmMTErAg/YWe69P6+3O96BLz/gG5CxzS+dM3ryEHvS7/t9A7aigGfJmUlsYMq/L seyA7kLZiyGK1IGOtiHp+A96n2k8CGWeBBJ++y+zE1IH8jU3+dNaLF/tZyV6yQbCRJJh LOvp/j7taLQ77x5I7xlxDYF8MMp7nTHAv8ez9X+CFt/sbSCwf3FE9Ve8mK1DzcaaZqe1 moPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=653wW3nxLwDc1GqFAR6Umq7l0o7whuAkrBQPp700DBU=; b=xbWyx6CUmvPB8lfGtqIWVyOfAf9nWGThVQBSLywc27/IxaenQBmJeJmJxqB6LjQZJx doyXKpr8yUyZxCfktyjhvaMeX0leDb5y59/DilV6gcSPfBYBZ1/NGRK2ufs4CQZEcvf+ QMG6IXAUV43nOdcJQawVp9Qq8X/iXQSJouT0jSlluOvpt8IoEy7TnraQHuNvnwq7nYIV CvMVozGHmYfKFYABW1YoM9B+0jYWS4TiEhaabeRDoAm/o59HKSAn383+utWzr3/VaKkE 3eyh1YHyJT9qnpchaQw3NpQfqIrxXVZWVrv8Ks5NfiTd4kyyhPSlnKWx6aEX8Sl3oOKq ztsA== X-Gm-Message-State: AOAM5310dNbD2IyfvI5ldjbQ/rngJLgrVuwDjDW1j83XyHS3LbwOsTA4 LfemMO063OcVbm3RDEkRTGY= X-Google-Smtp-Source: ABdhPJy0VQi0aTaWTkEr9RgNG//F4TZPWr+foXONB33NyuVM3xnKyJWdF3kvnKqNRGy/pDcaFdivWQ== X-Received: by 2002:a63:7542:: with SMTP id f2mr14562623pgn.147.1635180513689; Mon, 25 Oct 2021 09:48:33 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:b7cd:daa3:a144:1652]) by smtp.gmail.com with ESMTPSA id b3sm17052582pfm.54.2021.10.25.09.48.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Oct 2021 09:48:33 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell Subject: [PATCH v2 net-next 04/10] net: avoid dirtying sk->sk_rx_queue_mapping Date: Mon, 25 Oct 2021 09:48:19 -0700 Message-Id: <20211025164825.259415-5-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog In-Reply-To: <20211025164825.259415-1-eric.dumazet@gmail.com> References: <20211025164825.259415-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet sk_rx_queue_mapping is located in a cache line that should be kept read mostly. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- include/net/sock.h | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 99c4194cb61add848e3a35db0f952c4193f5ea1f..b4d3744b188ad869b4ec55f78e04236b710898de 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1916,10 +1916,8 @@ static inline void sk_rx_queue_set(struct sock *sk, const struct sk_buff *skb) if (skb_rx_queue_recorded(skb)) { u16 rx_queue = skb_get_rx_queue(skb); - if (WARN_ON_ONCE(rx_queue == NO_QUEUE_MAPPING)) - return; - - sk->sk_rx_queue_mapping = rx_queue; + if (unlikely(READ_ONCE(sk->sk_rx_queue_mapping) != rx_queue)) + WRITE_ONCE(sk->sk_rx_queue_mapping, rx_queue); } #endif } From patchwork Mon Oct 25 16:48:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12582371 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9FDAC433EF for ; Mon, 25 Oct 2021 16:48:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C458C601FF for ; Mon, 25 Oct 2021 16:48:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234098AbhJYQu6 (ORCPT ); Mon, 25 Oct 2021 12:50:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44664 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234101AbhJYQu5 (ORCPT ); Mon, 25 Oct 2021 12:50:57 -0400 Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60429C061745 for ; Mon, 25 Oct 2021 09:48:35 -0700 (PDT) Received: by mail-pg1-x529.google.com with SMTP id m21so11500060pgu.13 for ; Mon, 25 Oct 2021 09:48:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6T/8l40zJhUEXxcfn1C0FlUH4x2qZwvVYGnNQT7HqYY=; b=DW0FyiITdO/z4ikE1P9inF9yhPlyvQFle/i8HAN8i3ZKzEicEoupuhV15hGKiir/G2 BL9l+lxOFK47Kmna2uEIzbhSilcPU7iFsifvsNgtHUSOJqcs8geXPIapcj9ISUbV9l4U ZWoziDYMFNQr4nov3GweuClqgr2Jsqq3vAg8fO7vvDm3L+gz9AOpB/87YRNxkfbhkdXY SBSqM8MDhuHBlGt1ABjCaWQVDicHdVUfSg5wlhZCIt1sMgtj87UdfUi6N6ymOy93NZh2 Qxi0pvb79l8y6COMHRir3zJUA9Nt0M0Lmq8Pr9SiNXzFjoS7btkZUQNNjqmjqVKYnneA LxfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6T/8l40zJhUEXxcfn1C0FlUH4x2qZwvVYGnNQT7HqYY=; b=xGQMKCgn8ijnZ7KW5yhvIZ0xZwYyDqwANRi/GeWce/1Mix0moCttP735kA4IDh/TjP le4v9V2oMmJdSp/fAw15WC8Y1BkYzJlnyVrKJ0qhOlJuttPBEYhhGs0YgqvUMEC1NSgW b0fIAJH/Jy4RYOLWaSOSNtEOy9oiOehv1O6xvxQbz9qwn0shmem58ONffL3XxhPIsv+5 b3BxfR9qikG+oWW0Vk9lpi4eX7JEiSP6W+JuKDojuIL4bA4ZSGgtmovSiQ0WxYTlejF8 IqB2OuxUTQjkZmuhKQbxRmlmCeBNhdzLD2B2vzh1I1veGHJepK50khAK3+JDa/ijFYVP JGJg== X-Gm-Message-State: AOAM533OlOgpNcwDHY8HwJNNzKmZQzDylQATVL80cFeqoH9pYeSPL7I4 ervgOTkhMI9QJCSRoIqTjWE= X-Google-Smtp-Source: ABdhPJy82QGWtJzWNQ6A0sqCRR/WswVCp5se6wHcgiamgBewpeD1cKTMN6OLQ4o6lz4E3iwHYqwVrw== X-Received: by 2002:a05:6a00:1352:b0:44d:4573:3ac2 with SMTP id k18-20020a056a00135200b0044d45733ac2mr20306869pfu.12.1635180514903; Mon, 25 Oct 2021 09:48:34 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:b7cd:daa3:a144:1652]) by smtp.gmail.com with ESMTPSA id b3sm17052582pfm.54.2021.10.25.09.48.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Oct 2021 09:48:34 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell Subject: [PATCH v2 net-next 05/10] net: annotate accesses to sk->sk_rx_queue_mapping Date: Mon, 25 Oct 2021 09:48:20 -0700 Message-Id: <20211025164825.259415-6-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog In-Reply-To: <20211025164825.259415-1-eric.dumazet@gmail.com> References: <20211025164825.259415-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet sk->sk_rx_queue_mapping can be modified locklessly, add a couple of READ_ONCE()/WRITE_ONCE() to document this fact. Signed-off-by: Eric Dumazet --- include/net/sock.h | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index b4d3744b188ad869b4ec55f78e04236b710898de..b76be30674efc88434ed45d46b9c4600261b6271 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1925,15 +1925,19 @@ static inline void sk_rx_queue_set(struct sock *sk, const struct sk_buff *skb) static inline void sk_rx_queue_clear(struct sock *sk) { #ifdef CONFIG_SOCK_RX_QUEUE_MAPPING - sk->sk_rx_queue_mapping = NO_QUEUE_MAPPING; + WRITE_ONCE(sk->sk_rx_queue_mapping, NO_QUEUE_MAPPING); #endif } static inline int sk_rx_queue_get(const struct sock *sk) { #ifdef CONFIG_SOCK_RX_QUEUE_MAPPING - if (sk && sk->sk_rx_queue_mapping != NO_QUEUE_MAPPING) - return sk->sk_rx_queue_mapping; + if (sk) { + int res = READ_ONCE(sk->sk_rx_queue_mapping); + + if (res != NO_QUEUE_MAPPING) + return res; + } #endif return -1; From patchwork Mon Oct 25 16:48:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12582373 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8D9CC433F5 for ; Mon, 25 Oct 2021 16:48:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B192C601FF for ; Mon, 25 Oct 2021 16:48:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234108AbhJYQvA (ORCPT ); Mon, 25 Oct 2021 12:51:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44670 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234086AbhJYQu6 (ORCPT ); Mon, 25 Oct 2021 12:50:58 -0400 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B5A3C061745 for ; Mon, 25 Oct 2021 09:48:36 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id e65so11519645pgc.5 for ; Mon, 25 Oct 2021 09:48:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=s3s8ot4Es9NTH17Lse9dDj9nkiczkTxReRTzDV1hgqc=; b=Lvy6NyJQiqpXB5KvCN33Gw6wiMTdmOTBUrvVaMpUf8qkx/APUldCgwZouz3NQHHEjd uFzsAp/hn8vcNjiMxW7Go1SUqaYDe5sNLksbrtd1movgMm1IDokxvaaIciml6Ng0OmhP xQNCMyXkvwBltLiVLkrgrgX8btOL5NLIyJHmcwR4nffEcOo15KP92jl3DDpOgea8EiT0 h55e27r8zBzai8zX1VLsOb79vvROeeqvL5BlwfC6L4W7MfDUNCb8cHY/8edi1gOmn/f/ yVOT5SgWn9Ycp6gPWeUu+OfN9b1OH4x+JbEv3sdcrvd+HZ89klhi3BlsYy4an3YRWB8H u+yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=s3s8ot4Es9NTH17Lse9dDj9nkiczkTxReRTzDV1hgqc=; b=a1J39LFuNtZZT1Iitx8oz43FQf/kz+2kHKQd8pbzQju3O6tX6g2ipj8vlAP1lsQQow ETsdIF9pXYPaFMsG9MeXiPFY0tbV51JTuetxsS+s1ZLyF1YXUSBjIhL3NIw/pRBU9dY1 3FxnaEQEyg9Mhv3KCCGcxWMIvKtJBKOZcrfT9hg4TW+hvjq1WwYMOdwfI/oocj/nkAb6 NeyFu9tHTbR32tHDeYOYuRJD0eNVnXvPlSUznx/lL9Q1D/tkuBSPB9w40dVISD6eQWey W82xfhUYCnPPCV3Xxk59Wtunyfilahs7OoX4xSbvX7tUJ5UZoMkO6ruuuuEBrd7nHqAK GjvA== X-Gm-Message-State: AOAM530cOOJoM6J9Un3CXXZg9eiAi0Hc77ciufRSxZj80nTbLGHIIKX4 Z1k+vtfq+7RYjMAY/xnJhaM= X-Google-Smtp-Source: ABdhPJxv8Gh9J1CM2YOM4TgM8iIwAtYS3ZqzETs48WkBmbrlsIpElAYO0VfI7OVJQv0XZ/Htwtrvzw== X-Received: by 2002:a63:9844:: with SMTP id l4mr14789625pgo.271.1635180516195; Mon, 25 Oct 2021 09:48:36 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:b7cd:daa3:a144:1652]) by smtp.gmail.com with ESMTPSA id b3sm17052582pfm.54.2021.10.25.09.48.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Oct 2021 09:48:35 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell Subject: [PATCH v2 net-next 06/10] ipv6: annotate data races around np->min_hopcount Date: Mon, 25 Oct 2021 09:48:21 -0700 Message-Id: <20211025164825.259415-7-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog In-Reply-To: <20211025164825.259415-1-eric.dumazet@gmail.com> References: <20211025164825.259415-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet No report yet from KCSAN, yet worth documenting the races. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- net/ipv6/ipv6_sockglue.c | 5 ++++- net/ipv6/tcp_ipv6.c | 6 ++++-- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/net/ipv6/ipv6_sockglue.c b/net/ipv6/ipv6_sockglue.c index e4bdb09c558670f342f1abad5dfd8252f497aa68..9c3d28764b5c3a47a73491ea5d656867ece4fed2 100644 --- a/net/ipv6/ipv6_sockglue.c +++ b/net/ipv6/ipv6_sockglue.c @@ -950,7 +950,10 @@ static int do_ipv6_setsockopt(struct sock *sk, int level, int optname, goto e_inval; if (val < 0 || val > 255) goto e_inval; - np->min_hopcount = val; + /* tcp_v6_err() and tcp_v6_rcv() might read min_hopcount + * while we are changing it. + */ + WRITE_ONCE(np->min_hopcount, val); retv = 0; break; case IPV6_DONTFRAG: diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 50d9578e945bd2965247a46bc6d8b1adeb21d2f4..c93b2d48bb89a845c880679572c75a8791589525 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -414,7 +414,8 @@ static int tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt, if (sk->sk_state == TCP_CLOSE) goto out; - if (ipv6_hdr(skb)->hop_limit < tcp_inet6_sk(sk)->min_hopcount) { + /* min_hopcount can be changed concurrently from do_ipv6_setsockopt() */ + if (ipv6_hdr(skb)->hop_limit < READ_ONCE(tcp_inet6_sk(sk)->min_hopcount)) { __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); goto out; } @@ -1726,7 +1727,8 @@ INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb) return 0; } } - if (hdr->hop_limit < tcp_inet6_sk(sk)->min_hopcount) { + /* min_hopcount can be changed concurrently from do_ipv6_setsockopt() */ + if (hdr->hop_limit < READ_ONCE(tcp_inet6_sk(sk)->min_hopcount)) { __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); goto discard_and_relse; } From patchwork Mon Oct 25 16:48:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12582375 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6CB2C433F5 for ; Mon, 25 Oct 2021 16:48:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C31DB60EB9 for ; Mon, 25 Oct 2021 16:48:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234126AbhJYQvF (ORCPT ); Mon, 25 Oct 2021 12:51:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44674 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234101AbhJYQu7 (ORCPT ); Mon, 25 Oct 2021 12:50:59 -0400 Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4A59C061745 for ; Mon, 25 Oct 2021 09:48:37 -0700 (PDT) Received: by mail-pg1-x533.google.com with SMTP id 75so11530322pga.3 for ; Mon, 25 Oct 2021 09:48:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3sADqxOwcycN3Ol5Y822tWtM/xawBqRDa2NO7ajIG2Q=; b=Dh4N6tfp5zrCg2n5srQ+KZr1fV/7Pf1b7e+uzG7v4da4eAKFasI8irtvCmfAVXLf/2 maBAi0102ur12tCZIaly4CYEkmnXbZcOHWm5855Z/mPqId94NjJO8KiYvKf2kGIZsq3E qIt0WgAkY3pMvyOHWFKOzb6EWuQ+6ic9e74Ng0PEKwn80iZBNQcWqS6gkiFs0NpuRCNc PFnlYFriMhQOEEsgAxXTlCh6OdkTIsTJkuIqAw2EQ0th3D8RAPMNJEOynH6tPm7x4Ktn gaXM3xVzw6WS//rpULZid/h3sV8l6MS2c0FKpq1827hYe96hmQtBYMSSAiOHbLOWkIZC ShAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3sADqxOwcycN3Ol5Y822tWtM/xawBqRDa2NO7ajIG2Q=; b=LNP+QnTpoX4tQLgbdNw/VTPud/9i+/InDK0wsFmtfGXcGf35fyLnvQn/sf5apt6NK3 gEONeo5PxMTX3OW3npsExRkcoaJQ3YJg7iQLJPZNjA5qu28mxT9YZ7WSl5kOPpHxvAwM cjcOlzOZsJ2lWNT5uVtUSGio483MAnoKOzv5cpqiTfySgjcLbUBX5Tkc3CMPJgeGHhkf loZ6PTW/vnF7jdiU/R7d4jo7aA0xgUv/tTOxiZfxt32FCqrYA6mQQSNNVmSnJECU4/5j jh2isGP1gBuPKmiOibFeAiFp+ZCxMcasaGA8jo4aeIyyLjnV0NsVnJhG1AfMWznZOJYW 7hMw== X-Gm-Message-State: AOAM533FePjt69+0qy+qVITL9x42oezDeX5z7nLqZ+66CGOL+EaS1415 Gx1uqVC1bkGXuliy6MKjljU= X-Google-Smtp-Source: ABdhPJxwHrs6LMTuIMcJj89XYHUoF5UXalafCKN+WYOhAIlzQZouab2KKcfW0T+XDQqKbX/PupVBZw== X-Received: by 2002:a05:6a00:10c5:b0:47b:d112:96d6 with SMTP id d5-20020a056a0010c500b0047bd11296d6mr17253184pfu.59.1635180517260; Mon, 25 Oct 2021 09:48:37 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:b7cd:daa3:a144:1652]) by smtp.gmail.com with ESMTPSA id b3sm17052582pfm.54.2021.10.25.09.48.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Oct 2021 09:48:36 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell Subject: [PATCH v2 net-next 07/10] ipv6: guard IPV6_MINHOPCOUNT with a static key Date: Mon, 25 Oct 2021 09:48:22 -0700 Message-Id: <20211025164825.259415-8-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog In-Reply-To: <20211025164825.259415-1-eric.dumazet@gmail.com> References: <20211025164825.259415-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet RFC 5082 IPV6_MINHOPCOUNT is rarely used on hosts. Add a static key to remove from TCP fast path useless code, and potential cache line miss to fetch tcp_inet6_sk(sk)->min_hopcount Note that once ip6_min_hopcount static key has been enabled, it stays enabled until next boot. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- include/net/ipv6.h | 1 + net/ipv6/ipv6_sockglue.c | 6 ++++++ net/ipv6/tcp_ipv6.c | 21 +++++++++++++-------- 3 files changed, 20 insertions(+), 8 deletions(-) diff --git a/include/net/ipv6.h b/include/net/ipv6.h index f2d0ecc257bb28e6dd162d180c371e2b0487c8e3..c19bf51ded1d026e795a3f9ae0ff3be766fc174e 100644 --- a/include/net/ipv6.h +++ b/include/net/ipv6.h @@ -1092,6 +1092,7 @@ struct in6_addr *fl6_update_dst(struct flowi6 *fl6, /* * socket options (ipv6_sockglue.c) */ +DECLARE_STATIC_KEY_FALSE(ip6_min_hopcount); int ipv6_setsockopt(struct sock *sk, int level, int optname, sockptr_t optval, unsigned int optlen); diff --git a/net/ipv6/ipv6_sockglue.c b/net/ipv6/ipv6_sockglue.c index 9c3d28764b5c3a47a73491ea5d656867ece4fed2..41efca817db4228f265235a471449a3790075ce7 100644 --- a/net/ipv6/ipv6_sockglue.c +++ b/net/ipv6/ipv6_sockglue.c @@ -55,6 +55,8 @@ struct ip6_ra_chain *ip6_ra_chain; DEFINE_RWLOCK(ip6_ra_lock); +DEFINE_STATIC_KEY_FALSE(ip6_min_hopcount); + int ip6_ra_control(struct sock *sk, int sel) { struct ip6_ra_chain *ra, *new_ra, **rap; @@ -950,6 +952,10 @@ static int do_ipv6_setsockopt(struct sock *sk, int level, int optname, goto e_inval; if (val < 0 || val > 255) goto e_inval; + + if (val) + static_branch_enable(&ip6_min_hopcount); + /* tcp_v6_err() and tcp_v6_rcv() might read min_hopcount * while we are changing it. */ diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index c93b2d48bb89a845c880679572c75a8791589525..6945583c0fa48256db5510866342e4aab672fd71 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -414,10 +414,12 @@ static int tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt, if (sk->sk_state == TCP_CLOSE) goto out; - /* min_hopcount can be changed concurrently from do_ipv6_setsockopt() */ - if (ipv6_hdr(skb)->hop_limit < READ_ONCE(tcp_inet6_sk(sk)->min_hopcount)) { - __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); - goto out; + if (static_branch_unlikely(&ip6_min_hopcount)) { + /* min_hopcount can be changed concurrently from do_ipv6_setsockopt() */ + if (ipv6_hdr(skb)->hop_limit < READ_ONCE(tcp_inet6_sk(sk)->min_hopcount)) { + __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); + goto out; + } } tp = tcp_sk(sk); @@ -1727,10 +1729,13 @@ INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb) return 0; } } - /* min_hopcount can be changed concurrently from do_ipv6_setsockopt() */ - if (hdr->hop_limit < READ_ONCE(tcp_inet6_sk(sk)->min_hopcount)) { - __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); - goto discard_and_relse; + + if (static_branch_unlikely(&ip6_min_hopcount)) { + /* min_hopcount can be changed concurrently from do_ipv6_setsockopt() */ + if (hdr->hop_limit < READ_ONCE(tcp_inet6_sk(sk)->min_hopcount)) { + __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); + goto discard_and_relse; + } } if (!xfrm6_policy_check(sk, XFRM_POLICY_IN, skb)) From patchwork Mon Oct 25 16:48:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12582377 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2E74C433FE for ; Mon, 25 Oct 2021 16:48:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 99F1060EBB for ; Mon, 25 Oct 2021 16:48:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234104AbhJYQvG (ORCPT ); Mon, 25 Oct 2021 12:51:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234109AbhJYQvB (ORCPT ); Mon, 25 Oct 2021 12:51:01 -0400 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DFE45C061767 for ; Mon, 25 Oct 2021 09:48:38 -0700 (PDT) Received: by mail-pj1-x1035.google.com with SMTP id u12so4688445pjy.1 for ; Mon, 25 Oct 2021 09:48:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=U8PghNy/avxVQfHWNjAjlXaqoKopvj9ocRaUMCgv5H0=; b=mxBu7WhTI5eWJAWz0NSTeIjAWPQH7N6QuVUFKD6nRvOVyfKIh74zAIbBkX3gSOPHRH yfShXPPPfEEezBQ88W4kyOGdo/1/Y84BwIywqxnLMNwu52gNGKHXo/lJJtg75VC0CvAn XnoewJIJ8gpBeuq67EtOAV0WqMgPv55HEuiC/yMQSO7to5ABODYMxdLkZ+KHdB90bqzv b+9mIrGHuVIBdlnnXu+POB4JI+QXr/HWXUl2BWWR0UTOUTs57hSZ3TRNnVr119VpijEG a4XkhV4OaBRbrvvo+r2p/dBsuk9DDQb1F3Laghznf3XTDX+IexbIKvDq192k+4TCsZ2k TOPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=U8PghNy/avxVQfHWNjAjlXaqoKopvj9ocRaUMCgv5H0=; b=Rb+pGESpQRvORufOmRKeCODfIoru4MIh+wn415f+n9M2HEQq1ABvyNPG/aVxsi3V0l AmkPNZUrorH5yK5LOCVrK7XHsz+xDvmWeZDSUsoNaF/ZHEd4NoTZbU4bXav5PvZ7z0H1 vKCGzL/Ve9v1KFRuIIbORMUdX7BO6WxMrLAcXE8BJpx623vzRZ5StPQSAPFUwNHURj8Z A9CfcOIxjy9IiB9o5D/m0NGCGyPD2UW8ZWUu+UR83CENfCSthPKE6C25L10KmXItV8EL 9W9RLtyyEVoZ56kMfQFP7C9WEsvry947Y4XOTfTh8uQ9zRetf9aH6P+NbcztfrlCsAy0 e/XQ== X-Gm-Message-State: AOAM5329R+jeDbM728ASsBvMkpeBUo70vJY0egbHugWUzCzQw/IA/VSh b4qcqz1lEG5/thMENzyzyi0= X-Google-Smtp-Source: ABdhPJyJFZxsCioHc5ojYvDNgrEdexAhjIppLFzFOW2CFDewRl6Y3gDNFSs+RbWudeY143/5Nevk0Q== X-Received: by 2002:a17:902:aa03:b0:13f:a07e:da04 with SMTP id be3-20020a170902aa0300b0013fa07eda04mr17681273plb.80.1635180518478; Mon, 25 Oct 2021 09:48:38 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:b7cd:daa3:a144:1652]) by smtp.gmail.com with ESMTPSA id b3sm17052582pfm.54.2021.10.25.09.48.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Oct 2021 09:48:38 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell Subject: [PATCH v2 net-next 08/10] ipv4: annotate data races arount inet->min_ttl Date: Mon, 25 Oct 2021 09:48:23 -0700 Message-Id: <20211025164825.259415-9-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog In-Reply-To: <20211025164825.259415-1-eric.dumazet@gmail.com> References: <20211025164825.259415-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet No report yet from KCSAN, yet worth documenting the races. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- net/ipv4/ip_sockglue.c | 5 ++++- net/ipv4/tcp_ipv4.c | 7 +++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c index b297bb28556ec5cf383068f67ee910af38591cc3..d5487c8580674a01df8c7d8ce88f97c9add846b6 100644 --- a/net/ipv4/ip_sockglue.c +++ b/net/ipv4/ip_sockglue.c @@ -1352,7 +1352,10 @@ static int do_ip_setsockopt(struct sock *sk, int level, int optname, goto e_inval; if (val < 0 || val > 255) goto e_inval; - inet->min_ttl = val; + /* tcp_v4_err() and tcp_v4_rcv() might read min_ttl + * while we are changint it. + */ + WRITE_ONCE(inet->min_ttl, val); break; default: diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 2bdc32c1afb65bb123a27444d9f6e4d01a188074..a9cbc8e6b796207f4880b2b32ff9289321080068 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -508,7 +508,8 @@ int tcp_v4_err(struct sk_buff *skb, u32 info) if (sk->sk_state == TCP_CLOSE) goto out; - if (unlikely(iph->ttl < inet_sk(sk)->min_ttl)) { + /* min_ttl can be changed concurrently from do_ip_setsockopt() */ + if (unlikely(iph->ttl < READ_ONCE(inet_sk(sk)->min_ttl))) { __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); goto out; } @@ -2068,7 +2069,9 @@ int tcp_v4_rcv(struct sk_buff *skb) return 0; } } - if (unlikely(iph->ttl < inet_sk(sk)->min_ttl)) { + + /* min_ttl can be changed concurrently from do_ip_setsockopt() */ + if (unlikely(iph->ttl < READ_ONCE(inet_sk(sk)->min_ttl))) { __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); goto discard_and_relse; } From patchwork Mon Oct 25 16:48:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12582379 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D367C433F5 for ; Mon, 25 Oct 2021 16:48:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4EEC4601FF for ; Mon, 25 Oct 2021 16:48:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234114AbhJYQvL (ORCPT ); Mon, 25 Oct 2021 12:51:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44690 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234086AbhJYQvC (ORCPT ); Mon, 25 Oct 2021 12:51:02 -0400 Received: from mail-pj1-x1029.google.com (mail-pj1-x1029.google.com [IPv6:2607:f8b0:4864:20::1029]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EC36C061348 for ; Mon, 25 Oct 2021 09:48:40 -0700 (PDT) Received: by mail-pj1-x1029.google.com with SMTP id n11-20020a17090a2bcb00b001a1e7a0a6a6so8216488pje.0 for ; Mon, 25 Oct 2021 09:48:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UJz1lAu5C9MAqZ8KnyZ8z6mVntpBW4i3HKAaMu3gS2I=; b=njRZINMKN4HmIIQ/J68W9QM4fviQu+Jb+LxnjUVkyOmXiDznuaSdmLuMyoSOJAmFZJ m4Og5Re+MCSOnDGC9qpu3GwDh2SxLAuPwOcJmzFsc5K2lnVHsMzQsWAYHrxwb9wEK1G8 WKUEMBdX3kanKrt8U87GjHa9p4eJMwkIG8VRSt+skT0lEZsDLdHAnE4fLgqkQrbWi9Jb baiGDIvPs0+huVkaYGQy6+SnqS9/Q7bdjmEY2cdb4OZeT8eenTv4vz3SDDLjqtsc4hZa 5656rg/Fjz0wiRlsQqXn/WHNu912NdO/0FC7RTYml4vM0xJLEDkqVGubavbZzOYgLDTd B8nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UJz1lAu5C9MAqZ8KnyZ8z6mVntpBW4i3HKAaMu3gS2I=; b=OMsHmuScD/j0tkOxG/JC9JLRG1R+nWH1a9cB/DNEIEZaarRbLGFc31MWJ506F7KGFv wjj9lczh5s+On6dkA1YE8hYq96C5EhWqbD/pmTKpK++mx4hYCb+/w64oJ4s2tz6aQhc1 DiITp107AvYz2mmQvPDRwkMHEveaVoZQ9oN2dvirszMEgMb46cDB2mPQgZw7SM+7eNu1 KaSh0ieQLYvlEnnrXKk5F8znB5i7QCb87ADznhF+qZaskRju+fet8Fu7z7KklqAxRt6p eScT3fldB4Ef5fRVwpyQMPSZyM8bhw2QCDGW/Qj25E257EPElySl8G7FCv9HK9smJ7xQ yY+g== X-Gm-Message-State: AOAM5324v8N3RZz5mhIUsd52nAx/P6GrZ8n2AYc4ioJnBVXbTRkkuUlj AFb0D/8aNy6iBMM7ZfG+OFU= X-Google-Smtp-Source: ABdhPJz3FnQgzqPBX3W+RaBrkaWF88N2vIPdaJAS8wfKC0IR5X7L+W8bJX7ISvbMn5Oosn/6z8Czow== X-Received: by 2002:a17:902:9a41:b0:140:4ca:761e with SMTP id x1-20020a1709029a4100b0014004ca761emr17295656plv.51.1635180519656; Mon, 25 Oct 2021 09:48:39 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:b7cd:daa3:a144:1652]) by smtp.gmail.com with ESMTPSA id b3sm17052582pfm.54.2021.10.25.09.48.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Oct 2021 09:48:39 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell Subject: [PATCH v2 net-next 09/10] ipv4: guard IP_MINTTL with a static key Date: Mon, 25 Oct 2021 09:48:24 -0700 Message-Id: <20211025164825.259415-10-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog In-Reply-To: <20211025164825.259415-1-eric.dumazet@gmail.com> References: <20211025164825.259415-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet RFC 5082 IP_MINTTL option is rarely used on hosts. Add a static key to remove from TCP fast path useless code, and potential cache line miss to fetch inet_sk(sk)->min_ttl Note that once ip4_min_ttl static key has been enabled, it stays enabled until next boot. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- include/net/ip.h | 2 ++ net/ipv4/ip_sockglue.c | 6 ++++++ net/ipv4/tcp_ipv4.c | 20 ++++++++++++-------- 3 files changed, 20 insertions(+), 8 deletions(-) diff --git a/include/net/ip.h b/include/net/ip.h index cf229a53119428307da898af4b0dc23e1cecc053..b71e88507c4a0907011c41e1ed0148eb873b5186 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -24,6 +24,7 @@ #include #include #include +#include #include #include @@ -750,6 +751,7 @@ void ip_cmsg_recv_offset(struct msghdr *msg, struct sock *sk, struct sk_buff *skb, int tlen, int offset); int ip_cmsg_send(struct sock *sk, struct msghdr *msg, struct ipcm_cookie *ipc, bool allow_ipv6); +DECLARE_STATIC_KEY_FALSE(ip4_min_ttl); int ip_setsockopt(struct sock *sk, int level, int optname, sockptr_t optval, unsigned int optlen); int ip_getsockopt(struct sock *sk, int level, int optname, char __user *optval, diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c index d5487c8580674a01df8c7d8ce88f97c9add846b6..38d29b175ca6646c280e0626e8e935b348f00f08 100644 --- a/net/ipv4/ip_sockglue.c +++ b/net/ipv4/ip_sockglue.c @@ -886,6 +886,8 @@ static int compat_ip_mcast_join_leave(struct sock *sk, int optname, return ip_mc_leave_group(sk, &mreq); } +DEFINE_STATIC_KEY_FALSE(ip4_min_ttl); + static int do_ip_setsockopt(struct sock *sk, int level, int optname, sockptr_t optval, unsigned int optlen) { @@ -1352,6 +1354,10 @@ static int do_ip_setsockopt(struct sock *sk, int level, int optname, goto e_inval; if (val < 0 || val > 255) goto e_inval; + + if (val) + static_branch_enable(&ip4_min_ttl); + /* tcp_v4_err() and tcp_v4_rcv() might read min_ttl * while we are changint it. */ diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index a9cbc8e6b796207f4880b2b32ff9289321080068..13d868c43284584ee0c58ddfd411bb52c8b0c830 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -508,10 +508,12 @@ int tcp_v4_err(struct sk_buff *skb, u32 info) if (sk->sk_state == TCP_CLOSE) goto out; - /* min_ttl can be changed concurrently from do_ip_setsockopt() */ - if (unlikely(iph->ttl < READ_ONCE(inet_sk(sk)->min_ttl))) { - __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); - goto out; + if (static_branch_unlikely(&ip4_min_ttl)) { + /* min_ttl can be changed concurrently from do_ip_setsockopt() */ + if (unlikely(iph->ttl < READ_ONCE(inet_sk(sk)->min_ttl))) { + __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); + goto out; + } } tp = tcp_sk(sk); @@ -2070,10 +2072,12 @@ int tcp_v4_rcv(struct sk_buff *skb) } } - /* min_ttl can be changed concurrently from do_ip_setsockopt() */ - if (unlikely(iph->ttl < READ_ONCE(inet_sk(sk)->min_ttl))) { - __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); - goto discard_and_relse; + if (static_branch_unlikely(&ip4_min_ttl)) { + /* min_ttl can be changed concurrently from do_ip_setsockopt() */ + if (unlikely(iph->ttl < READ_ONCE(inet_sk(sk)->min_ttl))) { + __NET_INC_STATS(net, LINUX_MIB_TCPMINTTLDROP); + goto discard_and_relse; + } } if (!xfrm4_policy_check(sk, XFRM_POLICY_IN, skb)) From patchwork Mon Oct 25 16:48:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12582381 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1E3AC433EF for ; Mon, 25 Oct 2021 16:48:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C902E601FF for ; Mon, 25 Oct 2021 16:48:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234110AbhJYQvM (ORCPT ); Mon, 25 Oct 2021 12:51:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44694 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234115AbhJYQvD (ORCPT ); Mon, 25 Oct 2021 12:51:03 -0400 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2CA43C061243 for ; Mon, 25 Oct 2021 09:48:41 -0700 (PDT) Received: by mail-pg1-x52c.google.com with SMTP id m21so11500317pgu.13 for ; Mon, 25 Oct 2021 09:48:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yW6wdXGI1aTedsTHKwfr06XOVK5hGfaHSHYVe7iixmE=; b=AAK8ozfKOhjIB//MhFNly4p0EzPTQIFlmyVoA9jzygnGKe0zmF51BJgCsH1qM6vX6g PP+lsukjauvc497q/EFUzoiac5BeJjAR4beuHUZHTsNDNKVwijB4Q/yfuusoUKyXmeUZ VnX2AiXgI2a3zLdY/9/AWgFXVAj9Mg7IPWkZngYfyN9N63WsVYL7bI43XAyqepNs9mQf cJTlDxel5TDOWFa0xsjwnQOOt2eirTuBv+kI2muaJeu3Ue381B3VJVgLoJIYVVA17RL1 ifmXj7A5t99UK6qicZiYER/4CS3rivtUtjrSRQjDqP8WcFRdpKoG1i0Rt8gUcU0ae4n2 exRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yW6wdXGI1aTedsTHKwfr06XOVK5hGfaHSHYVe7iixmE=; b=MfJqj2jemmAlGxo7jjhKhuwAEhW7fzeRI4kuqQPJCjzIOLmyS3VBxGN1VaA45l6KFw E73vVaRLMt22D/vFBg/UZt0qUBR58NiH+Cij8TXPhQ7M9hdoODBUCHAbWIFFaxy9QKck T/VGuggIPU+k77CTvwvbuK9+o0KjMpGdzkMPAV7MUw6QJD6w8jcrrIseGgj1arYtiIOi wepSiMrEBVDVueyFRQmAgvX7ztDE6Dtn9LPXCho98Tjc5gbBnbQRJ2gbIGKB0Hajt9XA /2zcHbZCOg/y8SbcEIsZ79cRtCx+hF1FMXsd/yzwpqnfLcNrEQ9VefDb8Cr5p6KCRZ0R lnNQ== X-Gm-Message-State: AOAM532vL3Jif3u5+Pfk855OKMoQ4ruI9kxwpwr4H24p39HiY7kH5GRX ctLcCTKenKFhNDTpe3GCSd4= X-Google-Smtp-Source: ABdhPJzKbGa3YXdrHxHVv6mV77H3gkZnM8yV9I1celHS8YV8NPt1F7z1CBal6XWN/cePDwmSgTJedg== X-Received: by 2002:a05:6a00:1707:b0:44d:47e1:9ffe with SMTP id h7-20020a056a00170700b0044d47e19ffemr20013728pfc.53.1635180520774; Mon, 25 Oct 2021 09:48:40 -0700 (PDT) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:b7cd:daa3:a144:1652]) by smtp.gmail.com with ESMTPSA id b3sm17052582pfm.54.2021.10.25.09.48.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Oct 2021 09:48:40 -0700 (PDT) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell Subject: [PATCH v2 net-next 10/10] ipv6/tcp: small drop monitor changes Date: Mon, 25 Oct 2021 09:48:25 -0700 Message-Id: <20211025164825.259415-11-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.33.0.1079.g6e70778dc9-goog In-Reply-To: <20211025164825.259415-1-eric.dumazet@gmail.com> References: <20211025164825.259415-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Two kfree_skb() calls must be replaced by consume_skb() for skbs that are not technically dropped. Signed-off-by: Eric Dumazet Acked-by: Soheil Hassas Yeganeh --- net/ipv6/tcp_ipv6.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 6945583c0fa48256db5510866342e4aab672fd71..c678e778c1fb8f8cb7300c23cb876ef0d8e750c8 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -572,7 +572,7 @@ static int tcp_v6_send_synack(const struct sock *sk, struct dst_entry *dst, static void tcp_v6_reqsk_destructor(struct request_sock *req) { kfree(inet_rsk(req)->ipv6_opt); - kfree_skb(inet_rsk(req)->pktopts); + consume_skb(inet_rsk(req)->pktopts); } #ifdef CONFIG_TCP_MD5SIG @@ -1594,7 +1594,7 @@ static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb) } } - kfree_skb(opt_skb); + consume_skb(opt_skb); return 0; }