From patchwork Mon Nov 15 19:02:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620953 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E1D3C433F5 for ; Tue, 16 Nov 2021 00:28:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3598661A52 for ; Tue, 16 Nov 2021 00:28:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348912AbhKPAbU (ORCPT ); Mon, 15 Nov 2021 19:31:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345771AbhKOT3Q (ORCPT ); Mon, 15 Nov 2021 14:29:16 -0500 Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 443CCC0BC999 for ; Mon, 15 Nov 2021 11:03:00 -0800 (PST) Received: by mail-pj1-x1034.google.com with SMTP id np6-20020a17090b4c4600b001a90b011e06so631759pjb.5 for ; Mon, 15 Nov 2021 11:03:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9BWwpkrYxVGEuYLSkV9sKoqSnqttzgUOQ4sFuU/ZB1M=; b=e/p9faxlfzCt/RtOKXa4e6KuSFlM9508ZLgKOGpLDpC3SV16vBEhhrtgMti5COjZCH BL3RiMbj8wTVgZDoTFFzr9amYUGMj6oqbHhxolgclhsbmOtw+K/osgabUepMYb6EvelH KajIqwiJd632+oh1xuYg6DsqB8UOD2j6U+Ohdw3agCD24NWhncG7mmhrrs6bfgrP8yf+ ZX9Qqk8lYZBHwjKf1sBQ/QQeICsC9NH/zQFodsHP1whhpPLo0zR63l5OS11zTcnn9Dyu 7WqszppYzPtj3saI2jnzJaVaSkMgqnOAefwrVX59t01pBZDM6s3HOchbh93rdL2A/HJd fzug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9BWwpkrYxVGEuYLSkV9sKoqSnqttzgUOQ4sFuU/ZB1M=; b=nl65b8POZrfQCA++8+TJPxweq2b+c7YkC3boyzFa/AwRYh6x2M+huGiwNw122M/df5 ViqGjF6J7U7jJi0L4IFmtxUkfMplLm238dFYcGUhHUC2f7e29GvpASiodN3hTDfqB80G ohhU9yqbJVBY0jTNBt3BYCW72MrLti+/j09D/d5vT5dd84BvdU0XkUqzNsj5jWAxFmRt xSbVJbJLwxqBzQHvE7eEngRNVvixazzyplvXERjfxbUBO2hiJ9KPHScHGFZ8yDsy5qiN KGfOmQ7F1CyF2BaCA//FGG9FZGTVSN8nV8vAXYTlPnvjZtUuDEGS129PuGGa5j1EfUMb vtgQ== X-Gm-Message-State: AOAM5317TtSqsvuWUE/8tPUDZOpAjAfVY+XSDCsASR5uF6YJitO7oBTI B6WXqb9GqOjHfgnhtOhYI8kbiOEVjPw= X-Google-Smtp-Source: ABdhPJyDi+W+EK7J7x+xqUjcnqZiv4+GoRUBDskqOdHlRsAIowUL39grX3u3TpyCs8PCXl7kE+Xxaw== X-Received: by 2002:a17:902:b28a:b0:142:3e17:38d8 with SMTP id u10-20020a170902b28a00b001423e1738d8mr38367886plr.56.1637002979742; Mon, 15 Nov 2021 11:02:59 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.02.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:02:59 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 01/20] tcp: minor optimization in tcp_add_backlog() Date: Mon, 15 Nov 2021 11:02:30 -0800 Message-Id: <20211115190249.3936899-2-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet If packet is going to be coalesced, sk_sndbuf/sk_rcvbuf values are not used. Defer their access to the point we need them. Signed-off-by: Eric Dumazet --- net/ipv4/tcp_ipv4.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 13d868c43284584ee0c58ddfd411bb52c8b0c830..82a9e1b75405f1488d6bc5d56c11f9bc597ddb07 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -1800,8 +1800,7 @@ int tcp_v4_early_demux(struct sk_buff *skb) bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb) { - u32 limit = READ_ONCE(sk->sk_rcvbuf) + READ_ONCE(sk->sk_sndbuf); - u32 tail_gso_size, tail_gso_segs; + u32 limit, tail_gso_size, tail_gso_segs; struct skb_shared_info *shinfo; const struct tcphdr *th; struct tcphdr *thtail; @@ -1909,7 +1908,7 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb) * to reduce memory overhead, so add a little headroom here. * Few sockets backlog are possibly concurrently non empty. */ - limit += 64*1024; + limit = READ_ONCE(sk->sk_rcvbuf) + READ_ONCE(sk->sk_sndbuf) + 64*1024; if (unlikely(sk_add_backlog(sk, skb, limit))) { bh_unlock_sock(sk); From patchwork Mon Nov 15 19:02:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620911 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7AEAC433F5 for ; Tue, 16 Nov 2021 00:28:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D0D77619E3 for ; Tue, 16 Nov 2021 00:28:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345797AbhKPAbG (ORCPT ); Mon, 15 Nov 2021 19:31:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345779AbhKOT3V (ORCPT ); Mon, 15 Nov 2021 14:29:21 -0500 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B1B6C0BC9A0 for ; Mon, 15 Nov 2021 11:03:03 -0800 (PST) Received: by mail-pl1-x632.google.com with SMTP id p18so15197508plf.13 for ; Mon, 15 Nov 2021 11:03:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HUrBJZSDcvJUwz5P7+s6J9RNx1WuMqW7KYHmoyPKCEA=; b=Jj731c456blPGTO2Gph0c8OiZjCR4I65YJ3hJcNMbQOhGC0OjppnNirFFpH72+2m8t FzMLderViepUfYpWqWgT9M/eEFnXmNSObeR63rAd8qrWFGqBhRrqKM4ygRbhcPIgdZCK NpnxavLoXErQMJMeVIWfbBUBJCq7CS4U9cUTrlWPAPLEfxg9yaTIzUjcFra2Zg9H+spO eR5IgvXhl14E97Uy45kPR9hIj7lTHlnFGU3+leoIJMe1VI/9qrOSET4aEsbKNhZwdS4P +bOH/7lof95T1N73uq2zIcqn7ThC0/mkc2VLb3ybbK6WsCsSgip8VKUFulYBqP5LGOzB KeVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HUrBJZSDcvJUwz5P7+s6J9RNx1WuMqW7KYHmoyPKCEA=; b=SjfRQSJ1iidec7Lfd2x8FUIh+5DpuIWKLtoCjehV64vAHONiMP6vlmc0WXLK2Ay7Np z3JCOSpSQlXh7GNGkg3P7LnpOxO9s34vzx1ieMEXBTiP18S+xv1AFeJuyw9Kkfh9AWSZ Sc7krB1tR437lt3SlTxO2xTQOb5zQTkdERtg098RX1Z9KfGpjPlWZ9j9ENR6+xARZiRQ 5QTTHNDU3+A0YP9ob69U8FJdCYz+Ar8vbMzHMGh6HG3emCpqg3/nN0P/1xKZvBZ0WCUH /LrY9j203DY9rA7X9ANN7QX3nNelPdJJ47LHG5/gVs9Ukn+qoDzr8IM8gtLIs3uDpWq5 cF2A== X-Gm-Message-State: AOAM533WCoreL68POMxvaAA88AEaOsaMtNIwTAmRPamNGpf3y4SC+qQw 5mlg97jjcQ1cd+e0az5tUJGVeeXv02w= X-Google-Smtp-Source: ABdhPJyn8FqanG9/bcEtTtt8qvHpTjgo94s9cYqY01FhfT+Q1gNEhl0k85I6hsKKEjrBJ9GU50Dm2g== X-Received: by 2002:a17:902:da8e:b0:142:1142:b794 with SMTP id j14-20020a170902da8e00b001421142b794mr38249013plx.82.1637002982777; Mon, 15 Nov 2021 11:03:02 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:01 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 02/20] tcp: remove dead code in __tcp_v6_send_check() Date: Mon, 15 Nov 2021 11:02:31 -0800 Message-Id: <20211115190249.3936899-3-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet For some reason, I forgot to change __tcp_v6_send_check() at the same time I removed (ip_summed == CHECKSUM_PARTIAL) check in __tcp_v4_send_check() Fixes: 98be9b12096f ("tcp: remove dead code after CHECKSUM_PARTIAL adoption") Signed-off-by: Eric Dumazet --- include/net/ip6_checksum.h | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/include/net/ip6_checksum.h b/include/net/ip6_checksum.h index b3f4eaa88672a2e64ec3fbb3e77a60fe383e59d9..ea681910b7a3af3624b7248651ba8fdc587bafba 100644 --- a/include/net/ip6_checksum.h +++ b/include/net/ip6_checksum.h @@ -65,15 +65,9 @@ static inline void __tcp_v6_send_check(struct sk_buff *skb, { struct tcphdr *th = tcp_hdr(skb); - if (skb->ip_summed == CHECKSUM_PARTIAL) { - th->check = ~tcp_v6_check(skb->len, saddr, daddr, 0); - skb->csum_start = skb_transport_header(skb) - skb->head; - skb->csum_offset = offsetof(struct tcphdr, check); - } else { - th->check = tcp_v6_check(skb->len, saddr, daddr, - csum_partial(th, th->doff << 2, - skb->csum)); - } + th->check = ~tcp_v6_check(skb->len, saddr, daddr, 0); + skb->csum_start = skb_transport_header(skb) - skb->head; + skb->csum_offset = offsetof(struct tcphdr, check); } static inline void tcp_v6_gso_csum_prep(struct sk_buff *skb) From patchwork Mon Nov 15 19:02:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620919 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29ECEC433F5 for ; Tue, 16 Nov 2021 00:28:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0047D619E3 for ; Tue, 16 Nov 2021 00:28:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344645AbhKPAbL (ORCPT ); Mon, 15 Nov 2021 19:31:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345780AbhKOT3V (ORCPT ); Mon, 15 Nov 2021 14:29:21 -0500 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB72CC0BC9A1 for ; Mon, 15 Nov 2021 11:03:04 -0800 (PST) Received: by mail-pj1-x1035.google.com with SMTP id fv9-20020a17090b0e8900b001a6a5ab1392so647181pjb.1 for ; Mon, 15 Nov 2021 11:03:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xA3mbQCzAKWlSuH0DmW+A1gaMDFMbAKmKwd+iWxzBC0=; b=GW0HSdcghBGKTtSHZ3iQvlPc1s8tHjwG4PO9nDsrrG6c9S1MLOMmdZHb6MuGuxR5Wz WGVEpsGzc+0+blqJQdFsvQshAvVoivF7W+QzMdvdj+xYtxZvVGZWUEdx1195vb4Atqv5 HCgouz/SvhTSyR0Fjr1qmZ5avgkIRziDzmbC1hxcTDcTq2G1jngFCtN88sLXL3dP41lt A1/LxWdjKQtMgoHRZjetqr4vazQWqmBzv7obuvbCkX+kVxlYWtixeO65eGTGeIxkbWj8 tYhLJdl2NYSCnXlvk1YC7lQ9c9vQfnIZEYht/lx58Tr0/hC8rtKqFxcvyDKAhvrpurQV VVMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xA3mbQCzAKWlSuH0DmW+A1gaMDFMbAKmKwd+iWxzBC0=; b=ueuJxSD7+ceHGdAANgcyUzQULBtljZaTNsp9RkbHFF1CxpX41FtbfkWwCuxH8Bf7fT 5CV+PbiYqsXiACs900w9BJupOa0tqTpI06ZM+ctbnLx5R0Zv935RS/QSlMMj8pqmWrn8 IqKk/ZJAZfZbh6BVXJrwZnMQKkG4Ed4RjAZdfI7D7OI/ePjsxi9cbC3aJdaVVu2sygtS e1XUZV6WFhnKyC1btP5pWB+KL21yKYPU7Vdhq9DzdFkuym8/XdV2M/Gefd3kR/ATB7ZM E9P85Poev6fUL/v6aq5LyaAJCZr+YqCicHO5RijbUy6CvV7tUtmum4eKuyzTqWMJ5nCM x5qA== X-Gm-Message-State: AOAM533UU6IVyKgYbmLSTdtg96gk38UB8Lc1d64vFN+raasuZye6d4K5 I3OsLzxf40J52VkSTHpDTlQ= X-Google-Smtp-Source: ABdhPJzinRl9TuuR6kgew6FuPs0MoctCWRKcPX5He5k70HrUZD+YvYF+k2uTp1PZJEFne4D12eONlQ== X-Received: by 2002:a17:90b:3b8d:: with SMTP id pc13mr1031531pjb.112.1637002984345; Mon, 15 Nov 2021 11:03:04 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:03 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 03/20] tcp: small optimization in tcp_v6_send_check() Date: Mon, 15 Nov 2021 11:02:32 -0800 Message-Id: <20211115190249.3936899-4-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet For TCP flows, inet6_sk(sk)->saddr has the same value than sk->sk_v6_rcv_saddr. Using sk->sk_v6_rcv_saddr increases data locality. Signed-off-by: Eric Dumazet --- net/ipv6/tcp_ipv6.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 551fce49841d7f53a111b0435855634cece2b40a..1f1a89f096de9f77ab1bd2d871eb90a3f12e91e0 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -1893,9 +1893,7 @@ static struct timewait_sock_ops tcp6_timewait_sock_ops = { INDIRECT_CALLABLE_SCOPE void tcp_v6_send_check(struct sock *sk, struct sk_buff *skb) { - struct ipv6_pinfo *np = inet6_sk(sk); - - __tcp_v6_send_check(skb, &np->saddr, &sk->sk_v6_daddr); + __tcp_v6_send_check(skb, &sk->sk_v6_rcv_saddr, &sk->sk_v6_daddr); } const struct inet_connection_sock_af_ops ipv6_specific = { From patchwork Mon Nov 15 19:02:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620913 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00CF2C433FE for ; Tue, 16 Nov 2021 00:28:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DB35B619EA for ; Tue, 16 Nov 2021 00:28:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345762AbhKPAbI (ORCPT ); Mon, 15 Nov 2021 19:31:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345790AbhKOT3V (ORCPT ); Mon, 15 Nov 2021 14:29:21 -0500 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 638D5C0BC9A3 for ; Mon, 15 Nov 2021 11:03:06 -0800 (PST) Received: by mail-pl1-x630.google.com with SMTP id n8so15272739plf.4 for ; Mon, 15 Nov 2021 11:03:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IeJyargwm9M1G/oIDoLt8+Lyr5Du0x0yvF0DDZ4zfAs=; b=BiLYZ4KDbejfNIWtB5BV7Dy8WI3mwPGPw7iBVvuJDySejFXDHLMb/NvKbKbDJSuXhI q4Ohz+XmR5P3NApg6gTHfn91QsX0bBH4i7InP/TUjRav8TnMULewsOTJOqFVXOml/OaH tQdJN0Xr/s+lfVGYGhMhGPCS9H8IDkdDzGLYLfC+jHlxPLure94NRxnFvqTwfUZjgor5 0Vw11ntNynt/oSWeKlZ8fkGY0wwHq7kNiUsY969x1FP9wBFGXv1zVpOtGITnPjqMSNT7 jrZFCFbKC3youoaIXgDbkADbclFj2FUBmRhig/OBNNFnaq+jjie/mYGx0X13C0l6BKY+ tHlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IeJyargwm9M1G/oIDoLt8+Lyr5Du0x0yvF0DDZ4zfAs=; b=Bv/zMzWTCy1VIqTGPoprBHTqFmgyflMrP08GEQfUIYjY6hmSIdVBRbC9o5C4zGEP35 snInkdc8xiMPRmu8UjAPx1FNk7JUvP+urDNAG5hHkD/hbztQoKkO8BOPIMxAIjMURBx4 Az+2W+2t5gVmbcTMWXYIYKfTNEP8xfrO3YVOhcNfKrWNEOgPOwJT8GXEONy1axk07s/j 5NPslz5cZFBFhXYBxQi9NrUQcWFpkwq/IhccuDu5VCegX+iHAMmkLu3D7AjUIh5eQXvZ wYrjlM9pZ3J6R/YeV7y3Fs+91+Bh1Tmuh/nXw0PnyhZvRP5wMqcEVfM7REQPbcDwaqFX 90FA== X-Gm-Message-State: AOAM531LRw+2rGHsCTh+nCJ5yGMoJE2RaRtrjjssF1Ubr2ZCzZUDNJ03 5F+1u9cIT3VE1L5GNlN4qvA= X-Google-Smtp-Source: ABdhPJwgnJzjIo8HKmK8f7EvAOlMC8VnMIMoLci6pAcyno10ThUji80ePhMCpK3AsB79D0JfSOjwNA== X-Received: by 2002:a17:902:f784:b0:141:c9ce:6725 with SMTP id q4-20020a170902f78400b00141c9ce6725mr38174983pln.58.1637002985784; Mon, 15 Nov 2021 11:03:05 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:05 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 04/20] net: use sk_is_tcp() in more places Date: Mon, 15 Nov 2021 11:02:33 -0800 Message-Id: <20211115190249.3936899-5-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Move sk_is_tcp() to include/net/sock.h and use it where we can. Signed-off-by: Eric Dumazet --- include/linux/skmsg.h | 6 ------ include/net/sock.h | 5 +++++ net/core/skbuff.c | 6 ++---- net/core/sock.c | 6 ++---- 4 files changed, 9 insertions(+), 14 deletions(-) diff --git a/include/linux/skmsg.h b/include/linux/skmsg.h index 584d94be9c8b06e43dff0eecfcc25a63a17a652d..18a717fe62eb049758bc1502da97365cf7587ffd 100644 --- a/include/linux/skmsg.h +++ b/include/linux/skmsg.h @@ -507,12 +507,6 @@ static inline bool sk_psock_strp_enabled(struct sk_psock *psock) return !!psock->saved_data_ready; } -static inline bool sk_is_tcp(const struct sock *sk) -{ - return sk->sk_type == SOCK_STREAM && - sk->sk_protocol == IPPROTO_TCP; -} - static inline bool sk_is_udp(const struct sock *sk) { return sk->sk_type == SOCK_DGRAM && diff --git a/include/net/sock.h b/include/net/sock.h index b32906e1ab55527b5418f203d3de05853863f166..5bdeffdea5ecdb6069d13906bbf872d4479a1ce7 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -2638,6 +2638,11 @@ static inline void skb_setup_tx_timestamp(struct sk_buff *skb, __u16 tsflags) &skb_shinfo(skb)->tskey); } +static inline bool sk_is_tcp(const struct sock *sk) +{ + return sk->sk_type == SOCK_STREAM && sk->sk_protocol == IPPROTO_TCP; +} + /** * sk_eat_skb - Release a skb if it is no longer needed * @sk: socket to eat this skb from diff --git a/net/core/skbuff.c b/net/core/skbuff.c index ba2f38246f07e5ba5a4f97922b4be33bdb8ad6d6..d57796f38a0b4b0b78c513e6733580f9d4b56dc8 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -4849,8 +4849,7 @@ static void __skb_complete_tx_timestamp(struct sk_buff *skb, serr->header.h4.iif = skb->dev ? skb->dev->ifindex : 0; if (sk->sk_tsflags & SOF_TIMESTAMPING_OPT_ID) { serr->ee.ee_data = skb_shinfo(skb)->tskey; - if (sk->sk_protocol == IPPROTO_TCP && - sk->sk_type == SOCK_STREAM) + if (sk_is_tcp(sk)) serr->ee.ee_data -= sk->sk_tskey; } @@ -4919,8 +4918,7 @@ void __skb_tstamp_tx(struct sk_buff *orig_skb, if (tsonly) { #ifdef CONFIG_INET if ((sk->sk_tsflags & SOF_TIMESTAMPING_OPT_STATS) && - sk->sk_protocol == IPPROTO_TCP && - sk->sk_type == SOCK_STREAM) { + sk_is_tcp(sk)) { skb = tcp_get_timestamping_opt_stats(sk, orig_skb, ack_skb); opt_stats = true; diff --git a/net/core/sock.c b/net/core/sock.c index 8f2b2f2c0e7b1decdb4a5c8d86327ed7caa62c99..0be8e43f44b9e68678f4e20c3a86324ba1bfe03e 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -872,8 +872,7 @@ int sock_set_timestamping(struct sock *sk, int optname, if (val & SOF_TIMESTAMPING_OPT_ID && !(sk->sk_tsflags & SOF_TIMESTAMPING_OPT_ID)) { - if (sk->sk_protocol == IPPROTO_TCP && - sk->sk_type == SOCK_STREAM) { + if (sk_is_tcp(sk)) { if ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN)) return -EINVAL; @@ -1370,8 +1369,7 @@ int sock_setsockopt(struct socket *sock, int level, int optname, case SO_ZEROCOPY: if (sk->sk_family == PF_INET || sk->sk_family == PF_INET6) { - if (!((sk->sk_type == SOCK_STREAM && - sk->sk_protocol == IPPROTO_TCP) || + if (!(sk_is_tcp(sk) || (sk->sk_type == SOCK_DGRAM && sk->sk_protocol == IPPROTO_UDP))) ret = -ENOTSUPP; From patchwork Mon Nov 15 19:02:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620907 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFF8FC433FE for ; Tue, 16 Nov 2021 00:28:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D7CC9619E3 for ; Tue, 16 Nov 2021 00:28:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345761AbhKPAbF (ORCPT ); Mon, 15 Nov 2021 19:31:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345785AbhKOT3V (ORCPT ); Mon, 15 Nov 2021 14:29:21 -0500 Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0284C0BC9A5 for ; Mon, 15 Nov 2021 11:03:07 -0800 (PST) Received: by mail-pj1-x1034.google.com with SMTP id np3so13656370pjb.4 for ; Mon, 15 Nov 2021 11:03:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=n+6AYEM1krZbe6qqssF9bDseXCWrufJeh1nSPzlF2gg=; b=DfUL4CQ6pB2RcYtPNoXnrmxc6QXlULTjMlUEnv2HFWSiHw7YYKfPjHe1wuVnp5PgTw WktWMJbIgidYmz/rbH19GGp4JKAq1UTPrtPv6a8XAMS8S5n7L9ce8cAm8cVzv40zAToM QcRM0SwpmC8GrFgLu7quUnBEqA+T6b9cefGyQrFrfeFgvSVk2BQ8IZSrXIZKm4wzBrfs NzF9370ZgyLcvFxiMGriN8jOHNtjfbmtwD+jm95DRk/5NYwtsT2KpR4EHUOfGfDmo0SM junu3JrHnrmS87M1tie4qKTo45Av1pZgBnanQfX6nom8kjUJZfW2oCZCL4E8+gKdDreE f51A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=n+6AYEM1krZbe6qqssF9bDseXCWrufJeh1nSPzlF2gg=; b=1XKD3jRuoAVR6wO4pSqENm/Ci8gD4BHMyaAsenp0TTgOiVAfKoKxU6xUpobi+Yg1tI HquCqbfgGXLtHxuTHjTKk5gB4I9RshDiJmRfHVOjIW53pgSMLN8z/NUVITjDHuo79ZN8 sttptc6i8hwbFi3d66qjfUEDKrcSqREMoUJmvUpnL9aLs0ept3llPFh8LU0USf81wi2C Q7aKZF7Vwmda5xvZ+f/TgjvtbjYlxgOPfBoC5x4XKUUr5r7Vko9xh33rPGXy9R2gur8K 8DWxVBl9Xxel6ew+xr0I57ln4i7YoWCjKDmLTEd+5QwnSwlyOreQyaPSvnhUk60oTPgn kJuQ== X-Gm-Message-State: AOAM531tqP+euWUD7bOW4QXkqoRIXIMxZFQBFInKl820cWJoF9vIAYaI rAaZSWkzMlXL7KTWG9QsA1Q= X-Google-Smtp-Source: ABdhPJwFxrIFV1VJUpqK7HBXI8FJLGENkNBV6Y5vvs9q5tEVRDdkA5ni6iM/cKK24zqM4kvdy5TJeQ== X-Received: by 2002:a17:902:e353:b0:142:d33:9acd with SMTP id p19-20020a170902e35300b001420d339acdmr38468446plc.78.1637002987251; Mon, 15 Nov 2021 11:03:07 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:06 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 05/20] net: remove sk_route_forced_caps Date: Mon, 15 Nov 2021 11:02:34 -0800 Message-Id: <20211115190249.3936899-6-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet We were only using one bit, and we can replace it by sk_is_tcp() Signed-off-by: Eric Dumazet --- include/net/sock.h | 3 --- net/core/sock.c | 4 +++- net/ipv4/tcp.c | 1 - 3 files changed, 3 insertions(+), 5 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 5bdeffdea5ecdb6069d13906bbf872d4479a1ce7..ebad629dd9eda4bcec6f621cf2d4f783f293b7b7 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -285,8 +285,6 @@ struct bpf_local_storage; * @sk_no_check_rx: allow zero checksum in RX packets * @sk_route_caps: route capabilities (e.g. %NETIF_F_TSO) * @sk_route_nocaps: forbidden route capabilities (e.g NETIF_F_GSO_MASK) - * @sk_route_forced_caps: static, forced route capabilities - * (set in tcp_init_sock()) * @sk_gso_type: GSO type (e.g. %SKB_GSO_TCPV4) * @sk_gso_max_size: Maximum GSO segment size to build * @sk_gso_max_segs: Maximum number of GSO segments @@ -461,7 +459,6 @@ struct sock { struct page_frag sk_frag; netdev_features_t sk_route_caps; netdev_features_t sk_route_nocaps; - netdev_features_t sk_route_forced_caps; int sk_gso_type; unsigned int sk_gso_max_size; gfp_t sk_allocation; diff --git a/net/core/sock.c b/net/core/sock.c index 0be8e43f44b9e68678f4e20c3a86324ba1bfe03e..257b5fa604804ea671c0dbede4455ade8d65ede8 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2244,7 +2244,9 @@ void sk_setup_caps(struct sock *sk, struct dst_entry *dst) u32 max_segs = 1; sk_dst_set(sk, dst); - sk->sk_route_caps = dst->dev->features | sk->sk_route_forced_caps; + sk->sk_route_caps = dst->dev->features; + if (sk_is_tcp(sk)) + sk->sk_route_caps |= NETIF_F_GSO; if (sk->sk_route_caps & NETIF_F_GSO) sk->sk_route_caps |= NETIF_F_GSO_SOFTWARE; sk->sk_route_caps &= ~sk->sk_route_nocaps; diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index b7796b4cf0a099e9f14b28e50cb07367021a7cbf..4fa4b29260bd4c08da70b3fb199e3459013114f3 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -456,7 +456,6 @@ void tcp_init_sock(struct sock *sk) WRITE_ONCE(sk->sk_rcvbuf, sock_net(sk)->ipv4.sysctl_tcp_rmem[1]); sk_sockets_allocated_inc(sk); - sk->sk_route_forced_caps = NETIF_F_GSO; } EXPORT_SYMBOL(tcp_init_sock); From patchwork Mon Nov 15 19:02:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620327 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76100C433EF for ; Mon, 15 Nov 2021 20:09:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5B77261BD4 for ; Mon, 15 Nov 2021 20:09:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348494AbhKOULU (ORCPT ); Mon, 15 Nov 2021 15:11:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38280 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345792AbhKOT3V (ORCPT ); Mon, 15 Nov 2021 14:29:21 -0500 Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 123CEC0BC9A6 for ; Mon, 15 Nov 2021 11:03:09 -0800 (PST) Received: by mail-pj1-x1031.google.com with SMTP id fv9-20020a17090b0e8900b001a6a5ab1392so647361pjb.1 for ; Mon, 15 Nov 2021 11:03:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=o1Omkca/LYbQN9TgKjlnuoa43OtoyGHQvu/csyGiDLk=; b=ohUBpNZrwSr6dQNPTk8CkukK1KDYsNvWnwPBfCFhOQr715h1LbN0Ue0pUchowMsS4o 1mamjWyLI06MTy6RMDHfBsZlaXVg2qalwcSU5apYcMaRb5UdPKXqFpDB9gc7Yj3OtBbt p6afeqUPFNID7q7RK5i0WJ5VyM25mWOD6b7PbjlWagvprOkmv5mj/+7h+v7lW2Y9uLtg erVx4hlMOKCsHYuK/c1q7ZwMEyc3gsdzM4DeMJcR/LvDYHGjc03oSmCrCeOItP+D7yt2 QkXl9d8eUbadnIVBbaf0AwEVaL9LnBqq0YrsdqKwceS0b00upyy8LODnN6/HO1QyuwkH WG5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=o1Omkca/LYbQN9TgKjlnuoa43OtoyGHQvu/csyGiDLk=; b=7w71B/cX7sfwu/z5P2UxOK4H4xOkIxNz3XALDvNdKUqlrnIdnPd8xCLPbiLUaZ0DKB QSzepzhpHCac+jFSWhthoRvRUre94xHYmJ4KOi7xnHOEYI43eidwc/9zYbsj+JDBc9iD n4bW0cx3fK/C/iN/TLgd/Bn2Ae1siSQS53d8TqCmfkNNdPu6aQRukyis9ZCMsTt5xYgN VnxFnFxJPR3JjseVQNU/+uBWd9zGLMRk2GIXGluGY6M888egufrEcFxVRgA9Cr6OXLxi MQHLV4EpwPPb3YyONaDNRpfu442OIa6o/uGApIfQZRlXzMB4cj1rgBrzGabAKMjX442O VERg== X-Gm-Message-State: AOAM533lFWRN19Ys7s74PZ5DoFhwF4jKX+okv6+S3jAoC519zUKZ+oeb C7ZsiV/V14gtM4lSnXsvolM= X-Google-Smtp-Source: ABdhPJwwu6UrSLTbgrPCSYGlPZ1n0dQlFyk0Z9yBWNTP8Mc/8pihofTHctraS7QTzU7GoA8tXxY0Nw== X-Received: by 2002:a17:90b:4a0e:: with SMTP id kk14mr67199117pjb.42.1637002988500; Mon, 15 Nov 2021 11:03:08 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:08 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 06/20] net: remove sk_route_nocaps Date: Mon, 15 Nov 2021 11:02:35 -0800 Message-Id: <20211115190249.3936899-7-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Instead of using a full netdev_features_t, we can use a single bit, as sk_route_nocaps is only used to remove NETIF_F_GSO_MASK from sk->sk_route_cap. Signed-off-by: Eric Dumazet --- include/net/sock.h | 11 +++++------ net/core/sock.c | 3 ++- net/ipv4/tcp_ipv4.c | 4 ++-- net/ipv4/tcp_output.c | 2 +- net/ipv6/ip6_output.c | 2 +- 5 files changed, 11 insertions(+), 11 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index ebad629dd9eda4bcec6f621cf2d4f783f293b7b7..985ddcd335048068b78a0525500734ef96be44a0 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -284,7 +284,7 @@ struct bpf_local_storage; * @sk_no_check_tx: %SO_NO_CHECK setting, set checksum in TX packets * @sk_no_check_rx: allow zero checksum in RX packets * @sk_route_caps: route capabilities (e.g. %NETIF_F_TSO) - * @sk_route_nocaps: forbidden route capabilities (e.g NETIF_F_GSO_MASK) + * @sk_gso_disabled: if set, NETIF_F_GSO_MASK is forbidden. * @sk_gso_type: GSO type (e.g. %SKB_GSO_TCPV4) * @sk_gso_max_size: Maximum GSO segment size to build * @sk_gso_max_segs: Maximum number of GSO segments @@ -458,7 +458,6 @@ struct sock { unsigned long sk_max_pacing_rate; struct page_frag sk_frag; netdev_features_t sk_route_caps; - netdev_features_t sk_route_nocaps; int sk_gso_type; unsigned int sk_gso_max_size; gfp_t sk_allocation; @@ -468,7 +467,7 @@ struct sock { * Because of non atomicity rules, all * changes are protected by socket lock. */ - u8 sk_padding : 1, + u8 sk_gso_disabled : 1, sk_kern_sock : 1, sk_no_check_tx : 1, sk_no_check_rx : 1, @@ -2121,10 +2120,10 @@ static inline bool sk_can_gso(const struct sock *sk) void sk_setup_caps(struct sock *sk, struct dst_entry *dst); -static inline void sk_nocaps_add(struct sock *sk, netdev_features_t flags) +static inline void sk_gso_disable(struct sock *sk) { - sk->sk_route_nocaps |= flags; - sk->sk_route_caps &= ~flags; + sk->sk_gso_disabled = 1; + sk->sk_route_caps &= ~NETIF_F_GSO_MASK; } static inline int skb_do_copy_data_nocache(struct sock *sk, struct sk_buff *skb, diff --git a/net/core/sock.c b/net/core/sock.c index 257b5fa604804ea671c0dbede4455ade8d65ede8..99738e14224c44e5aa4b88857620fb162e9c265f 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2249,7 +2249,8 @@ void sk_setup_caps(struct sock *sk, struct dst_entry *dst) sk->sk_route_caps |= NETIF_F_GSO; if (sk->sk_route_caps & NETIF_F_GSO) sk->sk_route_caps |= NETIF_F_GSO_SOFTWARE; - sk->sk_route_caps &= ~sk->sk_route_nocaps; + if (unlikely(sk->sk_gso_disabled)) + sk->sk_route_caps &= ~NETIF_F_GSO_MASK; if (sk_can_gso(sk)) { if (dst->header_len && !xfrm_dst_offload_ok(dst)) { sk->sk_route_caps &= ~NETIF_F_GSO_MASK; diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 82a9e1b75405f1488d6bc5d56c11f9bc597ddb07..5ad81bfb27b2f8d9a3cfe11141160b48092cfa3a 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -1182,7 +1182,7 @@ int tcp_md5_do_add(struct sock *sk, const union tcp_md5_addr *addr, if (!md5sig) return -ENOMEM; - sk_nocaps_add(sk, NETIF_F_GSO_MASK); + sk_gso_disable(sk); INIT_HLIST_HEAD(&md5sig->head); rcu_assign_pointer(tp->md5sig_info, md5sig); } @@ -1620,7 +1620,7 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb, */ tcp_md5_do_add(newsk, addr, AF_INET, 32, l3index, key->flags, key->key, key->keylen, GFP_ATOMIC); - sk_nocaps_add(newsk, NETIF_F_GSO_MASK); + sk_gso_disable(newsk); } #endif diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 2e6e5a70168ebd037661dcee51595183b91f36f6..5079832af5c1090917a8fd5dfb1a3025e2d85ae0 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -1359,7 +1359,7 @@ static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, #ifdef CONFIG_TCP_MD5SIG /* Calculate the MD5 hash, as we have all we need now */ if (md5) { - sk_nocaps_add(sk, NETIF_F_GSO_MASK); + sk_gso_disable(sk); tp->af_specific->calc_md5_hash(opts.hash_location, md5, sk, skb); } diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 2f044a49afa8cf3586c36607c34073edecafc69c..007e433d4d4de7321e25db2a5fff83768dd8723a 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -977,7 +977,7 @@ int ip6_fragment(struct net *net, struct sock *sk, struct sk_buff *skb, fail_toobig: if (skb->sk && dst_allfrag(skb_dst(skb))) - sk_nocaps_add(skb->sk, NETIF_F_GSO_MASK); + sk_gso_disable(skb->sk); icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu); err = -EMSGSIZE; From patchwork Mon Nov 15 19:02:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620325 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3122C4167B for ; Mon, 15 Nov 2021 20:08:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D58C961B93 for ; Mon, 15 Nov 2021 20:08:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345562AbhKOULO (ORCPT ); Mon, 15 Nov 2021 15:11:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345787AbhKOT3V (ORCPT ); Mon, 15 Nov 2021 14:29:21 -0500 Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 403F5C0BC9A8 for ; Mon, 15 Nov 2021 11:03:10 -0800 (PST) Received: by mail-pl1-x634.google.com with SMTP id n8so15272879plf.4 for ; Mon, 15 Nov 2021 11:03:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LzvJ3uZpmPz8KIpiyyH1ezS3hd212SefPfGNUcXRiNk=; b=Q3O3+ksxvz6f897rrzvHxgsdFeQFejtLsiJgpoxOBH70UdsVZJvUnl5e87dTzr+92J TRbHV1UjwUiWQMgZ0nydGf+tHjuZ0oajrXKMHHv0GFmKJYqd17lzb8BUoBI7GSp1CsIA mMuI6naU1n1Iexz7Dd9JfzapKGXXYIkJxVBe2DosAnS2o67+RqgFetTjv8WIR9oxc3XD KxamAhAzCApv5Gk3WugdBtQfHqjBsgWp6P2yoOHk13w54Itl+JokaQugyf11j3VsHC+L 6RWbAZGnh+wrktu1VwjtyM0tHBdSYItkipq4m+w36IP0/2ae/PSninrHGNZNxd0/P9oA k4mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LzvJ3uZpmPz8KIpiyyH1ezS3hd212SefPfGNUcXRiNk=; b=DarwoXd00GlZs2qa1jRT1h/fI+Mp7ECnACuCwsqU4TWb3+WUJNwRNQKF9soCukHC0L eab1HBvM2xNyXC0i+CyIHTC2ltmK0cD0J3FArlY5pAgdkuLxGMtjE9Fk2rQFu9/F+l35 otQo5wUCYXA90tNrtFZ8rAjmewpV48uhR1wnQxsb6MLRijbk6I0D3a0nho08KoWy6Aky pld6gukQyrKc17LbIL4PhDty1PyQ3uSN4vFJqRxvebkkgndqmEyK35hNshbP4NhnvUCw Ijx0XZAYlPbSsoI86jtM50zbDnqe8cx9+8xYtomLF1aNI1Fy5I5uWznH6Sl5fJLdURxw vjyA== X-Gm-Message-State: AOAM533BIQ2toWfsdKYmezAhwQ5l3PTcntq12RMCzCd6gVxzvXockYys rTSInq925FqTC8WkuoKFdmU= X-Google-Smtp-Source: ABdhPJxq7/Mds7YWj+KBiMh2d/YhxAXKrQHrQlZe4d+9bqgkjnzQUH//96cXgzWlH/X9LsmGuKRAMQ== X-Received: by 2002:a17:90b:4a50:: with SMTP id lb16mr68287271pjb.37.1637002989824; Mon, 15 Nov 2021 11:03:09 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:09 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 07/20] ipv6: shrink struct ipcm6_cookie Date: Mon, 15 Nov 2021 11:02:36 -0800 Message-Id: <20211115190249.3936899-8-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet gso_size can be moved after tclass, to use an existing hole. (8 bytes saved on 64bit arches) Signed-off-by: Eric Dumazet --- include/net/ipv6.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/net/ipv6.h b/include/net/ipv6.h index c19bf51ded1d026e795a3f9ae0ff3be766fc174e..53ac7707ca70af513aef32389afd681bff3082f3 100644 --- a/include/net/ipv6.h +++ b/include/net/ipv6.h @@ -345,9 +345,9 @@ struct ipcm6_cookie { struct sockcm_cookie sockc; __s16 hlimit; __s16 tclass; - __s8 dontfrag; - struct ipv6_txoptions *opt; __u16 gso_size; + __s8 dontfrag; + struct ipv6_txoptions *opt; }; static inline void ipcm6_init(struct ipcm6_cookie *ipc6) From patchwork Mon Nov 15 19:02:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620915 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE108C433F5 for ; Tue, 16 Nov 2021 00:28:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CB913619F6 for ; Tue, 16 Nov 2021 00:28:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351840AbhKPAbJ (ORCPT ); Mon, 15 Nov 2021 19:31:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345775AbhKOT3V (ORCPT ); Mon, 15 Nov 2021 14:29:21 -0500 Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A42B0C0BC9AA for ; Mon, 15 Nov 2021 11:03:11 -0800 (PST) Received: by mail-pf1-x430.google.com with SMTP id x64so15889451pfd.6 for ; Mon, 15 Nov 2021 11:03:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sQgKlPPaZyaV7kL3H7TCeuNqjSO9ANGpYIIAyZja9Hk=; b=lnN3WOD2Ckzvxr52vbifCl4bfOE7TrecOhuoYtvrCmWbbcaWL7PW+CurHkDeUaUEjB K2dq73T4ZQ7rAQkGcCqf5KvZgeerbKcj380pl8U2beqNo31i2HRdQSmDfXBhXaZm8BL8 D4ndnKPtVcODYPs+NeDh8j8eu4bXr+/u7bhUJk6dJzwMyLcAlSDZhSqWfRN9/3Xj2aLI biAKY3/BHcX2iSP868OB16thDx+BAwypt+oj/zbTCZFwKySiH1d28zQYwBZe3evqHLCT A/gZy45KUnwyi5hjJnZu0zKezxKuHWLJ7VELESn5k0ZP6eGlhHWFWDrz0Ha254S6aL1/ Tjgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sQgKlPPaZyaV7kL3H7TCeuNqjSO9ANGpYIIAyZja9Hk=; b=tklNp5ttDnh5jF2bquUZ4jUfR9A2GAObYSmk8qMrgFrw97VzTYp8U1MNZdcxVPvNGT bY0bXbJECNOHYtGnaTxT2fgU+Lb/VfXwizvRdb7XqZ40wC2QBE0BkE666dRZWv5PZCHs oOdHoArr6NGA8KdePYFmiamPiR5wYerNQHeU3il23wb3XMvIF9WX3gWQaorckvogWzKC 1DDYLmLSgxSaEB40tlV1POBYlzSQCmLMRXWSF6YV58uOXKfXvcwxs6q7/TM16rqrX0FW p9HFzYckk14EVQFcvCwFOMO0mJebYSOxt22Vu3d2ormETzq3MthCJuk0Nmprx+qRx69C goOw== X-Gm-Message-State: AOAM532q2pDsmzLbn1UH8bhLaOrVBf+qGBU7wLFq4dkLQLaOtIHSSHLU 2JCykPifsayKF36spFKxdJw= X-Google-Smtp-Source: ABdhPJw6UNpZ66a8QBS48wN9m/kcrHoP999XBONMt1xD3xerqgeot/v51ZqlYI1AimgLNS4pUHfxZg== X-Received: by 2002:a65:6187:: with SMTP id c7mr744355pgv.317.1637002991295; Mon, 15 Nov 2021 11:03:11 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:10 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 08/20] net: shrink struct sock by 8 bytes Date: Mon, 15 Nov 2021 11:02:37 -0800 Message-Id: <20211115190249.3936899-9-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Move sk_bind_phc next to sk_peer_lock to fill a hole. Signed-off-by: Eric Dumazet --- include/net/sock.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/net/sock.h b/include/net/sock.h index 985ddcd335048068b78a0525500734ef96be44a0..2333ab08178903533cbc2dc1415a0de9545aa6db 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -489,6 +489,7 @@ struct sock { u16 sk_busy_poll_budget; #endif spinlock_t sk_peer_lock; + int sk_bind_phc; struct pid *sk_peer_pid; const struct cred *sk_peer_cred; @@ -498,7 +499,6 @@ struct sock { seqlock_t sk_stamp_seq; #endif u16 sk_tsflags; - int sk_bind_phc; u8 sk_shutdown; u32 sk_tskey; atomic_t sk_zckey; From patchwork Mon Nov 15 19:02:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620905 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 735AFC433FE for ; Tue, 16 Nov 2021 00:28:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5D3AA619F6 for ; Tue, 16 Nov 2021 00:28:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348429AbhKPAbE (ORCPT ); Mon, 15 Nov 2021 19:31:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345793AbhKOT3V (ORCPT ); Mon, 15 Nov 2021 14:29:21 -0500 Received: from mail-pg1-x52f.google.com (mail-pg1-x52f.google.com [IPv6:2607:f8b0:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00A00C0BC9AE for ; Mon, 15 Nov 2021 11:03:13 -0800 (PST) Received: by mail-pg1-x52f.google.com with SMTP id 200so15411474pga.1 for ; Mon, 15 Nov 2021 11:03:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=y8eHQYgHoIXIRLHiUQPADGg65tG82iTRKS/mqPl2NDw=; b=DI5sGUbbNb7Vo836n0dxW2T3Q7G3YNZpbLtoG0Y7Tg33KxhhdCq1Bj8IIXKrYuFJNJ wZ0TcydFGopW4HXChJT9pwqWPzx6bQq4u2zPgHlFxvVJNx/2BJ1F1de2ipHe5sW2GHHi njmCRvzCF1dssZEDx5p96U2ysK0VSFpeiM4ZV1cFZlT56F0NDz6x6aHvZqtSd38PznRm 0lhMVrN49c0fhT3+2UeTd5pYV8x5gvhK2rpcAsXeabWCR5ZFfM4usIG8uVNtS+OVJrj4 9WrBwr5yjyibFrWnATPPxF0cAfs40m7UZuxhHPvxJ+/J4jcJF6XaAEoEo/9jJt5myLcj D9QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=y8eHQYgHoIXIRLHiUQPADGg65tG82iTRKS/mqPl2NDw=; b=y2LaLbiL57spNeKBNibmybeMwAPYKfcrgcJRzl5ZtTzy+vXsd4/rXVxxhccJh833KW bTQH9tid/55BFUa9Fw5ojJuCHbys0pTKsfDl9isCTC6PDmq34bcvyO2+BGWY1MzYj/XD nLUuNmDIuvdFDy/89bFBi8pQnzwr74q+3o5CpFUP3JfxRSfiRznCLLWj+7A7z9z8uqiV vUNgiPoRRM8Oofz3hVbq0SIIkR9/GmnwFNKifJ5Zo3KLYEn1zf8uZaAmAA9J2S7ZHCmL zDgGR4COpPgvdXtAcUKplBbHZdZHQXEsrIfD13vGzkwq86gy40T4kEfhKPqAEB6fOiLL bK1g== X-Gm-Message-State: AOAM531idUAShpWqJQTtLGkVpd8ZztIP+dNjdL9E9EC2Vucegu/tAHC5 WxSOm3W5QAytIgY9fllyCzg= X-Google-Smtp-Source: ABdhPJyMUkImlILcmvr3q/PiVh7pHMzAWxF1iHxMjo4EjxTEv6WMo4gvm8BB7UMjPWjO2jCjhygVLw== X-Received: by 2002:aa7:8019:0:b0:44d:d761:6f79 with SMTP id j25-20020aa78019000000b0044dd7616f79mr35452759pfi.3.1637002992605; Mon, 15 Nov 2021 11:03:12 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:12 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 09/20] net: forward_alloc_get depends on CONFIG_MPTCP Date: Mon, 15 Nov 2021 11:02:38 -0800 Message-Id: <20211115190249.3936899-10-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet (struct proto)->sk_forward_alloc is currently only used by MPTCP. Signed-off-by: Eric Dumazet --- include/net/sock.h | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 2333ab08178903533cbc2dc1415a0de9545aa6db..cb97c448472aa5af3055916df844cbe422578190 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1206,7 +1206,9 @@ struct proto { unsigned int inuse_idx; #endif +#if IS_ENABLED(CONFIG_MPTCP) int (*forward_alloc_get)(const struct sock *sk); +#endif bool (*stream_memory_free)(const struct sock *sk, int wake); bool (*sock_is_readable)(struct sock *sk); @@ -1295,10 +1297,11 @@ INDIRECT_CALLABLE_DECLARE(bool tcp_stream_memory_free(const struct sock *sk, int static inline int sk_forward_alloc_get(const struct sock *sk) { - if (!sk->sk_prot->forward_alloc_get) - return sk->sk_forward_alloc; - - return sk->sk_prot->forward_alloc_get(sk); +#if IS_ENABLED(CONFIG_MPTCP) + if (sk->sk_prot->forward_alloc_get) + return sk->sk_prot->forward_alloc_get(sk); +#endif + return sk->sk_forward_alloc; } static inline bool __sk_stream_memory_free(const struct sock *sk, int wake) From patchwork Mon Nov 15 19:02:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620917 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75EEEC433F5 for ; Tue, 16 Nov 2021 00:28:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 62F45619E3 for ; Tue, 16 Nov 2021 00:28:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345701AbhKPAbK (ORCPT ); Mon, 15 Nov 2021 19:31:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345794AbhKOT3V (ORCPT ); Mon, 15 Nov 2021 14:29:21 -0500 Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6514CC0BC9B0 for ; Mon, 15 Nov 2021 11:03:14 -0800 (PST) Received: by mail-pj1-x102b.google.com with SMTP id j6-20020a17090a588600b001a78a5ce46aso86551pji.0 for ; Mon, 15 Nov 2021 11:03:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bmlR2S3oZD9NxWeZ2mafGLPdjBu6b2blaSlpRGXl3og=; b=MMhjxDt6skHg/0ST3ZizhL2lCzgcleL6d19aq1EvdkZUWL1kAW3rnE59lzceizSOGu zOoD5xJ0mtklR2cG6VpRAFRAHlWLH1nf+tL/VTticNTJGEABWW/eBw0TvS7Uek4eiqvh 42Fjdee3R4+vFppLasy0h5+mYIB06fJFNnPMb5eqfQALJoqEM47Z61iuTEfDLgA3qrd5 RdddGQLDZMLU4rHw1SfbXa1Hi64Wxgr84vvvgObOzGHW+1Fb1EDTtBQgVRZxB6DweSkX UYvgJaZeq6Lxtmv2vK5LGgMYgM4yj6qpxVGG0GfkWmcUK78kUpBKKQ4dMIKqnBXpbmkA ATDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bmlR2S3oZD9NxWeZ2mafGLPdjBu6b2blaSlpRGXl3og=; b=GnrqM7kwVHIDTzXb836kJDas5aBnx/c1DswetiA1R2S90l6nshXFg1/5S77u78Sw/5 VlChJ8hbE5mct0ObKbRYRBYBm6ZHbaj/TAIVaxB2i61KGGOuANRH6ui7A7pw4O7Gu/Uw aTYX7fMjjiA7/fcwzTlKUvxr1bYrY/9hhJhDDTphPld1DMwTu9eJB/EggeB9vnOOh1Y+ cr/O2MHoWSGTloMkhoDYIHePQhrsSPuusx5IKMnOGRzRZEQt2eneWr5HHwji3bT6mQ6r uVy+PPPiuMkYRsMdxX6FVC5pWf9VvVPf5bVF2Gmgwe1rLgwTrrfFEYdEld/GeuRmIJNn IQPQ== X-Gm-Message-State: AOAM532sqCDv2bmQW30kbgvvfC+Yxi/s+hBtKBiWbOSZoocxp0Tk6cx2 /VKHPsxKBgw9AjFjaHX+4aQZyIpzXS8= X-Google-Smtp-Source: ABdhPJy3dcg+sq4BDht6hU4FO4zvdBw/46sjkWUIBFxRTdLNj0MCM/bjOtkleW2I1cMwjrievvINfA== X-Received: by 2002:a17:903:245:b0:143:c5ba:8bd8 with SMTP id j5-20020a170903024500b00143c5ba8bd8mr13289957plh.64.1637002993819; Mon, 15 Nov 2021 11:03:13 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:13 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 10/20] net: cache align tcp_memory_allocated, tcp_sockets_allocated Date: Mon, 15 Nov 2021 11:02:39 -0800 Message-Id: <20211115190249.3936899-11-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet tcp_memory_allocated and tcp_sockets_allocated often share a common cache line, source of false sharing. Also take care of udp_memory_allocated and mptcp_sockets_allocated. Signed-off-by: Eric Dumazet --- net/ipv4/tcp.c | 4 ++-- net/ipv4/udp.c | 2 +- net/mptcp/protocol.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 4fa4b29260bd4c08da70b3fb199e3459013114f3..862e8cb8dda51e76300a427783a7d8c32e82cc7f 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -292,7 +292,7 @@ EXPORT_PER_CPU_SYMBOL_GPL(tcp_orphan_count); long sysctl_tcp_mem[3] __read_mostly; EXPORT_SYMBOL(sysctl_tcp_mem); -atomic_long_t tcp_memory_allocated; /* Current allocated memory. */ +atomic_long_t tcp_memory_allocated ____cacheline_aligned_in_smp; /* Current allocated memory. */ EXPORT_SYMBOL(tcp_memory_allocated); #if IS_ENABLED(CONFIG_SMC) @@ -303,7 +303,7 @@ EXPORT_SYMBOL(tcp_have_smc); /* * Current number of TCP sockets. */ -struct percpu_counter tcp_sockets_allocated; +struct percpu_counter tcp_sockets_allocated ____cacheline_aligned_in_smp; EXPORT_SYMBOL(tcp_sockets_allocated); /* diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index 319dd7bbfe33d64d25c36bc3a1cd3bd4caf0a779..d3cea98a5d5dbf4deea5b06bd95c38eebccd6a55 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -122,7 +122,7 @@ EXPORT_SYMBOL(udp_table); long sysctl_udp_mem[3] __read_mostly; EXPORT_SYMBOL(sysctl_udp_mem); -atomic_long_t udp_memory_allocated; +atomic_long_t udp_memory_allocated ____cacheline_aligned_in_smp; EXPORT_SYMBOL(udp_memory_allocated); #define MAX_UDP_PORTS 65536 diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index b7e32e316738b88d4b9f907f584b12785e396dae..6db93da59843a830c00e8707b7a35f90dfe8047d 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -48,7 +48,7 @@ enum { MPTCP_CMSG_TS = BIT(0), }; -static struct percpu_counter mptcp_sockets_allocated; +static struct percpu_counter mptcp_sockets_allocated ____cacheline_aligned_in_smp; static void __mptcp_destroy_sock(struct sock *sk); static void __mptcp_check_send_data_fin(struct sock *sk); From patchwork Mon Nov 15 19:02:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620333 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98CE7C43219 for ; Mon, 15 Nov 2021 20:09:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 825B461BE5 for ; Mon, 15 Nov 2021 20:09:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349281AbhKOUL7 (ORCPT ); Mon, 15 Nov 2021 15:11:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345801AbhKOT3Y (ORCPT ); Mon, 15 Nov 2021 14:29:24 -0500 Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C5FDC0BC9B2 for ; Mon, 15 Nov 2021 11:03:15 -0800 (PST) Received: by mail-pj1-x1033.google.com with SMTP id p18-20020a17090ad31200b001a78bb52876so46552pju.3 for ; Mon, 15 Nov 2021 11:03:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/TovmZ1K9tI8NqkLQ22dxDQKHjvouGKJ4E5JFP7fCMI=; b=LaTELH5xEJ7GuwtDwCpPRABIlLYpzMHtMFTcnJzjvv8mhwM1piwT8wN872jz993JFs lZ3uGPctLaor9B3wZwfNJSBdROVnEM0pbktSKtDRO4i1x14Zd+yJ9ztd/j2H4N0oQ3J2 BL6VxWSUAeoUKXIaChpE9J8KbXW4H0DPBrEWzNy7mfp3D4M6/JEPXseT3eg31vZdF/Ns nhHpXX2LZI6wDI3eJTCekHlucQY3A8PxoIdV9gGASUykjw1bPTS1KacumUqetCL+uFiK qti3ODBzSHwN3jLhBF0M17crwWDIiTGkc2p6e2E+vsYgQoWVT0J72qMEjc0i8iSmuX06 LU0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/TovmZ1K9tI8NqkLQ22dxDQKHjvouGKJ4E5JFP7fCMI=; b=elB1WS/Pv6K9720JqKSwJ42mCyuJDQm4IC3FKABzq/CXtMtONXe3ubfT0np1qAGTQ3 lJvXNXLloimc1l6mvpkfD8C+STiVKOABpyOdzzPgWvOruWMmI6sMTwcja6SORHo3TSzK ENWRebj6bCOUqkX1zCe19qdRfhDW9X4OUdcUuLkVpO6QvrBnDIR1Iq4CkK/sFkiH3ThS tCcoQZXwuidycm2nb9JJTZqFVAFwFSgX38DRlpuYDT7I3dewUheevdOemfopbr3cFvjF FMYzFgmlPxRZXCM8DbvwLRSOWn5lX1tN9tnqk978d+6LAw838P0+wp1SUs2NO7YfWimS IZrA== X-Gm-Message-State: AOAM531Ni6s9mK2JpmAZnhRt6xr0uaupL4htYq7lPs9t2immGv0Q+zke ncG7+4b+7YAxnE5ajLDKVL4= X-Google-Smtp-Source: ABdhPJyxbtPnWrfk5qbBYM656FNSH3TD/LAiC2K2Kba/d01i/1MEWglfXLEH9Vqds3sFW1Y0tSBncw== X-Received: by 2002:a17:90b:4f83:: with SMTP id qe3mr68157606pjb.56.1637002995217; Mon, 15 Nov 2021 11:03:15 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:14 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 11/20] tcp: small optimization in tcp recvmsg() Date: Mon, 15 Nov 2021 11:02:40 -0800 Message-Id: <20211115190249.3936899-12-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet When reading large chunks of data, incoming packets might be added to the backlog from BH. tcp recvmsg() detects the backlog queue is not empty, and uses a release_sock()/lock_sock() pair to process this backlog. We now have __sk_flush_backlog() to perform this a bit faster. Signed-off-by: Eric Dumazet --- net/ipv4/tcp.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 862e8cb8dda51e76300a427783a7d8c32e82cc7f..24d77a32c9cbcdf0e4380ec6d9aa3e42d2cf8730 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -2409,8 +2409,7 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, if (copied >= target) { /* Do not sleep, just process backlog. */ - release_sock(sk); - lock_sock(sk); + __sk_flush_backlog(sk); } else { sk_wait_data(sk, &timeo, last); } From patchwork Mon Nov 15 19:02:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620329 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C146AC43219 for ; Mon, 15 Nov 2021 20:09:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9F9BC61BE1 for ; Mon, 15 Nov 2021 20:09:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348878AbhKOULg (ORCPT ); Mon, 15 Nov 2021 15:11:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345802AbhKOT3Y (ORCPT ); Mon, 15 Nov 2021 14:29:24 -0500 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05B7CC0BC9B3 for ; Mon, 15 Nov 2021 11:03:17 -0800 (PST) Received: by mail-pj1-x1036.google.com with SMTP id v23so13659988pjr.5 for ; Mon, 15 Nov 2021 11:03:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LTc6Dee33uJ2iM5GtkCRCjbFA1CHZSBcy3vujWmv5u0=; b=KTfF3PIDGdoQGXweNs1CBj4XaRpu9S4ADSEdXfzBaljYYitpMtvvoFslOkR65Hy61Q QgDoJoiyNSBpiZbjGP/qmjkh+nB7PFIiasyMeqg1V1mjY9lcmfV5gesURwv/Xfx5IIGk iLrn0RmQtgj2Fnzi0I6rwkoPOYxEEaw6YyIrB4UPzewQNcyD5ihVHff+XtPLjiu+r7Uy S+3aR5iKC12NfQ1crPJ2s7K9nspn2W4dfVz7xpW1e/vTHC/Sp1yfd0f/B9vGNawbWUct LDLN84fzjYF3i+8I9x2MxCo7DXmM9VZmJ22hhWnGirXwneJP6ayr7duWequBpx3SQU/W iIww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LTc6Dee33uJ2iM5GtkCRCjbFA1CHZSBcy3vujWmv5u0=; b=KNC9IYqYuul68S7Taafnru8C3F57sMOHyhHNZfU3zGusfkCgzWjYtVmLa4Rd6/os21 Mg5A7FmBm7DAgymqNCpdKHb2bgMSZy8w7HDKHP4MIk9fDFFi0nSsRRcPVCn4sQtTrNmV EVa39KrYcpG2USpzx8u/EoXS83N9V+HHNKNp7PA3uOHtUXHaTSkSOzpT2lnhN4Wkyj5G nTTYPs2fjpGtDTtb5pg0NrR4g1Xuh6KHbfYtK6oI959XzHWKAxojjmXWeZDPQy8dMB03 hCDABwKsdmFqIIit/KPwHNx9fVJDqZGdBudPWTMBPcBajNvblBJU/CC4ewL8QdB6+UZr SkQg== X-Gm-Message-State: AOAM533HMY6xEg0S24QwnEGyXOjCbb4NSeyI+NGaRyqg6UycFvIPt166 z91Hb6QB6FbpMSjx6bLCyeYOjJZEMqU= X-Google-Smtp-Source: ABdhPJxrXDoSXowmwEsh0uWdK4tV3jhxkjqyfEcvZ4+102cy33gdrYOpUR75OQjoILkcAGRJf47MSQ== X-Received: by 2002:a17:902:74cb:b0:143:6fe9:ca4 with SMTP id f11-20020a17090274cb00b001436fe90ca4mr38689418plt.2.1637002996446; Mon, 15 Nov 2021 11:03:16 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:16 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 12/20] tcp: add RETPOLINE mitigation to sk_backlog_rcv Date: Mon, 15 Nov 2021 11:02:41 -0800 Message-Id: <20211115190249.3936899-13-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Use INDIRECT_CALL_INET() to avoid an indirect call when/if CONFIG_RETPOLINE=y Signed-off-by: Eric Dumazet --- include/net/sock.h | 8 +++++++- net/core/sock.c | 5 ++++- net/ipv6/tcp_ipv6.c | 5 +++-- 3 files changed, 14 insertions(+), 4 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index cb97c448472aa5af3055916df844cbe422578190..2d40fe4c7718ee702bf7e5a847ceff6f8f2f5b7d 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1018,12 +1018,18 @@ static inline __must_check int sk_add_backlog(struct sock *sk, struct sk_buff *s int __sk_backlog_rcv(struct sock *sk, struct sk_buff *skb); +INDIRECT_CALLABLE_DECLARE(int tcp_v4_do_rcv(struct sock *sk, struct sk_buff *skb)); +INDIRECT_CALLABLE_DECLARE(int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb)); + static inline int sk_backlog_rcv(struct sock *sk, struct sk_buff *skb) { if (sk_memalloc_socks() && skb_pfmemalloc(skb)) return __sk_backlog_rcv(sk, skb); - return sk->sk_backlog_rcv(sk, skb); + return INDIRECT_CALL_INET(sk->sk_backlog_rcv, + tcp_v6_do_rcv, + tcp_v4_do_rcv, + sk, skb); } static inline void sk_incoming_cpu_update(struct sock *sk) diff --git a/net/core/sock.c b/net/core/sock.c index 99738e14224c44e5aa4b88857620fb162e9c265f..c57d9883f62c75f522b7f6bc68451aaf8429dc83 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -327,7 +327,10 @@ int __sk_backlog_rcv(struct sock *sk, struct sk_buff *skb) BUG_ON(!sock_flag(sk, SOCK_MEMALLOC)); noreclaim_flag = memalloc_noreclaim_save(); - ret = sk->sk_backlog_rcv(sk, skb); + ret = INDIRECT_CALL_INET(sk->sk_backlog_rcv, + tcp_v6_do_rcv, + tcp_v4_do_rcv, + sk, skb); memalloc_noreclaim_restore(noreclaim_flag); return ret; diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 1f1a89f096de9f77ab1bd2d871eb90a3f12e91e0..f41f14b701233dd2d0f5ad464a623a5ba9774763 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -72,7 +72,7 @@ static void tcp_v6_send_reset(const struct sock *sk, struct sk_buff *skb); static void tcp_v6_reqsk_send_ack(const struct sock *sk, struct sk_buff *skb, struct request_sock *req); -static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb); +INDIRECT_CALLABLE_SCOPE int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb); static const struct inet_connection_sock_af_ops ipv6_mapped; const struct inet_connection_sock_af_ops ipv6_specific; @@ -1466,7 +1466,8 @@ INDIRECT_CALLABLE_DECLARE(struct dst_entry *ipv4_dst_check(struct dst_entry *, * This is because we cannot sleep with the original spinlock * held. */ -static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb) +INDIRECT_CALLABLE_SCOPE +int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb) { struct ipv6_pinfo *np = tcp_inet6_sk(sk); struct sk_buff *opt_skb = NULL; From patchwork Mon Nov 15 19:02:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620331 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78794C433FE for ; Mon, 15 Nov 2021 20:09:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5C59161B95 for ; Mon, 15 Nov 2021 20:09:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349075AbhKOULt (ORCPT ); Mon, 15 Nov 2021 15:11:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345804AbhKOT3Y (ORCPT ); Mon, 15 Nov 2021 14:29:24 -0500 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34CC4C06EDC0 for ; Mon, 15 Nov 2021 11:03:18 -0800 (PST) Received: by mail-pf1-x42d.google.com with SMTP id 8so3948971pfo.4 for ; Mon, 15 Nov 2021 11:03:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FooQCbUIJ+BdByiyKpi0sSy78LSDGfQV+hc2qyor+T8=; b=SGwyGH6qeokESF/Z7IQT6gReROC3o8xjNBCiMMEG+G4oVusmw0sm88sft568c9ifcN ERFZ9atwI45h8HtrSXLa1/F5Qy+2s3SLscwDlp9pOI0ix2G1Wj6VTu4TZiwmE0E13nWJ olJKj3BLeLB/hoJYYuH7vTbjnhz97GWz6X4jKIllGPN34ufgqbdMCdOrQRGVDc99oWxp XlXpwqjM/zAI/agfTa8x60gCoQtY/SWjuduC2i3BoG1PQmM9heDpv7DUpwcy3xOiPKsb vsIo2F4CNFm6EI5+xh6OPKlH01WC0OL+zNbPC5xER+d0CBQdQ4AUsyQ9VnnvkbTD4c0J YF+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FooQCbUIJ+BdByiyKpi0sSy78LSDGfQV+hc2qyor+T8=; b=uWoOzJsUmVIzmmXO+mUnEVk/4iOytMXwtyoaTzTcgsdFqw9JMFtFSMmpqpFlsJHBZY Ghg9TbG0eALnE7j6CFBm48TTMKV5TuyL2f7UfvBbyX4aUuONDQQED+dzgai6v4y08wn1 CQaOC9m6VDvLKVcguJxwKhkp2GYbAkUIH5Ep0XweUXyaK9S7VLohDsjLla8nAXTrrcVj MswYHkjuK2ZQ0nW1wX6vLZ3/G0X+QrUkd+qVvE70HPD/ud3p+hQYNc8mOfdj9gOLcqP8 6e8/rV5PT6vTBQGYDyeU5mNQT/KRw/O8bbF7bkw1mAPyUVfT7xlfHZDkEfLoby2kGC4U sKjw== X-Gm-Message-State: AOAM530w9I4tscf2oeOKel30bdCdXYBnjZz6V1HmA9OFFXsT4g5QCRzG P7b4RX1qmHZsG9/Gst4lyuE= X-Google-Smtp-Source: ABdhPJzEJnWrdVzZl7jdiAPjDlSWwNwnMF62JKv5Y0Ng6YYG7BLqdLEJLudLNO2kd8H3mBO7Yzb6QQ== X-Received: by 2002:a05:6a00:ac6:b029:374:a33b:a74 with SMTP id c6-20020a056a000ac6b0290374a33b0a74mr35437530pfl.51.1637002997725; Mon, 15 Nov 2021 11:03:17 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:17 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 13/20] tcp: annotate data-races on tp->segs_in and tp->data_segs_in Date: Mon, 15 Nov 2021 11:02:42 -0800 Message-Id: <20211115190249.3936899-14-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet tcp_segs_in() can be called from BH, while socket spinlock is held but socket owned by user, eventually reading these fields from tcp_get_info() Found by code inspection, no need to backport this patch to older kernels. Signed-off-by: Eric Dumazet --- include/net/tcp.h | 8 ++++++-- net/ipv4/tcp.c | 6 ++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 4da22b41bde688dec4a3741f510346dae0cf32e0..05c81677aaf782f23b8c63d6ed133df802b43064 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -2172,9 +2172,13 @@ static inline void tcp_segs_in(struct tcp_sock *tp, const struct sk_buff *skb) u16 segs_in; segs_in = max_t(u16, 1, skb_shinfo(skb)->gso_segs); - tp->segs_in += segs_in; + + /* We update these fields while other threads might + * read them from tcp_get_info() + */ + WRITE_ONCE(tp->segs_in, tp->segs_in + segs_in); if (skb->len > tcp_hdrlen(skb)) - tp->data_segs_in += segs_in; + WRITE_ONCE(tp->data_segs_in, tp->data_segs_in + segs_in); } /* diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 24d77a32c9cbcdf0e4380ec6d9aa3e42d2cf8730..267b2b18f048c4df4cabd819433a99bf8b3f2678 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -3769,10 +3769,12 @@ void tcp_get_info(struct sock *sk, struct tcp_info *info) tcp_get_info_chrono_stats(tp, info); info->tcpi_segs_out = tp->segs_out; - info->tcpi_segs_in = tp->segs_in; + + /* segs_in and data_segs_in can be updated from tcp_segs_in() from BH */ + info->tcpi_segs_in = READ_ONCE(tp->segs_in); + info->tcpi_data_segs_in = READ_ONCE(tp->data_segs_in); info->tcpi_min_rtt = tcp_min_rtt(tp); - info->tcpi_data_segs_in = tp->data_segs_in; info->tcpi_data_segs_out = tp->data_segs_out; info->tcpi_delivery_rate_app_limited = tp->rate_app_limited ? 1 : 0; From patchwork Mon Nov 15 19:02:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620909 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0AA6C4332F for ; Tue, 16 Nov 2021 00:28:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D7562619E3 for ; Tue, 16 Nov 2021 00:28:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232263AbhKPAbF (ORCPT ); Mon, 15 Nov 2021 19:31:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345806AbhKOT3Y (ORCPT ); Mon, 15 Nov 2021 14:29:24 -0500 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BF88C0BC9B4 for ; Mon, 15 Nov 2021 11:03:19 -0800 (PST) Received: by mail-pj1-x102c.google.com with SMTP id cq22-20020a17090af99600b001a9550a17a5so60199pjb.2 for ; Mon, 15 Nov 2021 11:03:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4ARyKQTpLRBIJL2Jucr9pSu+Ghn03sNWkhfxMBkGUTY=; b=cO+d2CSnB7UxweIOMHjoGZ+VzVrkL66UGF0iay4WKs6IWv5o0A9SUYaBCCwqf9bvVS 2m0WxW3u7xXpoC/8liQVzb8a4RDIxqwslhoLdrdcjEIwmva+oxPwDEMDZvVsRUG0qdZ0 PKk4OKNhc2SNZr4UucEQpsHdxbFKaKKS5uHKwHcXoHelmBjo/Tl8bSiQ5kI7LlehlXTW D8/lomua5NNADnxXWyArudVIF78f5YY2z/9Ve2khBXVdQzeIw5BA8hPMbGHRxUqcitdM gFz90olrbrRLTRryh3Ytu1t672U+N/mXBHt7QRqO2NRXAieHFdy1QrxuyWJ4LI2vvTPQ PwXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4ARyKQTpLRBIJL2Jucr9pSu+Ghn03sNWkhfxMBkGUTY=; b=dF5gf+Am2woxjnu5afCP8ea4uLjjEMSLx6Z5cAq6Abn5r3RMcrQN0BEX7WtJdtGGVD WptTH9jeJnjft5IZ8bkYkJzE8oPiKZM3kcbfzXSpGOhGO6GWZHileY0P4u5hKfy/gcP+ 8fj1dEW41zeYf57Tco9qnzT26MDuDS90N1IZqlKRqWugf/QcX/LBz2DIgrfV5e7cU5ND f37R7e2JWT4XNluPb2S/p0+d20j4v9zK2YhHOOVU/Sx8Xp2w7M3oGJUSyCUVi9OzgynU YsixiJRszROpCdTE00mf3M7KH53ULZxFeqLhMXY1k7CXj1RYs8olmKMwQ0Y8EDa37VGe IEmw== X-Gm-Message-State: AOAM533BUh5Q5SmKRcf0SmrFUU1t1rf0OZhhwhxMA1pqd2smJUlr0TuR S2QcHcGuB8b5jK4O2kJrfo0= X-Google-Smtp-Source: ABdhPJz7KsjZXLBh/vPVSj0gbGAfEZoR8LHBnek3RbbZs7dAjpiFFRveqvxoTBNzIJU03fYq2xBiUw== X-Received: by 2002:a17:90b:17cc:: with SMTP id me12mr952723pjb.179.1637002999035; Mon, 15 Nov 2021 11:03:19 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:18 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 14/20] tcp: annotate races around tp->urg_data Date: Mon, 15 Nov 2021 11:02:43 -0800 Message-Id: <20211115190249.3936899-15-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet tcp_poll() and tcp_ioctl() are reading tp->urg_data without socket lock owned. Also, it is faster to first check tp->urg_data in tcp_poll(), then tp->urg_seq == tp->copied_seq, because tp->urg_seq is located in a different/cold cache line. Signed-off-by: Eric Dumazet --- net/ipv4/tcp.c | 17 +++++++++-------- net/ipv4/tcp_input.c | 4 ++-- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 267b2b18f048c4df4cabd819433a99bf8b3f2678..313cf648c349a24ab7a04729180ec9b76b2f6aa2 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -545,10 +545,11 @@ __poll_t tcp_poll(struct file *file, struct socket *sock, poll_table *wait) if (state != TCP_SYN_SENT && (state != TCP_SYN_RECV || rcu_access_pointer(tp->fastopen_rsk))) { int target = sock_rcvlowat(sk, 0, INT_MAX); + u16 urg_data = READ_ONCE(tp->urg_data); - if (READ_ONCE(tp->urg_seq) == READ_ONCE(tp->copied_seq) && - !sock_flag(sk, SOCK_URGINLINE) && - tp->urg_data) + if (urg_data && + READ_ONCE(tp->urg_seq) == READ_ONCE(tp->copied_seq) && + !sock_flag(sk, SOCK_URGINLINE)) target++; if (tcp_stream_is_readable(sk, target)) @@ -573,7 +574,7 @@ __poll_t tcp_poll(struct file *file, struct socket *sock, poll_table *wait) } else mask |= EPOLLOUT | EPOLLWRNORM; - if (tp->urg_data & TCP_URG_VALID) + if (urg_data & TCP_URG_VALID) mask |= EPOLLPRI; } else if (state == TCP_SYN_SENT && inet_sk(sk)->defer_connect) { /* Active TCP fastopen socket with defer_connect @@ -607,7 +608,7 @@ int tcp_ioctl(struct sock *sk, int cmd, unsigned long arg) unlock_sock_fast(sk, slow); break; case SIOCATMARK: - answ = tp->urg_data && + answ = READ_ONCE(tp->urg_data) && READ_ONCE(tp->urg_seq) == READ_ONCE(tp->copied_seq); break; case SIOCOUTQ: @@ -1465,7 +1466,7 @@ static int tcp_recv_urg(struct sock *sk, struct msghdr *msg, int len, int flags) char c = tp->urg_data; if (!(flags & MSG_PEEK)) - tp->urg_data = TCP_URG_READ; + WRITE_ONCE(tp->urg_data, TCP_URG_READ); /* Read urgent data. */ msg->msg_flags |= MSG_OOB; @@ -2465,7 +2466,7 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, skip_copy: if (tp->urg_data && after(tp->copied_seq, tp->urg_seq)) { - tp->urg_data = 0; + WRITE_ONCE(tp->urg_data, 0); tcp_fast_path_check(sk); } @@ -2959,7 +2960,7 @@ int tcp_disconnect(struct sock *sk, int flags) tcp_clear_xmit_timers(sk); __skb_queue_purge(&sk->sk_receive_queue); WRITE_ONCE(tp->copied_seq, tp->rcv_nxt); - tp->urg_data = 0; + WRITE_ONCE(tp->urg_data, 0); tcp_write_queue_purge(sk); tcp_fastopen_active_disable_ofo_check(sk); skb_rbtree_purge(&tp->out_of_order_queue); diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 246ab7b5e857eb9e802c4805075e89c98cf00636..5ee07a337652696bdebb1117334ff39d88fd0276 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -5591,7 +5591,7 @@ static void tcp_check_urg(struct sock *sk, const struct tcphdr *th) } } - tp->urg_data = TCP_URG_NOTYET; + WRITE_ONCE(tp->urg_data, TCP_URG_NOTYET); WRITE_ONCE(tp->urg_seq, ptr); /* Disable header prediction. */ @@ -5617,7 +5617,7 @@ static void tcp_urg(struct sock *sk, struct sk_buff *skb, const struct tcphdr *t u8 tmp; if (skb_copy_bits(skb, ptr, &tmp, 1)) BUG(); - tp->urg_data = TCP_URG_VALID | tmp; + WRITE_ONCE(tp->urg_data, TCP_URG_VALID | tmp); if (!sock_flag(sk, SOCK_DEAD)) sk->sk_data_ready(sk); } From patchwork Mon Nov 15 19:02:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620335 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE7B1C433EF for ; Mon, 15 Nov 2021 20:09:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A390061BE5 for ; Mon, 15 Nov 2021 20:09:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348888AbhKOUMH (ORCPT ); Mon, 15 Nov 2021 15:12:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345807AbhKOT3Y (ORCPT ); Mon, 15 Nov 2021 14:29:24 -0500 Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F22AFC06EDC1 for ; Mon, 15 Nov 2021 11:03:20 -0800 (PST) Received: by mail-pl1-x62e.google.com with SMTP id y7so15380592plp.0 for ; Mon, 15 Nov 2021 11:03:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cE2uznwFrliA3ejhifpoOM8+JmuLRcxyYcIVXX2zOq8=; b=hUJ8O3orunQ3G0Dvlo7UYIp0/XV4rL0LosDFUvn3E5utWaIxCkAEy6mC/0Dcjq86JK xj/O13Go7TFS2saga+uIKt7vCpH9MYrt+O5vcr4G3AHOKifnf5y1uM2xx0aZOGxxenED TDg+ML3DY0slvy+qs5rDVqZJoQpsBYUWuPGhUIzLZ7Oj63qQkKKtuq64nNog+x0B3u8+ UBruytinXo+Xf15XStrAiyTAYD7HHKvpeFu0PZyOHGpIQkF2mP1hGeTZqQgCOsTrByBq 7qKYe9LbF/XRHAcc1+MnEREkDpyC4DouAFOZyXEk6wq9IqhPMEBzKFiqr4WuaeiehmOx Zn1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cE2uznwFrliA3ejhifpoOM8+JmuLRcxyYcIVXX2zOq8=; b=5V++DEXUxWYDTOGrtAyKYgl0fJKLoU+Nx9eISTRNubu8KP859I+q3QMy5/afppHX+e fTdXHWL5gggqnKOsp5pXUdfWJKXZvkghZpYLAstda7asSu0hdZwGrlKz6u85wqodvjpY gHcl03cjlQTNs3bT4imSS0MgkwSPkPgsR2nT4dZiPo4xZAqB1MWq31NEvRi8SXZOPAs2 XEJxT43Qm6cYWuS+xJVbUYJPzXHUWsUZbNPMf/lrPbU9iONmV45244NHTjlwAmQ0yQQ2 /lQRpwWyTokMqnNYi9WkOta9lutwqycqSgWf+2fK0PjPwUXEZ9HQY7AD/1LV6cy/dEDK UH2g== X-Gm-Message-State: AOAM5327auYnwHly2NndxwL69PF8DUzgcPo4q5R7w+dhidYdLagkH8zq mtDggZcdcDn9WW1nXiHEHMY= X-Google-Smtp-Source: ABdhPJxQg/DxPh000MK9J8csDabxIHZx/SSuBdi2BAZizn2Wx7u8STlYPctXBAuQfO7w8ok4LH0wMQ== X-Received: by 2002:a17:903:285:b0:142:7a83:6dd2 with SMTP id j5-20020a170903028500b001427a836dd2mr37381870plr.59.1637003000547; Mon, 15 Nov 2021 11:03:20 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:20 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 15/20] tcp: tp->urg_data is unlikely to be set Date: Mon, 15 Nov 2021 11:02:44 -0800 Message-Id: <20211115190249.3936899-16-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Use some unlikely() hints in the fast path. Signed-off-by: Eric Dumazet --- net/ipv4/tcp.c | 10 +++++----- net/ipv4/tcp_input.c | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 313cf648c349a24ab7a04729180ec9b76b2f6aa2..9175e0d729f5e65b5fa39acadc5bf9de715854ad 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -547,7 +547,7 @@ __poll_t tcp_poll(struct file *file, struct socket *sock, poll_table *wait) int target = sock_rcvlowat(sk, 0, INT_MAX); u16 urg_data = READ_ONCE(tp->urg_data); - if (urg_data && + if (unlikely(urg_data) && READ_ONCE(tp->urg_seq) == READ_ONCE(tp->copied_seq) && !sock_flag(sk, SOCK_URGINLINE)) target++; @@ -1633,7 +1633,7 @@ int tcp_read_sock(struct sock *sk, read_descriptor_t *desc, len = skb->len - offset; /* Stop reading if we hit a patch of urgent data */ - if (tp->urg_data) { + if (unlikely(tp->urg_data)) { u32 urg_offset = tp->urg_seq - seq; if (urg_offset < len) len = urg_offset; @@ -2326,7 +2326,7 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, u32 offset; /* Are we at urgent data? Stop if we have read anything or have SIGURG pending. */ - if (tp->urg_data && tp->urg_seq == *seq) { + if (unlikely(tp->urg_data) && tp->urg_seq == *seq) { if (copied) break; if (signal_pending(current)) { @@ -2431,7 +2431,7 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, used = len; /* Do we have urgent data here? */ - if (tp->urg_data) { + if (unlikely(tp->urg_data)) { u32 urg_offset = tp->urg_seq - *seq; if (urg_offset < used) { if (!urg_offset) { @@ -2465,7 +2465,7 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, tcp_rcv_space_adjust(sk); skip_copy: - if (tp->urg_data && after(tp->copied_seq, tp->urg_seq)) { + if (unlikely(tp->urg_data) && after(tp->copied_seq, tp->urg_seq)) { WRITE_ONCE(tp->urg_data, 0); tcp_fast_path_check(sk); } diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 5ee07a337652696bdebb1117334ff39d88fd0276..3658b9c3dd2b6cd4610603c78509c9af25ddcdbc 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -5604,11 +5604,11 @@ static void tcp_urg(struct sock *sk, struct sk_buff *skb, const struct tcphdr *t struct tcp_sock *tp = tcp_sk(sk); /* Check if we get a new urgent pointer - normally not. */ - if (th->urg) + if (unlikely(th->urg)) tcp_check_urg(sk, th); /* Do we wait for any urgent data? - normally not... */ - if (tp->urg_data == TCP_URG_NOTYET) { + if (unlikely(tp->urg_data == TCP_URG_NOTYET)) { u32 ptr = tp->urg_seq - ntohl(th->seq) + (th->doff * 4) - th->syn; From patchwork Mon Nov 15 19:02:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620901 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DA65C433F5 for ; Tue, 16 Nov 2021 00:28:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 047A6619E3 for ; Tue, 16 Nov 2021 00:28:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244391AbhKPAbB (ORCPT ); Mon, 15 Nov 2021 19:31:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345808AbhKOT3Y (ORCPT ); Mon, 15 Nov 2021 14:29:24 -0500 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 64582C06EDC2 for ; Mon, 15 Nov 2021 11:03:22 -0800 (PST) Received: by mail-pl1-x62f.google.com with SMTP id y8so9800521plg.1 for ; Mon, 15 Nov 2021 11:03:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cq6IvpouhA6CdLDUxp8Df/cpSEh3QpQotH8ssQAlWko=; b=SiQ/Ib+/tAoXV85NlA8oBj1/CuO6GsY4p2OxqIYb9v0XLVQm7TnIRuoFDgUtG62sQD sRrFkm5GRPAFU0w6pMyTEunSzzhlg/M/ZflWrpJX3sPNqCE/gDvBOI+Z1mGLdAtL6mqK vRzmjU9SAP9z8vq8MpdhsPaFXYtUIFqGwMoS2ysTOST2k8dcoa5HM41esNG18V2s1hOa UpyMV4SswUoSqx/qHO6l8s8Lrr2hEb/5hdf/kxuJuJd0gX0pT+eDNN+67NoCM1e/mMUM aMrf4A/CLLbN/JasT6JUXTQBdc/zBDOgjgF+zGSxpRlGwWM5OiLB7D9/Eqfa87IJ/zWK RfXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cq6IvpouhA6CdLDUxp8Df/cpSEh3QpQotH8ssQAlWko=; b=RB1OF7yqWFbZGesZoKp/TTGAs0gpsUkOBP3PC4TgFrHA15MT4Tx5qlbYiJEZV3pi0R rmAF8/s2u8GsDOTIhEBnUa3xoRbilJhH06J8quyhZpYugbnnYYVUTE+OU7xsOQV8/XrP SrYD6GfSXwQQntuZig4rWXgqXeby6A8yOMtDKDXlvVnxCZr4ZRJjGb35LTI0dTxPFwQB 2xjKN2RCe2QQmRcPbr+TcJzYpSWaSnBpxMnFfzrlp0lPH7tEAM/FDRNDYjdITrIAp9VB MhSkJ5506EfiYapuZ3T7v8TAEd5eZFRFHTK1ANPKhypCo/2FlG1GLmdIp7xrq/4cScfL YCiw== X-Gm-Message-State: AOAM533zCruLEmTRJVB051psm6KQ6SRmngE5zOZ8YeN1qcQnb68nquqZ 2nIPbyVQhK6OV8dN2Loz7sa0vUoR90A= X-Google-Smtp-Source: ABdhPJzHp+y+9zi+MVrnDP5IwM6N+FwNdUsitHOX7Kp38v2qji0desp94WI2mmq7nTwWwG/8TTOM6A== X-Received: by 2002:a17:902:c115:b0:142:2441:aa23 with SMTP id 21-20020a170902c11500b001422441aa23mr37072325pli.16.1637003001768; Mon, 15 Nov 2021 11:03:21 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:21 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 16/20] tcp: avoid indirect calls to sock_rfree Date: Mon, 15 Nov 2021 11:02:45 -0800 Message-Id: <20211115190249.3936899-17-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet TCP uses sk_eat_skb() when skbs can be removed from receive queue. However, the call so skb_orphan() from __kfree_skb() incurs an indirect call so sock_rfee(), which is more expensive than a direct call, especially for CONFIG_RETPOLINE=y. Add tcp_eat_recv_skb() function to make the call before __kfree_skb(). Signed-off-by: Eric Dumazet --- net/ipv4/tcp.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 9175e0d729f5e65b5fa39acadc5bf9de715854ad..4e7011672aa9a04370b7a03b972fe19cd48ea232 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1580,6 +1580,16 @@ void tcp_cleanup_rbuf(struct sock *sk, int copied) tcp_send_ack(sk); } +static void tcp_eat_recv_skb(struct sock *sk, struct sk_buff *skb) +{ + if (likely(skb->destructor == sock_rfree)) { + sock_rfree(skb); + skb->destructor = NULL; + skb->sk = NULL; + } + sk_eat_skb(sk, skb); +} + static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) { struct sk_buff *skb; @@ -1599,7 +1609,7 @@ static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) * splitted a fat GRO packet, while we released socket lock * in skb_splice_bits() */ - sk_eat_skb(sk, skb); + tcp_eat_recv_skb(sk, skb); } return NULL; } @@ -1665,11 +1675,11 @@ int tcp_read_sock(struct sock *sk, read_descriptor_t *desc, continue; } if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN) { - sk_eat_skb(sk, skb); + tcp_eat_recv_skb(sk, skb); ++seq; break; } - sk_eat_skb(sk, skb); + tcp_eat_recv_skb(sk, skb); if (!desc->count) break; WRITE_ONCE(tp->copied_seq, seq); @@ -2481,14 +2491,14 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN) goto found_fin_ok; if (!(flags & MSG_PEEK)) - sk_eat_skb(sk, skb); + tcp_eat_recv_skb(sk, skb); continue; found_fin_ok: /* Process the FIN. */ WRITE_ONCE(*seq, *seq + 1); if (!(flags & MSG_PEEK)) - sk_eat_skb(sk, skb); + tcp_eat_recv_skb(sk, skb); break; } while (len > 0); From patchwork Mon Nov 15 19:02:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620337 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AEFD5C433F5 for ; Mon, 15 Nov 2021 20:10:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 929C361BA9 for ; Mon, 15 Nov 2021 20:10:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242810AbhKOUMM (ORCPT ); Mon, 15 Nov 2021 15:12:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345820AbhKOT3Y (ORCPT ); Mon, 15 Nov 2021 14:29:24 -0500 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97DDFC06EDC4 for ; Mon, 15 Nov 2021 11:03:23 -0800 (PST) Received: by mail-pj1-x1032.google.com with SMTP id cq22-20020a17090af99600b001a9550a17a5so60330pjb.2 for ; Mon, 15 Nov 2021 11:03:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ro2ByOc87T3RNhfRZN4emKA5tVoBW1OfE+eXAwv54hM=; b=jhsX8d2KdJCLOaklsXSRf1alOfRDSu0E3HbgT+GzaQLP23C2z4By/bi7vBrljH5jrd fke2XgF/VdP3RiwbTSI05NXG5RdmnnnLVmHY4AOkFTGIKmLQR64PvvfgMVypm5tK6HIf VS9CKexJjcU0RWGaInz7WIJHBqyCx4gpl24ylEpc0YRpJd6JPOl4A1yt4C92hFP8YfNw LcYFfs+IVN2VL8CgJW/P9S/+QjoFkJHkMHJoIpJSIAF9t6P/7W5RNlVohag3X1c0QMKT Q8eVdR3gQ0IHuG4Qxtx7L+MlxAq86mL3GrI27X2PltKswOfFY+TvZGEZlGBKjIWhEl7t fBTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ro2ByOc87T3RNhfRZN4emKA5tVoBW1OfE+eXAwv54hM=; b=OaW0snda1WbXXEcO6ZueCijgtYMuZMA28IXpieHQHWgnguxam08t/RspBQRA5uiGMK 5o417EXfzE/S2mJZuD8YJdsJ9ZUqXZs8hU2P1bi/dfdG3Tdxtg3M4Jg4Ow+HEmqI68FN /HhepfxT61rdwnKxnfio9mZ1DGUaRJpj2LC55y1LlY0lpTbLVcTPZA6uGJXEr2TUkdAa 7lx3YURPEQuolYy1rgkPe82KfTuE6SwAZcVVZgmhng+YmkWqy4FTk4ZfSA7QwT1uSsxm z1Bah9yW39qq9Hl0ADi1FVAFMETHbls8yqPl4N3qSB1eCJOAn6j29ocU1hxGWs9A/sry +uXw== X-Gm-Message-State: AOAM531PUWS4klkCoGotDvNNcNYSP5gkRIRKTY7517Br6TBdouTlj8pJ Fqw6x8VUTF9XzKjxrXZwirk= X-Google-Smtp-Source: ABdhPJxEYO18HJFegFhlfjITIaVwNoT0hdlUDej/gt6Yl+2iamsHtpfk/IjKfS882ezDxJi7C7bdeQ== X-Received: by 2002:a17:90b:1185:: with SMTP id gk5mr969221pjb.113.1637003003065; Mon, 15 Nov 2021 11:03:23 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:22 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 17/20] tcp: defer skb freeing after socket lock is released Date: Mon, 15 Nov 2021 11:02:46 -0800 Message-Id: <20211115190249.3936899-18-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet tcp recvmsg() (or rx zerocopy) spends a fair amount of time freeing skbs after their payload has been consumed. A typical ~64KB GRO packet has to release ~45 page references, eventually going to page allocator for each of them. Currently, this freeing is performed while socket lock is held, meaning that there is a high chance that BH handler has to queue incoming packets to tcp socket backlog. This can cause additional latencies, because the user thread has to process the backlog at release_sock() time, and while doing so, additional frames can be added by BH handler. This patch adds logic to defer these frees after socket lock is released, or directly from BH handler if possible. Being able to free these skbs from BH handler helps a lot, because this avoids the usual alloc/free assymetry, when BH handler and user thread do not run on same cpu or NUMA node. One cpu can now be fully utilized for the kernel->user copy, and another cpu is handling BH processing and skb/page allocs/frees (assuming RFS is not forcing use of a single CPU) Tested: 100Gbit NIC Max throughput for one TCP_STREAM flow, over 10 runs MTU : 1500 Before: 55 Gbit After: 66 Gbit MTU : 4096+(headers) Before: 82 Gbit After: 95 Gbit Signed-off-by: Eric Dumazet --- include/linux/skbuff.h | 2 ++ include/net/sock.h | 3 +++ include/net/tcp.h | 10 ++++++++++ net/ipv4/tcp.c | 27 +++++++++++++++++++++++++-- net/ipv4/tcp_ipv4.c | 1 + net/ipv6/tcp_ipv6.c | 1 + 6 files changed, 42 insertions(+), 2 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 686a666d073d5106526f3c5c20d64f26131be72d..b8b806512e1615fad2bc9935baba3fff14996012 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -36,6 +36,7 @@ #include #include #include +#include #include #include #if IS_ENABLED(CONFIG_NF_CONNTRACK) @@ -743,6 +744,7 @@ struct sk_buff { }; struct rb_node rbnode; /* used in netem, ip4 defrag, and tcp stack */ struct list_head list; + struct llist_node ll_node; }; union { diff --git a/include/net/sock.h b/include/net/sock.h index 2d40fe4c7718ee702bf7e5a847ceff6f8f2f5b7d..2578d1f455a7af0d7f4ce5d3b4ac25ee41fdaeb4 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -63,6 +63,7 @@ #include #include #include +#include #include #include #include @@ -408,6 +409,8 @@ struct sock { struct sk_buff *head; struct sk_buff *tail; } sk_backlog; + struct llist_head defer_list; + #define sk_rmem_alloc sk_backlog.rmem_alloc int sk_forward_alloc; diff --git a/include/net/tcp.h b/include/net/tcp.h index 05c81677aaf782f23b8c63d6ed133df802b43064..44e442bf23f9ccc0a1a914345c3faf1fc9f99d5f 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -1368,6 +1368,16 @@ static inline bool tcp_checksum_complete(struct sk_buff *skb) } bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb); + +void __sk_defer_free_flush(struct sock *sk); + +static inline void sk_defer_free_flush(struct sock *sk) +{ + if (llist_empty(&sk->defer_list)) + return; + __sk_defer_free_flush(sk); +} + int tcp_filter(struct sock *sk, struct sk_buff *skb); void tcp_set_state(struct sock *sk, int state); void tcp_done(struct sock *sk); diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 4e7011672aa9a04370b7a03b972fe19cd48ea232..33cd9a1c199cef9822ec0ddb3aec91c1111754c7 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1580,14 +1580,34 @@ void tcp_cleanup_rbuf(struct sock *sk, int copied) tcp_send_ack(sk); } +void __sk_defer_free_flush(struct sock *sk) +{ + struct llist_node *head; + struct sk_buff *skb, *n; + + head = llist_del_all(&sk->defer_list); + llist_for_each_entry_safe(skb, n, head, ll_node) { + prefetch(n); + skb_mark_not_on_list(skb); + __kfree_skb(skb); + } +} +EXPORT_SYMBOL(__sk_defer_free_flush); + static void tcp_eat_recv_skb(struct sock *sk, struct sk_buff *skb) { + __skb_unlink(skb, &sk->sk_receive_queue); if (likely(skb->destructor == sock_rfree)) { sock_rfree(skb); skb->destructor = NULL; skb->sk = NULL; + if (!skb_queue_empty(&sk->sk_receive_queue) || + !llist_empty(&sk->defer_list)) { + llist_add(&skb->ll_node, &sk->defer_list); + return; + } } - sk_eat_skb(sk, skb); + __kfree_skb(skb); } static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off) @@ -2422,6 +2442,7 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, /* Do not sleep, just process backlog. */ __sk_flush_backlog(sk); } else { + sk_defer_free_flush(sk); sk_wait_data(sk, &timeo, last); } @@ -2540,6 +2561,7 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock, ret = tcp_recvmsg_locked(sk, msg, len, nonblock, flags, &tss, &cmsg_flags); release_sock(sk); + sk_defer_free_flush(sk); if (cmsg_flags && ret >= 0) { if (cmsg_flags & TCP_CMSG_TS) @@ -3065,7 +3087,7 @@ int tcp_disconnect(struct sock *sk, int flags) sk->sk_frag.page = NULL; sk->sk_frag.offset = 0; } - + sk_defer_free_flush(sk); sk_error_report(sk); return 0; } @@ -4194,6 +4216,7 @@ static int do_tcp_getsockopt(struct sock *sk, int level, err = BPF_CGROUP_RUN_PROG_GETSOCKOPT_KERN(sk, level, optname, &zc, &len, err); release_sock(sk); + sk_defer_free_flush(sk); if (len >= offsetofend(struct tcp_zerocopy_receive, msg_flags)) goto zerocopy_rcv_cmsg; switch (len) { diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 5ad81bfb27b2f8d9a3cfe11141160b48092cfa3a..3dd19a2bf06c483b43d7e60080c624f10bb2f63d 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -2102,6 +2102,7 @@ int tcp_v4_rcv(struct sk_buff *skb) sk_incoming_cpu_update(sk); + sk_defer_free_flush(sk); bh_lock_sock_nested(sk); tcp_segs_in(tcp_sk(sk), skb); ret = 0; diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index f41f14b701233dd2d0f5ad464a623a5ba9774763..3b7d6ede13649d2589f5a456c5a132409486880f 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -1758,6 +1758,7 @@ INDIRECT_CALLABLE_SCOPE int tcp_v6_rcv(struct sk_buff *skb) sk_incoming_cpu_update(sk); + sk_defer_free_flush(sk); bh_lock_sock_nested(sk); tcp_segs_in(tcp_sk(sk), skb); ret = 0; From patchwork Mon Nov 15 19:02:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620903 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B579AC433EF for ; Tue, 16 Nov 2021 00:28:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A36C5619F6 for ; Tue, 16 Nov 2021 00:28:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349808AbhKPAbD (ORCPT ); Mon, 15 Nov 2021 19:31:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345815AbhKOT3Y (ORCPT ); Mon, 15 Nov 2021 14:29:24 -0500 Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2F3AC06EDC5 for ; Mon, 15 Nov 2021 11:03:24 -0800 (PST) Received: by mail-pj1-x1033.google.com with SMTP id n15-20020a17090a160f00b001a75089daa3so64709pja.1 for ; Mon, 15 Nov 2021 11:03:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=prto76o5NaUG7Oy5ZXSTMOw0etNhHLG6g7CRl2GWtBU=; b=Z0ZceQzqvbbgYPPV8/5fI6Vk/9g6ViMIMLfMK68iUT2mmGkZKa/QYlMXmCF3mKPJDA LbQQmv1dG88LtuVSEO36048wFnQ1OeM9piMm59oFHh9pv+4fLQudPqdrVgHBkM4iU4Ht KEEpIqpXOMKITAdx0s1A1yILKgOtIklHSf4XZDTB2ruyE0LsZEwyn3JKKYVdCTz5C/my NPT9nyOyKi5Rgy8qR6wkURKeCxc7i5YpxFjGBtZkuzHGmVXAQfnuVZLnAhmokQ/TL5AV SLgzNmIv4ahQIg4Mkf0eeuJwLaw5oEJ8zoMHPLf4J0Y9B7ruRpAAXFXfnu8sCIJSUPgW nsQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=prto76o5NaUG7Oy5ZXSTMOw0etNhHLG6g7CRl2GWtBU=; b=7kKjNJYDrLwg7TiWRs+ECaXfqJkxschL+Qr2wRi1/ZfI0n4L8YPfTuYaoLLkHLcST1 BiwJPYo0nVWlIzBs5pvdn21Oa8wZiU8BncZnfIr2Ua4N18MlPilBqnSLi1cbrA+/tkGY icXpAoLUfrMauBSh+7prj0vkLk2iaKFiAMadbTN/1JCT61wOo70ybnuWhJqwidrG6cD/ kDusUVXIzdkGmW0NT2383GzQLbKL9ovJdx3UeCvfDQ1Pf7PZ/Lry234yYK+RQTnywmP7 NrlwyppD/LaFR9i5KHYT6ZKwpd3bK+WCX5SEvVp0rv/HPTzqKnhmXtx9XoMdKatql/qQ rnWg== X-Gm-Message-State: AOAM532Fk3zrYnbV4oHlscxBAVDFRcEcoLJVlEXsNSVJAM8w7yMETAm3 99W1+0xIy2qhO4ddf+kvySk= X-Google-Smtp-Source: ABdhPJx9EAjjvuWPNVJX0gD0uhnObu+2FH8XsbZrauUdwt+HJMTboyC23yUzHGNcjAJNzDp2lq+MhA== X-Received: by 2002:a17:90a:a083:: with SMTP id r3mr66481366pjp.55.1637003004353; Mon, 15 Nov 2021 11:03:24 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:24 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 18/20] tcp: check local var (timeo) before socket fields in one test Date: Mon, 15 Nov 2021 11:02:47 -0800 Message-Id: <20211115190249.3936899-19-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Testing timeo before sk_err/sk_state/sk_shutdown makes more sense. Modern applications use non-blocking IO, while a socket is terminated only once during its life time. Signed-off-by: Eric Dumazet --- net/ipv4/tcp.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 33cd9a1c199cef9822ec0ddb3aec91c1111754c7..7b1886103556e1295d84378d5bcb0f0346651de0 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -2399,10 +2399,10 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, break; if (copied) { - if (sk->sk_err || + if (!timeo || + sk->sk_err || sk->sk_state == TCP_CLOSE || (sk->sk_shutdown & RCV_SHUTDOWN) || - !timeo || signal_pending(current)) break; } else { From patchwork Mon Nov 15 19:02:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620341 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BFF0C4167D for ; Mon, 15 Nov 2021 20:10:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1C57661BD2 for ; Mon, 15 Nov 2021 20:10:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349309AbhKOUMn (ORCPT ); Mon, 15 Nov 2021 15:12:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38270 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345811AbhKOT3Y (ORCPT ); Mon, 15 Nov 2021 14:29:24 -0500 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 17782C0BC9B8 for ; Mon, 15 Nov 2021 11:03:26 -0800 (PST) Received: by mail-pj1-x102a.google.com with SMTP id cq22-20020a17090af99600b001a9550a17a5so60422pjb.2 for ; Mon, 15 Nov 2021 11:03:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AnNAUZppMhRZpSB2dP7oz0QMLOnCYASQg7WlLrqEU4E=; b=Np2WN3gujQUZufK82ijGoF2kREERe/STPXWL4kYZsMEZb0bnMGiWaf6BJfGOr6T83/ H43oK0P6b+/Ui4/DHq2ThGrVNj4+vzDL58tdLnYf9ZUF3LaUt9+iCr1gErVBPzy/++lC nZl0/86jW65jGSe334YRFIH0u5DjTjkATvwEW3AVfB5VziYAgwVNfrpYjgkHoI8bSpjG zT3kSsigGKfGku8Sw0NuHiZ1tAlLvV5TaDFc0ZWRzhDrOO0wqi8F51XnAIwD1VAQPo9E q2R5ACGaBZul6lMURXJ/6S3tPw7ZZlikbUHJE4EivoU+CrY5zobCcdj0hYXL2Ia+cKtU ClKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AnNAUZppMhRZpSB2dP7oz0QMLOnCYASQg7WlLrqEU4E=; b=jIlpo10aI6GtDbyjCp94AYiVQ0CY/XtBO+wfSYozbdWqO+JLSeh1vlhY6N1/W2iHNs /GjA+iaYWjxtdBMpobiiVL7njM+2dLPv0E/lfTr+FgIhWhVh8KzytUqpJ0HuZHvgUr8m KtpUFEMTeQ8J3v73h6jt1cYtk1GwfLxbr5RgyDkS536rjNCzpOTN9ifL9XC0zHh8eOwJ FW8Im5MANHEKHNQhP45A1S/eLt/usXz8T0nnDDyfj0xcHBda4CCGKuWF7+huezzpPpJj M8CUOGVueSJk2xlTCA8uj2RrrnQ5Y3lXh30MWMUZvnDKm/c4+TnWod/0NF3wpqxW8PoM qFvw== X-Gm-Message-State: AOAM532H/+AyEueI6Q4x5gxXOxMhMiv/RVFfGXGN6N23TkTplXQXHsg6 r64/nW+aF/wjYD4UjuOq/m8= X-Google-Smtp-Source: ABdhPJz4t5Z7MRu++nqT/IDyZRYswBRYFGwOP4QqvN9kG+Au30QnJOZtLxQ5z4lNgF5pVhVNuNDebQ== X-Received: by 2002:a17:90b:3a89:: with SMTP id om9mr977855pjb.99.1637003005588; Mon, 15 Nov 2021 11:03:25 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:25 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 19/20] tcp: do not call tcp_cleanup_rbuf() if we have a backlog Date: Mon, 15 Nov 2021 11:02:48 -0800 Message-Id: <20211115190249.3936899-20-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet Under pressure, tcp recvmsg() has logic to process the socket backlog, but calls tcp_cleanup_rbuf() right before. Avoiding sending ACK right before processing new segments makes a lot of sense, as this decrease the number of ACK packets, with no impact on effective ACK clocking. Signed-off-by: Eric Dumazet --- net/ipv4/tcp.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 7b1886103556e1295d84378d5bcb0f0346651de0..d1949fdb1462b97b87b99bc32ea132c574e9b647 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -2436,12 +2436,11 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, } } - tcp_cleanup_rbuf(sk, copied); - if (copied >= target) { /* Do not sleep, just process backlog. */ __sk_flush_backlog(sk); } else { + tcp_cleanup_rbuf(sk, copied); sk_defer_free_flush(sk); sk_wait_data(sk, &timeo, last); } From patchwork Mon Nov 15 19:02:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 12620339 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0784AC43219 for ; Mon, 15 Nov 2021 20:10:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E27546321A for ; Mon, 15 Nov 2021 20:10:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239728AbhKOUM0 (ORCPT ); Mon, 15 Nov 2021 15:12:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345821AbhKOT3Y (ORCPT ); Mon, 15 Nov 2021 14:29:24 -0500 Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E61DC0BC9B9 for ; Mon, 15 Nov 2021 11:03:27 -0800 (PST) Received: by mail-pl1-x635.google.com with SMTP id y7so15380768plp.0 for ; Mon, 15 Nov 2021 11:03:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ouXZBTAruen0qmI6aoDktc7jK1BqHvnZk0CNJ3/THFM=; b=ke4UK86FAUj+32OE8ArBE1LfWuwf9mHJqspuMEr3o1uLtJDkJ/Ek7T3WMLneRTiF8g d4SXbYJtspyLln2B4bdObZrISQ5rLmZ1F1OY6sj54UEif7sUZsqytCLEQetog798N8s2 50k+D87b0UoNr6gsPA/Rk0y222NDx8ZDw8ohdZ2PngKrHeEFQRnwEuB8EipBtQi5NnNq JYvDcit+VFn5a3+tVJkTAfFQe7oZHp6WhURn7pGUSsD+l3txTbPtBmB+vm5lgja5D7tN +Ck72CC5IkgxqgbqBwCaxpS8i2k546oSG3MzUVbYlifqEKy014SD6D2MHmNFy/kRUngV aHcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ouXZBTAruen0qmI6aoDktc7jK1BqHvnZk0CNJ3/THFM=; b=Tnv2r3QQYFpBdblCBUHStEruSH2J3/EG7ivejPm9czAqwKsOQCN1APVsZNTSoaQlBU 4ebbqg1agf/74cq9CFHwUp/1UDXDQqS6gIYI9LFR5pju5avi8d1mGXTaclBzpiAF/F1p HoyKY+o/5ejsRm4nNgVBLtZOd2LkaJpdHTqw1V7rW8dmVy7dJ28eQQFf/PvqE5RTWP+Y FXvMoiacxl8JPzCKmD8d7xWhmMHkyfrnoWJEhDOJkDdSGwT5Ue40anZHzJKN5FgX8ql7 +1/nEwm5pSBS6dgcjZ2/x5ZdEi+ByzSyg2v9n3m0B0H3dZrDbSoSKOUewNXAq+YV5cEN oyaw== X-Gm-Message-State: AOAM533N5cD/y8nJsIrWYAufgH5PPuYDRq8LhefoH5bv+hqgHWSXGWfv SK9LHRTxl78xTq9nDXVabo0= X-Google-Smtp-Source: ABdhPJxgZeGsPg2+1q/8AoQ9/uuzYFSpzczh4KkamemXqDK1ey8gfU55pZP0X8lkFWXjRz0Wp348sA== X-Received: by 2002:a17:90b:3908:: with SMTP id ob8mr989755pjb.57.1637003006747; Mon, 15 Nov 2021 11:03:26 -0800 (PST) Received: from edumazet1.svl.corp.google.com ([2620:15c:2c4:201:4994:f3d6:2eb1:61cb]) by smtp.gmail.com with ESMTPSA id f21sm11850834pfe.69.2021.11.15.11.03.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Nov 2021 11:03:26 -0800 (PST) From: Eric Dumazet To: "David S . Miller" , Jakub Kicinski Cc: netdev , Eric Dumazet , Eric Dumazet , Soheil Hassas Yeganeh , Neal Cardwell , Arjun Roy Subject: [PATCH net-next 20/20] net: move early demux fields close to sk_refcnt Date: Mon, 15 Nov 2021 11:02:49 -0800 Message-Id: <20211115190249.3936899-21-eric.dumazet@gmail.com> X-Mailer: git-send-email 2.34.0.rc1.387.gb447b232ab-goog In-Reply-To: <20211115190249.3936899-1-eric.dumazet@gmail.com> References: <20211115190249.3936899-1-eric.dumazet@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org From: Eric Dumazet sk_rx_dst/sk_rx_dst_ifindex/sk_rx_dst_cookie are read in early demux, and currently spans two cache lines. Moving them close to sk_refcnt makes more sense, as only one cache line is needed. New layout for this hot cache line is : struct sock { struct sock_common __sk_common; /* 0 0x88 */ /* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */ struct dst_entry * sk_rx_dst; /* 0x88 0x8 */ int sk_rx_dst_ifindex; /* 0x90 0x4 */ u32 sk_rx_dst_cookie; /* 0x94 0x4 */ socket_lock_t sk_lock; /* 0x98 0x20 */ atomic_t sk_drops; /* 0xb8 0x4 */ int sk_rcvlowat; /* 0xbc 0x4 */ /* --- cacheline 3 boundary (192 bytes) --- */ Signed-off-by: Eric Dumazet --- include/net/sock.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 2578d1f455a7af0d7f4ce5d3b4ac25ee41fdaeb4..95cc03bd3fac0f3f5ea49bfd540a1f0eda4ebf59 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -390,6 +390,11 @@ struct sock { #define sk_flags __sk_common.skc_flags #define sk_rxhash __sk_common.skc_rxhash + /* early demux fields */ + struct dst_entry *sk_rx_dst; + int sk_rx_dst_ifindex; + u32 sk_rx_dst_cookie; + socket_lock_t sk_lock; atomic_t sk_drops; int sk_rcvlowat; @@ -432,9 +437,6 @@ struct sock { #ifdef CONFIG_XFRM struct xfrm_policy __rcu *sk_policy[2]; #endif - struct dst_entry *sk_rx_dst; - int sk_rx_dst_ifindex; - u32 sk_rx_dst_cookie; struct dst_entry __rcu *sk_dst_cache; atomic_t sk_omem_alloc;