From patchwork Tue Jan 11 01:21:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709376 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD802C433EF for ; Tue, 11 Jan 2022 01:24:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346216AbiAKBYs (ORCPT ); Mon, 10 Jan 2022 20:24:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346209AbiAKBYq (ORCPT ); Mon, 10 Jan 2022 20:24:46 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B915C06173F; Mon, 10 Jan 2022 17:24:46 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id d18-20020a05600c251200b0034974323cfaso490517wma.4; Mon, 10 Jan 2022 17:24:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tq/iukreOG79JdYNe8HVV6T7x3H7rgMjUtsugvlEjlE=; b=pH6t7FlwLfwvt95KSgS+H8thrkFdfgDt4OTXvQAzPe6vleFfa/aF5R6Ief3Yqq5DN6 xb73XSqyFVZhfXNoZjud0iQ/IP9AOr04imo8r9GpXKDWUlgL7PQVxU/O4XbaQwCMx0q7 AZ53bEGEq7xZncNe6xK5DFpK1NJ2t8HKM/Jl22GAzx5w1CmS9bqwXMfEVqrmei+CuSnN s3A9FzwWpbn6iLQBE5HV4F/lzOA1ZXH2p9mZh6xkI45a0i2Z90YlEzoko8yXnbsv/9Ek a1G/w3Ee3AHX8kJ3k+pAVGlcTgLA/bF5u+bITNp1h0upctMvPyyy2rc8eWvhorkQm2Be 1FnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tq/iukreOG79JdYNe8HVV6T7x3H7rgMjUtsugvlEjlE=; b=j8IAwEz6MucwbEtnNRLuYnt6Q7Pj5rD3j3QBJbOodlYOQkM7/3Ry3I0dnlo7haLMaV 8PX4VQ2m7sDxSVDF7TDrjuPzoJ+6IENITCOdYMhJm8Q0De3EhHgAyqEFSTJRqyUblJxv qWDBFaCmKOo5yczRTiZbrxxYERfLX3kreU5EzTxu4K9eunpSKshMlwz/zFcv5QRY+kD2 3K14CsdmubH1iharDyjKMmNpN3F5mJqM3ng8vKZTu2q+BCy7UNJHtNxA4hPdfORBdLVZ VwtXY+mN77sWpR3QwOjtvR2zOdHNp6MYt/qDGsBqXjrL9mf5Nfm6YpUSY0XqOUqCrZ7i 5i9g== X-Gm-Message-State: AOAM532FfyF+mtSh2ZfTCPbtKebXSTi/CtyCoBsQVN/Ojo+IISG2wEQK F1pFpm2NM/ZaM8bBvo5Et9z9CpsaBNc= X-Google-Smtp-Source: ABdhPJwBq/cSc1x977yGly5mi7ozjF+H29JNvFK1yuBmYkHAPR/OP//iPlqYtFeqdFU5zOMvlbraZQ== X-Received: by 2002:a7b:c449:: with SMTP id l9mr336355wmi.160.1641864284825; Mon, 10 Jan 2022 17:24:44 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:44 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 01/14] ipv6: optimise dst referencing Date: Tue, 11 Jan 2022 01:21:33 +0000 Message-Id: <6ed8db52c96c8775fd311715d2b6d5e20d5dd9a5.1641863490.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org __ip6_make_skb() initialises skb's dst by taking an additional reference to cork->dst. However, cork->dst comes into the function holding a ref, which will be put shortly at the end of the function in ip6_cork_release(). Avoid this extra pair of get/put atomics by stealing cork->dst and NULL'ing the field, ip6_cork_release() already handles zero values. Signed-off-by: Pavel Begunkov --- net/ipv6/ip6_output.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 2995f8d89e7e..14d607ccfeea 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1807,6 +1807,15 @@ int ip6_append_data(struct sock *sk, } EXPORT_SYMBOL_GPL(ip6_append_data); +static void ip6_cork_steal_dst(struct sk_buff *skb, struct inet_cork_full *cork) +{ + struct dst_entry *dst = cork->base.dst; + + cork->base.dst = NULL; + cork->base.flags &= ~IPCORK_ALLFRAG; + skb_dst_set(skb, dst); +} + static void ip6_cork_release(struct inet_cork_full *cork, struct inet6_cork *v6_cork) { @@ -1889,7 +1898,7 @@ struct sk_buff *__ip6_make_skb(struct sock *sk, skb->tstamp = cork->base.transmit_time; - skb_dst_set(skb, dst_clone(&rt->dst)); + ip6_cork_steal_dst(skb, cork); IP6_UPD_PO_STATS(net, rt->rt6i_idev, IPSTATS_MIB_OUT, skb->len); if (proto == IPPROTO_ICMPV6) { struct inet6_dev *idev = ip6_dst_idev(skb_dst(skb)); From patchwork Tue Jan 11 01:21:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709378 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BABCC433EF for ; Tue, 11 Jan 2022 01:24:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346239AbiAKBYw (ORCPT ); Mon, 10 Jan 2022 20:24:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243319AbiAKBYs (ORCPT ); Mon, 10 Jan 2022 20:24:48 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58B74C06173F; Mon, 10 Jan 2022 17:24:47 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id t28so23330793wrb.4; Mon, 10 Jan 2022 17:24:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iXKHyNdsJkHtXxLMMhcsr/bT4YBYrcmkq9r0J3HN/VY=; b=Wyh0Bhtr8fyJg3IvYngkUcib5GcXPaO7i0Fdhbrfvs8FB3UFQHPnnlAPnKZy0HVP1y Cz0JTp6B3jNUN6Fh0SK2AkEDPch0+RFyo0XTEhtSQYBrofbcUHnyNnU9ZYPIRA5U9oxf Gro0YDhve01rAG1ZGHDwiiTHmDRbNXMEtAmpF6cazMEujL/SOiVGYZ7KRCW0sSJGgqvG LRC2ttoA4CdCbnhbg5/yYXZk9yyTTQwSbaMjb2fQP68H/mJJLMW9qn/vn3ySwzBZ/Hp+ Fb/TqjkgYXMR6OeMh7AMZAUqGS6nufCSLQ0cMCYDuZ7CGnlyz2lYNKGA4VYTqzK37x47 6kRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iXKHyNdsJkHtXxLMMhcsr/bT4YBYrcmkq9r0J3HN/VY=; b=WBqhrgjv8FtFNgL0QjlWzjXKa962ZbY6mmaUrONPX7bsUC0dL84w+jYvLp4hPEgc9i ahcfPfb+IHrYVBKVgV61ryyx7h/U2Y0h0xOp60eIspFNefWSvQWn68yCKGXxGf4uMVNm pA3RC+nlzwAkeEC/Q9NaBoP5kBO8YQJXIirtIPvA8QniUn3aNi3AyFKAuwA1nV4vLYcs QcuqP0ojOz8o1wQZKwQaJZCqzIzkwNo8YA6PYGyiIEMNUgNgGvzlb3iVFIKKyHeIDaUJ NN49XYIRl56DTTLMgHPQJVSQOO5XHnF/H50vL0U1ABYKH52fWDqoamgzkDaDTUDKrUhD 0gsw== X-Gm-Message-State: AOAM532NqBePeC/5pyz1v3OZGFQCnbKCIbNFu53l7yagGPMHY5wBojHR Vz8rt3Ae6XeB7qoqqxWWe3OhmRpw5YM= X-Google-Smtp-Source: ABdhPJwZRmBEgpMyBiAes8CLq7Oip1EJwcKI8iUuMXoSjdS9TNGhjbnef0VNV/GagU/spQXhEfyCYw== X-Received: by 2002:adf:e444:: with SMTP id t4mr1720893wrm.325.1641864285785; Mon, 10 Jan 2022 17:24:45 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:45 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 02/14] ipv6: shuffle up->pending AF_INET bits Date: Tue, 11 Jan 2022 01:21:34 +0000 Message-Id: <1f9024b0d64b9008cca1c6383c69c8e5854817da.1641863490.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Corked AF_INET for ipv6 socket doesn't appear to be the hottest case, so move it out of the common path under up->pending check to remove overhead. Signed-off-by: Pavel Begunkov --- net/ipv6/udp.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index df216268cb02..0c10ee0124b5 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -1363,9 +1363,6 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) } } - if (up->pending == AF_INET) - return udp_sendmsg(sk, msg, len); - /* Rough check on arithmetic overflow, better check is made in ip6_append_data(). */ @@ -1374,6 +1371,8 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) getfrag = is_udplite ? udplite_getfrag : ip_generic_getfrag; if (up->pending) { + if (up->pending == AF_INET) + return udp_sendmsg(sk, msg, len); /* * There are pending frames. * The socket lock must be held while it's corked. From patchwork Tue Jan 11 01:21:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709377 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC0E1C433EF for ; Tue, 11 Jan 2022 01:24:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346233AbiAKBYu (ORCPT ); Mon, 10 Jan 2022 20:24:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45012 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346218AbiAKBYs (ORCPT ); Mon, 10 Jan 2022 20:24:48 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39B3DC061748; Mon, 10 Jan 2022 17:24:48 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id t28so23330835wrb.4; Mon, 10 Jan 2022 17:24:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BT3XL2h14/pezcVsoOCWrtTWDzvTfgkCTHxovQFgrLg=; b=L7xwYZl32JRKpFu828tuty+GdXEMSomqvAPirfx/5RUwnJZr3a0LxXZLcTQkjvezE2 bMph9bVQmDgGWrC61bSUiQ92TE4geXhnPOmxdyOf9TutT3qdY+4SDbPRiJIUznhUFoSx 5VqmdLW33xwmnW1TfCBHYib8KY4OODYkevCLz54enC0+wg+rZuUsZHOItDMdBW3yvKc5 NecVZ2uzAljU6AylkKgPwMm2q5RlUUk0HcJUnDyHT9A0Mihx66BonVscJxeqt8onXQ8K zAY1XvB6RSbZeIwQzkeApYrJhI78j0wHiwKhWbclk3e+BFS/IJV45UQFU0mOvvs/WgGy qEzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BT3XL2h14/pezcVsoOCWrtTWDzvTfgkCTHxovQFgrLg=; b=DjetEe6sfapXFTZyF81xkWyAULm0bvD3QS9VKeedNkppe6/c/laUY6vhxn5XHB4wPj B5IhNgwFdaEuwIkiSdeb99HkVOBkFY5Xsd7Z7UrWVk84H8gnt29VO0iCDtoGegSTlumm RA3bLIxa/YnRL2Zyj8PxTiIFJSFG1+0qgMEUFEcgCoYLr9PICj57Ho15oarZL6Q+y6l9 0+pfLRkhSoE4N5MS4aSYPQeYsTKFDp8ziQGVpC6EjjgE2rbjoVhYaoPG6MTmpP7rONNh c7hm/R/Bjd8Z62AI47VmvB0PNyHdUwOMIQV2pLgkYpohTcMvJfwmt/NHPfKn8ZlAQmAF uNJw== X-Gm-Message-State: AOAM533OPyAPZ166sT071qOf2JPmzw/zx0S9t1bjCNoEhQqXWIYa9NvE 9kckx5z3OI9G9EobDmGeFSxJODWmgHw= X-Google-Smtp-Source: ABdhPJwTTNhB9HsNsiWQYR6v+29ko06w+rxlZvGfmmOzusUfJDh+IBgSas2xohKnmVVRxIYf2BKA0w== X-Received: by 2002:a5d:4588:: with SMTP id p8mr1709396wrq.649.1641864286635; Mon, 10 Jan 2022 17:24:46 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:46 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 03/14] ipv6: remove daddr temp buffer in __ip6_make_skb Date: Tue, 11 Jan 2022 01:21:35 +0000 Message-Id: <46d9179f641bd1349a68c6445378fad719aa0170.1641863490.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org __ip6_make_skb() doesn't actually need to keep an on-stack copy of fl6->daddr because even though ipv6_push_nfrag_opts() may return a different daddr it doesn't change the one that was passed in. Just set final_dst to fl6->daddr and get rid of the temp copy. Signed-off-by: Pavel Begunkov --- net/ipv6/ip6_output.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 14d607ccfeea..4acd577d5ec5 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1843,7 +1843,7 @@ struct sk_buff *__ip6_make_skb(struct sock *sk, { struct sk_buff *skb, *tmp_skb; struct sk_buff **tail_skb; - struct in6_addr final_dst_buf, *final_dst = &final_dst_buf; + struct in6_addr *final_dst; struct ipv6_pinfo *np = inet6_sk(sk); struct net *net = sock_net(sk); struct ipv6hdr *hdr; @@ -1873,9 +1873,9 @@ struct sk_buff *__ip6_make_skb(struct sock *sk, /* Allow local fragmentation. */ skb->ignore_df = ip6_sk_ignore_df(sk); - - *final_dst = fl6->daddr; __skb_pull(skb, skb_network_header_len(skb)); + + final_dst = &fl6->daddr; if (opt && opt->opt_flen) ipv6_push_frag_opts(skb, opt, &proto); if (opt && opt->opt_nflen) @@ -1895,7 +1895,6 @@ struct sk_buff *__ip6_make_skb(struct sock *sk, skb->priority = sk->sk_priority; skb->mark = cork->base.mark; - skb->tstamp = cork->base.transmit_time; ip6_cork_steal_dst(skb, cork); From patchwork Tue Jan 11 01:21:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709379 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 061DBC433EF for ; Tue, 11 Jan 2022 01:25:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346275AbiAKBY6 (ORCPT ); Mon, 10 Jan 2022 20:24:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346223AbiAKBYt (ORCPT ); Mon, 10 Jan 2022 20:24:49 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08A5AC06173F; Mon, 10 Jan 2022 17:24:49 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id p18so3201350wmg.4; Mon, 10 Jan 2022 17:24:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MVu4ZlnCBCVeXmBG2E3bbhj24V2QUrRwSLSKs/b5k4Y=; b=RY2UaPlIu2ZLXtF5wq4iDrUcRhU/hq0s/Ut2Ah76TSHS7hzBq4dyrNXIsz5P7d6xtj UIwrFaB68X9fJ9fNacw4DLof5TE5IVNjBf0R6Bur1sACvUdQUp6Y2KK/zqQH8pEoUn21 PQIVV+V2EQY+54AY//7QgXsOKGI9dbx7PU78BSYa7iexrsgcsNZmWGjmWYg/OJXsJ0EX qUB4E1uefG61gGjCKktu+hdbpC4RYYGO2wEE6qVEZnvsTol12eRODAZNtbFmmhY/ohpV +Uf6el5KQOjmmKl5ydaDrBjIs2BcRCw2jtcrY1YKneYzc3JkqXcp27i5rHgoIH6EKM7+ 6lEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MVu4ZlnCBCVeXmBG2E3bbhj24V2QUrRwSLSKs/b5k4Y=; b=HPl34G+IBNXjXbezFw3oTU78ocAlFvCB19iVVytugCQlBtlANkbOApVvo7p9o2DtGe z9b9hae+eqvBCPvfvsHwj5xuQGLSXvsC9NRi4Iu3hfXny0tLtMYKklkLHtclh1fmPnfw 8sH+52wcR+VoM7HlqiH8L1veC1m2iek/iV1OcOKwOE1UbSVbuF5ZOuvxbUf9vzLoJHhj ntFREoRPVM0i5Rsd5JjcG2rVoufg74KBlAPC88p4rDRl58FvaB14dW3/iB/ZRg+9mTek scUwrzoBTRhM8puxp+IZ+xs5ag3lO8KktCKUZq7t77pXg3MVUwC2YTe4pbOWVGVeEzsj 8g4A== X-Gm-Message-State: AOAM530ZbLWMWMQpG4yZo07I27iDafIh0J4frWGS36BmQKf+4ySslpSM GP43uc5X1+etBGgjgzYC+0DAK3hTl4A= X-Google-Smtp-Source: ABdhPJyhpv5Nw5z9W/M1+7t21+Cq0fH4GbgYhfe5w0dQXmV8XptfQErVxY/+FFHSJM1izV1V57lNxQ== X-Received: by 2002:a05:600c:a45:: with SMTP id c5mr318937wmq.127.1641864287529; Mon, 10 Jan 2022 17:24:47 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:47 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 04/14] ipv6: clean up cork setup/release Date: Tue, 11 Jan 2022 01:21:36 +0000 Message-Id: <1e6f9ddca160b1dc3d81ac32f10ddb302cb216a9.1641863490.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org A simple cleanup of ip6_setup_cork() and ip6_cork_release() adding a local variable for v6_cork->opt instead of retyping it many times. It serves as a preparation patch to make further work cleaner. Signed-off-by: Pavel Begunkov --- net/ipv6/ip6_output.c | 44 +++++++++++++++++++++---------------------- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 4acd577d5ec5..88349e49717a 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1354,7 +1354,7 @@ static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork, { struct ipv6_pinfo *np = inet6_sk(sk); unsigned int mtu; - struct ipv6_txoptions *opt = ipc6->opt; + struct ipv6_txoptions *nopt, *opt = ipc6->opt; /* * setup for corking @@ -1363,32 +1363,28 @@ static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork, if (WARN_ON(v6_cork->opt)) return -EINVAL; - v6_cork->opt = kzalloc(sizeof(*opt), sk->sk_allocation); - if (unlikely(!v6_cork->opt)) + nopt = v6_cork->opt = kzalloc(sizeof(*opt), sk->sk_allocation); + if (unlikely(!nopt)) return -ENOBUFS; - v6_cork->opt->tot_len = sizeof(*opt); - v6_cork->opt->opt_flen = opt->opt_flen; - v6_cork->opt->opt_nflen = opt->opt_nflen; + nopt->tot_len = sizeof(*opt); + nopt->opt_flen = opt->opt_flen; + nopt->opt_nflen = opt->opt_nflen; - v6_cork->opt->dst0opt = ip6_opt_dup(opt->dst0opt, - sk->sk_allocation); - if (opt->dst0opt && !v6_cork->opt->dst0opt) + nopt->dst0opt = ip6_opt_dup(opt->dst0opt, sk->sk_allocation); + if (opt->dst0opt && !nopt->dst0opt) return -ENOBUFS; - v6_cork->opt->dst1opt = ip6_opt_dup(opt->dst1opt, - sk->sk_allocation); - if (opt->dst1opt && !v6_cork->opt->dst1opt) + nopt->dst1opt = ip6_opt_dup(opt->dst1opt, sk->sk_allocation); + if (opt->dst1opt && !nopt->dst1opt) return -ENOBUFS; - v6_cork->opt->hopopt = ip6_opt_dup(opt->hopopt, - sk->sk_allocation); - if (opt->hopopt && !v6_cork->opt->hopopt) + nopt->hopopt = ip6_opt_dup(opt->hopopt, sk->sk_allocation); + if (opt->hopopt && !nopt->hopopt) return -ENOBUFS; - v6_cork->opt->srcrt = ip6_rthdr_dup(opt->srcrt, - sk->sk_allocation); - if (opt->srcrt && !v6_cork->opt->srcrt) + nopt->srcrt = ip6_rthdr_dup(opt->srcrt, sk->sk_allocation); + if (opt->srcrt && !nopt->srcrt) return -ENOBUFS; /* need source address above miyazawa*/ @@ -1820,11 +1816,13 @@ static void ip6_cork_release(struct inet_cork_full *cork, struct inet6_cork *v6_cork) { if (v6_cork->opt) { - kfree(v6_cork->opt->dst0opt); - kfree(v6_cork->opt->dst1opt); - kfree(v6_cork->opt->hopopt); - kfree(v6_cork->opt->srcrt); - kfree(v6_cork->opt); + struct ipv6_txoptions *opt = v6_cork->opt; + + kfree(opt->dst0opt); + kfree(opt->dst1opt); + kfree(opt->hopopt); + kfree(opt->srcrt); + kfree(opt); v6_cork->opt = NULL; } From patchwork Tue Jan 11 01:21:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709381 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F727C4332F for ; Tue, 11 Jan 2022 01:25:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346301AbiAKBZA (ORCPT ); Mon, 10 Jan 2022 20:25:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346230AbiAKBYu (ORCPT ); Mon, 10 Jan 2022 20:24:50 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9A8FC061756; Mon, 10 Jan 2022 17:24:49 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id v6so30028273wra.8; Mon, 10 Jan 2022 17:24:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CAcyrl4yFF10VZJC35lqjrTek3vjD8DPs+k+UlDzF8o=; b=dNsgW2pqYkHcV4tliHqD2/1yZ07xzfHAd/nGN3CSqtAuHUyJn3l3Fbf677mtw5/8oF WRQHiiN0z9nbd4sSicnOIicTPiEf0wgK3BUakBlhHkk/QP8lC2BvZu5GGxtPgqJkhfEf jyxxVqFRuhoHo2exaVnag4uPxN1c2/ye1saCpIS81seidXE5wXHBAZZfaoh7jJLb5J+g T4dJXAmYL3NmVy50zedX3Y0aww1gziqAy4Mhgqe2CViDz2jatvCAYYq/0x5Ty4ErI8BT zOSfaIgxPxkztwGYHity2ue/vX3iGszzwzujWRZZFpASltyEv2lW14PrAI+Tj2pCzXQ9 XULg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CAcyrl4yFF10VZJC35lqjrTek3vjD8DPs+k+UlDzF8o=; b=ZeQwNfkPnioy7tOOZKdMDbiMAbjaNnSWw+Jlp7KvcDLRy0tVmUwtDR76LbbG3d0mWS pU/2TqiM4EhfQuIphvT39JJR4VCLQCf7MVyFq+R3zKxcwKr3lLYp/qlK4BfvIHtvk1hu B/uIaPAs/bogaUrem3o1A76snP+/GIkAIDpUbmj+OnHSDu4oBCSdllqa/twm8E7BKtD9 8snWcW/IpdyLek61wnvpSJTqm0nSM4JUdIsfagDFCysiSjjtmKBhxH8oMIhNLrJdhrp+ Kg/gt+qzzQjdWXq/Y7ABoaDOpiO+vAICtDAUfF3hrMuMrzBBAdIJffnWiQmNLRq+kzC2 ZkFg== X-Gm-Message-State: AOAM532g0JF+1kMsqbmYlbDUVzuByRsQ4TSR5k+beGbvpXYyveiCZqh6 +yEDmJxWkbAgg7PJ+Nc5+Y16VdtYncc= X-Google-Smtp-Source: ABdhPJwIPAOximG+U7jQn8f5BT/isT4/o3WdWwtKIl6LRoLnhjlGtEM0oyGSTrXIJiF98kmck1ltfA== X-Received: by 2002:a05:6000:18c6:: with SMTP id w6mr1732499wrq.449.1641864288448; Mon, 10 Jan 2022 17:24:48 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:48 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 05/14] ipv6: don't zero cork's flowi after use Date: Tue, 11 Jan 2022 01:21:37 +0000 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org It doesn't appear there is any reason to zero cork->fl after use, i.e. in ip6_cork_release(), especially when cork struct is on-stack. Not only the memset accounts to 0.3-0.5% of total cycles (perf profiling), but also prevents other optimisations implemented in further patches. Also, now we can remove a relatively expensive flow copy in udp_v6_push_pending_frames(). Signed-off-by: Pavel Begunkov --- net/ipv6/ip6_output.c | 1 - net/ipv6/udp.c | 10 ++-------- 2 files changed, 2 insertions(+), 9 deletions(-) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 88349e49717a..b8fdda9ac797 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1831,7 +1831,6 @@ static void ip6_cork_release(struct inet_cork_full *cork, cork->base.dst = NULL; cork->base.flags &= ~IPCORK_ALLFRAG; } - memset(&cork->fl, 0, sizeof(cork->fl)); } struct sk_buff *__ip6_make_skb(struct sock *sk, diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 0c10ee0124b5..9a91b51d8e3f 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -1266,23 +1266,17 @@ static int udp_v6_push_pending_frames(struct sock *sk) { struct sk_buff *skb; struct udp_sock *up = udp_sk(sk); - struct flowi6 fl6; int err = 0; if (up->pending == AF_INET) return udp_push_pending_frames(sk); - /* ip6_finish_skb will release the cork, so make a copy of - * fl6 here. - */ - fl6 = inet_sk(sk)->cork.fl.u.ip6; - skb = ip6_finish_skb(sk); if (!skb) goto out; - err = udp_v6_send_skb(skb, &fl6, &inet_sk(sk)->cork.base); - + err = udp_v6_send_skb(skb, &inet_sk(sk)->cork.fl.u.ip6, + &inet_sk(sk)->cork.base); out: up->len = 0; up->pending = 0; From patchwork Tue Jan 11 01:21:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709382 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96DAAC433FE for ; Tue, 11 Jan 2022 01:25:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346323AbiAKBZC (ORCPT ); Mon, 10 Jan 2022 20:25:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346264AbiAKBY6 (ORCPT ); Mon, 10 Jan 2022 20:24:58 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C152DC061759; Mon, 10 Jan 2022 17:24:50 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id h10so19951237wrb.1; Mon, 10 Jan 2022 17:24:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WDqDCJ5NL4LuLdFrIrWhPcUdc5NRFY/X6Wj3QPHSD+0=; b=jUYWI+ZeIL9Eiqqj4RXjPaskEPvUgcrcURe2yNVDLyZRX1LnViGUVMjDsSC77TINMQ IrqQoFOg2TN1nHYrY8EM7Y8ti03U3rg74bgeP8kp0GzT4NNahvLzxdx0KnyHCCaBBNyW ChlkqU9CWK/qWwlkYKFwkK+D/8QbOeLyw7OVWkJAeQp1jdwKN5o4U42VEtfEWHLknuWz PPiLtK2X2WkicUrITJLx3K6p6T5VP6zRrlPyXC71pc5RzWn2+qnu6J1rLOkjvGr7X2lw IS6MYmr5b7dafDZfj3Dwrzs3ReaxpFAV6iMDb4sxipPa/FwWpd4MNyt3ZQyUvFP8w37/ U4Zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WDqDCJ5NL4LuLdFrIrWhPcUdc5NRFY/X6Wj3QPHSD+0=; b=QuiyUTlXqxrspaoQRp85+F0dq7L5bTPBtKgqrtjvaXQcP9NRb6iuTn8c0NeLCGzEhl ODWFF9V2s+T98bZw3E9/yVAW5B7KzQXxIuv8IWrh0fwPFJqXflq2kXyZxb2E3Rc1QFcm mQhfoyRjWRMa+qhxzs2iendHRQRYhL1LD+L80rPaCxQ1D7EBcd/vQw6q8bWAQxlTobBL xnZIJEY5BtQvAYxskGItEvKj6ENF/cfMLnCWhJZcqEyIt7AOSFE3o9uaA7rLec/L38kc Ypbw7d2AlYn4gD8BAogRT39FSJ5LVzbQWy7Gx17ybH8/+7cTNFfvWI2KoREFZnYJrYsM VCeg== X-Gm-Message-State: AOAM5321sOSdhPvlIi5yPwLeL5BaiXSComUTcPBiVC2KZBcwBA9pgjFw dWe1KSTQkIAvK470pvitgYuTlrlfXd0= X-Google-Smtp-Source: ABdhPJycpZT0wJ2aEQqt0aii/EvocEROihH3EMOvsKk1uk9kd6TX8js7d3DSqsahDk+Q3bcxdwCoYg== X-Received: by 2002:a5d:4c84:: with SMTP id z4mr1569710wrs.318.1641864289271; Mon, 10 Jan 2022 17:24:49 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:49 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 06/14] ipv6: pass full cork into __ip6_append_data() Date: Tue, 11 Jan 2022 01:21:38 +0000 Message-Id: <56c58d7537b65a5bcadf5d17f86e010dbba81485.1641863490.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Convert a struct inet_cork argument in __ip6_append_data() to struct inet_cork_full. As one struct contains another inet_cork is still can be accessed via ->base field. It's a preparation patch making further changes a bit cleaner. Signed-off-by: Pavel Begunkov --- net/ipv6/ip6_output.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index b8fdda9ac797..62da09819750 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1424,7 +1424,7 @@ static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork, static int __ip6_append_data(struct sock *sk, struct flowi6 *fl6, struct sk_buff_head *queue, - struct inet_cork *cork, + struct inet_cork_full *cork_full, struct inet6_cork *v6_cork, struct page_frag *pfrag, int getfrag(void *from, char *to, int offset, @@ -1433,6 +1433,7 @@ static int __ip6_append_data(struct sock *sk, unsigned int flags, struct ipcm6_cookie *ipc6) { struct sk_buff *skb, *skb_prev = NULL; + struct inet_cork *cork = &cork_full->base; unsigned int maxfraglen, fragheaderlen, mtu, orig_mtu, pmtu; struct ubuf_info *uarg = NULL; int exthdrlen = 0; @@ -1797,7 +1798,7 @@ int ip6_append_data(struct sock *sk, transhdrlen = 0; } - return __ip6_append_data(sk, fl6, &sk->sk_write_queue, &inet->cork.base, + return __ip6_append_data(sk, fl6, &sk->sk_write_queue, &inet->cork, &np->cork, sk_page_frag(sk), getfrag, from, length, transhdrlen, flags, ipc6); } @@ -1993,7 +1994,7 @@ struct sk_buff *ip6_make_skb(struct sock *sk, if (ipc6->dontfrag < 0) ipc6->dontfrag = inet6_sk(sk)->dontfrag; - err = __ip6_append_data(sk, fl6, &queue, &cork->base, &v6_cork, + err = __ip6_append_data(sk, fl6, &queue, cork, &v6_cork, ¤t->task_frag, getfrag, from, length + exthdrlen, transhdrlen + exthdrlen, flags, ipc6); From patchwork Tue Jan 11 01:21:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709383 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76002C433FE for ; Tue, 11 Jan 2022 01:25:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346328AbiAKBZD (ORCPT ); Mon, 10 Jan 2022 20:25:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346265AbiAKBY6 (ORCPT ); Mon, 10 Jan 2022 20:24:58 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6634C06175A; Mon, 10 Jan 2022 17:24:51 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id w26so4104506wmi.0; Mon, 10 Jan 2022 17:24:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZMPDzSjQOrUNWqC2VSg8brQGgBuhUXpA3LowQDg7QLw=; b=dRBROPBBDXjlIatKXI1B/6UOMPn5j/DxUxs0wV1kKWQnhHArRMCPXW6LJRB3bD2/KS 4z3aWKZQ4e2KS3IqGQEqxXhjM5VG6leJ0p1arNRxy3bwT+FnzyW9gHGt7xScqIPfKCLP 4yzD1DaSu9UP1d97toJ5sBNbCC7lhSp02ch7PdR9+7koi/UA0zmyKa419XoXOVqEQhb6 wwy/vNOlCVHw/6TnvEujzSC5uzuhX28egYD+Sd1HA4tApHEwWt1wKPe6C55NjxMJG0dK VndFaLpK0GHp1gHLjPlPgRpNaAO1oY+SZsTi8zw+KyJ0K7f3vvlgbx0Kj0ANLW/15vwG PS8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZMPDzSjQOrUNWqC2VSg8brQGgBuhUXpA3LowQDg7QLw=; b=NegXgMZNikzkjf+2L7PeTO6Y/p04GJgc0t9ApzWwlvpli3h0/f+gQEUOtgkXw0yPLc ugu5xG+BA1fUWgACGhZq3Bgz7pt5rsLh6lGIp95dWkFt4Kw8nV787yWt/PjNhf3zFTGu xUNuOJkE8WED3iRweTAL11exfgRovIZtIh7KhsDmrVKY5nJ5wludvJR+ByFklD15ZSG0 1YXBErF71WDmiU8tw0FSGGqNekHZ5NJervWsDs0lbwhPzxI6AP3AKfExhviQDwO+ddb+ thGxgrmxkCaDEccpJ8a8j2yd+zLksK96bBtK9VSw0DcgJVcfC6SDMW3TX4dje0jbwelI /j6g== X-Gm-Message-State: AOAM530h9RKgC8yh4R3RdiGDuI25N4dH7NyO0t0SSAlTKFQoEoGBNMXi x92r5QMlAcFJsdvN8hOwO/WJssTS1Rc= X-Google-Smtp-Source: ABdhPJwZQMIAah5MNkGLIHGtWLlMBOSlkCIxtcWgoT3edWp8cvUpZyvRGSIrY4RDJjA8yh2Tv0RIcw== X-Received: by 2002:a1c:f20e:: with SMTP id s14mr326748wmc.186.1641864290249; Mon, 10 Jan 2022 17:24:50 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:49 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 07/14] ipv6: pass flow in ip6_make_skb together with cork Date: Tue, 11 Jan 2022 01:21:39 +0000 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Another preparation patch. inet_cork_full already contains a field for iflow, so we can avoid passing a separate struct iflow6 into __ip6_append_data() and ip6_make_skb(), and use the flow stored in inet_cork_full. Make sure callers set cork->fl right, i.e. we init it in ip6_append_data() and right before the only ip6_make_skb() call. Signed-off-by: Pavel Begunkov --- include/net/ipv6.h | 2 +- net/ipv6/ip6_output.c | 20 +++++++++----------- net/ipv6/udp.c | 4 +++- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/include/net/ipv6.h b/include/net/ipv6.h index 3afcb128e064..5e0b56d66724 100644 --- a/include/net/ipv6.h +++ b/include/net/ipv6.h @@ -1020,7 +1020,7 @@ struct sk_buff *ip6_make_skb(struct sock *sk, int getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb), void *from, int length, int transhdrlen, - struct ipcm6_cookie *ipc6, struct flowi6 *fl6, + struct ipcm6_cookie *ipc6, struct rt6_info *rt, unsigned int flags, struct inet_cork_full *cork); diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 62da09819750..0cc490f2cfbf 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1350,7 +1350,7 @@ static void ip6_append_data_mtu(unsigned int *mtu, static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork, struct inet6_cork *v6_cork, struct ipcm6_cookie *ipc6, - struct rt6_info *rt, struct flowi6 *fl6) + struct rt6_info *rt) { struct ipv6_pinfo *np = inet6_sk(sk); unsigned int mtu; @@ -1391,7 +1391,6 @@ static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork, } dst_hold(&rt->dst); cork->base.dst = &rt->dst; - cork->fl.u.ip6 = *fl6; v6_cork->hop_limit = ipc6->hlimit; v6_cork->tclass = ipc6->tclass; if (rt->dst.flags & DST_XFRM_TUNNEL) @@ -1422,7 +1421,6 @@ static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork, } static int __ip6_append_data(struct sock *sk, - struct flowi6 *fl6, struct sk_buff_head *queue, struct inet_cork_full *cork_full, struct inet6_cork *v6_cork, @@ -1434,6 +1432,7 @@ static int __ip6_append_data(struct sock *sk, { struct sk_buff *skb, *skb_prev = NULL; struct inet_cork *cork = &cork_full->base; + struct flowi6 *fl6 = &cork_full->fl.u.ip6; unsigned int maxfraglen, fragheaderlen, mtu, orig_mtu, pmtu; struct ubuf_info *uarg = NULL; int exthdrlen = 0; @@ -1786,19 +1785,19 @@ int ip6_append_data(struct sock *sk, * setup for corking */ err = ip6_setup_cork(sk, &inet->cork, &np->cork, - ipc6, rt, fl6); + ipc6, rt); if (err) return err; + inet->cork.fl.u.ip6 = *fl6; exthdrlen = (ipc6->opt ? ipc6->opt->opt_flen : 0); length += exthdrlen; transhdrlen += exthdrlen; } else { - fl6 = &inet->cork.fl.u.ip6; transhdrlen = 0; } - return __ip6_append_data(sk, fl6, &sk->sk_write_queue, &inet->cork, + return __ip6_append_data(sk, &sk->sk_write_queue, &inet->cork, &np->cork, sk_page_frag(sk), getfrag, from, length, transhdrlen, flags, ipc6); } @@ -1967,9 +1966,8 @@ struct sk_buff *ip6_make_skb(struct sock *sk, int getfrag(void *from, char *to, int offset, int len, int odd, struct sk_buff *skb), void *from, int length, int transhdrlen, - struct ipcm6_cookie *ipc6, struct flowi6 *fl6, - struct rt6_info *rt, unsigned int flags, - struct inet_cork_full *cork) + struct ipcm6_cookie *ipc6, struct rt6_info *rt, + unsigned int flags, struct inet_cork_full *cork) { struct inet6_cork v6_cork; struct sk_buff_head queue; @@ -1986,7 +1984,7 @@ struct sk_buff *ip6_make_skb(struct sock *sk, cork->base.opt = NULL; cork->base.dst = NULL; v6_cork.opt = NULL; - err = ip6_setup_cork(sk, cork, &v6_cork, ipc6, rt, fl6); + err = ip6_setup_cork(sk, cork, &v6_cork, ipc6, rt); if (err) { ip6_cork_release(cork, &v6_cork); return ERR_PTR(err); @@ -1994,7 +1992,7 @@ struct sk_buff *ip6_make_skb(struct sock *sk, if (ipc6->dontfrag < 0) ipc6->dontfrag = inet6_sk(sk)->dontfrag; - err = __ip6_append_data(sk, fl6, &queue, cork, &v6_cork, + err = __ip6_append_data(sk, &queue, cork, &v6_cork, ¤t->task_frag, getfrag, from, length + exthdrlen, transhdrlen + exthdrlen, flags, ipc6); diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 9a91b51d8e3f..2580705431ea 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -1533,9 +1533,11 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) struct inet_cork_full cork; struct sk_buff *skb; + cork.fl.u.ip6 = fl6; + skb = ip6_make_skb(sk, getfrag, msg, ulen, sizeof(struct udphdr), &ipc6, - &fl6, (struct rt6_info *)dst, + (struct rt6_info *)dst, msg->msg_flags, &cork); err = PTR_ERR(skb); if (!IS_ERR_OR_NULL(skb)) From patchwork Tue Jan 11 01:21:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709380 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44BDCC433F5 for ; Tue, 11 Jan 2022 01:25:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346254AbiAKBZB (ORCPT ); Mon, 10 Jan 2022 20:25:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346266AbiAKBY6 (ORCPT ); Mon, 10 Jan 2022 20:24:58 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CFD29C06175B; Mon, 10 Jan 2022 17:24:52 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id s1so30124289wra.6; Mon, 10 Jan 2022 17:24:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GjvOLRey0Y2EctehLACcoBC0HbQUE5JRzwrEobDyrp8=; b=ZQtr0/tL2Htz+j4DEXszz4JyjiMKj5j38gnc3PX/AK1AUKbVqeH/m5rbEKDiB4ZLZ+ zGHNjhZMUzPApIm7vjINd3YNGGiOoGj1bBnniv1JIysWlgnIH/LsUVFmNnCgllb1HCxl c3N/E2MOdSbA22/MdCm3EYcnORZXqcxiSGCMUVFpjWPflpAbpAAUK7t+qIe4sYIcxVlX pTA+5lL60iXuxGUxM5IAM4W0rw9DzIJ8DK6GYy4CqRbp8PzR7GF+MW4PET8fHcXNrk/G LtiVY66X46ti36+S8DTwO1M9msFPZtjasOUin8eqUn0mQL997SSeGkKDYyjGICwUpW2u 7tsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GjvOLRey0Y2EctehLACcoBC0HbQUE5JRzwrEobDyrp8=; b=oPE/WcbMTWumBtew8ofZafPauJDQFTI9WgAY69wE+jXV0FIDVBo9arkb01Bw0Ueq7O 8sDSmgBohOeixNUZnRWeFdMwxLENB//EVWFeheoVC2vylRzE9tqDDH8Ld6otKwxCHGO3 InnoQw0h+xFJOLsER6ieQRfR3H6QsuGUmZ0ztSF0hx+g0LTAI/rMHfzJVlO8+fFhgpgM dL/Mwd+8MMl2tdwz6JK4PqZSagaYmuJ7JoC1A5hBRnkLxpzY+t7k3xFztfTSci5/c5p2 20CyfKnoXs7rf746BBxrs94qMqKXkae3ESVLFYdmqjDMnT476TcBfnrsS5tlL0w5wdEH z77A== X-Gm-Message-State: AOAM5332eGAMnLlikdyZAfvMU2t9tpI/xfanAbQEkhipp90uhmuo3irP EDbehzd0zlH1K0YpQ97iuWpWMMPKF3o= X-Google-Smtp-Source: ABdhPJwGbRLtdO/NE1WoM31km4eQ75oFlxdFUWy0ZWjzYbuLFwKge2HxRNz3R+QYMIghHpSFK9bMsA== X-Received: by 2002:a5d:59a7:: with SMTP id p7mr1760446wrr.258.1641864291205; Mon, 10 Jan 2022 17:24:51 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:50 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 08/14] ipv6/udp: don't make extra copies of iflow Date: Tue, 11 Jan 2022 01:21:40 +0000 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org struct flowi takes 88 bytes and copying it is relatively expensive. Currenly, udpv6_sendmsg() first initialises an on-stack struct flowi6 and then copies it into cork. Instead, directly initialise a flow in an on-stack cork, i.e. cork->fl, so corkless udp can avoid making an extra copy. Note: moving inet_cork_full instance shouldn't grow stack too much, it replaces 88 bytes for iflow with 160. Signed-off-by: Pavel Begunkov --- net/ipv6/udp.c | 85 +++++++++++++++++++++++++------------------------- 1 file changed, 42 insertions(+), 43 deletions(-) diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 2580705431ea..eec83e34ae27 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -1294,7 +1294,8 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) struct ipv6_txoptions *opt = NULL; struct ipv6_txoptions *opt_to_free = NULL; struct ip6_flowlabel *flowlabel = NULL; - struct flowi6 fl6; + struct inet_cork_full cork; + struct flowi6 *fl6 = &cork.fl.u.ip6; struct dst_entry *dst; struct ipcm6_cookie ipc6; int addr_len = msg->msg_namelen; @@ -1384,19 +1385,19 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) } ulen += sizeof(struct udphdr); - memset(&fl6, 0, sizeof(fl6)); + memset(fl6, 0, sizeof(*fl6)); if (sin6) { if (sin6->sin6_port == 0) return -EINVAL; - fl6.fl6_dport = sin6->sin6_port; + fl6->fl6_dport = sin6->sin6_port; daddr = &sin6->sin6_addr; if (np->sndflow) { - fl6.flowlabel = sin6->sin6_flowinfo&IPV6_FLOWINFO_MASK; - if (fl6.flowlabel&IPV6_FLOWLABEL_MASK) { - flowlabel = fl6_sock_lookup(sk, fl6.flowlabel); + fl6->flowlabel = sin6->sin6_flowinfo&IPV6_FLOWINFO_MASK; + if (fl6->flowlabel & IPV6_FLOWLABEL_MASK) { + flowlabel = fl6_sock_lookup(sk, fl6->flowlabel); if (IS_ERR(flowlabel)) return -EINVAL; } @@ -1413,24 +1414,24 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) if (addr_len >= sizeof(struct sockaddr_in6) && sin6->sin6_scope_id && __ipv6_addr_needs_scope_id(__ipv6_addr_type(daddr))) - fl6.flowi6_oif = sin6->sin6_scope_id; + fl6->flowi6_oif = sin6->sin6_scope_id; } else { if (sk->sk_state != TCP_ESTABLISHED) return -EDESTADDRREQ; - fl6.fl6_dport = inet->inet_dport; + fl6->fl6_dport = inet->inet_dport; daddr = &sk->sk_v6_daddr; - fl6.flowlabel = np->flow_label; + fl6->flowlabel = np->flow_label; connected = true; } - if (!fl6.flowi6_oif) - fl6.flowi6_oif = sk->sk_bound_dev_if; + if (!fl6->flowi6_oif) + fl6->flowi6_oif = sk->sk_bound_dev_if; - if (!fl6.flowi6_oif) - fl6.flowi6_oif = np->sticky_pktinfo.ipi6_ifindex; + if (!fl6->flowi6_oif) + fl6->flowi6_oif = np->sticky_pktinfo.ipi6_ifindex; - fl6.flowi6_uid = sk->sk_uid; + fl6->flowi6_uid = sk->sk_uid; if (msg->msg_controllen) { opt = &opt_space; @@ -1440,14 +1441,14 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) err = udp_cmsg_send(sk, msg, &ipc6.gso_size); if (err > 0) - err = ip6_datagram_send_ctl(sock_net(sk), sk, msg, &fl6, + err = ip6_datagram_send_ctl(sock_net(sk), sk, msg, fl6, &ipc6); if (err < 0) { fl6_sock_release(flowlabel); return err; } - if ((fl6.flowlabel&IPV6_FLOWLABEL_MASK) && !flowlabel) { - flowlabel = fl6_sock_lookup(sk, fl6.flowlabel); + if ((fl6->flowlabel&IPV6_FLOWLABEL_MASK) && !flowlabel) { + flowlabel = fl6_sock_lookup(sk, fl6->flowlabel); if (IS_ERR(flowlabel)) return -EINVAL; } @@ -1464,16 +1465,17 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) opt = ipv6_fixup_options(&opt_space, opt); ipc6.opt = opt; - fl6.flowi6_proto = sk->sk_protocol; - fl6.flowi6_mark = ipc6.sockc.mark; - fl6.daddr = *daddr; - if (ipv6_addr_any(&fl6.saddr) && !ipv6_addr_any(&np->saddr)) - fl6.saddr = np->saddr; - fl6.fl6_sport = inet->inet_sport; + fl6->flowi6_proto = sk->sk_protocol; + fl6->flowi6_mark = ipc6.sockc.mark; + fl6->daddr = *daddr; + if (ipv6_addr_any(&fl6->saddr) && !ipv6_addr_any(&np->saddr)) + fl6->saddr = np->saddr; + fl6->fl6_sport = inet->inet_sport; if (cgroup_bpf_enabled(CGROUP_UDP6_SENDMSG) && !connected) { err = BPF_CGROUP_RUN_PROG_UDP6_SENDMSG_LOCK(sk, - (struct sockaddr *)sin6, &fl6.saddr); + (struct sockaddr *)sin6, + &fl6->saddr); if (err) goto out_no_dst; if (sin6) { @@ -1489,32 +1491,32 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) err = -EINVAL; goto out_no_dst; } - fl6.fl6_dport = sin6->sin6_port; - fl6.daddr = sin6->sin6_addr; + fl6->fl6_dport = sin6->sin6_port; + fl6->daddr = sin6->sin6_addr; } } - if (ipv6_addr_any(&fl6.daddr)) - fl6.daddr.s6_addr[15] = 0x1; /* :: means loopback (BSD'ism) */ + if (ipv6_addr_any(&fl6->daddr)) + fl6->daddr.s6_addr[15] = 0x1; /* :: means loopback (BSD'ism) */ - final_p = fl6_update_dst(&fl6, opt, &final); + final_p = fl6_update_dst(fl6, opt, &final); if (final_p) connected = false; - if (!fl6.flowi6_oif && ipv6_addr_is_multicast(&fl6.daddr)) { - fl6.flowi6_oif = np->mcast_oif; + if (!fl6->flowi6_oif && ipv6_addr_is_multicast(&fl6->daddr)) { + fl6->flowi6_oif = np->mcast_oif; connected = false; - } else if (!fl6.flowi6_oif) - fl6.flowi6_oif = np->ucast_oif; + } else if (!fl6->flowi6_oif) + fl6->flowi6_oif = np->ucast_oif; - security_sk_classify_flow(sk, flowi6_to_flowi_common(&fl6)); + security_sk_classify_flow(sk, flowi6_to_flowi_common(fl6)); if (ipc6.tclass < 0) ipc6.tclass = np->tclass; - fl6.flowlabel = ip6_make_flowinfo(ipc6.tclass, fl6.flowlabel); + fl6->flowlabel = ip6_make_flowinfo(ipc6.tclass, fl6->flowlabel); - dst = ip6_sk_dst_lookup_flow(sk, &fl6, final_p, connected); + dst = ip6_sk_dst_lookup_flow(sk, fl6, final_p, connected); if (IS_ERR(dst)) { err = PTR_ERR(dst); dst = NULL; @@ -1522,7 +1524,7 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) } if (ipc6.hlimit < 0) - ipc6.hlimit = ip6_sk_dst_hoplimit(np, &fl6, dst); + ipc6.hlimit = ip6_sk_dst_hoplimit(np, fl6, dst); if (msg->msg_flags&MSG_CONFIRM) goto do_confirm; @@ -1530,18 +1532,15 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) /* Lockless fast path for the non-corking case */ if (!corkreq) { - struct inet_cork_full cork; struct sk_buff *skb; - cork.fl.u.ip6 = fl6; - skb = ip6_make_skb(sk, getfrag, msg, ulen, sizeof(struct udphdr), &ipc6, (struct rt6_info *)dst, msg->msg_flags, &cork); err = PTR_ERR(skb); if (!IS_ERR_OR_NULL(skb)) - err = udp_v6_send_skb(skb, &fl6, &cork.base); + err = udp_v6_send_skb(skb, fl6, &cork.base); goto out; } @@ -1563,7 +1562,7 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) ipc6.dontfrag = np->dontfrag; up->len += ulen; err = ip6_append_data(sk, getfrag, msg, ulen, sizeof(struct udphdr), - &ipc6, &fl6, (struct rt6_info *)dst, + &ipc6, fl6, (struct rt6_info *)dst, corkreq ? msg->msg_flags|MSG_MORE : msg->msg_flags); if (err) udp_v6_flush_pending_frames(sk); @@ -1598,7 +1597,7 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) do_confirm: if (msg->msg_flags & MSG_PROBE) - dst_confirm_neigh(dst, &fl6.daddr); + dst_confirm_neigh(dst, &fl6->daddr); if (!(msg->msg_flags&MSG_PROBE) || len) goto back_from_confirm; err = 0; From patchwork Tue Jan 11 01:21:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709389 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04097C433EF for ; Tue, 11 Jan 2022 01:25:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346385AbiAKBZg (ORCPT ); Mon, 10 Jan 2022 20:25:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346278AbiAKBY6 (ORCPT ); Mon, 10 Jan 2022 20:24:58 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A3FFC061201; Mon, 10 Jan 2022 17:24:54 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id e9so28605860wra.2; Mon, 10 Jan 2022 17:24:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WLXW72viCdTqGqSUN8B/wJ3y6rx1ShnvlX/6mgA/9Xo=; b=MkjnzgdbarY2G24kMTnfTMWw6YO8KJWOBRn54o2VeMv0jtN+dexJXKD+8Z1DK919ru q/3brEep39XVdvqSuZgYNScFXx9oaAL8rONzamI7ygfiE5Oo2W/UURrzZM14YVgg36LS ZweIIOzK5IXbjmsgnJx3F9RP5hCysuEQmZkwB8lfamfkY5T7RykQ5jIUPdCA0q/vZzJT tctGCY/6KEHzZcFUt8U1Qv5yIvVNbmeudk0JzRZMG45aDbiITS9ggJDRhQdg+4OgYga9 sHz/+n0SaqvT0xHInAcW0GyYS3Ip4tI6pVRrE7iopfmrx6mHouoowfYm0KvH3e2044fk 9zDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WLXW72viCdTqGqSUN8B/wJ3y6rx1ShnvlX/6mgA/9Xo=; b=Nuol8Ou7Ulw0L89tgPhL4wOzKG7n3/DT3WnliAHyX519eJ80O1uwqqx4g+GBuAEkV1 NabQ+8ALn5Bj/ICwQkQdtjDGu9ZRvV3jmd1PlOIC6ZfsHhKYlTfAMpXXmlBQZDQvormr /EH9rvIOdpCkoHmVZbDxMwGgnY8W15PyJwh5576vAjJCp3CW4XO7UuOaiER/JA81RlfL kg+sr0CoW7I/bPyVK3opX3Ab0PkF2G9dJUPZtoaIXdhmuN5B0z69iOL1NaeBl5miu4Xm tLzMG5u2l+RL7IJpvOJnNpFg90plWPjzp59/qOsxL6TL606Olysm8+O36lg5CCTbtsge Ygmg== X-Gm-Message-State: AOAM530JqpHggTfEPM9OBgUNQ3w7Xv3WxGWC7agb+L+A8gpXMc+MGvi7 8SrBq8dijHKKe1pKXlZjxloT3SRZUO8= X-Google-Smtp-Source: ABdhPJyEJu/8rM2j1jKQ83BmKsi1GeVilEWi6/P7o3ocKooP95U+H6dO2+/HMuP86/309RGcBqKr+A== X-Received: by 2002:a05:6000:c8:: with SMTP id q8mr1723397wrx.611.1641864292947; Mon, 10 Jan 2022 17:24:52 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:52 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 09/14] ipv6: hand dst refs to cork setup Date: Tue, 11 Jan 2022 01:21:42 +0000 Message-Id: <07031c43d3e5c005fbfc76b60a58e30c66d7c620.1641863490.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org During cork->dst setup, ip6_make_skb() gets an additional reference to a passed in dst. However, udpv6_sendmsg() doesn't need dst after calling ip6_make_skb(), and so we can save two additional atomics by passing dst references to ip6_make_skb(). udpv6_sendmsg() is the only caller, so it's enough to make sure it doesn't use dst afterwards. Signed-off-by: Pavel Begunkov --- net/ipv6/ip6_output.c | 9 ++++++--- net/ipv6/udp.c | 3 ++- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 0cc490f2cfbf..6a7bba4dd04d 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1356,6 +1356,8 @@ static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork, unsigned int mtu; struct ipv6_txoptions *nopt, *opt = ipc6->opt; + cork->base.dst = &rt->dst; + /* * setup for corking */ @@ -1389,8 +1391,6 @@ static int ip6_setup_cork(struct sock *sk, struct inet_cork_full *cork, /* need source address above miyazawa*/ } - dst_hold(&rt->dst); - cork->base.dst = &rt->dst; v6_cork->hop_limit = ipc6->hlimit; v6_cork->tclass = ipc6->tclass; if (rt->dst.flags & DST_XFRM_TUNNEL) @@ -1784,6 +1784,7 @@ int ip6_append_data(struct sock *sk, /* * setup for corking */ + dst_hold(&rt->dst); err = ip6_setup_cork(sk, &inet->cork, &np->cork, ipc6, rt); if (err) @@ -1974,8 +1975,10 @@ struct sk_buff *ip6_make_skb(struct sock *sk, int exthdrlen = (ipc6->opt ? ipc6->opt->opt_flen : 0); int err; - if (flags & MSG_PROBE) + if (flags & MSG_PROBE) { + dst_release(&rt->dst); return NULL; + } __skb_queue_head_init(&queue); diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index eec83e34ae27..3039dff7fe64 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -1541,7 +1541,8 @@ int udpv6_sendmsg(struct sock *sk, struct msghdr *msg, size_t len) err = PTR_ERR(skb); if (!IS_ERR_OR_NULL(skb)) err = udp_v6_send_skb(skb, fl6, &cork.base); - goto out; + /* ip6_make_skb steals dst reference */ + goto out_no_dst; } lock_sock(sk); From patchwork Tue Jan 11 01:21:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709388 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AF57C433EF for ; Tue, 11 Jan 2022 01:25:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346342AbiAKBZf (ORCPT ); Mon, 10 Jan 2022 20:25:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346277AbiAKBY6 (ORCPT ); Mon, 10 Jan 2022 20:24:58 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 55CF5C061212; Mon, 10 Jan 2022 17:24:55 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id r28so1717663wrc.3; Mon, 10 Jan 2022 17:24:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=u7C5kpsi9JrnwpLE6F+x2Sk2xvwwtL0J8QqF8lH/geA=; b=mcVOCfu6sXLiAEXb5zG8Oc7E4y3xSWA+m3yGfqSbGHoARMcRkCk7TmJq3I8ANy1UI0 ftVeblEbsDXvBg8cSxdtQP7M2ZI+IXIvorJWwQMuGH208MOQJYMKQm8IYsEGLRidypi7 rva6J/FLcxjL/b1yuwYMSoTH1DFw4uY2zGvdzVUHSQWFvYiWFxGcD3Vp+DizLU7HBcBA KetFGDE7oGTTjOTZ7p+R5GxbrGVuGRfRllVGehSFxgycvLo4HLXM8zF+SkKbg6UB0+pV ZOpL5SDs7G5RW574KRYgEl/shQH/v1TttJ/MwOBxXoyGz+0A3SBX+suUFc9IrxUSW1+9 xTWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=u7C5kpsi9JrnwpLE6F+x2Sk2xvwwtL0J8QqF8lH/geA=; b=o8vRnKOHJ4JQfqAFoymGxOKPexqB/FUrSifwhUvf/8WoRjRHZ/s4IHW8z9XjFwaT0a Dkxh64Y4u6FXl8+8K201Q+5XsWqTgmeTqy+wbIqhddOrtTnR1qB3Cb7Jycl1jdO5ib71 xttAA/VCjIns29gC3FDTxM3QYKH9wctsWbiLbDmGZFteBl5OA8D4CwQUnKnJI1RzutHA grXNPLX//IKNXkM0mULHGvN6BX7TdV3nrMELTBp0ErupPFkm5aSrwQQQb6h2JYGGzqpC oo2qhx5Gx17z7SzuCgwCFag2Ues3/dYlbCf8Zzap9eqx0KAGHbgjZO0LKLTiVM7ORGu+ yEnQ== X-Gm-Message-State: AOAM531pn+gsd5epayNlvN4Po2O+AGhqxcv6ri1/spMVikE/v7ycoG2J keK/uJDFJsKOcKYFujYdTvf+qIVAx5w= X-Google-Smtp-Source: ABdhPJzFs8FnSx0Oq52b7ssQdi+Ukdaebs5ctHAkxFP4q+FDLn9BGECXCe0p5PXVcICt7eUmA+4CGg== X-Received: by 2002:adf:e410:: with SMTP id g16mr1666142wrm.17.1641864293833; Mon, 10 Jan 2022 17:24:53 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:53 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 10/14] skbuff: drop zero check from skb_zcopy_set Date: Tue, 11 Jan 2022 01:21:43 +0000 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Only two skb_zcopy_set() callers may pass a NULL skb, so kill the zero check from inside the function, which can't be compiled out, and place it where needed. It's also needed by the following patch. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 2 +- net/ipv4/ip_output.c | 3 ++- net/ipv6/ip6_output.c | 3 ++- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 642acb0d1646..8a7d0d03a100 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1499,7 +1499,7 @@ static inline void skb_zcopy_init(struct sk_buff *skb, struct ubuf_info *uarg) static inline void skb_zcopy_set(struct sk_buff *skb, struct ubuf_info *uarg, bool *have_ref) { - if (skb && uarg && !skb_zcopy(skb)) { + if (uarg && !skb_zcopy(skb)) { if (unlikely(have_ref && *have_ref)) *have_ref = false; else diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 57c1d8431386..87d4472545a5 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1010,7 +1010,8 @@ static int __ip_append_data(struct sock *sk, paged = true; } else { uarg->zerocopy = 0; - skb_zcopy_set(skb, uarg, &extra_uref); + if (skb) + skb_zcopy_set(skb, uarg, &extra_uref); } } diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 6a7bba4dd04d..9881b61da493 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1522,7 +1522,8 @@ static int __ip6_append_data(struct sock *sk, paged = true; } else { uarg->zerocopy = 0; - skb_zcopy_set(skb, uarg, &extra_uref); + if (skb) + skb_zcopy_set(skb, uarg, &extra_uref); } } From patchwork Tue Jan 11 01:21:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709387 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93C02C433EF for ; Tue, 11 Jan 2022 01:25:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346462AbiAKBZW (ORCPT ); Mon, 10 Jan 2022 20:25:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346285AbiAKBY7 (ORCPT ); Mon, 10 Jan 2022 20:24:59 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 457EFC0611FF; Mon, 10 Jan 2022 17:24:56 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id s1so30124407wra.6; Mon, 10 Jan 2022 17:24:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=m6AI07V5V9bxuzbKOOkycd9Y2cUTCObOibCgtj03Nis=; b=LXz32GX2FsenK8ogWOx5xOgq7KcpiJxFHgmyj6jS4UUJuu+yV4Folo9Pz2uwpKs2ND jsUjWlAA94+XO7eLG1W74qJUPN8Z2KDmhbLw5TDQ90rHZJo2PR8+BxE04BV6BbLMwAfy H41XjFt6UZEr6kqnvO/O/WLF/yB8J5HGpxgvboBA1P7UcgE3BYI3UcC+Cq8ymHk4Bgs+ yOljL0SUj1pIndc8u9PyuTHdeJ3nKuVC7ujEz5vEBqdmzwUcL7MHGHn8XMxgX/wOEYi/ TVZKrZ22ezyJiEkhIYoC0yVzqWwb+fYp9PdFGGhgCrAfMousd7cgdupeygcrJdskRCNe 9AmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=m6AI07V5V9bxuzbKOOkycd9Y2cUTCObOibCgtj03Nis=; b=dk4KXXzsaf1z5eLqjnCWOWVA4Amrz3edXUKaNs2LdcNTrKadPRGo55g1YEW/eWWTuO Itfl5E7OukO8njpLu6Lh8eYkKmtzQ6qDI8GTZvsGZje2rXsQs2Nw/YNyaWAo42fmSGBu azDUFfAr0O50rVLTk25QNB/jjU4rTMhEOow0ZC7tOckwwwfMn36QlFJnXxacXyL6C2Gh ZJ9dTOjp8RQQeTAwOH249AZTVE2M9HUqZBlBxGY659Nee4iatq3Vcxeo+PtkBu9YUxDD nmoEITZPNcoTQkNvIYcgJoUHZ7cMXivB9ZShpj3zcUSmufaZFXwXybbbhACT/F12hg0H MGzA== X-Gm-Message-State: AOAM530X9AVSYANf7E8G16rTiQ2deN/gHZgPFncbcvEem81mc8vgYFtw babvBf/RGyyq+7uZodSALYnJK6vCh9w= X-Google-Smtp-Source: ABdhPJy/ZUGcCJS0Lon/Dpa8hvO13IMCkzTAgnkRG64EN426gYS89LzrnHhzAeiD2rJLi8K3AFKPPA== X-Received: by 2002:a05:6000:1e15:: with SMTP id bj21mr1832686wrb.118.1641864294702; Mon, 10 Jan 2022 17:24:54 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:54 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 11/14] skbuff: drop null check from skb_zcopy Date: Tue, 11 Jan 2022 01:21:44 +0000 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org skb_zcopy() is used all around the networkong code with many of calls sitting in generic not necessarily zerocopy paths. Many of callers don't ever pass a NULL skb, however a NULL check inside skb_zcopy() can't be optimised out. As with previous patch, move the check out of the helper to a few places where it's needed. It removes a bunch of extra ifs in non-zerocopy paths, which is nice. E.g. before and after: text data bss dec hex filename 8521472 0 0 8521472 820700 arch/x86/boot/bzImage 8521056 0 0 8521056 820560 arch/x86/boot/bzImage delta=416B Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 2 +- net/core/dev.c | 2 +- net/core/skbuff.c | 3 ++- net/ipv4/ip_output.c | 7 +++++-- net/ipv4/tcp.c | 5 ++++- net/ipv6/ip6_output.c | 7 +++++-- 6 files changed, 18 insertions(+), 8 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 8a7d0d03a100..7fd2b44aada0 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1469,7 +1469,7 @@ static inline struct skb_shared_hwtstamps *skb_hwtstamps(struct sk_buff *skb) static inline struct ubuf_info *skb_zcopy(struct sk_buff *skb) { - bool is_zcopy = skb && skb_shinfo(skb)->flags & SKBFL_ZEROCOPY_ENABLE; + bool is_zcopy = skb_shinfo(skb)->flags & SKBFL_ZEROCOPY_ENABLE; return is_zcopy ? skb_uarg(skb) : NULL; } diff --git a/net/core/dev.c b/net/core/dev.c index 83a4089990a0..877ebc0f72bd 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -2239,7 +2239,7 @@ void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev) } out_unlock: if (pt_prev) { - if (!skb_orphan_frags_rx(skb2, GFP_ATOMIC)) + if (!skb2 || !skb_orphan_frags_rx(skb2, GFP_ATOMIC)) pt_prev->func(skb2, skb->dev, pt_prev, skb->dev); else kfree_skb(skb2); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index e514a36bcffc..a9b8ac38dc1a 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -890,7 +890,8 @@ EXPORT_SYMBOL(skb_dump); */ void skb_tx_error(struct sk_buff *skb) { - skb_zcopy_clear(skb, true); + if (skb) + skb_zcopy_clear(skb, true); } EXPORT_SYMBOL(skb_tx_error); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 87d4472545a5..b63f307cc5ab 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1001,10 +1001,13 @@ static int __ip_append_data(struct sock *sk, csummode = CHECKSUM_PARTIAL; if (flags & MSG_ZEROCOPY && length && sock_flag(sk, SOCK_ZEROCOPY)) { - uarg = msg_zerocopy_realloc(sk, length, skb_zcopy(skb)); + if (skb) + uarg = skb_zcopy(skb); + extra_uref = !uarg; /* only ref on new uarg */ + + uarg = msg_zerocopy_realloc(sk, length, uarg); if (!uarg) return -ENOBUFS; - extra_uref = !skb_zcopy(skb); /* only ref on new uarg */ if (rt->dst.dev->features & NETIF_F_SG && csummode == CHECKSUM_PARTIAL) { paged = true; diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 3b75836db19b..f35e49ea08ec 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1188,7 +1188,10 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (flags & MSG_ZEROCOPY && size && sock_flag(sk, SOCK_ZEROCOPY)) { skb = tcp_write_queue_tail(sk); - uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); + if (skb) + uarg = skb_zcopy(skb); + + uarg = msg_zerocopy_realloc(sk, size, uarg); if (!uarg) { err = -ENOBUFS; goto out_err; diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 9881b61da493..41abe83c3419 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1513,10 +1513,13 @@ static int __ip6_append_data(struct sock *sk, csummode = CHECKSUM_PARTIAL; if (flags & MSG_ZEROCOPY && length && sock_flag(sk, SOCK_ZEROCOPY)) { - uarg = msg_zerocopy_realloc(sk, length, skb_zcopy(skb)); + if (skb) + uarg = skb_zcopy(skb); + extra_uref = !uarg; /* only ref on new uarg */ + + uarg = msg_zerocopy_realloc(sk, length, uarg); if (!uarg) return -ENOBUFS; - extra_uref = !skb_zcopy(skb); /* only ref on new uarg */ if (rt->dst.dev->features & NETIF_F_SG && csummode == CHECKSUM_PARTIAL) { paged = true; From patchwork Tue Jan 11 01:21:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709386 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DC6CC433F5 for ; Tue, 11 Jan 2022 01:25:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346296AbiAKBZR (ORCPT ); Mon, 10 Jan 2022 20:25:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346289AbiAKBZA (ORCPT ); Mon, 10 Jan 2022 20:25:00 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A339C034001; Mon, 10 Jan 2022 17:24:57 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id r28so1717747wrc.3; Mon, 10 Jan 2022 17:24:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/NURB/5FWWBfbG+TKgRZPMIkRC+TZfJeR4s5cDZLj40=; b=YKteNh9VRvpLU0D49Yyc9iBZ9ms1N1Q1lbFMkySHT/m3mdPP+zEIGjhgT1/c0KrN2F a52P3MzGmmgLfctYbxgpQQXpVwCqaH4v+623ioHT9TjadUbF3MqK2GaQ86SqhhT2QROb H7fzSeMS2QxC3MeTHRhje+DOwBREihpuAbpQnxQUsFG1j1IOsaj5mNVNSyMeartxRJ7E IfLvL78K+deP1zIO0Ab7QIN5oex/3K9HihCOtUHTVeFEAsSDj/mA4dGA52JCO70sIImW cDWRx8XY65H2ZeklpMb3qLi/XcZifly1Y4Dl6teFdUwuIuE9EqwZPAvrnihJi0cu2/u8 6htA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/NURB/5FWWBfbG+TKgRZPMIkRC+TZfJeR4s5cDZLj40=; b=czMRn3goU4r8KXb3hgry+qX4ZGbUBz0VfveQtZ37bcd9Q5KgNGcnNeZqopwyX+t0G2 Evp3q13E4igZJbcqr1CeJ6xtqbzqbcliqMW94GSLJURq3BsHQSGp+70+9HH4wpABStgk qdjfdPsPgHznVMuoBK36tRjqTbkp56jpAQObnPSFzParPOoASsn9SxI6vRYAM6kC1ybK /4alMJMQPZC+93OIQoXOXCY90o+XB5C5HxxFUQdg5aOLV9+eYD2bcmlgKcbCXmElPzy9 NGOKvWo/N2XeGm5CwK6iwPH96nQxPGsBc9XdDKtRjbE4BZf4RwZMlBg8QzSc1G2h3FRq Spcw== X-Gm-Message-State: AOAM530gqfsSHFsHxe/vDMBWvNFrzVq2Bj5B1vfPC1qrP9HL+Oxmy2Ct Wu7brn2tO3DWXLxPMXqtu0YGO4BHQNU= X-Google-Smtp-Source: ABdhPJzDmxRIhnN5EpCoFwAfvYWiP+8E7sayOOP0REBpL4YMU9CebM6CG8+DKyiiUSQA04Nw2VA7rw== X-Received: by 2002:a5d:5988:: with SMTP id n8mr1753420wri.309.1641864295565; Mon, 10 Jan 2022 17:24:55 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:55 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 12/14] skbuff: optimise alloc_skb_with_frags() Date: Tue, 11 Jan 2022 01:21:45 +0000 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Many users of alloc_skb_with_frags() pass zero datalen, e.g. all callers sock_alloc_send_skb() including udp. Extract and inline a part of it doing skb allocation. BTW, do a minor cleanup, e.g. don't set errcode in advance as it can't be optimised. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 41 ++++++++++++++++++++++++++++++++++++----- net/core/skbuff.c | 31 ++++++++++++------------------- 2 files changed, 48 insertions(+), 24 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 7fd2b44aada0..8ea145101b56 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1130,11 +1130,42 @@ static inline struct sk_buff *alloc_skb(unsigned int size, return __alloc_skb(size, priority, 0, NUMA_NO_NODE); } -struct sk_buff *alloc_skb_with_frags(unsigned long header_len, - unsigned long data_len, - int max_page_order, - int *errcode, - gfp_t gfp_mask); +struct sk_buff *alloc_skb_frags(struct sk_buff *skb, + unsigned long data_len, + int max_page_order, + int *errcode, + gfp_t gfp_mask); + +/** + * alloc_skb_with_frags - allocate skb with page frags + * + * @header_len: size of linear part + * @data_len: needed length in frags + * @max_page_order: max page order desired. + * @errcode: pointer to error code if any + * @gfp_mask: allocation mask + * + * This can be used to allocate a paged skb, given a maximal order for frags. + */ +static inline struct sk_buff *alloc_skb_with_frags(unsigned long header_len, + unsigned long data_len, + int max_page_order, + int *errcode, + gfp_t gfp_mask) +{ + struct sk_buff *skb; + + skb = alloc_skb(header_len, gfp_mask); + if (unlikely(!skb)) { + *errcode = -ENOBUFS; + return NULL; + } + + if (!data_len) + return skb; + return alloc_skb_frags(skb, data_len, max_page_order, errcode, gfp_mask); +} + struct sk_buff *alloc_skb_for_msg(struct sk_buff *first); /* Layout of fast clones : [skb1][skb2][fclone_ref] */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a9b8ac38dc1a..7811dde22f26 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -5922,40 +5922,32 @@ int skb_mpls_dec_ttl(struct sk_buff *skb) EXPORT_SYMBOL_GPL(skb_mpls_dec_ttl); /** - * alloc_skb_with_frags - allocate skb with page frags + * alloc_skb_frags - allocate page frags for skb * - * @header_len: size of linear part + * @skb: buffer * @data_len: needed length in frags * @max_page_order: max page order desired. * @errcode: pointer to error code if any * @gfp_mask: allocation mask * - * This can be used to allocate a paged skb, given a maximal order for frags. + * This can be used to allocate pages for skb, given a maximal order for frags. */ -struct sk_buff *alloc_skb_with_frags(unsigned long header_len, - unsigned long data_len, - int max_page_order, - int *errcode, - gfp_t gfp_mask) +struct sk_buff *alloc_skb_frags(struct sk_buff *skb, + unsigned long data_len, + int max_page_order, + int *errcode, + gfp_t gfp_mask) { int npages = (data_len + (PAGE_SIZE - 1)) >> PAGE_SHIFT; unsigned long chunk; - struct sk_buff *skb; struct page *page; int i; - *errcode = -EMSGSIZE; /* Note this test could be relaxed, if we succeed to allocate * high order pages... */ - if (npages > MAX_SKB_FRAGS) - return NULL; - - *errcode = -ENOBUFS; - skb = alloc_skb(header_len, gfp_mask); - if (!skb) - return NULL; - + if (unlikely(npages > MAX_SKB_FRAGS)) + goto failure; skb->truesize += npages << PAGE_SHIFT; for (i = 0; npages > 0; i++) { @@ -5989,9 +5981,10 @@ struct sk_buff *alloc_skb_with_frags(unsigned long header_len, failure: kfree_skb(skb); + *errcode = -EMSGSIZE; return NULL; } -EXPORT_SYMBOL(alloc_skb_with_frags); +EXPORT_SYMBOL(alloc_skb_frags); /* carve out the first off bytes from skb when off < headlen */ static int pskb_carve_inside_header(struct sk_buff *skb, const u32 off, From patchwork Tue Jan 11 01:21:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709385 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 345CFC433FE for ; Tue, 11 Jan 2022 01:25:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346434AbiAKBZT (ORCPT ); Mon, 10 Jan 2022 20:25:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346291AbiAKBZA (ORCPT ); Mon, 10 Jan 2022 20:25:00 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F4044C06173F; Mon, 10 Jan 2022 17:24:57 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id l25so19233499wrb.13; Mon, 10 Jan 2022 17:24:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BjXWOH5wrWztd6i7p0jL5/v39iYb4LIPGSHy9klVSRw=; b=g69qarQTFT4uz2Fj4M4Nxle/qtQwXnwMv2lw7EZtVrGyxrlfrOXt0+eMVcaXEfR4eJ 4Fl4q8Dp2467IRttdetDoN2MKH1s+f0KddnCfRW7ezRIwCk9CMeyki6jqBxkyEmyN0Mn SI0RHbK56xgO+ZHuIebRzXuhQAPNZ+HTDzCxK3afR7GLnEfRb5GDLoEXebjZyNmaSqtt /0DuOlTbRvTo7srcmPqrg26mX+tKZV4Zlxt+8NmBNyPvZFSHj257HmGua2joh+U78iIg B8NNqxaSSHPoXQmE7c/vutkQLV+rKY3PF/bU7CnqV6H/CmvFANlihlnDJP/5y1tA4Ewy rshQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BjXWOH5wrWztd6i7p0jL5/v39iYb4LIPGSHy9klVSRw=; b=j1jowB7njXo7bq8ZMfY2LNQbYdU08aMQCUr1tisekBnSoZjVtDTPaOEEI7xHSdUUcj dBsaH4Xempz4w0OR2tVmclsbrA3Z6uO9t0t3BqScdNO/jZnDNuXqKrHH3ZIrnA8+SRgl /6csBk+oKBOBVVdnhgS01v9TZk8+bLEEnhnFtpnv7z5SqBmOsOVpnRAx3hUNx1G+mTbt naXAIpgkmS3TPYiMQA0YlORJX8aNAAxKDlN22PWA/AmkHcGsI6Wy/QRMxvc0eEVrw5Zf LoEk9LzmKVo8vgIGUxRH/O7/0CDSSxyzQbWgAxCmxRAlXHaGvIweH87LQACD53uC+Ci8 0PDA== X-Gm-Message-State: AOAM531/xjDZqURe+ZCqlZzXdRFvnmLL5MLlX2AEJEzrtbqC2vfg0hPw gsBWK3TrN7beAkJLRFe3S6S+vL2k/x0= X-Google-Smtp-Source: ABdhPJw35QsuQoC2G7905+UuKYpKBDkhY4MVXmEH6xzrX9/YClSJxBE1aBm6x2XWfosWAyul3B6Dfw== X-Received: by 2002:a05:6000:15c5:: with SMTP id y5mr1688211wry.473.1641864296428; Mon, 10 Jan 2022 17:24:56 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:56 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 13/14] net: inline part of skb_csum_hwoffload_help Date: Tue, 11 Jan 2022 01:21:46 +0000 Message-Id: <0bc041d2d38a08064a642c05ca8cceb0ca165f88.1641863490.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Inline a HW csum'ed part of skb_csum_hwoffload_help(). Signed-off-by: Pavel Begunkov --- include/linux/netdevice.h | 16 ++++++++++++++-- net/core/dev.c | 13 +++---------- 2 files changed, 17 insertions(+), 12 deletions(-) diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 3213c7227b59..fbe6c764ce57 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -4596,8 +4596,20 @@ void netdev_rss_key_fill(void *buffer, size_t len); int skb_checksum_help(struct sk_buff *skb); int skb_crc32c_csum_help(struct sk_buff *skb); -int skb_csum_hwoffload_help(struct sk_buff *skb, - const netdev_features_t features); +int __skb_csum_hwoffload_help(struct sk_buff *skb, + const netdev_features_t features); + +static inline int skb_csum_hwoffload_help(struct sk_buff *skb, + const netdev_features_t features) +{ + if (unlikely(skb_csum_is_sctp(skb))) + return !!(features & NETIF_F_SCTP_CRC) ? 0 : + skb_crc32c_csum_help(skb); + + if (features & NETIF_F_HW_CSUM) + return 0; + return __skb_csum_hwoffload_help(skb, features); +} struct sk_buff *__skb_gso_segment(struct sk_buff *skb, netdev_features_t features, bool tx_path); diff --git a/net/core/dev.c b/net/core/dev.c index 877ebc0f72bd..e65a3b311810 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -3513,16 +3513,9 @@ static struct sk_buff *validate_xmit_vlan(struct sk_buff *skb, return skb; } -int skb_csum_hwoffload_help(struct sk_buff *skb, - const netdev_features_t features) +int __skb_csum_hwoffload_help(struct sk_buff *skb, + const netdev_features_t features) { - if (unlikely(skb_csum_is_sctp(skb))) - return !!(features & NETIF_F_SCTP_CRC) ? 0 : - skb_crc32c_csum_help(skb); - - if (features & NETIF_F_HW_CSUM) - return 0; - if (features & (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM)) { switch (skb->csum_offset) { case offsetof(struct tcphdr, check): @@ -3533,7 +3526,7 @@ int skb_csum_hwoffload_help(struct sk_buff *skb, return skb_checksum_help(skb); } -EXPORT_SYMBOL(skb_csum_hwoffload_help); +EXPORT_SYMBOL(__skb_csum_hwoffload_help); static struct sk_buff *validate_xmit_skb(struct sk_buff *skb, struct net_device *dev, bool *again) { From patchwork Tue Jan 11 01:21:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12709384 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82487C433EF for ; Tue, 11 Jan 2022 01:25:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346288AbiAKBZR (ORCPT ); Mon, 10 Jan 2022 20:25:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346297AbiAKBZA (ORCPT ); Mon, 10 Jan 2022 20:25:00 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BAC6CC061751; Mon, 10 Jan 2022 17:24:58 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id v6so30028683wra.8; Mon, 10 Jan 2022 17:24:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cLHZNKxSVLlIqt82iN+00oFOdp+Zw3lomAl6fzMW3gQ=; b=E97YACHjCfA9o7nbBNMoNuGTlk5ww1E/pGt2T80i6nTf4voJ2+m973msMByvGKfZCg /r6Ed2lI9aPTOLEvnqghjaiORTGYlmiuSRHrcfmihkwlN8klBoSoE0zINvH7S+6C26pd LJxNq5+VJWLOGrUpynL3eY6qvEUM4MlKIPDH5L/aPvv5EGMUgl7mRnbCg1jARTmCNGqN 30SNuXC8K9Fhf8i6+FcOx01SAVQEnkd9kdhqD76voNUgqV5JsFkAxugZ8cqQkvsUdZM6 ZwacmybXvQzyv+lfOe8r6IRu1+SbFNy9ZnuyR+eNqhOIwofkSlwkB75efOAJxLALBSrm kuwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cLHZNKxSVLlIqt82iN+00oFOdp+Zw3lomAl6fzMW3gQ=; b=3lDCfBnpJuAaL8LD9SvM8hg8LB1nExcq4RHZxGW9v1wPDSDuulJBFAyD2aFO9ZH92I L93j0k8hOxREYiGlmB63y5whIBv3p5268ShZVQf18yJ8iK1PjRrenwuW3x6vHgp20RE7 X2Md/JmOF7wLE6yJptiMvj2LN2+TAlOoi6waPDkgjVSlD5sid0xikYUdPKpqOXlo3w2X Zf2Q1PZnNkfNcQb/9BjshhjA670dPX6Rwi9a4k+vyZ5lq4kFDQPdmRE39zptTV8QSlHb KqdlmugY3C2xH+gjDxbvpbrb+qJg4NasDkMOC9K2GAgtGkWk1Ylg8dEk+pVwvmgsjfbk NuRA== X-Gm-Message-State: AOAM530kgevx0dVPsIVDk3B+qVa6WMernXVcEP01i3Wulkaj0vWk2N/r yanXUkSO1VaugNzTOcijOzJ6PQYQBN8= X-Google-Smtp-Source: ABdhPJycmpPYwVnA2PBfXK/arNpJX02Fu1XAxJuzT/SY1+MVe/Q1mQCyMv4OBHocREiUHjedv1S+9g== X-Received: by 2002:a05:6000:1866:: with SMTP id d6mr1702705wri.704.1641864297269; Mon, 10 Jan 2022 17:24:57 -0800 (PST) Received: from 127.0.0.1localhost ([148.252.129.73]) by smtp.gmail.com with ESMTPSA id i8sm709886wru.26.2022.01.10.17.24.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Jan 2022 17:24:57 -0800 (PST) From: Pavel Begunkov To: netdev@vger.kernel.org, "David S . Miller" , Jakub Kicinski Cc: Hideaki YOSHIFUJI , David Ahern , Eric Dumazet , Willem de Bruijn , linux-kernel@vger.kernel.org, Pavel Begunkov Subject: [PATCH 14/14] net: inline sock_alloc_send_skb Date: Tue, 11 Jan 2022 01:21:47 +0000 Message-Id: <00f1789794462df0bd2f23bd9672cc7ddc740494.1641863490.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org sock_alloc_send_skb() is simple and just proxying to another function, so we can inline it and cut associated overhead. Signed-off-by: Pavel Begunkov --- include/net/sock.h | 10 ++++++++-- net/core/sock.c | 7 ------- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/include/net/sock.h b/include/net/sock.h index 7b4b4237e6e0..cde35481a152 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1810,11 +1810,17 @@ int sock_getsockopt(struct socket *sock, int level, int op, char __user *optval, int __user *optlen); int sock_gettstamp(struct socket *sock, void __user *userstamp, bool timeval, bool time32); -struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size, - int noblock, int *errcode); struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, unsigned long data_len, int noblock, int *errcode, int max_page_order); + +static inline struct sk_buff *sock_alloc_send_skb(struct sock *sk, + unsigned long size, + int noblock, int *errcode) +{ + return sock_alloc_send_pskb(sk, size, 0, noblock, errcode, 0); +} + void *sock_kmalloc(struct sock *sk, int size, gfp_t priority); void sock_kfree_s(struct sock *sk, void *mem, int size); void sock_kzfree_s(struct sock *sk, void *mem, int size); diff --git a/net/core/sock.c b/net/core/sock.c index e21485ab285d..25a266a429d4 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2592,13 +2592,6 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, } EXPORT_SYMBOL(sock_alloc_send_pskb); -struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size, - int noblock, int *errcode) -{ - return sock_alloc_send_pskb(sk, size, 0, noblock, errcode, 0); -} -EXPORT_SYMBOL(sock_alloc_send_skb); - int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg, struct sockcm_cookie *sockc) {