From patchwork Wed Apr 10 15:34:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Gobert X-Patchwork-Id: 13624760 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4714B16EBEF; Wed, 10 Apr 2024 15:35:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763320; cv=none; b=YvoJVuyR/cToTaSIGfNzlZ8ixa8xZeufDdvoHvwlb+63v0+bsa1tFjbw3gl4Ey8VNWcOl9noBcoUxYx92qa91v2Lyhvf9hnneGqanyYJwuwYNZSNpe4BqfdkMAYt09+7OVSstUW2SiNYrNjFH5WktubeUjLapQloHBcsyDfTqVU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763320; c=relaxed/simple; bh=vWx8hjmgbGaYijZPOHAFQF3Diarftvn5nTcDZWxUfWQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NC6D1ojATcEyfcdbg3hJeozwzB7auAZ9nnyGPXWTdX9lwTjp2QuCeTZmE7yxXTE4BPpyZYPr/7gearDz3iEttWLEipp2/67rBsAlz4Vm5iHPsVh6cjaxmXV2V+nkml2BRCIl3HEiw1sayr30ieQmF0OpEuomF+dW7XONHXtQxgw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lUmhn/29; arc=none smtp.client-ip=209.85.221.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lUmhn/29" Received: by mail-wr1-f50.google.com with SMTP id ffacd0b85a97d-343e46ec237so4116290f8f.2; Wed, 10 Apr 2024 08:35:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712763317; x=1713368117; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=68baX53kYRdqiSV3OuL3sThzmNqdOFLBEOv/P9LxiMo=; b=lUmhn/29r1jhMngW5rBuyxAxpzyouSBQc7BhiLnU7QSuAUUY1IoLnwGvoXFNAPS9+o MCJdoX/M25J6nnBSL1OPsbV8uFAfleloNrDW0fS1LvX9MnIN5dAP9xWOrYuPsJbrJKsF x6fyq87Yh0K74Iq9kXx3c79Ecm3RZpybAFRqMkmzMladSRu3CWx7CeSIqM8rqpZ3CIVU PcuSpRkUMwr8fTf34+Ukjpti6l02M/r3vyzMkmzCMDgnNCpPrjllt2EKQEOCuTNj72Q+ yUL5gVKH1OApCTyfEIDo+IXJUJ9c5bYGZ1pqwToVySlZPS1GX3EJjGW0f/1QIE9Zqmmb p9nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712763317; x=1713368117; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=68baX53kYRdqiSV3OuL3sThzmNqdOFLBEOv/P9LxiMo=; b=Ht7Xk4I6LLPsj4VP8DKMyaYcjxiGa2Xx4VYjyJOrvnx6ev53eT1qJXrORHyLsx74cD J1mhbJc9WN1ubd0bA0MbUJHVtbSfSNDRPx2CvHdHECD6PKqeggwwoxoqPjkN7i5f42Z/ Z87nXFq3dvNYyhP7zqXEo7ozdSyMAhMjaNBxaEU8TnzMz9+XoO9HHn9fDpDODo3RP3PS fczEHlgW73flRNDEI/9S6t3ZHWqh1Inn3Jgxn+ZYIFcSnHUn1PzSq50MDtMwh8vJnXV+ SXklDiMbmoGraXxV5oKVD9i3vsvtb0q3/Sw11MBQ7a9lEfqLF3/2kT5ZZ7R+F629C8Bb CNKw== X-Forwarded-Encrypted: i=1; AJvYcCXV4j+hWfgALjTUmAzGFSPkaxg2SCNCV/8QfwDpk+ogErJn4DPn+hBCngSGJeBzJwqpu1RDoWnhJJZP8tc6tNDejPPX6MMwrAL1lvBbYWD5Iq7pY73UjXYTKESy4Ba42nTBQy1od9sSGC0WcTclXQqaqOkQyOafl7Tp2/JodVGIkm/YiRFO X-Gm-Message-State: AOJu0YyjkoayZurNJwFXP+7850ONUUEaQmV51scBoHn4ToTsYhXNNpxF YuvEdsDR3rket2pVbAVN8k4ysc6IgW2tlHAYaEJq460zRCPcmBfg X-Google-Smtp-Source: AGHT+IFpCTXftuYdAbu9wXWKxlU/Jww9KocYRa4/MBt6ILtJAroKsGboJFb5WMAzrx/I3nDy7QQrCg== X-Received: by 2002:adf:b312:0:b0:346:b6e:395 with SMTP id j18-20020adfb312000000b003460b6e0395mr2143793wrd.9.1712763317311; Wed, 10 Apr 2024 08:35:17 -0700 (PDT) Received: from localhost ([146.70.204.204]) by smtp.gmail.com with ESMTPSA id y14-20020a5d4ace000000b00341dbb4a3a7sm14043505wrs.86.2024.04.10.08.35.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 08:35:16 -0700 (PDT) From: Richard Gobert To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, willemdebruijn.kernel@gmail.com, shuah@kernel.org, dsahern@kernel.org, aduyck@mirantis.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Richard Gobert Subject: [PATCH net-next v6 1/6] net: gro: add flush check in udp_gro_receive_segment Date: Wed, 10 Apr 2024 17:34:18 +0200 Message-Id: <20240410153423.107381-2-richardbgobert@gmail.com> In-Reply-To: <20240410153423.107381-1-richardbgobert@gmail.com> References: <20240410153423.107381-1-richardbgobert@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org GRO-GSO path is supposed to be transparent and as such L3 flush checks are relevant to all flows which call skb_gro_receive. This patch uses the same logic and code from tcp_gro_receive but in the relevant flow path in udp_gro_receive_segment. Fixes: 36707061d6ba ("udp: allow forwarding of plain (non-fraglisted) UDP GRO packets") Signed-off-by: Richard Gobert --- net/ipv4/udp_offload.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/net/ipv4/udp_offload.c b/net/ipv4/udp_offload.c index 3498dd1d0694..1f4e08f43c4b 100644 --- a/net/ipv4/udp_offload.c +++ b/net/ipv4/udp_offload.c @@ -471,6 +471,7 @@ static struct sk_buff *udp_gro_receive_segment(struct list_head *head, struct sk_buff *p; unsigned int ulen; int ret = 0; + int flush; /* requires non zero csum, for symmetry with GSO */ if (!uh->check) { @@ -528,7 +529,17 @@ static struct sk_buff *udp_gro_receive_segment(struct list_head *head, skb_gro_postpull_rcsum(skb, uh, sizeof(struct udphdr)); - ret = skb_gro_receive(p, skb); + flush = NAPI_GRO_CB(p)->flush; + + if (NAPI_GRO_CB(p)->flush_id != 1 || + NAPI_GRO_CB(p)->count != 1 || + !NAPI_GRO_CB(p)->is_atomic) + flush |= NAPI_GRO_CB(p)->flush_id; + else + NAPI_GRO_CB(p)->is_atomic = false; + + if (flush || skb_gro_receive(p, skb)) + ret = 1; } } From patchwork Wed Apr 10 15:34:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Gobert X-Patchwork-Id: 13624761 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 699CD16F0D8; Wed, 10 Apr 2024 15:35:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763329; cv=none; b=qheiA23TBmrz93fVCEc6dUrB8i0IwrvL/Q36+ZJXH/qYDBIiQH9f9kufrVaQo3v1xU5HdumkfnYkQk72/FjRUEFvLQ0G6zxltjY5zAKXlOlZk8nf/j9jxfTHC/UhNVPCa0zlTn7oumqQWQyt6991007pa5Qh/q4aAEzD4KuRCs0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763329; c=relaxed/simple; bh=bjUcEut3WQ26NBLs6jb1FUz99Rz8McKWANPmfgo4mJA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rNXaaPhipi/fRjww9lU1P/wqKBLeRjadILO3OrYMjAToFBQlaepS3ChVNCSmQXzF0562JOUXcZzfhbZMSYVOG3qY+d//B9LSxvkDEp7ivcfG/WRZ/DHTB/5VwdM4XEg0Ac8TBemEMcp7q/Lq0ldyhrDijkL5gPU4Kl1h3ysgei8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aW+YCPJp; arc=none smtp.client-ip=209.85.221.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aW+YCPJp" Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-343eb6cc46bso3548011f8f.2; Wed, 10 Apr 2024 08:35:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712763325; x=1713368125; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x/7TjPxwoxAQbRrLaAjmIUiQ1mftYc30wuJsOkEmeZ8=; b=aW+YCPJpOODg7+g4/jb9VVfZQa+t2/meYITd/tt2DwFARHrCqPzBgFSpoV0e4WoxXC 7bjkiEWP2fngU7jpBhRYE2hqLboj9ktbtZIaltdE5sc/mNB4JFUIZLz8VBbp39Bhlv73 tw0rTO3HK6OhqXgXQSKekPMq3NT5lwtDoiXrYXzvZK1WRdc8TTx6Rt7iq+KAA8AP9rrW tWp+AB9oXdsDC7Tuq5G7KVo98e6b9yCbIl2QxnCHIJ0dFEMHGympt42WbQZ79jJfY+V3 mPJ/B72tb2GkB/FeHzG+qgFbqrcjvC/gxvY+Ba3SrzXPhCtl1HxAs473ElQV1mz5Llva hZ3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712763325; x=1713368125; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x/7TjPxwoxAQbRrLaAjmIUiQ1mftYc30wuJsOkEmeZ8=; b=KU9tIdqnwHkdEefobC9epbWYH8U2PIEN8mU9rzMT5FSjtAzcK/8rgiw32k5DvG+vYj xox9Yw7MdvNTnkB7nAyJfJBosMKD/OerwIjc3nFtB7ushCeTRJuZjm08ln2p+8VaeA3k 42+JJWzTiSO+cF6hKGFjUDMuJ1ioS9o8FlDBK/v3t/vuP8HDgQYIZVmoBbs+SL0q/6gT fJCWq6SKmuty/xka0cIBMyltnWhUrG9RpEB19vSupqmk1O693nHnen/tzkjCuDm6efAC snudGYX15jtv5dR2ereOZ3xWbCkie+dMU/bxPOs03B0bIHoHyVlh0m3TWh5mrhryAN+H Gazw== X-Forwarded-Encrypted: i=1; AJvYcCVwxz3WQze3+cJvFk8jDfKBJqU9vDw3o5D3ca0rezp5LKDAzK9k8UCrrZTLOCbe32uiA2Jk4DG6hepjBdzvBmJg8CAjy1aG4iJdhuxxlMtBRk+uGMi3UU6ei2o6wMt0N+mX5R0I6iSbDkdYl0ClHKMNWJKBI6doADt5l0rOEaKDIma4l6Ui X-Gm-Message-State: AOJu0YzCunB9bCJWrFJSMQ627xNRKZPLpjnc+jmWJBQpOa2t5K0KgWvB 6BeAu1LNH0j7o0Uop06zjHWTsGh2kJWSYfHdBjAM9I4IUHuJBbDa X-Google-Smtp-Source: AGHT+IEmQcf/3ilqiJLz0QokNnzPAAoFBvkWZj3KUtAGJiBu4Ozp5gxrnbSoHJC30ab7mbd1CgCmug== X-Received: by 2002:a5d:5185:0:b0:343:7cdc:458b with SMTP id k5-20020a5d5185000000b003437cdc458bmr2139759wrv.7.1712763324590; Wed, 10 Apr 2024 08:35:24 -0700 (PDT) Received: from localhost ([146.70.204.204]) by smtp.gmail.com with ESMTPSA id g12-20020a5d540c000000b00343f662327bsm12848738wrv.77.2024.04.10.08.35.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 08:35:24 -0700 (PDT) From: Richard Gobert To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, willemdebruijn.kernel@gmail.com, shuah@kernel.org, dsahern@kernel.org, aduyck@mirantis.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Richard Gobert Subject: [PATCH net-next v6 2/6] net: gro: add p_off param in *_gro_complete Date: Wed, 10 Apr 2024 17:34:19 +0200 Message-Id: <20240410153423.107381-3-richardbgobert@gmail.com> In-Reply-To: <20240410153423.107381-1-richardbgobert@gmail.com> References: <20240410153423.107381-1-richardbgobert@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Commits a602456 ("udp: Add GRO functions to UDP socket") and 57c67ff ("udp: additional GRO support") introduce incorrect usage of {ip,ipv6}_hdr in the complete phase of gro. The functions always return skb->network_header, which in the case of encapsulated packets at the gro complete phase, is always set to the innermost L3 of the packet. That means that calling {ip,ipv6}_hdr for skbs which completed the GRO receive phase (both in gro_list and *_gro_complete) when parsing an encapsulated packet's _outer_ L3/L4 may return an unexpected value. This incorrect usage leads to a bug in GRO's UDP socket lookup. udp{4,6}_lib_lookup_skb functions use ip_hdr/ipv6_hdr respectively. These *_hdr functions return network_header which will point to the innermost L3, resulting in the wrong offset being used in __udp{4,6}_lib_lookup with encapsulated packets. To fix this issue p_off param is used in *_gro_complete to pass off the offset of the previous layer. Reproduction example: Endpoint configuration example (fou + local address bind) # ip fou add port 6666 ipproto 4 # ip link add name tun1 type ipip remote 2.2.2.1 local 2.2.2.2 encap fou encap-dport 5555 encap-sport 6666 mode ipip # ip link set tun1 up # ip a add 1.1.1.2/24 dev tun1 Netperf TCP_STREAM result on net-next before patch is applied: net-next main, GRO enabled: $ netperf -H 1.1.1.2 -t TCP_STREAM -l 5 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 131072 16384 16384 5.28 2.37 net-next main, GRO disabled: $ netperf -H 1.1.1.2 -t TCP_STREAM -l 5 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 131072 16384 16384 5.01 2745.06 patch applied, GRO enabled: $ netperf -H 1.1.1.2 -t TCP_STREAM -l 5 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 131072 16384 16384 5.01 2877.38 Fixes: 57c67ff4bd92 ("udp: additional GRO support") Suggested-by: Eric Dumazet Signed-off-by: Richard Gobert --- drivers/net/geneve.c | 7 ++++--- drivers/net/vxlan/vxlan_core.c | 11 +++++++---- include/linux/etherdevice.h | 2 +- include/linux/netdevice.h | 3 ++- include/linux/udp.h | 2 +- include/net/gro.h | 11 ++++++----- include/net/inet_common.h | 2 +- include/net/tcp.h | 6 ++++-- include/net/udp.h | 8 ++++---- include/net/udp_tunnel.h | 2 +- net/8021q/vlan_core.c | 4 ++-- net/core/gro.c | 2 +- net/ethernet/eth.c | 4 ++-- net/ipv4/af_inet.c | 8 ++++---- net/ipv4/fou_core.c | 9 +++++---- net/ipv4/gre_offload.c | 5 +++-- net/ipv4/tcp_offload.c | 7 ++++--- net/ipv4/udp.c | 3 ++- net/ipv4/udp_offload.c | 26 ++++++++++++++------------ net/ipv6/ip6_offload.c | 22 ++++++++++++---------- net/ipv6/tcpv6_offload.c | 7 ++++--- net/ipv6/udp.c | 3 ++- net/ipv6/udp_offload.c | 13 +++++++------ 23 files changed, 93 insertions(+), 74 deletions(-) diff --git a/drivers/net/geneve.c b/drivers/net/geneve.c index 163f94a5a58f..9c18a39b0d0c 100644 --- a/drivers/net/geneve.c +++ b/drivers/net/geneve.c @@ -555,7 +555,7 @@ static struct sk_buff *geneve_gro_receive(struct sock *sk, } static int geneve_gro_complete(struct sock *sk, struct sk_buff *skb, - int nhoff) + int p_off, int nhoff) { struct genevehdr *gh; struct packet_offload *ptype; @@ -569,11 +569,12 @@ static int geneve_gro_complete(struct sock *sk, struct sk_buff *skb, /* since skb->encapsulation is set, eth_gro_complete() sets the inner mac header */ if (likely(type == htons(ETH_P_TEB))) - return eth_gro_complete(skb, nhoff + gh_len); + return eth_gro_complete(skb, p_off, nhoff + gh_len); ptype = gro_find_complete_by_type(type); if (ptype) - err = ptype->callbacks.gro_complete(skb, nhoff + gh_len); + err = ptype->callbacks.gro_complete(skb, p_off, + nhoff + gh_len); skb_set_inner_mac_header(skb, nhoff + gh_len); diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c index 72ecf6cf809c..6fb182d9d6e7 100644 --- a/drivers/net/vxlan/vxlan_core.c +++ b/drivers/net/vxlan/vxlan_core.c @@ -767,15 +767,17 @@ static struct sk_buff *vxlan_gpe_gro_receive(struct sock *sk, return pp; } -static int vxlan_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff) +static int vxlan_gro_complete(struct sock *sk, struct sk_buff *skb, + int p_off, int nhoff) { /* Sets 'skb->inner_mac_header' since we are always called with * 'skb->encapsulation' set. */ - return eth_gro_complete(skb, nhoff + sizeof(struct vxlanhdr)); + return eth_gro_complete(skb, p_off, nhoff + sizeof(struct vxlanhdr)); } -static int vxlan_gpe_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff) +static int vxlan_gpe_gro_complete(struct sock *sk, struct sk_buff *skb, + int p_off, int nhoff) { struct vxlanhdr *vh = (struct vxlanhdr *)(skb->data + nhoff); const struct packet_offload *ptype; @@ -786,7 +788,8 @@ static int vxlan_gpe_gro_complete(struct sock *sk, struct sk_buff *skb, int nhof return err; ptype = gro_find_complete_by_type(protocol); if (ptype) - err = ptype->callbacks.gro_complete(skb, nhoff + sizeof(struct vxlanhdr)); + err = ptype->callbacks.gro_complete(skb, p_off, nhoff + + sizeof(struct vxlanhdr)); return err; } diff --git a/include/linux/etherdevice.h b/include/linux/etherdevice.h index 224645f17c33..b081b43d9686 100644 --- a/include/linux/etherdevice.h +++ b/include/linux/etherdevice.h @@ -64,7 +64,7 @@ struct net_device *devm_alloc_etherdev_mqs(struct device *dev, int sizeof_priv, #define devm_alloc_etherdev(dev, sizeof_priv) devm_alloc_etherdev_mqs(dev, sizeof_priv, 1, 1) struct sk_buff *eth_gro_receive(struct list_head *head, struct sk_buff *skb); -int eth_gro_complete(struct sk_buff *skb, int nhoff); +int eth_gro_complete(struct sk_buff *skb, int p_off, int nhoff); /* Reserved Ethernet Addresses per IEEE 802.1Q */ static const u8 eth_reserved_addr_base[ETH_ALEN] __aligned(2) = diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index d45f330d083d..daff430d4d26 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -2710,7 +2710,8 @@ struct offload_callbacks { netdev_features_t features); struct sk_buff *(*gro_receive)(struct list_head *head, struct sk_buff *skb); - int (*gro_complete)(struct sk_buff *skb, int nhoff); + int (*gro_complete)(struct sk_buff *skb, int nhoff, + int thoff); }; struct packet_offload { diff --git a/include/linux/udp.h b/include/linux/udp.h index 17539d089666..ffb6af9f9404 100644 --- a/include/linux/udp.h +++ b/include/linux/udp.h @@ -82,7 +82,7 @@ struct udp_sock { struct sk_buff *skb); int (*gro_complete)(struct sock *sk, struct sk_buff *skb, - int nhoff); + int nhoff, int thoff); /* udp_recvmsg try to use this before splicing sk_receive_queue */ struct sk_buff_head reader_queue ____cacheline_aligned_in_smp; diff --git a/include/net/gro.h b/include/net/gro.h index 50f1e403dbbb..ebead1d642b4 100644 --- a/include/net/gro.h +++ b/include/net/gro.h @@ -383,18 +383,18 @@ static inline void skb_gro_flush_final_remcsum(struct sk_buff *skb, INDIRECT_CALLABLE_DECLARE(struct sk_buff *ipv6_gro_receive(struct list_head *, struct sk_buff *)); -INDIRECT_CALLABLE_DECLARE(int ipv6_gro_complete(struct sk_buff *, int)); +INDIRECT_CALLABLE_DECLARE(int ipv6_gro_complete(struct sk_buff *, int, int)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *inet_gro_receive(struct list_head *, struct sk_buff *)); -INDIRECT_CALLABLE_DECLARE(int inet_gro_complete(struct sk_buff *, int)); +INDIRECT_CALLABLE_DECLARE(int inet_gro_complete(struct sk_buff *, int, int)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *udp4_gro_receive(struct list_head *, struct sk_buff *)); -INDIRECT_CALLABLE_DECLARE(int udp4_gro_complete(struct sk_buff *, int)); +INDIRECT_CALLABLE_DECLARE(int udp4_gro_complete(struct sk_buff *, int, int)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *udp6_gro_receive(struct list_head *, struct sk_buff *)); -INDIRECT_CALLABLE_DECLARE(int udp6_gro_complete(struct sk_buff *, int)); +INDIRECT_CALLABLE_DECLARE(int udp6_gro_complete(struct sk_buff *, int, int)); #define indirect_call_gro_receive_inet(cb, f2, f1, head, skb) \ ({ \ @@ -405,7 +405,8 @@ INDIRECT_CALLABLE_DECLARE(int udp6_gro_complete(struct sk_buff *, int)); struct sk_buff *udp_gro_receive(struct list_head *head, struct sk_buff *skb, struct udphdr *uh, struct sock *sk); -int udp_gro_complete(struct sk_buff *skb, int nhoff, udp_lookup_t lookup); +int udp_gro_complete(struct sk_buff *skb, int nhoff, int thoff, + udp_lookup_t lookup); static inline struct udphdr *udp_gro_udphdr(struct sk_buff *skb) { diff --git a/include/net/inet_common.h b/include/net/inet_common.h index f50a644d87a9..605f917c830c 100644 --- a/include/net/inet_common.h +++ b/include/net/inet_common.h @@ -64,7 +64,7 @@ int inet_recv_error(struct sock *sk, struct msghdr *msg, int len, int *addr_len); struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb); -int inet_gro_complete(struct sk_buff *skb, int nhoff); +int inet_gro_complete(struct sk_buff *skb, int nhoff, int thoff); struct sk_buff *inet_gso_segment(struct sk_buff *skb, netdev_features_t features); diff --git a/include/net/tcp.h b/include/net/tcp.h index 9ab5b37e9d53..2dd45c1dfa7d 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -2196,9 +2196,11 @@ void tcp_v4_destroy_sock(struct sock *sk); struct sk_buff *tcp_gso_segment(struct sk_buff *skb, netdev_features_t features); struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb); -INDIRECT_CALLABLE_DECLARE(int tcp4_gro_complete(struct sk_buff *skb, int thoff)); +INDIRECT_CALLABLE_DECLARE(int tcp4_gro_complete(struct sk_buff *skb, int nhoff, + int thoff)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)); -INDIRECT_CALLABLE_DECLARE(int tcp6_gro_complete(struct sk_buff *skb, int thoff)); +INDIRECT_CALLABLE_DECLARE(int tcp6_gro_complete(struct sk_buff *skb, int nhoff, + int thoff)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb)); #ifdef CONFIG_INET void tcp_gro_complete(struct sk_buff *skb); diff --git a/include/net/udp.h b/include/net/udp.h index 488a6d2babcc..601d1c3b677a 100644 --- a/include/net/udp.h +++ b/include/net/udp.h @@ -166,8 +166,8 @@ static inline void udp_csum_pull_header(struct sk_buff *skb) UDP_SKB_CB(skb)->cscov -= sizeof(struct udphdr); } -typedef struct sock *(*udp_lookup_t)(const struct sk_buff *skb, __be16 sport, - __be16 dport); +typedef struct sock *(*udp_lookup_t)(const struct sk_buff *skb, int nhoff, + __be16 sport, __be16 dport); void udp_v6_early_demux(struct sk_buff *skb); INDIRECT_CALLABLE_DECLARE(int udpv6_rcv(struct sk_buff *)); @@ -301,7 +301,7 @@ struct sock *udp4_lib_lookup(struct net *net, __be32 saddr, __be16 sport, struct sock *__udp4_lib_lookup(struct net *net, __be32 saddr, __be16 sport, __be32 daddr, __be16 dport, int dif, int sdif, struct udp_table *tbl, struct sk_buff *skb); -struct sock *udp4_lib_lookup_skb(const struct sk_buff *skb, +struct sock *udp4_lib_lookup_skb(const struct sk_buff *skb, int nhoff, __be16 sport, __be16 dport); struct sock *udp6_lib_lookup(struct net *net, const struct in6_addr *saddr, __be16 sport, @@ -312,7 +312,7 @@ struct sock *__udp6_lib_lookup(struct net *net, const struct in6_addr *daddr, __be16 dport, int dif, int sdif, struct udp_table *tbl, struct sk_buff *skb); -struct sock *udp6_lib_lookup_skb(const struct sk_buff *skb, +struct sock *udp6_lib_lookup_skb(const struct sk_buff *skb, int nhoff, __be16 sport, __be16 dport); int udp_read_skb(struct sock *sk, skb_read_actor_t recv_actor); diff --git a/include/net/udp_tunnel.h b/include/net/udp_tunnel.h index a93dc51f6323..ffedf2121532 100644 --- a/include/net/udp_tunnel.h +++ b/include/net/udp_tunnel.h @@ -75,7 +75,7 @@ typedef struct sk_buff *(*udp_tunnel_gro_receive_t)(struct sock *sk, struct list_head *head, struct sk_buff *skb); typedef int (*udp_tunnel_gro_complete_t)(struct sock *sk, struct sk_buff *skb, - int nhoff); + int nhoff, int thoff); struct udp_tunnel_sock_cfg { void *sk_user_data; /* user data used by encap_rcv call back */ diff --git a/net/8021q/vlan_core.c b/net/8021q/vlan_core.c index f00158234505..247704cf70af 100644 --- a/net/8021q/vlan_core.c +++ b/net/8021q/vlan_core.c @@ -510,7 +510,7 @@ static struct sk_buff *vlan_gro_receive(struct list_head *head, return pp; } -static int vlan_gro_complete(struct sk_buff *skb, int nhoff) +static int vlan_gro_complete(struct sk_buff *skb, int p_off, int nhoff) { struct vlan_hdr *vhdr = (struct vlan_hdr *)(skb->data + nhoff); __be16 type = vhdr->h_vlan_encapsulated_proto; @@ -521,7 +521,7 @@ static int vlan_gro_complete(struct sk_buff *skb, int nhoff) if (ptype) err = INDIRECT_CALL_INET(ptype->callbacks.gro_complete, ipv6_gro_complete, inet_gro_complete, - skb, nhoff + sizeof(*vhdr)); + skb, p_off, nhoff + sizeof(*vhdr)); return err; } diff --git a/net/core/gro.c b/net/core/gro.c index 83f35d99a682..b129cd201937 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -252,7 +252,7 @@ static void napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb) err = INDIRECT_CALL_INET(ptype->callbacks.gro_complete, ipv6_gro_complete, inet_gro_complete, - skb, 0); + skb, 0, 0); break; } rcu_read_unlock(); diff --git a/net/ethernet/eth.c b/net/ethernet/eth.c index 2edc8b796a4e..7515e6bcbb7d 100644 --- a/net/ethernet/eth.c +++ b/net/ethernet/eth.c @@ -453,7 +453,7 @@ struct sk_buff *eth_gro_receive(struct list_head *head, struct sk_buff *skb) } EXPORT_SYMBOL(eth_gro_receive); -int eth_gro_complete(struct sk_buff *skb, int nhoff) +int eth_gro_complete(struct sk_buff *skb, int p_off, int nhoff) { struct ethhdr *eh = (struct ethhdr *)(skb->data + nhoff); __be16 type = eh->h_proto; @@ -467,7 +467,7 @@ int eth_gro_complete(struct sk_buff *skb, int nhoff) if (ptype != NULL) err = INDIRECT_CALL_INET(ptype->callbacks.gro_complete, ipv6_gro_complete, inet_gro_complete, - skb, nhoff + sizeof(*eh)); + skb, p_off, nhoff + sizeof(*eh)); return err; } diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index 55bd72997b31..5a747d91cd0a 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -1641,7 +1641,7 @@ int inet_recv_error(struct sock *sk, struct msghdr *msg, int len, int *addr_len) } EXPORT_SYMBOL(inet_recv_error); -int inet_gro_complete(struct sk_buff *skb, int nhoff) +int inet_gro_complete(struct sk_buff *skb, int prior_off, int nhoff) { struct iphdr *iph = (struct iphdr *)(skb->data + nhoff); const struct net_offload *ops; @@ -1667,17 +1667,17 @@ int inet_gro_complete(struct sk_buff *skb, int nhoff) */ err = INDIRECT_CALL_2(ops->callbacks.gro_complete, tcp4_gro_complete, udp4_gro_complete, - skb, nhoff + sizeof(*iph)); + skb, nhoff, nhoff + sizeof(*iph)); out: return err; } -static int ipip_gro_complete(struct sk_buff *skb, int nhoff) +static int ipip_gro_complete(struct sk_buff *skb, int prior_off, int nhoff) { skb->encapsulation = 1; skb_shinfo(skb)->gso_type |= SKB_GSO_IPXIP4; - return inet_gro_complete(skb, nhoff); + return inet_gro_complete(skb, prior_off, nhoff); } int inet_ctl_sock_create(struct sock **sk, unsigned short family, diff --git a/net/ipv4/fou_core.c b/net/ipv4/fou_core.c index a8494f796dca..7cf214d0b96d 100644 --- a/net/ipv4/fou_core.c +++ b/net/ipv4/fou_core.c @@ -260,7 +260,7 @@ static struct sk_buff *fou_gro_receive(struct sock *sk, } static int fou_gro_complete(struct sock *sk, struct sk_buff *skb, - int nhoff) + int p_off, int nhoff) { const struct net_offload __rcu **offloads; u8 proto = fou_from_sock(sk)->protocol; @@ -272,7 +272,7 @@ static int fou_gro_complete(struct sock *sk, struct sk_buff *skb, if (WARN_ON(!ops || !ops->callbacks.gro_complete)) goto out; - err = ops->callbacks.gro_complete(skb, nhoff); + err = ops->callbacks.gro_complete(skb, p_off, nhoff); skb_set_inner_mac_header(skb, nhoff); @@ -445,7 +445,8 @@ static struct sk_buff *gue_gro_receive(struct sock *sk, return pp; } -static int gue_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff) +static int gue_gro_complete(struct sock *sk, struct sk_buff *skb, + int p_off, int nhoff) { struct guehdr *guehdr = (struct guehdr *)(skb->data + nhoff); const struct net_offload __rcu **offloads; @@ -480,7 +481,7 @@ static int gue_gro_complete(struct sock *sk, struct sk_buff *skb, int nhoff) if (WARN_ON(!ops || !ops->callbacks.gro_complete)) goto out; - err = ops->callbacks.gro_complete(skb, nhoff + guehlen); + err = ops->callbacks.gro_complete(skb, p_off, nhoff + guehlen); skb_set_inner_mac_header(skb, nhoff + guehlen); diff --git a/net/ipv4/gre_offload.c b/net/ipv4/gre_offload.c index 5028c72d494a..d4520c3f7c09 100644 --- a/net/ipv4/gre_offload.c +++ b/net/ipv4/gre_offload.c @@ -233,7 +233,7 @@ static struct sk_buff *gre_gro_receive(struct list_head *head, return pp; } -static int gre_gro_complete(struct sk_buff *skb, int nhoff) +static int gre_gro_complete(struct sk_buff *skb, int p_off, int nhoff) { struct gre_base_hdr *greh = (struct gre_base_hdr *)(skb->data + nhoff); struct packet_offload *ptype; @@ -253,7 +253,8 @@ static int gre_gro_complete(struct sk_buff *skb, int nhoff) ptype = gro_find_complete_by_type(type); if (ptype) - err = ptype->callbacks.gro_complete(skb, nhoff + grehlen); + err = ptype->callbacks.gro_complete(skb, p_off, + nhoff + grehlen); skb_set_inner_mac_header(skb, nhoff + grehlen); diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index fab0973f995b..7f045b881dd4 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -328,10 +328,11 @@ struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb) return tcp_gro_receive(head, skb); } -INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) +INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int nhoff, + int thoff) { - const struct iphdr *iph = ip_hdr(skb); - struct tcphdr *th = tcp_hdr(skb); + const struct iphdr *iph = (const struct iphdr *)(skb->data + nhoff); + struct tcphdr *th = (struct tcphdr *)(skb->data + thoff); th->check = ~tcp_v4_check(skb->len - thoff, iph->saddr, iph->daddr, 0); diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index 7613daa339b0..2c7cea1cbd7e 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -530,9 +530,10 @@ static inline struct sock *__udp4_lib_lookup_skb(struct sk_buff *skb, } struct sock *udp4_lib_lookup_skb(const struct sk_buff *skb, + int nhoff, __be16 sport, __be16 dport) { - const struct iphdr *iph = ip_hdr(skb); + const struct iphdr *iph = (const struct iphdr *)(skb->data + nhoff); struct net *net = dev_net(skb->dev); int iif, sdif; diff --git a/net/ipv4/udp_offload.c b/net/ipv4/udp_offload.c index 1f4e08f43c4b..ad4c88fe7d15 100644 --- a/net/ipv4/udp_offload.c +++ b/net/ipv4/udp_offload.c @@ -689,18 +689,19 @@ static int udp_gro_complete_segment(struct sk_buff *skb) return 0; } -int udp_gro_complete(struct sk_buff *skb, int nhoff, +int udp_gro_complete(struct sk_buff *skb, int nhoff, int thoff, udp_lookup_t lookup) { - __be16 newlen = htons(skb->len - nhoff); - struct udphdr *uh = (struct udphdr *)(skb->data + nhoff); + struct udphdr *uh = (struct udphdr *)(skb->data + thoff); + __be16 newlen = htons(skb->len - thoff); struct sock *sk; int err; uh->len = newlen; sk = INDIRECT_CALL_INET(lookup, udp6_lib_lookup_skb, - udp4_lib_lookup_skb, skb, uh->source, uh->dest); + udp4_lib_lookup_skb, skb, nhoff, uh->source, + uh->dest); if (sk && udp_sk(sk)->gro_complete) { skb_shinfo(skb)->gso_type = uh->check ? SKB_GSO_UDP_TUNNEL_CSUM : SKB_GSO_UDP_TUNNEL; @@ -714,8 +715,8 @@ int udp_gro_complete(struct sk_buff *skb, int nhoff, * functions to make them set up the inner offsets. */ skb->encapsulation = 1; - err = udp_sk(sk)->gro_complete(sk, skb, - nhoff + sizeof(struct udphdr)); + err = udp_sk(sk)->gro_complete(sk, skb, nhoff, + thoff + sizeof(struct udphdr)); } else { err = udp_gro_complete_segment(skb); } @@ -727,14 +728,15 @@ int udp_gro_complete(struct sk_buff *skb, int nhoff, } EXPORT_SYMBOL(udp_gro_complete); -INDIRECT_CALLABLE_SCOPE int udp4_gro_complete(struct sk_buff *skb, int nhoff) +INDIRECT_CALLABLE_SCOPE int udp4_gro_complete(struct sk_buff *skb, int nhoff, + int thoff) { - const struct iphdr *iph = ip_hdr(skb); - struct udphdr *uh = (struct udphdr *)(skb->data + nhoff); + const struct iphdr *iph = (const struct iphdr *)(skb->data + nhoff); + struct udphdr *uh = (struct udphdr *)(skb->data + thoff); /* do fraglist only if there is no outer UDP encap (or we already processed it) */ if (NAPI_GRO_CB(skb)->is_flist && !NAPI_GRO_CB(skb)->encap_mark) { - uh->len = htons(skb->len - nhoff); + uh->len = htons(skb->len - thoff); skb_shinfo(skb)->gso_type |= (SKB_GSO_FRAGLIST|SKB_GSO_UDP_L4); skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count; @@ -745,10 +747,10 @@ INDIRECT_CALLABLE_SCOPE int udp4_gro_complete(struct sk_buff *skb, int nhoff) } if (uh->check) - uh->check = ~udp_v4_check(skb->len - nhoff, iph->saddr, + uh->check = ~udp_v4_check(skb->len - thoff, iph->saddr, iph->daddr, 0); - return udp_gro_complete(skb, nhoff, udp4_lib_lookup_skb); + return udp_gro_complete(skb, nhoff, thoff, udp4_lib_lookup_skb); } int __init udpv4_offload_init(void) diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c index b41e35af69ea..10ddbbc0e46d 100644 --- a/net/ipv6/ip6_offload.c +++ b/net/ipv6/ip6_offload.c @@ -346,12 +346,14 @@ static struct sk_buff *ip4ip6_gro_receive(struct list_head *head, return inet_gro_receive(head, skb); } -INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) +INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, + int p_off, int nhoff) { const struct net_offload *ops; struct ipv6hdr *iph; int err = -ENOSYS; u32 payload_len; + int nhlen; if (skb->encapsulation) { skb_set_inner_protocol(skb, cpu_to_be16(ETH_P_IPV6)); @@ -387,36 +389,36 @@ INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) iph->payload_len = htons(payload_len); } - nhoff += sizeof(*iph) + ipv6_exthdrs_len(iph, &ops); - if (WARN_ON(!ops || !ops->callbacks.gro_complete)) + nhlen = sizeof(*iph) + ipv6_exthdrs_len(iph, &ops); + if (WARN_ON_ONCE(!ops || !ops->callbacks.gro_complete)) goto out; err = INDIRECT_CALL_L4(ops->callbacks.gro_complete, tcp6_gro_complete, - udp6_gro_complete, skb, nhoff); + udp6_gro_complete, skb, nhoff, nhoff + nhlen); out: return err; } -static int sit_gro_complete(struct sk_buff *skb, int nhoff) +static int sit_gro_complete(struct sk_buff *skb, int p_off, int nhoff) { skb->encapsulation = 1; skb_shinfo(skb)->gso_type |= SKB_GSO_IPXIP4; - return ipv6_gro_complete(skb, nhoff); + return ipv6_gro_complete(skb, p_off, nhoff); } -static int ip6ip6_gro_complete(struct sk_buff *skb, int nhoff) +static int ip6ip6_gro_complete(struct sk_buff *skb, int p_off, int nhoff) { skb->encapsulation = 1; skb_shinfo(skb)->gso_type |= SKB_GSO_IPXIP6; - return ipv6_gro_complete(skb, nhoff); + return ipv6_gro_complete(skb, p_off, nhoff); } -static int ip4ip6_gro_complete(struct sk_buff *skb, int nhoff) +static int ip4ip6_gro_complete(struct sk_buff *skb, int p_off, int nhoff) { skb->encapsulation = 1; skb_shinfo(skb)->gso_type |= SKB_GSO_IPXIP6; - return inet_gro_complete(skb, nhoff); + return inet_gro_complete(skb, p_off, nhoff); } diff --git a/net/ipv6/tcpv6_offload.c b/net/ipv6/tcpv6_offload.c index 4b07d1e6c952..91a55372dcd4 100644 --- a/net/ipv6/tcpv6_offload.c +++ b/net/ipv6/tcpv6_offload.c @@ -27,10 +27,11 @@ struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb) return tcp_gro_receive(head, skb); } -INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff) +INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, + int nhoff, int thoff) { - const struct ipv6hdr *iph = ipv6_hdr(skb); - struct tcphdr *th = tcp_hdr(skb); + const struct ipv6hdr *iph = (const struct ipv6hdr *)(skb->data + nhoff); + struct tcphdr *th = (struct tcphdr *)(skb->data + thoff); th->check = ~tcp_v6_check(skb->len - thoff, &iph->saddr, &iph->daddr, 0); diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index a9466f881f1b..fac6b6cd36e9 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -271,9 +271,10 @@ static struct sock *__udp6_lib_lookup_skb(struct sk_buff *skb, } struct sock *udp6_lib_lookup_skb(const struct sk_buff *skb, + int nhoff, __be16 sport, __be16 dport) { - const struct ipv6hdr *iph = ipv6_hdr(skb); + const struct ipv6hdr *iph = (const struct ipv6hdr *)(skb->data + nhoff); struct net *net = dev_net(skb->dev); int iif, sdif; diff --git a/net/ipv6/udp_offload.c b/net/ipv6/udp_offload.c index bbd347de00b4..078055665397 100644 --- a/net/ipv6/udp_offload.c +++ b/net/ipv6/udp_offload.c @@ -162,14 +162,15 @@ struct sk_buff *udp6_gro_receive(struct list_head *head, struct sk_buff *skb) return NULL; } -INDIRECT_CALLABLE_SCOPE int udp6_gro_complete(struct sk_buff *skb, int nhoff) +INDIRECT_CALLABLE_SCOPE int udp6_gro_complete(struct sk_buff *skb, int nhoff, + int thoff) { - const struct ipv6hdr *ipv6h = ipv6_hdr(skb); - struct udphdr *uh = (struct udphdr *)(skb->data + nhoff); + const struct ipv6hdr *ipv6h = (const struct ipv6hdr *)(skb->data + nhoff); + struct udphdr *uh = (struct udphdr *)(skb->data + thoff); /* do fraglist only if there is no outer UDP encap (or we already processed it) */ if (NAPI_GRO_CB(skb)->is_flist && !NAPI_GRO_CB(skb)->encap_mark) { - uh->len = htons(skb->len - nhoff); + uh->len = htons(skb->len - thoff); skb_shinfo(skb)->gso_type |= (SKB_GSO_FRAGLIST|SKB_GSO_UDP_L4); skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count; @@ -180,10 +181,10 @@ INDIRECT_CALLABLE_SCOPE int udp6_gro_complete(struct sk_buff *skb, int nhoff) } if (uh->check) - uh->check = ~udp_v6_check(skb->len - nhoff, &ipv6h->saddr, + uh->check = ~udp_v6_check(skb->len - thoff, &ipv6h->saddr, &ipv6h->daddr, 0); - return udp_gro_complete(skb, nhoff, udp6_lib_lookup_skb); + return udp_gro_complete(skb, nhoff, thoff, udp6_lib_lookup_skb); } int __init udpv6_offload_init(void) From patchwork Wed Apr 10 15:34:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Gobert X-Patchwork-Id: 13624762 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9C6B116F0EA; Wed, 10 Apr 2024 15:35:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763334; cv=none; b=DB1pdHMi9N/J66CMFfxaxk+iOdaXw1sz3blDjYQ56G2gF4gmaZiaMMAti+P5diiGdc1MPFgMZNVwnA1AwZX99rU2sN3rfds/vdpKyWrv8hB9NXJgVNEa3DqhbNAcSBPMXu8DMY7Nkt4LdOopXRpVWwt3GJAyyE5ilmIVbgkuaaw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763334; c=relaxed/simple; bh=56AsyIsWC5H0JL0LkAtUWQlFUGlkXpdQfBm/2Ob/Ksg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rhAwXw3Y3Zu+mcwFCyT9QJ+ZF7zzraIUta2smDcIbKH2TI60b24E6fPRKewuS2okUEvJm7XXIrQqLfBWIWwj311J74OSdFO6jWfSRbDVaJw4kDo+00MiZM85P/d93AkcNQTccej3JbDiw+QkYwEYoyNWm8cGcnhAAM43rJCxrLA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=N3A5tOq9; arc=none smtp.client-ip=209.85.128.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="N3A5tOq9" Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-4167fce0a41so18003545e9.0; Wed, 10 Apr 2024 08:35:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712763331; x=1713368131; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YB+sQJeuFBV/HMhVbtgA8BmimTyrV33+UXXBE7KSJuo=; b=N3A5tOq9/nediaNh+/qoZM/mIxP5PlN/lDKRY/heOHF3bX0DTW28cRli9S2YesTb5t F6tobDv5E+OVo6tvTsVaysJEpPrZpWq/bUKx5mE+m+j0Zl3iI/qvLmg0VAw3OoU5uvW4 O3JQrMwyDPcd3Bm+hjTC5wrtpM5CZ7PK1rqxWF41LthKN11p942m7fUpycfiLGLPnbTm bSl1TphG94Gml3TqldvPLrlmuft/cQzi51kRQLpzHyxbG6QXQiazwjNVtF78JGO7zy9H uvbMcjiSfhmPJ0wzmBMHI2GYlEaUqOFuomB0XyGs8ZmF5f5FGZn2QTS5gRxKQlyN//t2 T2JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712763331; x=1713368131; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YB+sQJeuFBV/HMhVbtgA8BmimTyrV33+UXXBE7KSJuo=; b=UHxklMtXTcypxDcjhhyE54TjAadPWZ6tGzjfwb3Yz8Z2WBJbtmF9Ud/h7GtU3VxE56 03bU1Jvitn8aU0U/rc2BCXxlA2OONDPuQYfSvQbPbd3ZRRWw9lt/TamKhxSTCigcWdo8 9G5GIxToq/VyYP9qM11DhTD8OMEXO7PYXpbccZUcPF3WlNSC79TitYpsdG6wozA+xeeb qVy6D9C1k5Vjt70zohCPaadJWcZ33Mp/kB2DXFhmyd7OCk8xhPVXQS58EZvwMgaau+oa pdN+j95RY5o7qgTvJvRk05em5FSdimj6PBNxJzM3V+D9nb7MTBmFnsdHBZz61clv7cI0 kcTg== X-Forwarded-Encrypted: i=1; AJvYcCWIcQuETvXZ+tjet3Bdm045Z/Y913ClFiU5H2bUdDCnaVlW8DC4I3ZbJxPcbCQrVh0Jb41+L/jgq5+McA6iwi3EnigZcpWmRsg1MU2N+43iHQWPDCRbC+ZfkaiQZv8oHiVthO+/w/xpyYdaVuLa39aPmNl5tjyWpOgshlb61svJjm0NtrIW X-Gm-Message-State: AOJu0Yy3ix9jJkT59CANBxkFvdzha3d3z9aJ1B8Rap+kZSOa5l9CSqzj YBDIf5Td3RMpKIJWw1g44Ew4KV3wHQQfk7Hl1Weis+pXP4VDGJgD X-Google-Smtp-Source: AGHT+IF3QnEZThB8P2DtBhWrRps8ug/RqQ/K7aYUrZf+/dfCJ17PbmCFm1GoTRpQfLx4ODCTDKPw9Q== X-Received: by 2002:a05:6000:4595:b0:343:b5dd:fe1d with SMTP id gb21-20020a056000459500b00343b5ddfe1dmr4881133wrb.28.1712763330629; Wed, 10 Apr 2024 08:35:30 -0700 (PDT) Received: from localhost ([146.70.204.204]) by smtp.gmail.com with ESMTPSA id dm14-20020a0560000bce00b003437a76565asm13963985wrb.25.2024.04.10.08.35.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 08:35:30 -0700 (PDT) From: Richard Gobert To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, willemdebruijn.kernel@gmail.com, shuah@kernel.org, dsahern@kernel.org, aduyck@mirantis.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Richard Gobert Subject: [PATCH net-next v6 3/6] selftests/net: add local address bind in vxlan selftest Date: Wed, 10 Apr 2024 17:34:20 +0200 Message-Id: <20240410153423.107381-4-richardbgobert@gmail.com> In-Reply-To: <20240410153423.107381-1-richardbgobert@gmail.com> References: <20240410153423.107381-1-richardbgobert@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Add local address bind support to existing udpgro_fwd.sh vxlan selftest, to ensure UDP socket lookup in GRO is working. Signed-off-by: Richard Gobert --- tools/testing/selftests/net/udpgro_fwd.sh | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/net/udpgro_fwd.sh b/tools/testing/selftests/net/udpgro_fwd.sh index 83ed987cff34..52c2de895abe 100755 --- a/tools/testing/selftests/net/udpgro_fwd.sh +++ b/tools/testing/selftests/net/udpgro_fwd.sh @@ -62,11 +62,13 @@ create_vxlan_endpoint() { local -r bm_rem_addr=$3 local -r vxlan_dev=$4 local -r vxlan_id=$5 + local -r bm_local_addr=$6 local -r vxlan_port=4789 ip -n $netns link set dev $bm_dev up ip -n $netns link add dev $vxlan_dev type vxlan id $vxlan_id \ - dstport $vxlan_port remote $bm_rem_addr + dstport $vxlan_port local $bm_local_addr \ + remote $bm_rem_addr ip -n $netns link set dev $vxlan_dev up } @@ -77,11 +79,13 @@ create_vxlan_pair() { for ns in $SRC $DST; do # note that 3 - $SRC == $DST and 3 - $DST == $SRC - create_vxlan_endpoint $BASE$ns veth$ns $BM_NET_V4$((3 - $ns)) vxlan$ns 4 + create_vxlan_endpoint $BASE$ns veth$ns $BM_NET_V4$((3 - $ns)) \ + vxlan$ns 4 $BM_NET_V4$ns ip -n $BASE$ns addr add dev vxlan$ns $OL_NET_V4$ns/24 done for ns in $SRC $DST; do - create_vxlan_endpoint $BASE$ns veth$ns $BM_NET_V6$((3 - $ns)) vxlan6$ns 6 + create_vxlan_endpoint $BASE$ns veth$ns $BM_NET_V6$((3 - $ns)) \ + vxlan6$ns 6 $BM_NET_V6$ns ip -n $BASE$ns addr add dev vxlan6$ns $OL_NET_V6$ns/24 nodad done From patchwork Wed Apr 10 15:34:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Gobert X-Patchwork-Id: 13624763 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0EF116F0D8; Wed, 10 Apr 2024 15:35:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763340; cv=none; b=bs4eMYIeaqOY1jrIA9EB2P790FtFzkOeiPz2sy2YahXP+2mj61lNs2a7+MjhJ/Wj+y45GjzT7PUza8+4eZ5ptdxB9BVrYNtNSseg0/H7KLPTcFh7WOSnZV9HRJCjRg4BEaV7NR/eqFK8/TZzFZfy/rgG5iLwH9oVxKjiWdWRyc0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763340; c=relaxed/simple; bh=pD4qzKqcpUuSaHV7ouYV9btcuo6kjLpBp9xdnY0xgig=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gecIRqwSvRkTRBMZocWyQIJfytsByxplMEYC2A22BB2WWNBrTovLTD7m82QxHU9bLsD8asC4uC/QVFgjnowEl/pQUFDkz4gMWLvuiq9yXyAHjjgHaEaTlKcPx8USSjKIsyYZMi4a7LVQMihv1ANsEp2jhrbtFQOoVA04IYb06Jw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YzBCYgFT; arc=none smtp.client-ip=209.85.128.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YzBCYgFT" Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-417c5763d3fso2136175e9.1; Wed, 10 Apr 2024 08:35:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712763337; x=1713368137; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D4x4IBe/lFW1aP700dLN9QduvfOFnUhVQ5zJ3YbImhg=; b=YzBCYgFT8Rgew9SntbVqNCtr9z0aEDyrOmJocber1OxsK3bbR2VWqX06uNeEkvvtM9 6Iq4q+TpHEL91sNXx6YRE45YhJTu5u3m2jf6iAyl7PF8I5FnAYvUtuyfIZ7ZF181gdes VtUs+9vGBx1izb3UwTUeZ1b7kN1ormIghM5QlxTJOlrtzV0j7XItJ6TtKNBfZTaj2nCp nF36CFW/sFUr5FG2OOuyjdAo5xfvSosPFuL/u9p1NVmuaOp/eUGTn2gp+llEkDk4AocV wLhAN3ZX1Bj7zSQEWrXMfrnp0iZ3p+A0emQ71Z0okm6TqKC/9SGJVuca0avJWYCy6+kD y6+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712763337; x=1713368137; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D4x4IBe/lFW1aP700dLN9QduvfOFnUhVQ5zJ3YbImhg=; b=gylwuf9i4ciLY0khRpJrnXMn8Dq2E8zWW1DiPDH9oPmOWuXgx2JMGtVaYiNw1oFAyw X6kDVQijg/7R+dzmJr7EAftYWzirsAOHukT6tv+NArk6iv0CqmSCE1W7KuIlckCOrmXO PfCu1n3Hb7qEppbPSSLEG5Zin163T6q7zydsqoorqZvKXXk+CbW/BZ9G7eAz5Fw1riNH hnddw2p8OXTUY7pvRUc9JFN1zSDjp+s89Qg61g6rbyYtmpu1iLI7jUl/PDbMh3t3C4RK FbsYbt5n+661tq2px8ArU86gh9rV2AiEQQJNdwveClQMTM7+yn+RiipECx1r546YOVVP 9xEA== X-Forwarded-Encrypted: i=1; AJvYcCU44n1kxE4N8j5U3SIdZ3eb7WNoC0i1Oc7p8lViakFjh+Kp3QygT/M7tXikuT47hGp/GKNvi2/aremdR9TLiuabY7mWix47h9GvlVr34di1SJLp5n6p/WVFk+/Z6+gk2Bem4SKFcWHSkQLF3icDnVmJbMmGfyMPiE4w+s2c6JIe+HuIPWVQ X-Gm-Message-State: AOJu0YzWRTEt86zjA1NZQnHuOIqpQiM3p0yrqcf1UcinC4YZBiXKnA3B pfH6dzSKxS5KmzMSE75gqrEPSdlMzNv6kOGKeLgAcyJGo58pLqbm X-Google-Smtp-Source: AGHT+IHoqFgdZSiILnYwhl9dCzr/qZl4oRB6baJ8FwkK1pf+Jc3CknSdJ+abTMN289u4B5jcrOJwoA== X-Received: by 2002:a05:600c:a4b:b0:414:726:87d9 with SMTP id c11-20020a05600c0a4b00b00414072687d9mr2852609wmq.12.1712763336943; Wed, 10 Apr 2024 08:35:36 -0700 (PDT) Received: from localhost ([146.70.204.204]) by smtp.gmail.com with ESMTPSA id f18-20020a05600c4e9200b0041632fcf272sm2630547wmq.22.2024.04.10.08.35.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 08:35:36 -0700 (PDT) From: Richard Gobert To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, willemdebruijn.kernel@gmail.com, shuah@kernel.org, dsahern@kernel.org, aduyck@mirantis.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Richard Gobert Subject: [PATCH net-next v6 4/6] net: gro: add {inner_}network_offset to napi_gro_cb Date: Wed, 10 Apr 2024 17:34:21 +0200 Message-Id: <20240410153423.107381-5-richardbgobert@gmail.com> In-Reply-To: <20240410153423.107381-1-richardbgobert@gmail.com> References: <20240410153423.107381-1-richardbgobert@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org This patch adds network_offset and inner_network_offset to napi_gro_cb, and makes sure both are set correctly. In the common path there's only one write (skb_gro_reset_offset). Signed-off-by: Richard Gobert --- drivers/net/geneve.c | 1 + drivers/net/vxlan/vxlan_core.c | 1 + include/net/gro.h | 18 ++++++++++++++++-- net/8021q/vlan_core.c | 2 ++ net/core/gro.c | 1 + net/ethernet/eth.c | 1 + net/ipv4/af_inet.c | 5 +---- net/ipv4/gre_offload.c | 1 + net/ipv6/ip6_offload.c | 8 ++++---- 9 files changed, 28 insertions(+), 10 deletions(-) diff --git a/drivers/net/geneve.c b/drivers/net/geneve.c index 9c18a39b0d0c..a6256ea1f5bc 100644 --- a/drivers/net/geneve.c +++ b/drivers/net/geneve.c @@ -545,6 +545,7 @@ static struct sk_buff *geneve_gro_receive(struct sock *sk, if (!ptype) goto out; + NAPI_GRO_CB(skb)->inner_network_offset = hlen; pp = call_gro_receive(ptype->callbacks.gro_receive, head, skb); flush = 0; diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c index 6fb182d9d6e7..9fb93c3953c1 100644 --- a/drivers/net/vxlan/vxlan_core.c +++ b/drivers/net/vxlan/vxlan_core.c @@ -754,6 +754,7 @@ static struct sk_buff *vxlan_gpe_gro_receive(struct sock *sk, vh = vxlan_gro_prepare_receive(sk, head, skb, &grc); if (vh) { + NAPI_GRO_CB(skb)->inner_network_offset = skb_gro_offset(skb); if (!vxlan_parse_gpe_proto(vh, &protocol)) goto out; ptype = gro_find_receive_by_type(protocol); diff --git a/include/net/gro.h b/include/net/gro.h index ebead1d642b4..a1cc8e8c2ebd 100644 --- a/include/net/gro.h +++ b/include/net/gro.h @@ -87,6 +87,15 @@ struct napi_gro_cb { /* used to support CHECKSUM_COMPLETE for tunneling protocols */ __wsum csum; + + /* L3 offsets */ + union { + struct { + u16 network_offset; + u16 inner_network_offset; + }; + u16 network_offsets[2]; + }; }; #define NAPI_GRO_CB(skb) ((struct napi_gro_cb *)(skb)->cb) @@ -172,12 +181,17 @@ static inline void *skb_gro_header(struct sk_buff *skb, unsigned int hlen, return ptr; } +static inline int skb_gro_network_offset(const struct sk_buff *skb) +{ + return NAPI_GRO_CB(skb)->network_offsets[NAPI_GRO_CB(skb)->encap_mark]; +} + static inline void *skb_gro_network_header(const struct sk_buff *skb) { if (skb_gro_may_pull(skb, skb_gro_offset(skb))) - return skb_gro_header_fast(skb, skb_network_offset(skb)); + return skb_gro_header_fast(skb, skb_gro_network_offset(skb)); - return skb_network_header(skb); + return skb->data + skb_gro_network_offset(skb); } static inline __wsum inet_gro_compute_pseudo(const struct sk_buff *skb, diff --git a/net/8021q/vlan_core.c b/net/8021q/vlan_core.c index 247704cf70af..355cafe23329 100644 --- a/net/8021q/vlan_core.c +++ b/net/8021q/vlan_core.c @@ -478,6 +478,8 @@ static struct sk_buff *vlan_gro_receive(struct list_head *head, if (unlikely(!vhdr)) goto out; + NAPI_GRO_CB(skb)->network_offsets[NAPI_GRO_CB(skb)->encap_mark] = hlen; + type = vhdr->h_vlan_encapsulated_proto; ptype = gro_find_receive_by_type(type); diff --git a/net/core/gro.c b/net/core/gro.c index b129cd201937..b2156e6cc4ad 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -371,6 +371,7 @@ static inline void skb_gro_reset_offset(struct sk_buff *skb, u32 nhoff) const skb_frag_t *frag0; unsigned int headlen; + NAPI_GRO_CB(skb)->network_offset = 0; NAPI_GRO_CB(skb)->data_offset = 0; headlen = skb_headlen(skb); NAPI_GRO_CB(skb)->frag0 = skb->data; diff --git a/net/ethernet/eth.c b/net/ethernet/eth.c index 7515e6bcbb7d..e3eca605bcc7 100644 --- a/net/ethernet/eth.c +++ b/net/ethernet/eth.c @@ -441,6 +441,7 @@ struct sk_buff *eth_gro_receive(struct list_head *head, struct sk_buff *skb) skb_gro_pull(skb, sizeof(*eh)); skb_gro_postpull_rcsum(skb, eh, sizeof(*eh)); + NAPI_GRO_CB(skb)->inner_network_offset = hlen; pp = indirect_call_gro_receive_inet(ptype->callbacks.gro_receive, ipv6_gro_receive, inet_gro_receive, diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index 5a747d91cd0a..6546bf376b24 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -1568,10 +1568,6 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb) NAPI_GRO_CB(skb)->is_atomic = !!(iph->frag_off & htons(IP_DF)); NAPI_GRO_CB(skb)->flush |= flush; - skb_set_network_header(skb, off); - /* The above will be needed by the transport layer if there is one - * immediately following this IP hdr. - */ /* Note : No need to call skb_gro_postpull_rcsum() here, * as we already checked checksum over ipv4 header was 0 @@ -1597,6 +1593,7 @@ static struct sk_buff *ipip_gro_receive(struct list_head *head, } NAPI_GRO_CB(skb)->encap_mark = 1; + NAPI_GRO_CB(skb)->inner_network_offset = skb_gro_offset(skb); return inet_gro_receive(head, skb); } diff --git a/net/ipv4/gre_offload.c b/net/ipv4/gre_offload.c index d4520c3f7c09..ae596285d78c 100644 --- a/net/ipv4/gre_offload.c +++ b/net/ipv4/gre_offload.c @@ -224,6 +224,7 @@ static struct sk_buff *gre_gro_receive(struct list_head *head, /* Adjusted NAPI_GRO_CB(skb)->csum after skb_gro_pull()*/ skb_gro_postpull_rcsum(skb, greh, grehlen); + NAPI_GRO_CB(skb)->inner_network_offset = hlen; pp = call_gro_receive(ptype->callbacks.gro_receive, head, skb); flush = 0; diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c index 10ddbbc0e46d..ba41939537f2 100644 --- a/net/ipv6/ip6_offload.c +++ b/net/ipv6/ip6_offload.c @@ -67,7 +67,7 @@ static int ipv6_gro_pull_exthdrs(struct sk_buff *skb, int off, int proto) off += len; } - skb_gro_pull(skb, off - skb_network_offset(skb)); + skb_gro_pull(skb, off - skb_gro_network_offset(skb)); return proto; } @@ -236,8 +236,6 @@ INDIRECT_CALLABLE_SCOPE struct sk_buff *ipv6_gro_receive(struct list_head *head, if (unlikely(!iph)) goto out; - skb_set_network_header(skb, off); - flush += ntohs(iph->payload_len) != skb->len - hlen; proto = iph->nexthdr; @@ -259,7 +257,7 @@ INDIRECT_CALLABLE_SCOPE struct sk_buff *ipv6_gro_receive(struct list_head *head, NAPI_GRO_CB(skb)->proto = proto; flush--; - nlen = skb_network_header_len(skb); + nlen = skb_gro_offset(skb) - off; list_for_each_entry(p, head, list) { const struct ipv6hdr *iph2; @@ -327,6 +325,7 @@ static struct sk_buff *sit_ip6ip6_gro_receive(struct list_head *head, } NAPI_GRO_CB(skb)->encap_mark = 1; + NAPI_GRO_CB(skb)->inner_network_offset = skb_gro_offset(skb); return ipv6_gro_receive(head, skb); } @@ -342,6 +341,7 @@ static struct sk_buff *ip4ip6_gro_receive(struct list_head *head, } NAPI_GRO_CB(skb)->encap_mark = 1; + NAPI_GRO_CB(skb)->inner_network_offset = skb_gro_offset(skb); return inet_gro_receive(head, skb); } From patchwork Wed Apr 10 15:34:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Gobert X-Patchwork-Id: 13624764 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0ADE8176FC8; Wed, 10 Apr 2024 15:35:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763350; cv=none; b=G7tH9kIXkJt4mQqntJze2j0GR61tDCIq9dB5BPj6nkKYHXcYT0Nwot6BmYYRTST+X/WJ2X6krkk3QS24i2Sv7cJUTimHcsd0byhW0kc/rtbSCADRbQgHmgzRgtmIkApzjHGLiy31dC774y2DVkmdIdULTqTG2T/B19zNKx/dTho= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763350; c=relaxed/simple; bh=Hr8YHaTGoei0aoyswr+qYCaHrLJXYO3y1I6JtRFpITM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GWu7TSVRBqFtNO3NhS/Yg/Q/CEz0XSACoXQ4YfP3u1KLatcJuQsd7TRGnWrfUa6HYAuHi1b30145DodJQsXrDnOwp4i81BOWp+eVvvhu+5IQ9oV+PxIeK9Tgxjkl+IdjZJZ+tHYfDs5pzMbpi9IApwvfQDxdhn68EQogD9C4NmE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dACZU0kH; arc=none smtp.client-ip=209.85.128.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dACZU0kH" Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-416422e2acbso22711555e9.1; Wed, 10 Apr 2024 08:35:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712763346; x=1713368146; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/pudm4Qj5I3CWyz7oEzjDZpZS7D4UqigIUQcMjBXv8Q=; b=dACZU0kHgRumA94XMsdKP+aONtozt5e0nccS8cLUaAMVYXG3kd6R9KdQfn3qbPMj8N TAdsUpKew9FZ89eRxU/pJV0wGnTUHP8a97rvchwkQXtvi1ppw2efBN7H52qtwkG2H5SI o/zU0FjtyCRB7pE3f++wRhZbLbwEuIcY0WP8PaKI8BLp5jqVThkj8DOZm4lz872Rltcl DrtS2j3resaWQkhKpjHFiTdf87F1/hlL2gHNltKcdqZuiqBHX+w7/DLoZGb9xO2nB4nH 7TVnhMotD5aaIxjka/d7NDujhd6Us8GPx34x+G9xqpRM8SOYcVJJonJmniTDTvjyaMKR 4/lA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712763346; x=1713368146; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/pudm4Qj5I3CWyz7oEzjDZpZS7D4UqigIUQcMjBXv8Q=; b=OMYHATLSz0UkD8rPrkdazsoWOSEUIn+agmK1HcayM9k0uHqc3VShLQH3MmaoTwQNI+ TK5t2a1pXfGItY7SNqNOIOP6TTx09Wdqn9IbQRJnsebmgcYG4lX71i94X/nDsx6ayqOE d3nhM43bw/YuFYceRJ/ZktirjNQhPVGk+CPQJ+HVnlfgKUIY2i9ap7A6nbGJL5c+rK/Q sLpnxACejd/8M2y4p3bf6RpkRuCsCgd08zKNOq0tvx3JEmD8M5Rz550P4SX0p1CTYIGG zT0w995GnWpBr/i50FiDe3ZYG9PBDNnz7z9S5EtcIQzlicJ7dsQsNNNdwMmtqsS9R3x3 ewog== X-Forwarded-Encrypted: i=1; AJvYcCVJ+LBNJnW3wx/WiO3SZDRXNB0y4QL7Cgu9VV8bbGMCXSs3QH1gjHLJvYl+4oOVRA8eqSw1LmDU+94PstFs7KrJP+g2Vz+d/Sbn23f5YVIV/uGg/S8QbHZKuu5PSjw3GG0QVXXKfzIvFCRRPg+dhUxZuUtRl2iI6iO1pqRLi6uEtzLwxval X-Gm-Message-State: AOJu0YxfHpqNGoDfjwC3byTuWNPsNffLZtq7/C84oJ/ziPhOOncbpvv8 a8NZdgjMNqtxQ51XSz/TOKuVN8YvGHyhVtH7k7FG/cM6QVRAzDqFV3MU0dIQ X-Google-Smtp-Source: AGHT+IEcT2y6TucCnrc7G1/FcafJ6/+NkPKgE76AKprrrCrG+mKW6tYWsJVBdVL8X9+8AuZ0knLMmw== X-Received: by 2002:a05:600c:1c04:b0:416:350c:db48 with SMTP id j4-20020a05600c1c0400b00416350cdb48mr10696wms.9.1712763346270; Wed, 10 Apr 2024 08:35:46 -0700 (PDT) Received: from localhost ([146.70.204.204]) by smtp.gmail.com with ESMTPSA id d2-20020a05600c34c200b004155387c08esm2625240wmq.27.2024.04.10.08.35.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 08:35:46 -0700 (PDT) From: Richard Gobert To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, willemdebruijn.kernel@gmail.com, shuah@kernel.org, dsahern@kernel.org, aduyck@mirantis.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Richard Gobert Subject: [PATCH net-next v6 5/6] net: gro: move L3 flush checks to tcp_gro_receive and udp_gro_receive_segment Date: Wed, 10 Apr 2024 17:34:22 +0200 Message-Id: <20240410153423.107381-6-richardbgobert@gmail.com> In-Reply-To: <20240410153423.107381-1-richardbgobert@gmail.com> References: <20240410153423.107381-1-richardbgobert@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org {inet,ipv6}_gro_receive functions perform flush checks (ttl, flags, iph->id, ...) against all packets in a loop. These flush checks are used currently only in tcp flows in GRO. These checks need to be done only once in tcp_gro_receive and only against the found p skb, since they only affect flush and not same_flow. Leveraging the previous commit in the series, in which correct network header offsets are saved for both outer and inner network headers - allowing these checks to be done only once, in tcp_gro_receive. As a result, NAPI_GRO_CB(p)->flush is not used at all. In addition, flush_id checks are more declarative and contained in inet_gro_flush, thus removing the need for flush_id in napi_gro_cb. This results in less parsing code for UDP flows and non-loop flush tests for TCP flows. To make sure results are not within noise range - I've made netfilter drop all TCP packets, and measured CPU performance in GRO (in this case GRO is responsible for about 50% of the CPU utilization). gro_network_flush is compiled inline to tcp_gro_receive. perf top while replaying 64 parallel IP/TCP streams merging in GRO net-next: 6.94% [kernel] [k] inet_gro_receive 3.02% [kernel] [k] tcp_gro_receive patch applied: 4.27% [kernel] [k] tcp_gro_receive 4.22% [kernel] [k] inet_gro_receive perf top while replaying 64 parallel IP/IP/TCP streams merging in GRO (same results for any encapsulation, in this case inet_gro_receive is top offender in net-next) net-next: 10.09% [kernel] [k] inet_gro_receive 2.08% [kernel] [k] tcp_gro_receive patch applied: 6.97% [kernel] [k] inet_gro_receive 3.68% [kernel] [k] tcp_gro_receive perf top -g while running 64 IP/UDP netperf connections NOT merging in GRO (udp_gro_receive is included because of -g, in this case GRO is just overhead) net-next: 1.26% [kernel] [k] inet_gro_receive patch applied: 0.85% [kernel] [k] inet_gro_receive Signed-off-by: Richard Gobert --- include/net/gro.h | 66 ++++++++++++++++++++++++++++++++++++++---- net/core/gro.c | 4 --- net/ipv4/af_inet.c | 41 +------------------------- net/ipv4/tcp_offload.c | 15 ++-------- net/ipv4/udp_offload.c | 16 +++------- net/ipv6/ip6_offload.c | 11 ------- 6 files changed, 67 insertions(+), 86 deletions(-) diff --git a/include/net/gro.h b/include/net/gro.h index a1cc8e8c2ebd..c1f80f1156d6 100644 --- a/include/net/gro.h +++ b/include/net/gro.h @@ -36,15 +36,15 @@ struct napi_gro_cb { /* This is non-zero if the packet cannot be merged with the new skb. */ u16 flush; - /* Save the IP ID here and check when we get to the transport layer */ - u16 flush_id; - /* Number of segments aggregated. */ u16 count; /* Used in ipv6_gro_receive() and foo-over-udp and esp-in-udp */ u16 proto; + /* used to support CHECKSUM_COMPLETE for tunneling protocols */ + __wsum csum; + /* Used in napi_gro_cb::free */ #define NAPI_GRO_FREE 1 #define NAPI_GRO_FREE_STOLEN_HEAD 2 @@ -85,9 +85,6 @@ struct napi_gro_cb { u8 is_flist:1; ); - /* used to support CHECKSUM_COMPLETE for tunneling protocols */ - __wsum csum; - /* L3 offsets */ union { struct { @@ -443,6 +440,63 @@ static inline __wsum ip6_gro_compute_pseudo(const struct sk_buff *skb, skb_gro_len(skb), proto, 0)); } +static inline int inet_gro_flush(const struct iphdr *iph, const struct iphdr *iph2, + struct sk_buff *p, bool outer) +{ + const u32 id = ntohl(*(__be32 *)&iph->id); + const u32 id2 = ntohl(*(__be32 *)&iph2->id); + const u16 flush_id = (id >> 16) - (id2 >> 16); + const u16 count = NAPI_GRO_CB(p)->count; + const u32 df = id & IP_DF; + u32 is_atomic; + int flush; + + /* All fields must match except length and checksum. */ + flush = (iph->ttl ^ iph2->ttl) | (iph->tos ^ iph2->tos) | (df ^ (id2 & IP_DF)); + + if (outer && df) + return flush; + + /* When we receive our second frame we can make a decision on if we + * continue this flow as an atomic flow with a fixed ID or if we use + * an incrementing ID. + */ + NAPI_GRO_CB(p)->is_atomic |= (count == 1 && df && flush_id == 0); + is_atomic = (df && NAPI_GRO_CB(p)->is_atomic) - 1; + + return flush | (flush_id ^ (count & is_atomic)); +} + +static inline int ipv6_gro_flush(const struct ipv6hdr *iph, const struct ipv6hdr *iph2) +{ + /* */ + __be32 first_word = *(__be32 *)iph ^ *(__be32 *)iph2; + + /* Flush if Traffic Class fields are different. */ + return !!((first_word & htonl(0x0FF00000)) | + (__force __be32)(iph->hop_limit ^ iph2->hop_limit)); +} + +static inline int gro_network_flush(const void *th, const void *th2, struct sk_buff *p, int off) +{ + const bool encap_mark = NAPI_GRO_CB(p)->encap_mark; + int flush = 0; + int i; + + for (i = 0; i <= encap_mark; i++) { + const u16 diff = off - NAPI_GRO_CB(p)->network_offsets[i]; + const void *nh = th - diff; + const void *nh2 = th2 - diff; + + if (((struct iphdr *)nh)->version == 6) + flush |= ipv6_gro_flush(nh, nh2); + else + flush |= inet_gro_flush(nh, nh2, p, i != encap_mark); + } + + return flush; +} + int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb); /* Pass the currently batched GRO_NORMAL SKBs up to the stack. */ diff --git a/net/core/gro.c b/net/core/gro.c index b2156e6cc4ad..3bfdfefe4a24 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -89,7 +89,6 @@ void dev_remove_offload(struct packet_offload *po) } EXPORT_SYMBOL(dev_remove_offload); - int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb) { struct skb_shared_info *pinfo, *skbinfo = skb_shinfo(skb); @@ -330,8 +329,6 @@ static void gro_list_prepare(const struct list_head *head, list_for_each_entry(p, head, list) { unsigned long diffs; - NAPI_GRO_CB(p)->flush = 0; - if (hash != skb_get_hash_raw(p)) { NAPI_GRO_CB(p)->same_flow = 0; continue; @@ -471,7 +468,6 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff sizeof(u32))); /* Avoid slow unaligned acc */ *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0; NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb); - NAPI_GRO_CB(skb)->is_atomic = 1; NAPI_GRO_CB(skb)->count = 1; if (unlikely(skb_is_gso(skb))) { NAPI_GRO_CB(skb)->count = skb_shinfo(skb)->gso_segs; diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c index 6546bf376b24..af094aecf38c 100644 --- a/net/ipv4/af_inet.c +++ b/net/ipv4/af_inet.c @@ -1481,7 +1481,6 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb) struct sk_buff *p; unsigned int hlen; unsigned int off; - unsigned int id; int flush = 1; int proto; @@ -1507,13 +1506,10 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb) goto out; NAPI_GRO_CB(skb)->proto = proto; - id = ntohl(*(__be32 *)&iph->id); - flush = (u16)((ntohl(*(__be32 *)iph) ^ skb_gro_len(skb)) | (id & ~IP_DF)); - id >>= 16; + flush = (u16)((ntohl(*(__be32 *)iph) ^ skb_gro_len(skb)) | (ntohl(*(__be32 *)&iph->id) & ~IP_DF)); list_for_each_entry(p, head, list) { struct iphdr *iph2; - u16 flush_id; if (!NAPI_GRO_CB(p)->same_flow) continue; @@ -1530,43 +1526,8 @@ struct sk_buff *inet_gro_receive(struct list_head *head, struct sk_buff *skb) NAPI_GRO_CB(p)->same_flow = 0; continue; } - - /* All fields must match except length and checksum. */ - NAPI_GRO_CB(p)->flush |= - (iph->ttl ^ iph2->ttl) | - (iph->tos ^ iph2->tos) | - ((iph->frag_off ^ iph2->frag_off) & htons(IP_DF)); - - NAPI_GRO_CB(p)->flush |= flush; - - /* We need to store of the IP ID check to be included later - * when we can verify that this packet does in fact belong - * to a given flow. - */ - flush_id = (u16)(id - ntohs(iph2->id)); - - /* This bit of code makes it much easier for us to identify - * the cases where we are doing atomic vs non-atomic IP ID - * checks. Specifically an atomic check can return IP ID - * values 0 - 0xFFFF, while a non-atomic check can only - * return 0 or 0xFFFF. - */ - if (!NAPI_GRO_CB(p)->is_atomic || - !(iph->frag_off & htons(IP_DF))) { - flush_id ^= NAPI_GRO_CB(p)->count; - flush_id = flush_id ? 0xFFFF : 0; - } - - /* If the previous IP ID value was based on an atomic - * datagram we can overwrite the value and ignore it. - */ - if (NAPI_GRO_CB(skb)->is_atomic) - NAPI_GRO_CB(p)->flush_id = flush_id; - else - NAPI_GRO_CB(p)->flush_id |= flush_id; } - NAPI_GRO_CB(skb)->is_atomic = !!(iph->frag_off & htons(IP_DF)); NAPI_GRO_CB(skb)->flush |= flush; /* Note : No need to call skb_gro_postpull_rcsum() here, diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index 7f045b881dd4..1b10ab3b0f6a 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -232,9 +232,7 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) goto out_check_final; found: - /* Include the IP ID check below from the inner most IP hdr */ - flush = NAPI_GRO_CB(p)->flush; - flush |= (__force int)(flags & TCP_FLAG_CWR); + flush = (__force int)(flags & TCP_FLAG_CWR); flush |= (__force int)((flags ^ tcp_flag_word(th2)) & ~(TCP_FLAG_CWR | TCP_FLAG_FIN | TCP_FLAG_PSH)); flush |= (__force int)(th->ack_seq ^ th2->ack_seq); @@ -242,16 +240,7 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) flush |= *(u32 *)((u8 *)th + i) ^ *(u32 *)((u8 *)th2 + i); - /* When we receive our second frame we can made a decision on if we - * continue this flow as an atomic flow with a fixed ID or if we use - * an incrementing ID. - */ - if (NAPI_GRO_CB(p)->flush_id != 1 || - NAPI_GRO_CB(p)->count != 1 || - !NAPI_GRO_CB(p)->is_atomic) - flush |= NAPI_GRO_CB(p)->flush_id; - else - NAPI_GRO_CB(p)->is_atomic = false; + flush |= gro_network_flush(th, th2, p, off); mss = skb_shinfo(p)->gso_size; diff --git a/net/ipv4/udp_offload.c b/net/ipv4/udp_offload.c index ad4c88fe7d15..c5a5155904cf 100644 --- a/net/ipv4/udp_offload.c +++ b/net/ipv4/udp_offload.c @@ -466,12 +466,12 @@ static struct sk_buff *udp_gro_receive_segment(struct list_head *head, struct sk_buff *skb) { struct udphdr *uh = udp_gro_udphdr(skb); + int off = skb_gro_offset(skb); struct sk_buff *pp = NULL; struct udphdr *uh2; struct sk_buff *p; unsigned int ulen; int ret = 0; - int flush; /* requires non zero csum, for symmetry with GSO */ if (!uh->check) { @@ -529,17 +529,9 @@ static struct sk_buff *udp_gro_receive_segment(struct list_head *head, skb_gro_postpull_rcsum(skb, uh, sizeof(struct udphdr)); - flush = NAPI_GRO_CB(p)->flush; - - if (NAPI_GRO_CB(p)->flush_id != 1 || - NAPI_GRO_CB(p)->count != 1 || - !NAPI_GRO_CB(p)->is_atomic) - flush |= NAPI_GRO_CB(p)->flush_id; - else - NAPI_GRO_CB(p)->is_atomic = false; - - if (flush || skb_gro_receive(p, skb)) - ret = 1; + ret = gro_network_flush(uh, uh2, p, off); + if (!ret) + ret = skb_gro_receive(p, skb); } } diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c index ba41939537f2..c9a6bc1afc9a 100644 --- a/net/ipv6/ip6_offload.c +++ b/net/ipv6/ip6_offload.c @@ -288,19 +288,8 @@ INDIRECT_CALLABLE_SCOPE struct sk_buff *ipv6_gro_receive(struct list_head *head, nlen - sizeof(struct ipv6hdr))) goto not_same_flow; } - /* flush if Traffic Class fields are different */ - NAPI_GRO_CB(p)->flush |= !!((first_word & htonl(0x0FF00000)) | - (__force __be32)(iph->hop_limit ^ iph2->hop_limit)); - NAPI_GRO_CB(p)->flush |= flush; - - /* If the previous IP ID value was based on an atomic - * datagram we can overwrite the value and ignore it. - */ - if (NAPI_GRO_CB(skb)->is_atomic) - NAPI_GRO_CB(p)->flush_id = 0; } - NAPI_GRO_CB(skb)->is_atomic = true; NAPI_GRO_CB(skb)->flush |= flush; skb_gro_postpull_rcsum(skb, iph, nlen); From patchwork Wed Apr 10 15:34:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Gobert X-Patchwork-Id: 13624765 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 08C4B1779AF; Wed, 10 Apr 2024 15:35:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763356; cv=none; b=YAriZSDLwIFUjngVWyNxzx4of6NWW0skYi/lRawDBWIzVRtsNoYvu0cFDKMUS9FbbH+sHBgr1abOSeAzI0q9QbdaKd0Y3qNwbazHIofcyhZ6+fvQKGMtRiMIv2+xwFC9B6M1jLbnl/s4XsBbfYuTjC3pvgRzVMPXPF0c1udfK44= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712763356; c=relaxed/simple; bh=9Zp8iPX0jQlUJwfcjhF0I+ShRBkh39BZ3ToTtQd5BH0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Qoo4NL8cqQ9fajX5LdWb2A+0Wg60Ibqostq6/N+I6RlyIkANx0m4jfCGKGWlUkkukPCXHJjK2NlNbM8rmPZA49QKc/HPrgTsUuZWjZt3vUrEE4Z2n2IgzaHDtbmKFZQ8yrZz6cO/byGwXAv+i5se/mOpcSZyD3ZOU7GUbRF9RKQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jpSIBqsa; arc=none smtp.client-ip=209.85.128.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jpSIBqsa" Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-4173f9e5e05so4191605e9.3; Wed, 10 Apr 2024 08:35:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712763353; x=1713368153; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=79Cy7Ko+UcnCG7nOcHmA1+y+Nd7j5NW8NL0jgsxVudM=; b=jpSIBqsavTd1UU/pZoN4MFO0LNOjtr0/HA1RvdLM9i95jmuP9qPGBdsvarQBRGqA1M jowBKl/iZrp/e7pHUJ/INK4CgRDEIiLPDSOSWRU/YxWTQrMUtI11rUcsYBWRGqGddeQF Du2eazTLMWgNIGvQSEWdtYlYdNB6fw3nCMAYdp5LJZfPoqcvMvDLbI+FuSpfYRgVSiwG 56Iwxa1wIsrzo/uoP6eUL4ivwYnJoSowGfp3cgTziG1LxbUkbF85LmRmTyccmB1bPG1R vxMqb2BG6YPO/C5r0QwvPNK9NApyPI61V04glqmbchK8rDMS+QnQe1pqm539Q6yop3MT Q3wQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712763353; x=1713368153; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=79Cy7Ko+UcnCG7nOcHmA1+y+Nd7j5NW8NL0jgsxVudM=; b=JoqBDmdJBHI9nejRtslmBCfyUSJfrSjfRMuqZ9aUNvRyUAplOrBWsjWD4fEmp5crzT 7jEW4fUUJhxAi8Wtyi/tJdRtcNCMzvvqvcyXO4KuXIS6sWCquuByWvxD1bMDicyJsg3V 0ltjYtLzUYCpyKsZfCXDeYcALISG4rRn4Uk/a5t4blnLKA6APMR0gbpZVLy1F8B1KVCL UF6LkXq5YJZDUgX+L5+GAnX2ohq0H9hfXf7jK/hR1CvCinY585tf3pvxgkSoLqeh8oBt 9N9k/fo1HueRp6GoK3laqWQDZWBqpNtglsBDYXvypvIwtNn2P7CGLfLZVdr8p3dDEh1h rRoQ== X-Forwarded-Encrypted: i=1; AJvYcCWpA46TZOilwCEYUUlfwstAprGTQQI5gCugUAnmeSkXuVbh19B2958nRgl/GNc2gOFGyYImlvON/iKHRDR0WA6VG3Hh5Vy42O4fg0ANnAU7TL+u1uFX7uuaWv2DYCebNBGxEUbH2UQ97b2DpyR6PSl4pkPf8r6kUsOm96BQxD0O5jm4Yo++ X-Gm-Message-State: AOJu0YwCk3z0c3U6ah4j4HOAXYVABdgY79Zp3Gsna8qP/QcZ/gLakCNg L9bJag8HH4bVWFfs8SIswp6B+trODKM+mDfes86i6MC/MFX0EgN72wpIz+cp X-Google-Smtp-Source: AGHT+IHwUOss4eOPGbcTTFqRl4GRlRAHFdSjwFWuxW5e7ZrqVh7llGW9PJ8P0Ig6TNN1rErwomDyBg== X-Received: by 2002:a05:600c:4446:b0:417:36c5:7c64 with SMTP id v6-20020a05600c444600b0041736c57c64mr1374948wmn.21.1712763353353; Wed, 10 Apr 2024 08:35:53 -0700 (PDT) Received: from localhost ([146.70.204.204]) by smtp.gmail.com with ESMTPSA id bg25-20020a05600c3c9900b004162d06768bsm2631946wmb.21.2024.04.10.08.35.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 08:35:53 -0700 (PDT) From: Richard Gobert To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, willemdebruijn.kernel@gmail.com, shuah@kernel.org, dsahern@kernel.org, aduyck@mirantis.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org Cc: Richard Gobert Subject: [PATCH net-next v6 6/6] selftests/net: add flush id selftests Date: Wed, 10 Apr 2024 17:34:23 +0200 Message-Id: <20240410153423.107381-7-richardbgobert@gmail.com> In-Reply-To: <20240410153423.107381-1-richardbgobert@gmail.com> References: <20240410153423.107381-1-richardbgobert@gmail.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Added flush id selftests to test different cases where DF flag is set or unset and id value changes in the following packets. All cases where the packets should coalesce or should not coalesce are tested. Signed-off-by: Richard Gobert --- tools/testing/selftests/net/gro.c | 144 ++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) diff --git a/tools/testing/selftests/net/gro.c b/tools/testing/selftests/net/gro.c index 353e1e867fbb..74ab06953c38 100644 --- a/tools/testing/selftests/net/gro.c +++ b/tools/testing/selftests/net/gro.c @@ -617,6 +617,120 @@ static void add_ipv6_exthdr(void *buf, void *optpkt, __u8 exthdr_type, char *ext iph->payload_len = htons(ntohs(iph->payload_len) + MIN_EXTHDR_SIZE); } +static void fix_ip4_checksum(struct iphdr *iph) +{ + iph->check = 0; + iph->check = checksum_fold(iph, sizeof(struct iphdr), 0); +} + +static void send_flush_id_case(int fd, struct sockaddr_ll *daddr, int tcase) +{ + bool send_three = false; + static char buf1[MAX_HDR_LEN + PAYLOAD_LEN]; + static char buf2[MAX_HDR_LEN + PAYLOAD_LEN]; + static char buf3[MAX_HDR_LEN + PAYLOAD_LEN]; + + create_packet(buf1, 0, 0, PAYLOAD_LEN, 0); + create_packet(buf2, PAYLOAD_LEN, 0, PAYLOAD_LEN, 0); + create_packet(buf3, PAYLOAD_LEN * 2, 0, PAYLOAD_LEN, 0); + + struct iphdr *iph1 = (struct iphdr *)(buf1 + ETH_HLEN); + struct iphdr *iph2 = (struct iphdr *)(buf2 + ETH_HLEN); + struct iphdr *iph3 = (struct iphdr *)(buf3 + ETH_HLEN); + + switch (tcase) { + case 0: /* DF=1, Incrementing - should coalesce */ + iph1->frag_off |= htons(IP_DF); + iph1->id = htons(8); + fix_ip4_checksum(iph1); + + iph2->frag_off |= htons(IP_DF); + iph2->id = htons(9); + fix_ip4_checksum(iph2); + break; + + case 1: /* DF=1, Fixed - should coalesce */ + iph1->frag_off |= htons(IP_DF); + iph1->id = htons(8); + fix_ip4_checksum(iph1); + + iph2->frag_off |= htons(IP_DF); + iph2->id = htons(8); + fix_ip4_checksum(iph2); + break; + + case 2: /* DF=0, Incrementing - should coalesce */ + iph1->frag_off &= ~htons(IP_DF); + iph1->id = htons(8); + fix_ip4_checksum(iph1); + + iph2->frag_off &= ~htons(IP_DF); + iph2->id = htons(9); + fix_ip4_checksum(iph2); + break; + + case 3: /* DF=0, Fixed - should not coalesce */ + iph1->frag_off &= ~htons(IP_DF); + iph1->id = htons(8); + fix_ip4_checksum(iph1); + + iph2->frag_off &= ~htons(IP_DF); + iph2->id = htons(8); + fix_ip4_checksum(iph2); + break; + + case 4: /* DF=1, two packets incrementing, and one fixed - should + * coalesce only the first two packets + */ + iph1->frag_off |= htons(IP_DF); + iph1->id = htons(8); + fix_ip4_checksum(iph1); + + iph2->frag_off |= htons(IP_DF); + iph2->id = htons(9); + fix_ip4_checksum(iph2); + + iph3->frag_off |= htons(IP_DF); + iph3->id = htons(9); + fix_ip4_checksum(iph3); + send_three = true; + break; + + case 5: /* DF=1, two packets fixed, and one incrementing - should + * coalesce only the first two packets + */ + iph1->frag_off |= htons(IP_DF); + iph1->id = htons(8); + fix_ip4_checksum(iph1); + + iph2->frag_off |= htons(IP_DF); + iph2->id = htons(8); + fix_ip4_checksum(iph2); + + iph3->frag_off |= htons(IP_DF); + iph3->id = htons(9); + fix_ip4_checksum(iph3); + send_three = true; + break; + } + + write_packet(fd, buf1, total_hdr_len + PAYLOAD_LEN, daddr); + write_packet(fd, buf2, total_hdr_len + PAYLOAD_LEN, daddr); + + if (send_three) + write_packet(fd, buf3, total_hdr_len + PAYLOAD_LEN, daddr); +} + +static void test_flush_id(int fd, struct sockaddr_ll *daddr, char *fin_pkt) +{ + for (int i = 0; i < 6; i++) { + sleep(1); + send_flush_id_case(fd, daddr, i); + sleep(1); + write_packet(fd, fin_pkt, total_hdr_len, daddr); + } +} + static void send_ipv6_exthdr(int fd, struct sockaddr_ll *daddr, char *ext_data1, char *ext_data2) { static char buf[MAX_HDR_LEN + PAYLOAD_LEN]; @@ -935,6 +1049,8 @@ static void gro_sender(void) send_fragment4(txfd, &daddr); sleep(1); write_packet(txfd, fin_pkt, total_hdr_len, &daddr); + + test_flush_id(txfd, &daddr, fin_pkt); } else if (proto == PF_INET6) { sleep(1); send_fragment6(txfd, &daddr); @@ -1061,6 +1177,34 @@ static void gro_receiver(void) printf("fragmented ip4 doesn't coalesce: "); check_recv_pkts(rxfd, correct_payload, 2); + + /* is_atomic checks */ + printf("DF=1, Incrementing - should coalesce: "); + correct_payload[0] = PAYLOAD_LEN * 2; + check_recv_pkts(rxfd, correct_payload, 1); + + printf("DF=1, Fixed - should coalesce: "); + correct_payload[0] = PAYLOAD_LEN * 2; + check_recv_pkts(rxfd, correct_payload, 1); + + printf("DF=0, Incrementing - should coalesce: "); + correct_payload[0] = PAYLOAD_LEN * 2; + check_recv_pkts(rxfd, correct_payload, 1); + + printf("DF=0, Fixed - should not coalesce: "); + correct_payload[0] = PAYLOAD_LEN; + correct_payload[1] = PAYLOAD_LEN; + check_recv_pkts(rxfd, correct_payload, 2); + + printf("DF=1, 2 Incrementing and one fixed - should coalesce only first 2 packets: "); + correct_payload[0] = PAYLOAD_LEN * 2; + correct_payload[1] = PAYLOAD_LEN; + check_recv_pkts(rxfd, correct_payload, 2); + + printf("DF=1, 2 Fixed and one incrementing - should coalesce only first 2 packets: "); + correct_payload[0] = PAYLOAD_LEN * 2; + correct_payload[1] = PAYLOAD_LEN; + check_recv_pkts(rxfd, correct_payload, 2); } else if (proto == PF_INET6) { /* GRO doesn't check for ipv6 hop limit when flushing. * Hence no corresponding test to the ipv4 case.