From patchwork Wed Apr 24 18:04:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Fietkau X-Patchwork-Id: 13642298 X-Patchwork-Delegate: kuba@kernel.org Received: from nbd.name (nbd.name [46.4.11.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8754B16D9B4; Wed, 24 Apr 2024 18:05:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=46.4.11.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713981908; cv=none; b=QUqzXIFQ7YfjbTLPGU0ELjJw6foTY9XDbyYAcvgRBwYPs+J4ZhPyNqwB4nMKXljleIiICUj3YMeA0y1TcB4mMz71gMW7fvxxLehMoUX7hYuXcsTMPrKXWtwR13MW1Vk5aqx846NPTsSiJ8igTRS0JlbJg0hI96y3yTtX5+c7y3g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713981908; c=relaxed/simple; bh=IXZq04hiu+Pn+sF7BKtV+tVHTOSfOBndsfo4YDuDbeo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BRVRyIA6Ir+pB9RdIyI49yuI7Qy6VOS6hoQ0ZpHxg0GJkiqtevhILl8D2dr+g2PPLT0uuBAspsSK2vZWzWf81Mr+k37cuidMu38RPon6FlieobiNTfMj+oP+vFGSxn91ejWO4DWk8zAJF5GMB3N9r8lSmRAvin86ufGdb3dT+vo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name; spf=none smtp.mailfrom=nbd.name; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b=UvFn6rmZ; arc=none smtp.client-ip=46.4.11.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=nbd.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b="UvFn6rmZ" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=nbd.name; s=20160729; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=3H6dBRGk+BHU+y8mRcNlZ506UX4DS949WCdH9l5DVuI=; b=UvFn6rmZheJj5CMt6PXevzwOZs M0qTYxH0R7Yn5FdcYXj36vvMQQRcEGP/YQdAImQ0fjvvtjLkhSgkMSRhMri4ysR4ym4QUfF4/kNjv f7qogJNz8YhgbxWnMnq78rkzZrwludynpTumIqX0Eyx7PXwLvlqrl98wbR07UMHR2cKo=; Received: from p200300daa70d8400593e4be1bb3506c9.dip0.t-ipconnect.de ([2003:da:a70d:8400:593e:4be1:bb35:6c9] helo=Maecks.lan) by ds12 with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (Exim 4.96) (envelope-from ) id 1rzgzM-006rUX-0o; Wed, 24 Apr 2024 20:05:00 +0200 From: Felix Fietkau To: netdev@vger.kernel.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , David Ahern Cc: willemdebruijn.kernel@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH net-next 1/4] net: move skb_gro_receive_list from udp to core Date: Wed, 24 Apr 2024 20:04:53 +0200 Message-ID: <20240424180458.56211-2-nbd@nbd.name> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240424180458.56211-1-nbd@nbd.name> References: <20240424180458.56211-1-nbd@nbd.name> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org This helper function will be used for TCP fraglist GRO support Signed-off-by: Felix Fietkau --- include/net/gro.h | 1 + net/core/gro.c | 27 +++++++++++++++++++++++++++ net/ipv4/udp_offload.c | 27 --------------------------- 3 files changed, 28 insertions(+), 27 deletions(-) diff --git a/include/net/gro.h b/include/net/gro.h index 50f1e403dbbb..ca8e4b3de044 100644 --- a/include/net/gro.h +++ b/include/net/gro.h @@ -429,6 +429,7 @@ static inline __wsum ip6_gro_compute_pseudo(const struct sk_buff *skb, } int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb); +int skb_gro_receive_list(struct sk_buff *p, struct sk_buff *skb); /* Pass the currently batched GRO_NORMAL SKBs up to the stack. */ static inline void gro_normal_list(struct napi_struct *napi) diff --git a/net/core/gro.c b/net/core/gro.c index 2459ab697f7f..268c6c826d09 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -231,6 +231,33 @@ int skb_gro_receive(struct sk_buff *p, struct sk_buff *skb) return 0; } +int skb_gro_receive_list(struct sk_buff *p, struct sk_buff *skb) +{ + if (unlikely(p->len + skb->len >= 65536)) + return -E2BIG; + + if (NAPI_GRO_CB(p)->last == p) + skb_shinfo(p)->frag_list = skb; + else + NAPI_GRO_CB(p)->last->next = skb; + + skb_pull(skb, skb_gro_offset(skb)); + + NAPI_GRO_CB(p)->last = skb; + NAPI_GRO_CB(p)->count++; + p->data_len += skb->len; + + /* sk ownership - if any - completely transferred to the aggregated packet */ + skb->destructor = NULL; + skb->sk = NULL; + p->truesize += skb->truesize; + p->len += skb->len; + + NAPI_GRO_CB(skb)->same_flow = 1; + + return 0; +} + static void napi_gro_complete(struct napi_struct *napi, struct sk_buff *skb) { diff --git a/net/ipv4/udp_offload.c b/net/ipv4/udp_offload.c index 3498dd1d0694..a3cd546a1aea 100644 --- a/net/ipv4/udp_offload.c +++ b/net/ipv4/udp_offload.c @@ -433,33 +433,6 @@ static struct sk_buff *udp4_ufo_fragment(struct sk_buff *skb, return segs; } -static int skb_gro_receive_list(struct sk_buff *p, struct sk_buff *skb) -{ - if (unlikely(p->len + skb->len >= 65536)) - return -E2BIG; - - if (NAPI_GRO_CB(p)->last == p) - skb_shinfo(p)->frag_list = skb; - else - NAPI_GRO_CB(p)->last->next = skb; - - skb_pull(skb, skb_gro_offset(skb)); - - NAPI_GRO_CB(p)->last = skb; - NAPI_GRO_CB(p)->count++; - p->data_len += skb->len; - - /* sk ownership - if any - completely transferred to the aggregated packet */ - skb->destructor = NULL; - skb->sk = NULL; - p->truesize += skb->truesize; - p->len += skb->len; - - NAPI_GRO_CB(skb)->same_flow = 1; - - return 0; -} - #define UDP_GRO_CNT_MAX 64 static struct sk_buff *udp_gro_receive_segment(struct list_head *head, From patchwork Wed Apr 24 18:04:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Fietkau X-Patchwork-Id: 13642299 X-Patchwork-Delegate: kuba@kernel.org Received: from nbd.name (nbd.name [46.4.11.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 533B216D4E5; Wed, 24 Apr 2024 18:05:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=46.4.11.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713981909; cv=none; b=Wqi+hWvB8FAKgWi8xBqs2p1o4MepKTIIwMJ3oxLYxQ91ff+SQlTsspbYKhXN/KmhhSH1s0c3hCIZ6icNQf8ae2ulIoNyv1W3jOf8FzNe5t9QDF2R1X+Gc3C/f8tarC5nF8sQtJilm0cB1i5zJWNCu9p44Q033cV1Tk+mAR1fOhk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713981909; c=relaxed/simple; bh=7cYcqFWyS5ILf8FSC42LtAhAyiCrjlIaapO4c6lZXBg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EgkgCNBReznkgdLkpS6xi9GdTc1RiRk9MDzSapKaXf0wCs68vHAm9D0xcXAXeYq7Iy9QeQQazoGinZb/9yG2S18uAUSmpyT9qNorAv6ibZFOZaN1vvra+pS8FL1b4M1Hd2kl+54gXO+jzUY04l8z4MAMwhPQUAS1EKX3hnhK2z4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name; spf=none smtp.mailfrom=nbd.name; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b=knt0vgFr; arc=none smtp.client-ip=46.4.11.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=nbd.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b="knt0vgFr" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=nbd.name; s=20160729; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=EI1eLVntxcTn53NTpTv6PcIFtjylSdx6l7qTZk+FoL4=; b=knt0vgFr4/irmYJquuO7mvbZFs Ah1taKuRKDqS0CTaMNoDmf1QWlJYfo64r8HqF/a/rJVJg5TZRTqAV9vJEQjVNoK0MX5exu/UWdPOp vf0wsikEF4tRECCtYcYnP/VSSaiGmrqJAJRCSFQoYEReDLLzGYsfQS1sTR1CfU2Jml5g=; Received: from p200300daa70d8400593e4be1bb3506c9.dip0.t-ipconnect.de ([2003:da:a70d:8400:593e:4be1:bb35:6c9] helo=Maecks.lan) by ds12 with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (Exim 4.96) (envelope-from ) id 1rzgzN-006rUX-04; Wed, 24 Apr 2024 20:05:01 +0200 From: Felix Fietkau To: netdev@vger.kernel.org, Eric Dumazet , "David S. Miller" , David Ahern , Jakub Kicinski , Paolo Abeni Cc: willemdebruijn.kernel@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH net-next 2/4] net: add support for segmenting TCP fraglist GSO packets Date: Wed, 24 Apr 2024 20:04:54 +0200 Message-ID: <20240424180458.56211-3-nbd@nbd.name> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240424180458.56211-1-nbd@nbd.name> References: <20240424180458.56211-1-nbd@nbd.name> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org Preparation for adding TCP fraglist GRO support. It expects packets to be combined in a similar way as UDP fraglist GSO packets. One difference is the fact that this code assumes that the TCP flags of all packets have the same value. This allows simple handling of flags mutations. For IPv4 packets, NAT is handled in the same way as UDP fraglist GSO. Signed-off-by: Felix Fietkau --- net/ipv4/tcp_offload.c | 74 ++++++++++++++++++++++++++++++++++++++++ net/ipv6/tcpv6_offload.c | 37 ++++++++++++++++++++ 2 files changed, 111 insertions(+) diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index fab0973f995b..06dbb2e2b2f3 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -28,6 +28,77 @@ static void tcp_gso_tstamp(struct sk_buff *skb, unsigned int ts_seq, } } +static void __tcpv4_gso_segment_csum(struct sk_buff *seg, + __be32 *oldip, __be32 *newip, + __be16 *oldport, __be16 *newport) +{ + struct tcphdr *th; + struct iphdr *iph; + + if (*oldip == *newip && *oldport == *newport) + return; + + th = tcp_hdr(seg); + iph = ip_hdr(seg); + + inet_proto_csum_replace4(&th->check, seg, *oldip, *newip, true); + inet_proto_csum_replace2(&th->check, seg, *oldport, *newport, false); + *oldport = *newport; + + csum_replace4(&iph->check, *oldip, *newip); + *oldip = *newip; +} + +static struct sk_buff *__tcpv4_gso_segment_list_csum(struct sk_buff *segs) +{ + struct sk_buff *seg; + struct tcphdr *th, *th2; + struct iphdr *iph, *iph2; + __be32 flags, flags2; + + seg = segs; + th = tcp_hdr(seg); + iph = ip_hdr(seg); + flags = tcp_flag_word(th); + flags2 = tcp_flag_word(tcp_hdr(seg->next)); + + if ((tcp_hdr(seg)->dest == tcp_hdr(seg->next)->dest) && + (tcp_hdr(seg)->source == tcp_hdr(seg->next)->source) && + (ip_hdr(seg)->daddr == ip_hdr(seg->next)->daddr) && + (ip_hdr(seg)->saddr == ip_hdr(seg->next)->saddr) && + (flags == flags2)) + return segs; + + while ((seg = seg->next)) { + th2 = tcp_hdr(seg); + iph2 = ip_hdr(seg); + + __tcpv4_gso_segment_csum(seg, + &iph2->saddr, &iph->saddr, + &th2->source, &th->source); + __tcpv4_gso_segment_csum(seg, + &iph2->daddr, &iph->daddr, + &th2->dest, &th->dest); + if (flags == flags2) + continue; + + inet_proto_csum_replace4(&th2->check, seg, flags2, flags, false); + tcp_flag_word(th2) = flags; + } + + return segs; +} + +static struct sk_buff *__tcp_gso_segment_list(struct sk_buff *skb, + netdev_features_t features) +{ + skb = skb_segment_list(skb, features, skb_mac_header_len(skb)); + if (IS_ERR(skb)) + return skb; + + return __tcpv4_gso_segment_list_csum(skb); +} + static struct sk_buff *tcp4_gso_segment(struct sk_buff *skb, netdev_features_t features) { @@ -37,6 +108,9 @@ static struct sk_buff *tcp4_gso_segment(struct sk_buff *skb, if (!pskb_may_pull(skb, sizeof(struct tcphdr))) return ERR_PTR(-EINVAL); + if (skb_shinfo(skb)->gso_type & SKB_GSO_FRAGLIST) + return __tcp_gso_segment_list(skb, features); + if (unlikely(skb->ip_summed != CHECKSUM_PARTIAL)) { const struct iphdr *iph = ip_hdr(skb); struct tcphdr *th = tcp_hdr(skb); diff --git a/net/ipv6/tcpv6_offload.c b/net/ipv6/tcpv6_offload.c index 4b07d1e6c952..12fe79cb2c10 100644 --- a/net/ipv6/tcpv6_offload.c +++ b/net/ipv6/tcpv6_offload.c @@ -40,6 +40,40 @@ INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff) return 0; } +static struct sk_buff *__tcpv6_gso_segment_list_csum(struct sk_buff *segs) +{ + struct tcphdr *th, *th2; + __be32 flags, flags2; + struct sk_buff *seg; + + seg = segs; + th = tcp_hdr(seg); + flags = tcp_flag_word(th); + flags2 = tcp_flag_word(tcp_hdr(seg->next)); + + if (flags == flags2) + return segs; + + while ((seg = seg->next)) { + th2 = tcp_hdr(seg); + + inet_proto_csum_replace4(&th2->check, seg, flags2, flags, false); + tcp_flag_word(th2) = flags; + } + + return segs; +} + +static struct sk_buff *__tcp_gso_segment_list(struct sk_buff *skb, + netdev_features_t features) +{ + skb = skb_segment_list(skb, features, skb_mac_header_len(skb)); + if (IS_ERR(skb)) + return skb; + + return __tcpv6_gso_segment_list_csum(skb); +} + static struct sk_buff *tcp6_gso_segment(struct sk_buff *skb, netdev_features_t features) { @@ -51,6 +85,9 @@ static struct sk_buff *tcp6_gso_segment(struct sk_buff *skb, if (!pskb_may_pull(skb, sizeof(*th))) return ERR_PTR(-EINVAL); + if (skb_shinfo(skb)->gso_type & SKB_GSO_FRAGLIST) + return __tcp_gso_segment_list(skb, features); + if (unlikely(skb->ip_summed != CHECKSUM_PARTIAL)) { const struct ipv6hdr *ipv6h = ipv6_hdr(skb); struct tcphdr *th = tcp_hdr(skb); From patchwork Wed Apr 24 18:04:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Fietkau X-Patchwork-Id: 13642300 X-Patchwork-Delegate: kuba@kernel.org Received: from nbd.name (nbd.name [46.4.11.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A0B916D9BE; Wed, 24 Apr 2024 18:05:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=46.4.11.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713981910; cv=none; b=EyVeiayBI4OavG5UIg1vOPDOQ6qHPyIlhhpttKV/an2yS1z9n7ShHr1ZGCfDe1PNy/AiQsfbqsJcnDnentdYG8WhVIpMMkOYyTKUyZ4A2L9V0J62SitkONlm3d76ooLk7XPZ2ci+UHlgUswdE0wMFnSFpArNNQ5bWDRNm5SgAmM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713981910; c=relaxed/simple; bh=09c7uP1ORdbCFiEPT2xKyCC6xooD2nv5BIgHym+HUj8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K3s1JoivTN3I5wyn7cArumTZpbeTNAF5o+SaDikyIndOosHQfRJX/t5vx9ZAVda543luKo4/kSzeVJQbUwk93+ZsMYrd6ogTyvGh8v58SuKUKsgtDfcQ/64wJMkqV0yUkY5eyBv288rFjfSk7MsZh6hYNacmMlM29PfPzq9ZlDU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name; spf=none smtp.mailfrom=nbd.name; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b=bIUvkJ0V; arc=none smtp.client-ip=46.4.11.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=nbd.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b="bIUvkJ0V" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=nbd.name; s=20160729; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=mptlG2kbHgQHXt+1cYb67vWWHauxjwdCKy7YK+knXSQ=; b=bIUvkJ0VbdATezSoXnbZMuqXih cPqVkkI1fAa9vJb8qjNzgcY2O/ckAfNP/sZlKwtBE5H25WA6BkU9pHKJdaLGLTjawbxejJEgOGCvn GUYPqBA84Q434MDvgz/NtuuuIDs0YoF0o1MoTsaegWx6x/Md4O80m0mTO5vLgGo4hhCA=; Received: from p200300daa70d8400593e4be1bb3506c9.dip0.t-ipconnect.de ([2003:da:a70d:8400:593e:4be1:bb35:6c9] helo=Maecks.lan) by ds12 with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (Exim 4.96) (envelope-from ) id 1rzgzO-006rUX-2A; Wed, 24 Apr 2024 20:05:02 +0200 From: Felix Fietkau To: netdev@vger.kernel.org, Eric Dumazet , "David S. Miller" , Jakub Kicinski , Paolo Abeni , David Ahern Cc: willemdebruijn.kernel@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH net-next 3/4] net: add code for TCP fraglist GRO Date: Wed, 24 Apr 2024 20:04:55 +0200 Message-ID: <20240424180458.56211-4-nbd@nbd.name> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240424180458.56211-1-nbd@nbd.name> References: <20240424180458.56211-1-nbd@nbd.name> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org This implements fraglist GRO similar to how it's handled in UDP, however no functional changes are added yet. The next change adds a heuristic for using fraglist GRO instead of regular GRO. Signed-off-by: Felix Fietkau --- include/net/tcp.h | 3 ++- net/ipv4/tcp_offload.c | 29 +++++++++++++++++++++++++++-- net/ipv6/tcpv6_offload.c | 11 ++++++++++- 3 files changed, 39 insertions(+), 4 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index b935e1ae4caf..875cda53a7c9 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -2194,7 +2194,8 @@ void tcp_v4_destroy_sock(struct sock *sk); struct sk_buff *tcp_gso_segment(struct sk_buff *skb, netdev_features_t features); -struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb); +struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb, + bool fraglist); INDIRECT_CALLABLE_DECLARE(int tcp4_gro_complete(struct sk_buff *skb, int thoff)); INDIRECT_CALLABLE_DECLARE(struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb)); INDIRECT_CALLABLE_DECLARE(int tcp6_gro_complete(struct sk_buff *skb, int thoff)); diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index 06dbb2e2b2f3..6294e7a5c099 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -252,7 +252,8 @@ struct sk_buff *tcp_gso_segment(struct sk_buff *skb, return segs; } -struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) +struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb, + bool fraglist) { struct sk_buff *pp = NULL; struct sk_buff *p; @@ -289,6 +290,7 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) len = skb_gro_len(skb); flags = tcp_flag_word(th); + NAPI_GRO_CB(skb)->is_flist = fraglist; list_for_each_entry(p, head, list) { if (!NAPI_GRO_CB(p)->same_flow) continue; @@ -308,6 +310,7 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) found: /* Include the IP ID check below from the inner most IP hdr */ flush = NAPI_GRO_CB(p)->flush; + flush |= fraglist != NAPI_GRO_CB(p)->is_flist; flush |= (__force int)(flags & TCP_FLAG_CWR); flush |= (__force int)((flags ^ tcp_flag_word(th2)) & ~(TCP_FLAG_CWR | TCP_FLAG_FIN | TCP_FLAG_PSH)); @@ -341,6 +344,19 @@ struct sk_buff *tcp_gro_receive(struct list_head *head, struct sk_buff *skb) flush |= (ntohl(th2->seq) + skb_gro_len(p)) ^ ntohl(th->seq); flush |= skb_cmp_decrypted(p, skb); + if (fraglist) { + flush |= (__force int)(flags ^ tcp_flag_word(th2)); + flush |= skb->ip_summed != p->ip_summed; + flush |= skb->csum_level != p->csum_level; + flush |= !pskb_may_pull(skb, skb_gro_offset(skb)); + flush |= NAPI_GRO_CB(p)->count >= 64; + + if (flush || skb_gro_receive_list(p, skb)) + mss = 1; + + goto out_check_final; + } + if (flush || skb_gro_receive(p, skb)) { mss = 1; goto out_check_final; @@ -399,7 +415,7 @@ struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb) return NULL; } - return tcp_gro_receive(head, skb); + return tcp_gro_receive(head, skb, false); } INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) @@ -407,6 +423,15 @@ INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) const struct iphdr *iph = ip_hdr(skb); struct tcphdr *th = tcp_hdr(skb); + if (NAPI_GRO_CB(skb)->is_flist) { + skb_shinfo(skb)->gso_type |= SKB_GSO_FRAGLIST | SKB_GSO_TCPV4; + skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count; + + __skb_incr_checksum_unnecessary(skb); + + return 0; + } + th->check = ~tcp_v4_check(skb->len - thoff, iph->saddr, iph->daddr, 0); diff --git a/net/ipv6/tcpv6_offload.c b/net/ipv6/tcpv6_offload.c index 12fe79cb2c10..239588557dc4 100644 --- a/net/ipv6/tcpv6_offload.c +++ b/net/ipv6/tcpv6_offload.c @@ -24,7 +24,7 @@ struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb) return NULL; } - return tcp_gro_receive(head, skb); + return tcp_gro_receive(head, skb, false); } INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff) @@ -32,6 +32,15 @@ INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff) const struct ipv6hdr *iph = ipv6_hdr(skb); struct tcphdr *th = tcp_hdr(skb); + if (NAPI_GRO_CB(skb)->is_flist) { + skb_shinfo(skb)->gso_type |= SKB_GSO_FRAGLIST | SKB_GSO_TCPV6; + skb_shinfo(skb)->gso_segs = NAPI_GRO_CB(skb)->count; + + __skb_incr_checksum_unnecessary(skb); + + return 0; + } + th->check = ~tcp_v6_check(skb->len - thoff, &iph->saddr, &iph->daddr, 0); skb_shinfo(skb)->gso_type |= SKB_GSO_TCPV6; From patchwork Wed Apr 24 18:04:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Fietkau X-Patchwork-Id: 13642301 X-Patchwork-Delegate: kuba@kernel.org Received: from nbd.name (nbd.name [46.4.11.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4401316D9D9; Wed, 24 Apr 2024 18:05:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=46.4.11.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713981910; cv=none; b=iYJYnlpBb2RzXIlDonR0kj0t+iIN/aD3m9VaDJJIoII7dNhBSloh/ZOL7cosnzAbdxLnr7QgKKiFBtgtoXt7sz/lkY2k/LpV5y++MpjHgs+0flYuqPQz8Zvqq89MvGjfKwb1DY29yBhdVeLEE8R5kcx8dSfmY3XaVzeLO9/9kCM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713981910; c=relaxed/simple; bh=koRc4LMtG9MduWD+18W71wTDcAdZFuQSmz/k0s0PY7E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=p/acoDZf0YUItHA9RDk4vKDpNAqqCu9xq86ALAFPnOMllidFoQnfkoVO7rAq5zNvEgL13e4E03gRHaArMGW6Xifvl2CwaKEmJjoa7xgdL7i8wdsIY3WZQnUqhjn+CSk/A20KPf/kubFCaJ5OXTln1xueK4beaC0hKl+E1hbL8gs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name; spf=none smtp.mailfrom=nbd.name; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b=k/ZjWole; arc=none smtp.client-ip=46.4.11.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=nbd.name Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=nbd.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b="k/ZjWole" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=nbd.name; s=20160729; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=pea9y4EsSmtuSay+TRktXrd0LL/ZcJS+6ZAWLAVY6k4=; b=k/ZjWolehXdgCwLDZXcctnK8JK ZRjBTCHXfWZFiUtco9JgLg0mTR6/iwEQ/JyR6dz5tudk0pXB1QJFrQi5x2FIV/yaurzJns+IJcC0t I93jhizNI6uqeEtiu2el8PI/TN+bua+RhuG80Xqng+IOnqrjEplwzGlzwrS59uxT8zfU=; Received: from p200300daa70d8400593e4be1bb3506c9.dip0.t-ipconnect.de ([2003:da:a70d:8400:593e:4be1:bb35:6c9] helo=Maecks.lan) by ds12 with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (Exim 4.96) (envelope-from ) id 1rzgzP-006rUX-1U; Wed, 24 Apr 2024 20:05:03 +0200 From: Felix Fietkau To: netdev@vger.kernel.org, Eric Dumazet , "David S. Miller" , David Ahern , Jakub Kicinski , Paolo Abeni Cc: willemdebruijn.kernel@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH net-next 4/4] net: add heuristic for enabling TCP fraglist GRO Date: Wed, 24 Apr 2024 20:04:56 +0200 Message-ID: <20240424180458.56211-5-nbd@nbd.name> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240424180458.56211-1-nbd@nbd.name> References: <20240424180458.56211-1-nbd@nbd.name> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org When forwarding TCP after GRO, software segmentation is very expensive, especially when the checksum needs to be recalculated. One case where that's currently unavoidable is when routing packets over PPPoE. Performance improves significantly when using fraglist GRO implemented in the same way as for UDP. When NETIF_F_GRO_FRAGLIST is enabled, perform a lookup for an established socket in the same netns as the receiving device. While this may not cover all relevant use cases in multi-netns configurations, it should be good enough for most configurations that need this. Here's a measurement of running 2 TCP streams through a MediaTek MT7622 device (2-core Cortex-A53), which runs NAT with flow offload enabled from one ethernet port to PPPoE on another ethernet port + cake qdisc set to 1Gbps. rx-gro-list off: 630 Mbit/s, CPU 35% idle rx-gro-list on: 770 Mbit/s, CPU 40% idle Signe-off-by: Felix Fietkau --- net/ipv4/tcp_offload.c | 45 ++++++++++++++++++++++++++++++++++++++- net/ipv6/tcpv6_offload.c | 46 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 89 insertions(+), 2 deletions(-) diff --git a/net/ipv4/tcp_offload.c b/net/ipv4/tcp_offload.c index 6294e7a5c099..f987e2d8423a 100644 --- a/net/ipv4/tcp_offload.c +++ b/net/ipv4/tcp_offload.c @@ -404,6 +404,49 @@ void tcp_gro_complete(struct sk_buff *skb) } EXPORT_SYMBOL(tcp_gro_complete); +static bool tcp4_check_fraglist_gro(struct sk_buff *skb) +{ + const struct iphdr *iph = skb_gro_network_header(skb); + struct net *net = dev_net(skb->dev); + unsigned int off, hlen, thlen; + struct tcphdr *th; + struct sock *sk; + int iif, sdif; + + if (!(skb->dev->features & NETIF_F_GRO_FRAGLIST)) + return false; + + inet_get_iif_sdif(skb, &iif, &sdif); + + off = skb_gro_offset(skb); + hlen = off + sizeof(*th); + th = skb_gro_header(skb, hlen, off); + if (unlikely(!th)) + return false; + + thlen = th->doff * 4; + if (thlen < sizeof(*th)) + return false; + + hlen = off + thlen; + if (!skb_gro_may_pull(skb, hlen)) { + th = skb_gro_header_slow(skb, hlen, off); + if (unlikely(!th)) + return false; + } + + sk = __inet_lookup_established(net, net->ipv4.tcp_death_row.hashinfo, + iph->saddr, th->source, + iph->daddr, ntohs(th->dest), + iif, sdif); + if (!sk) + return true; + + sock_put(sk); + + return false; +} + INDIRECT_CALLABLE_SCOPE struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb) { @@ -415,7 +458,7 @@ struct sk_buff *tcp4_gro_receive(struct list_head *head, struct sk_buff *skb) return NULL; } - return tcp_gro_receive(head, skb, false); + return tcp_gro_receive(head, skb, tcp4_check_fraglist_gro(skb)); } INDIRECT_CALLABLE_SCOPE int tcp4_gro_complete(struct sk_buff *skb, int thoff) diff --git a/net/ipv6/tcpv6_offload.c b/net/ipv6/tcpv6_offload.c index 239588557dc4..c214f5cfe595 100644 --- a/net/ipv6/tcpv6_offload.c +++ b/net/ipv6/tcpv6_offload.c @@ -7,12 +7,56 @@ */ #include #include +#include #include #include #include #include #include "ip6_offload.h" +static bool tcp6_check_fraglist_gro(struct sk_buff *skb) +{ + const struct ipv6hdr *hdr = skb_gro_network_header(skb); + struct net *net = dev_net(skb->dev); + unsigned int off, hlen, thlen; + struct tcphdr *th; + struct sock *sk; + int iif, sdif; + + if (!(skb->dev->features & NETIF_F_GRO_FRAGLIST)) + return false; + + inet6_get_iif_sdif(skb, &iif, &sdif); + + off = skb_gro_offset(skb); + hlen = off + sizeof(*th); + th = skb_gro_header(skb, hlen, off); + if (unlikely(!th)) + return false; + + thlen = th->doff * 4; + if (thlen < sizeof(*th)) + return false; + + hlen = off + thlen; + if (!skb_gro_may_pull(skb, hlen)) { + th = skb_gro_header_slow(skb, hlen, off); + if (unlikely(!th)) + return false; + } + + sk = __inet6_lookup_established(net, net->ipv4.tcp_death_row.hashinfo, + &hdr->saddr, th->source, + &hdr->daddr, ntohs(th->dest), + iif, sdif); + if (!sk) + return true; + + sock_put(sk); + + return false; +} + INDIRECT_CALLABLE_SCOPE struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb) { @@ -24,7 +68,7 @@ struct sk_buff *tcp6_gro_receive(struct list_head *head, struct sk_buff *skb) return NULL; } - return tcp_gro_receive(head, skb, false); + return tcp_gro_receive(head, skb, tcp6_check_fraglist_gro(skb)); } INDIRECT_CALLABLE_SCOPE int tcp6_gro_complete(struct sk_buff *skb, int thoff)