From patchwork Fri Jun 14 13:49:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Neil Hellfeldt X-Patchwork-Id: 13698724 X-Patchwork-Delegate: kuba@kernel.org Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2136.outbound.protection.outlook.com [40.107.244.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BAC1881E for ; Fri, 14 Jun 2024 13:49:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.136 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718372956; cv=fail; b=CLLvx2OoqYNkPmzWW5fI3nrtSsQAn/+dstVv0LiyKQnqQmaXyEBNTRSgEOT6so9h8+7EhJG+t3K2JutHuoVir+VDjsM/QnykuoWnmpK2XYQXg1Myj55nhVjZIjdpRhAMITqaB0q9MFT/E5DWX5gvgJi/gk2Rd3ua+xEwfOpSE8Y= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718372956; c=relaxed/simple; bh=jGDSpEvAr6kg3ZdILSzLjgVaz/3o9z00izF8tUkLwcQ=; h=Message-ID:Date:To:From:Subject:Content-Type:MIME-Version; b=LhjMUBQEp1vcJIDnSNxAjTG4SDV65IfsERgJGhoOsMfLvPQ6DL74gBLqQ1bRuTVSmH5bqRq0p7DMrh6Sag5Ux0VyjsYt8h4M+a/C044AsKHLHo6NHtSPvu0elOvB+1fn98sXm6+EajNvx3u7ccJzJdtlEq64hi2gjzuN+zWTsPE= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=esteem.com; spf=pass smtp.mailfrom=esteem.com; dkim=pass (1024-bit key) header.d=esteem.com header.i=@esteem.com header.b=xFjyNsPx; arc=fail smtp.client-ip=40.107.244.136 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=esteem.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=esteem.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=esteem.com header.i=@esteem.com header.b="xFjyNsPx" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HXKlawSK8BacR1m1CSq4BCiIkme9K5WUR7H/SGsBC6/oyGRUOdDytjJvGAI/ArUJJ81C0cLM7+M7+8YjHNDy2ljPXJ/sdVyZpilfPq5BaHZxQSH0iVR+26GfnmMRplhLn6Z+NzyfN7ZTgSnyqIq+WRGl2m46Jm7jzEM3B9k3qCAVGlPmUq16wXQj2WV6Q6/jyDt8dQIKGtVkuzYWK/kievRb9hoY9xgrz7Iw2Vw2QJwUbaR1Qr5D3BCjpXT+MKoZUqU39135UNnQouJEbaL8i60ZeZVj6AkNgahKUVj5uWA92w/DvAbCPrUyWZ+gy643Ef/owbAXAXm/T2JCPM020Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=EQ6w5gEFo0k1baXP+gwMlTEVDrO2OB0CWbtQ7pSWsjs=; b=jc2Ym1xI8fWWE/gPV0ke+O7gM2W1F3PH0Xp8P/Azb5607V2HqxjYGMKu80xSKoQAXI1QCrRuMgWrN1jaFDRba9RAvzRFpNFkDvVTydNaB+mT0wjXDeSo97b4r0lfNMOIt3KXQb9k3X4d5fxIrKLHhca8bUvmTTRsudqcJ8QUvSjncWYpQaFYZDoFQNWRKmFLm4WJ5wPvuBJ3x8wyiWG5m1t1E6qOTS9KpuhJci85QkkYor7+HKfGob4GAo0V4i/pWNRX/KsywzD4qiPSkMaK7NM2oiYURKVTZCogOX8WgPlVlTmQssAbXbGLYihe7R21DHfDWOHbR1ugk4iBPURoUA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=esteem.com; dmarc=pass action=none header.from=esteem.com; dkim=pass header.d=esteem.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=esteem.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EQ6w5gEFo0k1baXP+gwMlTEVDrO2OB0CWbtQ7pSWsjs=; b=xFjyNsPxMK7y+I7xUvY4UMK9MxDJDojW+wLhf2DVb2Fo/UnbVoj5kqIYyNQKeBiutelNwxFDXQfxJfFP7FcL3KOfIZhunal6hW4NllQMyKeXfepZLYee7k9MsZjEEQiLozetCAajhh41vdehNrMnXdQqB7MxwV3E36LX42hqbKE= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=esteem.com; Received: from DM4PR12MB6373.namprd12.prod.outlook.com (2603:10b6:8:a4::7) by DS0PR12MB9347.namprd12.prod.outlook.com (2603:10b6:8:193::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7677.24; Fri, 14 Jun 2024 13:49:10 +0000 Received: from DM4PR12MB6373.namprd12.prod.outlook.com ([fe80::12f7:eff:380b:589f]) by DM4PR12MB6373.namprd12.prod.outlook.com ([fe80::12f7:eff:380b:589f%6]) with mapi id 15.20.7677.024; Fri, 14 Jun 2024 13:49:10 +0000 Message-ID: Date: Fri, 14 Jun 2024 06:49:08 -0700 User-Agent: Mozilla Thunderbird Content-Language: en-US To: netdev@vger.kernel.org From: Neil Hellfeldt Subject: Throughtput regression due to [v2,net,2/2] tcp: fix delayed ACKs for MSS boundary condition X-ClientProxiedBy: MW4PR03CA0187.namprd03.prod.outlook.com (2603:10b6:303:b8::12) To DM4PR12MB6373.namprd12.prod.outlook.com (2603:10b6:8:a4::7) Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR12MB6373:EE_|DS0PR12MB9347:EE_ X-MS-Office365-Filtering-Correlation-Id: b4796876-c6b3-42de-c518-08dc8c78c46a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230037|366013|376011|1800799021; X-Microsoft-Antispam-Message-Info: =?utf-8?q?oldeT7e2rz3RTDOrz0A5Qkk5vA7CwmC?= =?utf-8?q?7wLkDX0mSAmPrvEkJsL0XS+UdLBOhE0/qTS//MXSFiZOoMl1UJXfFz/ym8InqMq+6?= =?utf-8?q?Rmh2kDmZ1U8I5W6zdSh/2aa2NI4NtpN6k3eQ87Qdb8KSKY+1z/yQusqt2wI0et3ve?= =?utf-8?q?KkFzijyhU27ty1TM24nKrkib5KYe/J/l6CaxzFTkhL0BLzzC5x1E30dpLBNYR24dQ?= =?utf-8?q?jQrTtcQQHSgEi2iK4++X26EEwblABtKwPDxsiXP6B2MKW1wyiDtwPcUWEdB5LJqw1?= =?utf-8?q?s93eDkO8VFq6OK4thXnZa5HaGWvKOywJ61gQC5pfoZo+sSaBlciklhQGNGAn3CxFT?= =?utf-8?q?6JEy0lczA8fyvHG47POlEDdnk5IfdAr15qkTSFuetZcJ+bhFLHf/0m1V4V6iERuR2?= =?utf-8?q?ehQa/z3IP0Po2v1lw/6CJuLsbrjZqELIqZ+w1o+K1EwH3twvcIoonWNl1qWE4EULW?= =?utf-8?q?uXdOasNkyhO2TzLXYLCyPSdbFalk2ZUOxilb/vI0D1CvMbejTINTVkjuyeVJGgfqW?= =?utf-8?q?EfrKEPhDf5phaPB8c2ljX/tZbNjVYWNq2G5HayauHhnEfm/mN9mkNgU1zSkJ/o03b?= =?utf-8?q?b1X6jCNHCYt3M2YXDWOoqwQHj3yYxulp378dXrFBEBEnJ2iNVTQ1UNt9DB1AqkIWf?= =?utf-8?q?cAYNqJ3StjUUusrIR4anuOJXsnmsQVm+y5ohP+8rTDjdpC40cqxD0fnQ9mbcGIjWo?= =?utf-8?q?PZh39etgtbXaOUThFwu6Ie5Qon11sxEmLdUWe2OZY9TRcamYTSeyvNKVh4GyIGrST?= =?utf-8?q?UUPFPi7+LjVMtUpQmo1g7RBwI6MNitFIhvShlZQcPaw1ZazvD4Bm41h0xdowUGkmM?= =?utf-8?q?UwsMFDeNJ16jLCq52aYZavN4UTVjx50FRo+CgP38wvwkd1P2nLD7/k3FPGG11MgPZ?= =?utf-8?q?6MM/kbBb4Qel90048MjArg1CFBkHDTb1bEKkxvP/tZqpXoFiVVkDkF5SLqAPsg8pI?= =?utf-8?q?kuOjcSHrcFL0cfplXbcs8j3W5J6eaC58ngBX6P31vbmFzU0nd0P0omz2oEQKghh1I?= =?utf-8?q?i+M9FtZ4jO4KTZkNWZdXk2wg45TyCrYPqqyp+Y7ObkQNNhFM91RXSDQ+3tCvm80No?= =?utf-8?q?UdEgi0rQ6UAiI0rffyPxGvQ66POtqigGB4bC6D1nsFRdRSCU4goP9FLCfr5Clu9xJ?= =?utf-8?q?rmwyfB1MBX0zB1DNiQr9Pv7pg34IZRCkTYAZEOEpf/KhcLlTDCjqunjfT7C4BWn5o?= =?utf-8?q?2wBj1Kt5DOzRmc6CSEUOU8HnLccRscfPy7SlB4LaO41iylw12lSpot8yuVXN7RLfu?= =?utf-8?q?MSkOtaRY+qDxC?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM4PR12MB6373.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230037)(366013)(376011)(1800799021);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?gP7+7i7G91qRC17XjAy6HRI5A3y+?= =?utf-8?q?s+4/NBph9IRy2sUvwwlG3EMsDVB9YM3cP0eMv4EDso1XeEmgj+m/WzwyTFjrpfRx9?= =?utf-8?q?QLDvyidLOiUNJq9/ZGuJvhhgP1HX34vzm89FlyaBUxWY/261bjtjNrmorDSEhi7od?= =?utf-8?q?C1h7rGnAC5TUwLxUsJZbY5tcEOUleJ9lz4dpyfnHoh9WnMqftX+YIzXH+gwY9bVT0?= =?utf-8?q?MCYDq/uD4T90928GktiFCDT4absVlwfnFYArT1/laOQcRyggpa/DK7rydiHaJb5xu?= =?utf-8?q?rYnNvv65FjzcbBHQKol6bw9hx9MVQ+5SOenXkCCSRL4LBikflAPuTjo+3NzsQS+o6?= =?utf-8?q?k9vUl44SiXNDszl/Ygm3GkJ1jmT0z4VQg2i5V6l+TLzyRTBSETxpTIKqG+GHHcNIs?= =?utf-8?q?omseySLAAf/jx6Y5elP27CAp9Y1PQP4YCHBPk/4W5sVAvMmEXPMDcmFRs6Bb5NU1K?= =?utf-8?q?aKGTNm8+yExa/TcimEx1c+ubzEjcJIqSNWg2JqAzC3M/0W1+7Sn+htahY706QaFRw?= =?utf-8?q?UWDebiq1hg9KZJ6iFJjHRv9RYG5FDlCXwR4oHRNrjh4rlgbSICBK9TZLHi/qmCXDv?= =?utf-8?q?vz/Lx7mPhz/iJFqO5BAJs789iAo2Wdp2/VkVzKf8J2wt6f1ap+T4mengXqnakHX5T?= =?utf-8?q?V/mFGe7MjkURYdlCaMVZnMOVS3pcW1AqA7AIjdklitrzoAceOuYNVjPMeE/Ovz9RR?= =?utf-8?q?+7U7w6XCtDPXQ3g3ZgJu4gZf9qIEI4wpBHqxbpxVgupVBQHLy7h+Eg5ZMWw5gt5Ou?= =?utf-8?q?/b3xl/q6mQK9MSOjp9syUfR7ZrAd0ova9L6DsnpLVmLhMfW2cvFGMbzRu5Xj1BKq9?= =?utf-8?q?aF1JZWiWdRWmkWl7oOKWXCOv7hxNqCuo1wRGL27cF8wLwjE73gX11ReklbYma+8h5?= =?utf-8?q?JcSzOaWzyNfyujbEYKVGtC3CJMl/9zCrULg/Syjn8duCwPlqLxAzGlDDQtQXYr9bq?= =?utf-8?q?rc+UfZqO3q+t+KUIHRU1kY9jc6/XevAOg12n7x+UIzkuICdFFqeD8TfBCfVN2d3da?= =?utf-8?q?nwbQAc7v21h1ccA0sIaBlKgPdHvPYl5EIwquOVjQ2K2MYRicXc0RGYPCSbQN7FtA4?= =?utf-8?q?AW9WogROebNne89IQoJq3SkSddX92ErLZxLhiQDDM/N8CCfhv4qhIvk74hen0CtyC?= =?utf-8?q?SSCk5rKzgcRvKYg9VXX/qsrTJqXsnfhFNeW0astfKyWIGhSxX52Pf6T+LN7eIJU0Y?= =?utf-8?q?BhQfzWoVGWMBEoo4wXG4FiX/BmqNNbBqDbENlmmxGH8yjVnLr7YQoB6zBiveQMno0?= =?utf-8?q?i5bj3zbuk+WQ8MOwVK21JP4f/FA6aa95G/J54RCaLEN9kEcD95lc+mz54OTs1U2sa?= =?utf-8?q?wJCsGkywXl//1HnXe4Ub5uwmy14iEUBNeDeQURylx+7q9qemdOT7CrZQ21FwxsvzV?= =?utf-8?q?d/aR5tO/XrabX8nuIu9qgzJZLXhuO/N5vlMYgYC3H/YL8caHjrQhIPeqJYNn19xB9?= =?utf-8?q?zZD/lO/J4GyMKaw450oKNqQxXCMa3bkeYok2zLkrkaC+FnFS6MXpbLLmdzM4GIp/x?= =?utf-8?q?ucCg0lUkfBtr?= X-OriginatorOrg: esteem.com X-MS-Exchange-CrossTenant-Network-Message-Id: b4796876-c6b3-42de-c518-08dc8c78c46a X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB6373.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jun 2024 13:49:10.1390 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: f32e9c2f-8342-451b-8efe-15edc71a9887 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VlcHdg+Q6xE4j75hIi3bBw8Z2jFR+v7o7jThjqbE4ryMmyLZMkviF6eGkJu8rrJfd9ntRNhtL6pmd56AcMP9Bw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB9347 Hi, So I believe I found a regression due to: patch: [v2,net,2/2] tcp: fix delayed ACKs for MSS boundary condition commit: 4720852ed9afb1c5ab84e96135cb5b73d5afde6f I recently upgraded our production machines from Ubuntu 16.04 all the way up to 24.04. In the process I noticed that iperf3 was no longer able to get the throughput that it was able to on 16.04 I found that Ubuntu 22.04 is when it broke. Then I found that Ubuntu's kernel version 5.15.0-92 worked fine and version 5.15.0-93 did not. After that I narrowed it down to the patch: patch: [v2,net,2/2] tcp: fix delayed ACKs for MSS boundary condition commit: 4720852ed9afb1c5ab84e96135cb5b73d5afde6f After I removed the patches I was able to get the expected speeds. I then reverted the patched on most current version of the kernel for Ubuntu which is 6.8.0-35 and I was able to get the expected speeds again. The device unit under test is a embedded device with lwip and built in iperf3 server. It has 100mbit network port. It is also a low data rate wireless radio. The expected rate for the Ethernet is ~52000Kbps avg with the patch applied it was getting ~38000Kbps. The expected throughput for wireless is ~328kbps with the patch applied were are getting ~292Kbps. That a ~27% throughput regression for Ethernet and a ~11% for the wireless. command used iperf3 -c 172.18.8.134 -P 1 -i 1 -f m -t 10 -4 -w 64K -R Iperf version is 3.0.11 from Ubuntu. The newer version of iperf3 3.16 show the correct speeds but shows a saw tooth plot for the wireless. diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 06fe1cf645d5a..8afb0950a6979 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -253,6 +253,19 @@ static void tcp_measure_rcv_mss(struct sock *sk, const struct sk_buff *skb) if (unlikely(len > icsk->icsk_ack.rcv_mss + MAX_TCP_OPTION_SPACE)) tcp_gro_dev_warn(sk, skb, len); + /* If the skb has a len of exactly 1*MSS and has the PSH bit + * set then it is likely the end of an application write. So + * more data may not be arriving soon, and yet the data sender + * may be waiting for an ACK if cwnd-bound or using TX zero + * copy. So we set ICSK_ACK_PUSHED here so that + * tcp_cleanup_rbuf() will send an ACK immediately if the app + * reads all of the data and is not ping-pong. If len > MSS + * then this logic does not matter (and does not hurt) because + * tcp_cleanup_rbuf() will always ACK immediately if the app + * reads data and there is more than an MSS of unACKed data. + */ + if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_PSH) + icsk->icsk_ack.pending |= ICSK_ACK_PUSHED; } else { /* Otherwise, we make more careful check taking into account, * that SACKs block is variable.