From patchwork Fri Dec 13 12:27:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yunsheng Lin X-Patchwork-Id: 13906961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1AC6E7717D for ; Fri, 13 Dec 2024 12:34:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52E286B007B; Fri, 13 Dec 2024 07:34:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DEEB6B0082; Fri, 13 Dec 2024 07:34:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A6456B0083; Fri, 13 Dec 2024 07:34:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 1A3796B007B for ; Fri, 13 Dec 2024 07:34:35 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B8DA780D07 for ; Fri, 13 Dec 2024 12:34:34 +0000 (UTC) X-FDA: 82889878218.27.D98C6F7 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf14.hostedemail.com (Postfix) with ESMTP id 77866100012 for ; Fri, 13 Dec 2024 12:34:03 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734093255; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references; bh=pzVh31aWs2NP8LhqbDMfDSjESjxfi4CnNp83YjhQt4Q=; b=DK18kh5aPBn0djNnNEqyPF2tmOwWqCgyRBHLVSyQuAsAxkiyHnugXMi3oj0+gDsLZvHVRG oom/ZFfgzq3jS1C6BhsxmVXODkp00yfWc54OCFhXwncNjlp/8jOSxnipbLFeLRoXDmSGPp Dr50tx3OxnpdFJz/F6cRkxP3FvuD7dc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of linyunsheng@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=linyunsheng@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734093255; a=rsa-sha256; cv=none; b=VlKJooDntDYA+ciZCDxLJurJXffGLGP9bmv0TA+RE7z4EzeE6ut/51dH/ExTAX5j3gsE1w Seq0hz8qJigK7/9Xd87y1icVA0LUNzYika83iF6F1NVX2P4NtIJEbbFoQW2dAcjOQ5aRsb 8+iuATG+9+Pu2oKJp4t9q+p4YQYPcTI= Received: from mail.maildlp.com (unknown [172.19.88.194]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4Y8pcZ33WXzhZTt; Fri, 13 Dec 2024 20:31:58 +0800 (CST) Received: from dggpemf200006.china.huawei.com (unknown [7.185.36.61]) by mail.maildlp.com (Postfix) with ESMTPS id 538CD140123; Fri, 13 Dec 2024 20:34:25 +0800 (CST) Received: from localhost.localdomain (10.90.30.45) by dggpemf200006.china.huawei.com (7.185.36.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Fri, 13 Dec 2024 20:34:25 +0800 From: Yunsheng Lin To: , , CC: , , , , Yunsheng Lin , Alexander Lobakin , Robin Murphy , Alexander Duyck , Andrew Morton , IOMMU , MM , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Matthias Brugger , AngeloGioacchino Del Regno , , , , , , Subject: [PATCH RFCv5 0/8] fix two bugs related to page_pool Date: Fri, 13 Dec 2024 20:27:31 +0800 Message-ID: <20241213122739.4050137-1-linyunsheng@huawei.com> X-Mailer: git-send-email 2.30.0 MIME-Version: 1.0 X-Originating-IP: [10.90.30.45] X-ClientProxiedBy: dggems702-chm.china.huawei.com (10.3.19.179) To dggpemf200006.china.huawei.com (7.185.36.61) X-Rspamd-Server: rspam05 X-Stat-Signature: jexk56gs1smmnz3z8zhwd4rzmzhigy3j X-Rspamd-Queue-Id: 77866100012 X-Rspam-User: X-HE-Tag: 1734093243-136571 X-HE-Meta: U2FsdGVkX18/Js+fC4mRrQXOCNcBkMrgqXg7IpJ2GXbtzbEV8XEPzGdYOTIAWk0WM3wVIU7gQzw1e3oNH/uRre3tMGx9RXzUFwyRIkJMdusMBSV6Jjd4IpI9COxJABcc4HbN0/Yf805Kn47qUZdQWgLtwneoLEb8SxihSYxTmZHIFihqrbjFhmJ++b0udq8B1iPbPDZFZVgaGD5HvybDwb6/CNS3NhUgCtf7ICCIn+w0iqh7iz18OykY55lFZQnE3eUGt/bX2Jh0X0ZK5vKVYxyYoxj8nywn7aVioI4+8Q7ikvQCFySstGUCe+/v5L/mRQvwV32mrNjCWl7KbKqMujjbHyIMAijojW8L4aVl+6YCV/RTNVGXpPI6R/2HN3qYi4rOCiHPgVhf40Z+6pJyJuG4Rn0V6YzyzTaY5IRefWJi8AVXLhrYh2znDGuW5/rA8FnRLKKsXzpf3x6bS5wMq2/6ZzQj6mgAuvVyCsUrmEFqPPTkMO/ZXs7cheVdJr8SXKK2zhdN+BICJNyHR8sUCtTU35qBNsU5i10QnM1YJ0haJwtXZAn21wZ7Iq8vm8pDsfN++2Xbutjc0vtl6Q8LxtNs9NIn3mH4Vxfl4ecSotXIJW79jvYysavp0IBW7Ntmk25xJb6Q8PeDCVDk2dQClZSunQTny+Ll97cXIjWu1l2XzLZXtqmlhkYmwXzrIG4jt0RbCF62n0jOsnwkBvCe/+wbsFXP9XBos8doQak3qCZxOLhYB/PoHPgl/Jf44A5bq0CRkpHuBCqPYmJYW9xxh60HmzGvRlVFBXTCfSK5uZyNqfbFIw1jpYK0n/Im8w+d5rWBaL3dVNWqD+ecF2pramogad2IaMUREFhFFhMMNsK5X7WltciQQovtH5p72VBo0SF8+lyve8dSsL+Ksh1UdSFOznKG7NZUysibm+lHEEcCiWn+dPXDQk9c+/FGWSX1A3MwhS1CVr6enrED5KL /AcEpW5Y iAau4nvU2XOztLu0o9tfSIeG5GdRop3Fo395A9EJnl7aAQdaVOY+EmF7sqHbKQtH/OV1L7F1Ovp/ac8oIkERuL8JQHf264/AEHLgMNaJOYpxIOTFDJxjiaiZf9JDYUJZeCAeowDuciV6WrJqRhpnhwfJN8sviiyCrcGkWMzxbuLHrmU/FBHPsfnWr0auFr54UYn/dB03XDfblAbooGWglZPvUqiel3tWqHMV22SR8eIDDpQmtzPYx18xHMQg4NpPe03Gtxs519li7mBfdkx46KD0s/1GGaO4Qe0uVS3J84HDG38zpdn5CyM5BB7CASL+E3t6IQkh6oeZHDga7MQKdfTpPpdAXV5beqnV3dGrXzZtN9fXblhw/98hu5zVKvFQmsiKwoUftBToKdBn0sl+vddSWuQO+oUF4IzYDVQqj6TdFZSn800qSqxD3WqoPhhTlV7pN9A93lCfV2L4CRwcBdtztjAhdJbdbKmV3yS/Hc8Wntu8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This version is mainly to see if using page_pool_item metadata to keep track of all pages is the correct way to fix the dma API misuse problem. Note, it is not based on the latest net-next tree yet, but based on the below commit in net-next: commit da4fa00abe56 ("Merge branch 'mitigate-the-two-reallocations-issue-for-iptunnels'") From the below performance data, the overhead is avoided as much as possible for time_bench_page_pool01_fast_path() and time_bench_page_pool02_ptr_ring, and there is about 20ns overhead for time_bench_page_pool03_slow() for fixing the bug. Before this patchset: root@(none)$ insmod bench_page_pool_simple.ko root@(none)$ insmod bench_page_pool_simple.ko [ 67.667196] bench_page_pool_simple: Loaded [ 67.748321] time_bench: Type:for_loop Per elem: 0 cycles(tsc) 0.769 ns (step:0) - (measurement period time:0.076977910 sec time_interval:76977910) - (invoke count:100000000 tsc_interval:7697783) [ 69.851812] time_bench: Type:atomic_inc Per elem: 2 cycles(tsc) 20.862 ns (step:0) - (measurement period time:2.086207700 sec time_interval:2086207700) - (invoke count:100000000 tsc_interval:208620764) [ 70.019852] time_bench: Type:lock Per elem: 1 cycles(tsc) 15.015 ns (step:0) - (measurement period time:0.150151600 sec time_interval:150151600) - (invoke count:10000000 tsc_interval:15015154) [ 70.691100] time_bench: Type:rcu Per elem: 0 cycles(tsc) 6.541 ns (step:0) - (measurement period time:0.654142450 sec time_interval:654142450) - (invoke count:100000000 tsc_interval:65414239) [ 70.708119] bench_page_pool_simple: time_bench_page_pool01_fast_path(): Cannot use page_pool fast-path [ 70.975262] time_bench: Type:no-softirq-page_pool01 Per elem: 2 cycles(tsc) 25.785 ns (step:0) - (measurement period time:0.257850110 sec time_interval:257850110) - (invoke count:10000000 tsc_interval:25785005) [ 70.993931] bench_page_pool_simple: time_bench_page_pool02_ptr_ring(): Cannot use page_pool fast-path [ 71.575053] time_bench: Type:no-softirq-page_pool02 Per elem: 5 cycles(tsc) 57.191 ns (step:0) - (measurement period time:0.571916900 sec time_interval:571916900) - (invoke count:10000000 tsc_interval:57191684) [ 71.593722] bench_page_pool_simple: time_bench_page_pool03_slow(): Cannot use page_pool fast-path [ 73.384560] time_bench: Type:no-softirq-page_pool03 Per elem: 17 cycles(tsc) 178.197 ns (step:0) - (measurement period time:1.781979820 sec time_interval:1781979820) - (invoke count:10000000 tsc_interval:178197975) [ 73.403581] bench_page_pool_simple: pp_tasklet_handler(): in_serving_softirq fast-path [ 73.411485] bench_page_pool_simple: time_bench_page_pool01_fast_path(): in_serving_softirq fast-path [ 73.678410] time_bench: Type:tasklet_page_pool01_fast_path Per elem: 2 cycles(tsc) 25.780 ns (step:0) - (measurement period time:0.257807630 sec time_interval:257807630) - (invoke count:10000000 tsc_interval:25780758) [ 73.697686] bench_page_pool_simple: time_bench_page_pool02_ptr_ring(): in_serving_softirq fast-path [ 74.242807] time_bench: Type:tasklet_page_pool02_ptr_ring Per elem: 5 cycles(tsc) 53.608 ns (step:0) - (measurement period time:0.536089620 sec time_interval:536089620) - (invoke count:10000000 tsc_interval:53608957) [ 74.261996] bench_page_pool_simple: time_bench_page_pool03_slow(): in_serving_softirq fast-path [ 76.115762] time_bench: Type:tasklet_page_pool03_slow Per elem: 18 cycles(tsc) 184.508 ns (step:0) - (measurement period time:1.845082100 sec time_interval:1845082100) - (invoke count:10000000 tsc_interval:184508203) After this patchset: root@(none)$ insmod bench_page_pool_simple.ko [ 88.665991] bench_page_pool_simple: Loaded [ 88.747105] time_bench: Type:for_loop Per elem: 0 cycles(tsc) 0.769 ns (step:0) - (measurement period time:0.076981170 sec time_interval:76981170) - (invoke count:100000000 tsc_interval:7698109) [ 91.585102] time_bench: Type:atomic_inc Per elem: 2 cycles(tsc) 28.206 ns (step:0) - (measurement period time:2.820699360 sec time_interval:2820699360) - (invoke count:100000000 tsc_interval:282069929) [ 91.753048] time_bench: Type:lock Per elem: 1 cycles(tsc) 15.005 ns (step:0) - (measurement period time:0.150057320 sec time_interval:150057320) - (invoke count:10000000 tsc_interval:15005727) [ 92.424306] time_bench: Type:rcu Per elem: 0 cycles(tsc) 6.541 ns (step:0) - (measurement period time:0.654151520 sec time_interval:654151520) - (invoke count:100000000 tsc_interval:65415145) [ 92.441325] bench_page_pool_simple: time_bench_page_pool01_fast_path(): Cannot use page_pool fast-path [ 92.696225] time_bench: Type:no-softirq-page_pool01 Per elem: 2 cycles(tsc) 24.560 ns (step:0) - (measurement period time:0.245607210 sec time_interval:245607210) - (invoke count:10000000 tsc_interval:24560715) [ 92.714893] bench_page_pool_simple: time_bench_page_pool02_ptr_ring(): Cannot use page_pool fast-path [ 93.336550] time_bench: Type:no-softirq-page_pool02 Per elem: 6 cycles(tsc) 61.245 ns (step:0) - (measurement period time:0.612451380 sec time_interval:612451380) - (invoke count:10000000 tsc_interval:61245127) [ 93.355219] bench_page_pool_simple: time_bench_page_pool03_slow(): Cannot use page_pool fast-path [ 95.402370] time_bench: Type:no-softirq-page_pool03 Per elem: 20 cycles(tsc) 203.828 ns (step:0) - (measurement period time:2.038286740 sec time_interval:2038286740) - (invoke count:10000000 tsc_interval:203828660) [ 95.421395] bench_page_pool_simple: pp_tasklet_handler(): in_serving_softirq fast-path [ 95.429301] bench_page_pool_simple: time_bench_page_pool01_fast_path(): in_serving_softirq fast-path [ 95.684025] time_bench: Type:tasklet_page_pool01_fast_path Per elem: 2 cycles(tsc) 24.560 ns (step:0) - (measurement period time:0.245605490 sec time_interval:245605490) - (invoke count:10000000 tsc_interval:24560544) [ 95.703301] bench_page_pool_simple: time_bench_page_pool02_ptr_ring(): in_serving_softirq fast-path [ 96.401877] time_bench: Type:tasklet_page_pool02_ptr_ring Per elem: 6 cycles(tsc) 68.954 ns (step:0) - (measurement period time:0.689544160 sec time_interval:689544160) - (invoke count:10000000 tsc_interval:68954410) [ 96.421065] bench_page_pool_simple: time_bench_page_pool03_slow(): in_serving_softirq fast-path [ 98.496283] time_bench: Type:tasklet_page_pool03_slow Per elem: 20 cycles(tsc) 206.653 ns (step:0) - (measurement period time:2.066533210 sec time_interval:2066533210) - (invoke count:10000000 tsc_interval:206653316) 1. https://lore.kernel.org/lkml/8067f204-1380-4d37-8ffd-007fc6f26738@kernel.org/T/ CC: Alexander Lobakin CC: Robin Murphy CC: Alexander Duyck CC: Andrew Morton CC: IOMMU CC: MM Change log: V5: 1. Support unlimit inflight pages. 2. Add some optimization to avoid the overhead of fixing bug. V4: 1. use scanning to do the unmapping 2. spilt dma sync skipping into separate patch V3: 1. Target net-next tree instead of net tree. 2. Narrow the rcu lock as the discussion in v2. 3. Check the ummapping cnt against the inflight cnt. V2: 1. Add a item_full stat. 2. Use container_of() for page_pool_to_pp(). Yunsheng Lin (8): page_pool: introduce page_pool_to_pp() API page_pool: fix timing for checking and disabling napi_local page_pool: fix IOMMU crash when driver has already unbound page_pool: support unlimited number of inflight pages page_pool: skip dma sync operation for inflight pages page_pool: use list instead of ptr_ring for ring cache page_pool: batch refilling pages to reduce atomic operation page_pool: use list instead of array for alloc cache drivers/net/ethernet/freescale/fec_main.c | 8 +- .../ethernet/google/gve/gve_buffer_mgmt_dqo.c | 2 +- drivers/net/ethernet/intel/iavf/iavf_txrx.c | 6 +- drivers/net/ethernet/intel/idpf/idpf_txrx.c | 14 +- drivers/net/ethernet/intel/libeth/rx.c | 2 +- .../net/ethernet/mellanox/mlx5/core/en/xdp.c | 3 +- drivers/net/netdevsim/netdev.c | 6 +- drivers/net/wireless/mediatek/mt76/mt76.h | 2 +- include/linux/mm_types.h | 2 +- include/linux/skbuff.h | 1 + include/net/libeth/rx.h | 3 +- include/net/netmem.h | 10 +- include/net/page_pool/helpers.h | 11 + include/net/page_pool/types.h | 57 +- net/core/devmem.c | 4 +- net/core/netmem_priv.h | 5 +- net/core/page_pool.c | 660 ++++++++++++++---- net/core/page_pool_priv.h | 12 +- net/core/skbuff.c | 3 +- net/core/xdp.c | 3 +- 20 files changed, 650 insertions(+), 164 deletions(-)