From patchwork Mon Dec 11 03:52:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13486641 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A727BC4167B for ; Mon, 11 Dec 2023 03:53:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 212A56B0098; Sun, 10 Dec 2023 22:53:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C22C6B0099; Sun, 10 Dec 2023 22:53:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 062346B009A; Sun, 10 Dec 2023 22:53:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E66686B0098 for ; Sun, 10 Dec 2023 22:53:17 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id ACEB0160587 for ; Mon, 11 Dec 2023 03:53:17 +0000 (UTC) X-FDA: 81553167234.25.8AF7CC2 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf11.hostedemail.com (Postfix) with ESMTP id C285040017 for ; Mon, 11 Dec 2023 03:53:15 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MmMq8ve7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702266795; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0gkCO+cmYizzxtKpmJJFZh1hseverSK4jjEXH+6EVns=; b=VuxfP8pVIyj4OkJzyoirA/X1Y5ikakJowH9vA0oDNzWhrYF9QkxDUu7mIZqpZuZQQ95kKd g37G4PYGIjBEIt96i4MOY6aQ6A/0zM4+9SXhwfKib3WDEioe166+YL0jsNxtele1avDurn RP4INBXO5YxcefyyuYFEBVkFn5+8buk= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MmMq8ve7; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702266795; a=rsa-sha256; cv=none; b=SLJ7QbDYUVWpbThYo6aAjh+jRQeuiD9SOxLnq6blvjba0uiX/dyCbvP42lBMs/fib/Qw9H NuiCYujdTBk67JVJ4KfYjiPRu2op+mqXekmBdNq35ZhWa45W39KumCXJKffLQs2cXSqPSg fpMeFCFfHazuRhsOk/FKLMbivM052VY= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-1d06d42a58aso37528025ad.0 for ; Sun, 10 Dec 2023 19:53:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702266794; x=1702871594; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0gkCO+cmYizzxtKpmJJFZh1hseverSK4jjEXH+6EVns=; b=MmMq8ve7SW2RZVGXjtz40sAFh1Q9oPoBLiELfRc2rIEjE6NPlVzKO7dzonA49A+2xT Qpy1sS27jB4u0SoiAMb5Xoil+3umBwjGTXVxBe3UF97gzrBZLYVPxjvyOpaW+E1c5yNs Dl/iU3FXESId7MeBksO3z27ORWbIwVmq5HY9DT4H/6AOHzhXQ2/bb8sse/8MQr8tzlVt eK9JGdOGh0N2kAuBgFYGeX1QTvXyIxtQ/JM/air6EXjpk+5U8iFuTkGAZu/tiOL9Vq5C JLOOVwoRb+Gzb4O5J5dTpX9cRxoQlxzX65QgTMGjHlCx5eDoKAAiIWAzQFqkOTIlTBnY F0Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702266794; x=1702871594; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0gkCO+cmYizzxtKpmJJFZh1hseverSK4jjEXH+6EVns=; b=ukezJ1NHYAejDGLFXmowO7scCRRAzh/L2+PXqk/IWVyQii6DvcjvdIJAQKtmS6ex9j 2clX64nSfOFOrFOAALtnsOankH3OxCRLikWHhcBcFfg7ViHzaLRvTVzFxloXOCWQKKux /LGCctXUG2ZQpTKKqW1JWKTxJ50NsiWqhE66D6XZnlEcbQiAIlb2T2POKxZKLoiXxw6P OZgd9l700ds7Lb7t5TUVANUXPH2LkaX+huN8ZPXEJYSs5HDGNtvPZrfOx5e7y3QfnZN5 YnVZWnx9NBebsaf4HqcvJakwC7he99zINxr2WTpuTdlUncTJCp34GEixojABlkrkrTYc xHgA== X-Gm-Message-State: AOJu0YxzC2G3f94KyanvWLb2OHcPrweTRM6GfkKDWfB0eFR4X0iQ3OGP vufgOkDEMkm15NUq6UzHDfo= X-Google-Smtp-Source: AGHT+IEXf6IbJoR8Iz9MNTsR6P223Ynd+dcUtTQl8p+o+6SoIZRPusRJS8EGhxpR+3WfEFg3O3977A== X-Received: by 2002:a17:902:a5c7:b0:1d0:7844:5086 with SMTP id t7-20020a170902a5c700b001d078445086mr3242096plq.7.1702266794499; Sun, 10 Dec 2023 19:53:14 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id b8-20020a170903228800b001d052d1aaf2sm5411491plh.101.2023.12.10.19.53.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 19:53:13 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, jasowang@redhat.com, almasrymina@google.com, liangchen.linux@gmail.com Subject: [PATCH net-next v8 1/4] page_pool: transition to reference count management after page draining Date: Mon, 11 Dec 2023 11:52:40 +0800 Message-Id: <20231211035243.15774-2-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231211035243.15774-1-liangchen.linux@gmail.com> References: <20231211035243.15774-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C285040017 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: p6wqf138kaqba4qapqyjiqkxmy6k3ofu X-HE-Tag: 1702266795-235898 X-HE-Meta: U2FsdGVkX1+sUNh1BjxHuW7h9bp/kyDGVRBabn0Rkz7b9Ojc16uHZK8Q6PjWxStXFhZP5ZpgXcHd4OAepGPtDoxrRijxzrM85ipZANbEYKS/RLA8oGH4M1Dd0oTRH6LXbPcr8uFoabENLZGNielc4VbmLC2utuWI0Sa8NMmtM8eaP4p54eflz7VcXhUOZ/cvb4nISCb81b/ywi9U7wzAJwWvuBDyu/pbo1mOLT5jT6h0G92THnfF7rpdzuT5CPq+LaKU7trku9raYj09Eho8s/xqLm46HKWvi5u9Y+l/npCmgR0x6+JA4RBZqEvfVTPeG1W70vuYIaARKR2cxvJ0VpUaTvZ6dUtBr3Gyz42ke4MepdXaRLLydhNVCqV/WO/EudYW3rxgW7LX6jFU7tuETnk96hdHneAQganWVGoFW9gm3DRhqU+a+7ZWP2X6VMYfvEKPlD8KkP7Eyrmgg/qDnCSvsXhoZBZ+f1gBJxx/QsBmrqyK08bCwNRmKHveNUN16+lXxc237xI5tM/0kx069AEyWJqBZnhMb9NJNwyXVmxBrykernICIkyMgSxjtiKVb8oSDZbFnrSgpqWJaRHjsS/GcTbJPLZ/oyM7E5kXbiuDQV3lVkLrUVJPdm15cMHoNL2mr6QQE6CUF3Rm17L9br6isUZ2+boRAlCVK6J6VKT7b5jiSn61/l3hy86Eb3MOzEG/cERz9Wx+En+Tlng89RITW8UTW08U+VK/fqrEmG62Z7GMzRHKBZmSqB63l5hBIpF5qB0Hbgdz0+Xo5h4uLxYF55qq9Qg6Qlho1Tac/ZJJfFbTjwQjqTzE2zEPpFD4FKM1KaOZ4hUFH68uoHif4OIdFY+Ss04uPAxJBwxYb45t9Mr7tezbfL/wSAhZ+VtrdT/2o4CNMJfrJXRzsK6AINqoFS06UZlKhccvibZyWMc7U84HhKDSSXg+tIGvCjdjvKnv/yMj+l1g64rh3x/ 64ls7ttz 4qjdbw4S/klF3AQz0goHiAJdksvNOqlsENIXKX12cD2HBHKdGZqhXT9olycd9IDilJ8FZ6EZMDE3bcT38bt0A9vCut/hNLvn5UweVXu9UG/GUFhIKzKy/b61x4R823NP1hdNRegpZnuM4oKDtX59t3rdWmRPoWIbdNHwtE90+nNU0CIDyh+j/sJjC/EtFqX1LHchxGpFHAjOS8ioJgDU3ZooE5hTgrGrhr8KzKfZnj/z4Pmb9L2ZvbaLLkdCRZ3zjzFzTwORDAglGEmobBi+EEtsOK70DCgBvyFsueL2o9jbK46s30tZHyDa3jXywwLGmbrqipKZOJWSmMQslzWbOl5Kb+j4jOti8/OBwOzv8NnSDQxvKPBkxnawurQpcPyqnWZbyB17Z6/2n/3q2jwce2Y3a5d/HKxh4cFJvmrDX+a4719TjvduDN0sC0Ky734pp/u0ZAIEyB/7wGRwzgVjlsmAYC/2HJu7uTjhA5SqO7yB+B5gGZLxsEE0lZWsWt4DuTf8yemGMzwByKeYu4ccNCLcLoH5W8RWSZIsBih1egmWruE/8W5pgIPy1/x2Jyi9giKXBP4tcHISzw30d/EIcc1D0mffroc9QX8gyr6ry2dIo3a6nhOP8lLiW+OGD0/BVugFzg562BoJKzRDx0OGEWNn7eRMmicGdLGQ586BRN/liTwU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To support multiple users referencing the same fragment, 'pp_frag_count' is renamed to 'pp_ref_count', transitioning pp pages from fragment management to reference count management after draining based on the suggestion from [1]. The idea is that the concept of fragmenting exists before the page is drained, and all related functions retain their current names. However, once the page is drained, its management shifts to being governed by 'pp_ref_count'. Therefore, all functions associated with that lifecycle stage of a pp page are renamed. [1] http://lore.kernel.org/netdev/f71d9448-70c8-8793-dc9a-0eb48a570300@huawei.com Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin Reviewed-by: Ilias Apalodimas --- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 4 +- include/linux/mm_types.h | 2 +- include/net/page_pool/helpers.h | 60 +++++++++++-------- include/net/page_pool/types.h | 6 +- net/core/page_pool.c | 12 ++-- 5 files changed, 46 insertions(+), 38 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 8d9743a5e42c..98d33ac7ec64 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -298,8 +298,8 @@ static void mlx5e_page_release_fragmented(struct mlx5e_rq *rq, u16 drain_count = MLX5E_PAGECNT_BIAS_MAX - frag_page->frags; struct page *page = frag_page->page; - if (page_pool_defrag_page(page, drain_count) == 0) - page_pool_put_defragged_page(rq->page_pool, page, -1, true); + if (page_pool_unref_page(page, drain_count) == 0) + page_pool_put_unrefed_page(rq->page_pool, page, -1, true); } static inline int mlx5e_get_rx_frag(struct mlx5e_rq *rq, diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 957ce38768b2..64e4572ef06d 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -125,7 +125,7 @@ struct page { struct page_pool *pp; unsigned long _pp_mapping_pad; unsigned long dma_addr; - atomic_long_t pp_frag_count; + atomic_long_t pp_ref_count; }; struct { /* Tail pages of compound page */ unsigned long compound_head; /* Bit zero is set */ diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 4ebd544ae977..d0c5e7e6857a 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -29,7 +29,7 @@ * page allocated from page pool. Page splitting enables memory saving and thus * avoids TLB/cache miss for data access, but there also is some cost to * implement page splitting, mainly some cache line dirtying/bouncing for - * 'struct page' and atomic operation for page->pp_frag_count. + * 'struct page' and atomic operation for page->pp_ref_count. * * The API keeps track of in-flight pages, in order to let API users know when * it is safe to free a page_pool object, the API users must call @@ -214,69 +214,77 @@ inline enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool) return pool->p.dma_dir; } -/* pp_frag_count represents the number of writers who can update the page - * either by updating skb->data or via DMA mappings for the device. - * We can't rely on the page refcnt for that as we don't know who might be - * holding page references and we can't reliably destroy or sync DMA mappings - * of the fragments. +/** + * page_pool_fragment_page() - split a fresh page into fragments + * @page: page to split + * @nr: references to set + * + * pp_ref_count represents the number of outstanding references to the page, + * which will be freed using page_pool APIs (rather than page allocator APIs + * like put_page()). Such references are usually held by page_pool-aware + * objects like skbs marked for page pool recycling. * - * When pp_frag_count reaches 0 we can either recycle the page if the page - * refcnt is 1 or return it back to the memory allocator and destroy any - * mappings we have. + * This helper allows the caller to take (set) multiple references to a + * freshly allocated page. The page must be freshly allocated (have a + * pp_ref_count of 1). This is commonly done by drivers and + * "fragment allocators" to save atomic operations - either when they know + * upfront how many references they will need; or to take MAX references and + * return the unused ones with a single atomic dec(), instead of performing + * multiple atomic inc() operations. */ static inline void page_pool_fragment_page(struct page *page, long nr) { - atomic_long_set(&page->pp_frag_count, nr); + atomic_long_set(&page->pp_ref_count, nr); } -static inline long page_pool_defrag_page(struct page *page, long nr) +static inline long page_pool_unref_page(struct page *page, long nr) { long ret; - /* If nr == pp_frag_count then we have cleared all remaining + /* If nr == pp_ref_count then we have cleared all remaining * references to the page: * 1. 'n == 1': no need to actually overwrite it. * 2. 'n != 1': overwrite it with one, which is the rare case - * for pp_frag_count draining. + * for pp_ref_count draining. * * The main advantage to doing this is that not only we avoid a atomic * update, as an atomic_read is generally a much cheaper operation than * an atomic update, especially when dealing with a page that may be - * partitioned into only 2 or 3 pieces; but also unify the pp_frag_count + * referenced by only 2 or 3 users; but also unify the pp_ref_count * handling by ensuring all pages have partitioned into only 1 piece * initially, and only overwrite it when the page is partitioned into * more than one piece. */ - if (atomic_long_read(&page->pp_frag_count) == nr) { + if (atomic_long_read(&page->pp_ref_count) == nr) { /* As we have ensured nr is always one for constant case using * the BUILD_BUG_ON(), only need to handle the non-constant case - * here for pp_frag_count draining, which is a rare case. + * here for pp_ref_count draining, which is a rare case. */ BUILD_BUG_ON(__builtin_constant_p(nr) && nr != 1); if (!__builtin_constant_p(nr)) - atomic_long_set(&page->pp_frag_count, 1); + atomic_long_set(&page->pp_ref_count, 1); return 0; } - ret = atomic_long_sub_return(nr, &page->pp_frag_count); + ret = atomic_long_sub_return(nr, &page->pp_ref_count); WARN_ON(ret < 0); - /* We are the last user here too, reset pp_frag_count back to 1 to + /* We are the last user here too, reset pp_ref_count back to 1 to * ensure all pages have been partitioned into 1 piece initially, * this should be the rare case when the last two fragment users call - * page_pool_defrag_page() currently. + * page_pool_unref_page() currently. */ if (unlikely(!ret)) - atomic_long_set(&page->pp_frag_count, 1); + atomic_long_set(&page->pp_ref_count, 1); return ret; } -static inline bool page_pool_is_last_frag(struct page *page) +static inline bool page_pool_is_last_ref(struct page *page) { - /* If page_pool_defrag_page() returns 0, we were the last user */ - return page_pool_defrag_page(page, 1) == 0; + /* If page_pool_unref_page() returns 0, we were the last user */ + return page_pool_unref_page(page, 1) == 0; } /** @@ -301,10 +309,10 @@ static inline void page_pool_put_page(struct page_pool *pool, * allow registering MEM_TYPE_PAGE_POOL, but shield linker. */ #ifdef CONFIG_PAGE_POOL - if (!page_pool_is_last_frag(page)) + if (!page_pool_is_last_ref(page)) return; - page_pool_put_defragged_page(pool, page, dma_sync_size, allow_direct); + page_pool_put_unrefed_page(pool, page, dma_sync_size, allow_direct); #endif } diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index e1bb92c192de..6a5323619f6e 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -224,9 +224,9 @@ static inline void page_pool_put_page_bulk(struct page_pool *pool, void **data, } #endif -void page_pool_put_defragged_page(struct page_pool *pool, struct page *page, - unsigned int dma_sync_size, - bool allow_direct); +void page_pool_put_unrefed_page(struct page_pool *pool, struct page *page, + unsigned int dma_sync_size, + bool allow_direct); static inline bool is_page_pool_compiled_in(void) { diff --git a/net/core/page_pool.c b/net/core/page_pool.c index df2a06d7da52..106220b1f89c 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -650,8 +650,8 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, return NULL; } -void page_pool_put_defragged_page(struct page_pool *pool, struct page *page, - unsigned int dma_sync_size, bool allow_direct) +void page_pool_put_unrefed_page(struct page_pool *pool, struct page *page, + unsigned int dma_sync_size, bool allow_direct) { page = __page_pool_put_page(pool, page, dma_sync_size, allow_direct); if (page && !page_pool_recycle_in_ring(pool, page)) { @@ -660,7 +660,7 @@ void page_pool_put_defragged_page(struct page_pool *pool, struct page *page, page_pool_return_page(pool, page); } } -EXPORT_SYMBOL(page_pool_put_defragged_page); +EXPORT_SYMBOL(page_pool_put_unrefed_page); /** * page_pool_put_page_bulk() - release references on multiple pages @@ -687,7 +687,7 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data, struct page *page = virt_to_head_page(data[i]); /* It is not the last user for the page frag case */ - if (!page_pool_is_last_frag(page)) + if (!page_pool_is_last_ref(page)) continue; page = __page_pool_put_page(pool, page, -1, false); @@ -729,7 +729,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool, long drain_count = BIAS_MAX - pool->frag_users; /* Some user is still using the page frag */ - if (likely(page_pool_defrag_page(page, drain_count))) + if (likely(page_pool_unref_page(page, drain_count))) return NULL; if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) { @@ -750,7 +750,7 @@ static void page_pool_free_frag(struct page_pool *pool) pool->frag_page = NULL; - if (!page || page_pool_defrag_page(page, drain_count)) + if (!page || page_pool_unref_page(page, drain_count)) return; page_pool_return_page(pool, page); From patchwork Mon Dec 11 03:52:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13486642 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8699BC4167B for ; Mon, 11 Dec 2023 03:53:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 177656B0083; Sun, 10 Dec 2023 22:53:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1261F6B0093; Sun, 10 Dec 2023 22:53:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2F186B009A; Sun, 10 Dec 2023 22:53:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DE88A6B0083 for ; Sun, 10 Dec 2023 22:53:23 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B9538A1957 for ; Mon, 11 Dec 2023 03:53:23 +0000 (UTC) X-FDA: 81553167486.01.17EC603 Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) by imf24.hostedemail.com (Postfix) with ESMTP id F0882180014 for ; Mon, 11 Dec 2023 03:53:21 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iKb0H1Kq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702266802; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J730iItaW8fpD9bcYiCc3WSDtwgBtsyA2Jb0R83edv4=; b=AQnl3dXKf8p4VTshq9iREnE0CViE6jAav7Hs7ycPrHOxo8FwhfA5yjiOpYQddCkoLEa1m6 XfNQKngEb9wuWtdOnVfrN1dE73grDzmrFIBaY18ZGlI1/fDNNdQVETL/ijylLsiD4zrxdu XAaB88lIW8+ovN0PRsLn0HDCxO3fRkE= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=iKb0H1Kq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf24.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702266802; a=rsa-sha256; cv=none; b=LoKyHCH1Q8eIOzTdq1aLPM18L0khdbbJ3mLaX9Y+XIe5LI5ra2M2kWB4TtWaB5hxHja2ki sY7/eB+kHV3uby3J/WZZpaSPNli5Qb3mUc9NAtqHvD85wJWcxot+9zR3+ZMYRfWzvQQLM7 ehMWWBrJOiZtOOWHAV9uBWoB4c1+DHg= Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-5cfc3a48ab2so33079457b3.0 for ; Sun, 10 Dec 2023 19:53:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702266801; x=1702871601; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J730iItaW8fpD9bcYiCc3WSDtwgBtsyA2Jb0R83edv4=; b=iKb0H1Kqysh+J8YHDAWXvLZM/5j+IXgUlaDYrPzWGBFR0mqDDJStbc1BKASQaABSGM hCDKLS/Y5qukyoIlYsqwz8X0iUlHQHqdzcZhlNr+f7MWKUyzyp1+8NB3BDBr54BiA+FM OjJG5TtfI8fhUJsSHLbAoT4JDOH3t7VI0Ur4jmhkt0zE51i6xHDwe8/yKei4Xc69f5uv sgPImJ90vx0V4YmMnNustOlp3kTUhudNeLg7eTqaX6ef6DOXKglK18mcIm32Z9/5/Mb+ 0N1OKEuuS91VDdxumqc8xrxYO5oaEkDqTXLf/mqRMLlEU+cIVLpXP6/ZmIgdQ5qlLUat 7TTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702266801; x=1702871601; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J730iItaW8fpD9bcYiCc3WSDtwgBtsyA2Jb0R83edv4=; b=hxFdXtIY3NYUddPr1qFjkf0ZkRVzokjgcrftrKmAySw1mzOW7I6QvXM1qgZzrb9ltm XY5Q82Q/9icbXePqAl8x9auPDtWbvb5/y0YL21U9WoLrTvMyrIiy0t1tVuO9MWsm53l8 etjPXbAtfZ5qPWiNGpi1cucJgkYKmOvICFTTxM+XnGsyEA76+R44HNlc0oPeVx7mVo1g S6C1tnt0Kfjit66eRW0pPhavvhb/AWrlfacYdsmjrrYzT+/3+7qB1QcivS/zESc1JC1/ e6vhBYxuVC3EvRbLhmWZuvCGvXOBXRNb0JkrmOTQnsVqAwFpUeLn7/s135NN16ODBo/G pXjg== X-Gm-Message-State: AOJu0YzocWcsKqGCBHkoweFuuvyvQDJOvMM9dIvupb7sISGRL+Y8epNT ZTmW8qruM0jAbyc2uWGawQo= X-Google-Smtp-Source: AGHT+IHcK2/Q8KqUlgqgd9YgADevE8eOhbq+FQOnrZxm7i37nZvpcynhBH5Vgt0k8DumJ+fezNSkrA== X-Received: by 2002:a81:8406:0:b0:5d4:2ab1:9f0a with SMTP id u6-20020a818406000000b005d42ab19f0amr2593146ywf.42.1702266800986; Sun, 10 Dec 2023 19:53:20 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id b8-20020a170903228800b001d052d1aaf2sm5411491plh.101.2023.12.10.19.53.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 19:53:20 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, jasowang@redhat.com, almasrymina@google.com, liangchen.linux@gmail.com Subject: [PATCH net-next v8 2/4] page_pool: halve BIAS_MAX for multiple user references of a fragment Date: Mon, 11 Dec 2023 11:52:41 +0800 Message-Id: <20231211035243.15774-3-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231211035243.15774-1-liangchen.linux@gmail.com> References: <20231211035243.15774-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: F0882180014 X-Stat-Signature: 9q4mkw9pzomc8f6orhz3697qr6fjpqyj X-Rspam-User: X-HE-Tag: 1702266801-354388 X-HE-Meta: U2FsdGVkX19PPcQEL+B+3Kxp1Cy3vn57t613pqA63cnypsx4l1WvjeZR/NLwqqGyaqhHECz9/M4H3mZfES1in+uEOHkGW5mKI3m/EnN3Kz+TcXRMSnXGntEqZWz2+mPdCTbhOWmQUzANFQd5mgGlaFn3wC574g3ZJDcvtf11Np8+ugyIAdM+Dof4Sdzl0k4JSh1eHlbeqq2thyCrwndSdIdL4WXdoC+WrQOUbYp4Me2hVKKRez6wGAZjbczWnXpMRRfS3nQ1TJJibxsjXviQ18D+GsQkJzYPpTMHTOQpIzZ41VbCaoTjiFlJsW4GbUg+0z6YbymVp8bgSHgx9oMfeXkxJLWU+sy8BKGXhX/3S/ZjxMB1Zbhgi7JkjjWWzu7RR6YcNsQTtJSOZr1KluhwOKh+qMMWYqSRwtECay49MgTK0IF21Rp/QbgRCadXMGalHhL5HYh/KSqyRqbovb4XQlo89aL8Z6etkT6qccK6Ca+j9Wl/rFoQckK+GpBsWku9NiCWeNMBuFU6aWscuRrfo03AILfTIYPjOtUNn16f3ICJpJDFcplfhFlT96zN5ahxWzstmQsMXkNBvPOS0EYeTujAqW006qWArA8KLD+0RIrQJPfNs9t+Iogujc+4TCm+rUFV9Vwd9AIXXbYWhc7t1PgNj00G5Pq0UOBLatRgIhLd14XvEYt+0hLpHrzrGtcLFrD0KKlW0SWoKhM27zU2At7JyzapPQUWhbqz2irb6H/kkHbk+yiZ8D8rj+UVNxpJMh2sh+ab3HxrbeVhBt9/kaIWtx4HkBhFNvrmSd7LTRKwtQkz6Q2+y0k0oGl2iUoh34BsU25XOsxuMaUIV+Hwc6YjFS3VGpUBI+CTooaVvFHbAS/mmxTWt2cN7QbHsKhwti5dXkEwJ/hQfRpG5+qATxwRGIoGKF3ohRnHOLXKgzdNIOKFXlOclHs7dvnsi5p7gqQE68e95gmYhMklBzk 6C/TlpvY JJqGvjSXCB0xGnkngOEF2O3yUdBVuqwxb5BjYX1vt6/9jY47Wun4KUkfIM7mMqvhEU035lWLvk+VvV4KetjEHSEnGx1gE+K/mA27a3Cv83YZ38QZmg1rs8lEZL1FvURMe9/OzY5RfKZkxtdiqGWuxPymWzeCl7D33+fMm5bqPlFb7h5R+HBLat+OWNFQB9B90H/7gzIiW3Xyx4epBwMbwq7dP1tIlAYjs24ady4MXYoGJL9wQVzAnWc/1rgvJw7Wfnijo/E2HPruHQATcJhiMEMHdjAZV5kRJBidzgVzqznreN+VKF5gvao2XaYq3auv7QZGRVHV1lpKb2V9ZOjqHFlkKOnkCZQe/FnIq+IBlT+t1wazNtmcThqGGQCQ88T9is+glwbgKfgQ7mP1HJ5WiLAQ3gnYk+MldSkNAmvBCxXy+mtbnFCLAR464qik+f1Yp/GoZQ6JGcHKSJJm/yzZYlIjVCo9o08NrEzL4Q3iOsiAUPW/XV3iHe5O1F1weZ6m9waEAw/I7UXPYB5Dpmo/HKqvRH4afArvb6Z6NXNQFdnup1aBqGe+pSGtwIvSLJT9N8fSFmHo6Q5DUX0i8jYqmspF0j7Z5mrsj5TqXUtTepAU4gzhnHMOPqbTdTBG5OsHSZZorj7YgNd9GVRnQTOKVM6QHMhHahFKKlJOmTKzXaL87y+2H3JDlaMXdOZDHFoPIt8jTs2jnbR3UXRZhpVLlVhnVWWlbt+HfzHDvtwmntWs/SxHOQoTJsSm4zr1vaaLOopmD X-Bogosity: Ham, tests=bogofilter, spamicity=0.029419, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Referring to patch [1], in order to support multiple users referencing the same fragment and prevent overflow from pp_ref_count growing, the initial value of pp_ref_count is halved, leaving room for pp_ref_count to increment before the page is drained. [1] https://lore.kernel.org/all/20211009093724.10539-3-linyunsheng@huawei.com/ Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin --- net/core/page_pool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 106220b1f89c..436f7ffea7b4 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -26,7 +26,7 @@ #define DEFER_TIME (msecs_to_jiffies(1000)) #define DEFER_WARN_INTERVAL (60 * HZ) -#define BIAS_MAX LONG_MAX +#define BIAS_MAX (LONG_MAX >> 1) #ifdef CONFIG_PAGE_POOL_STATS /* alloc_stat_inc is intended to be used in softirq context */ From patchwork Mon Dec 11 03:52:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13486643 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7ED6C4167B for ; Mon, 11 Dec 2023 03:53:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 791036B009C; Sun, 10 Dec 2023 22:53:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7432D6B009E; Sun, 10 Dec 2023 22:53:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E1966B009D; Sun, 10 Dec 2023 22:53:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4A90D6B007D for ; Sun, 10 Dec 2023 22:53:30 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 228B51C0E47 for ; Mon, 11 Dec 2023 03:53:30 +0000 (UTC) X-FDA: 81553167780.01.A76EF96 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf15.hostedemail.com (Postfix) with ESMTP id 56602A0004 for ; Mon, 11 Dec 2023 03:53:28 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=F40GoeHf; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf15.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.216.50 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702266808; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G2d8n0WXue9ygrbxloUCx8WAoE9HhkQ6PXxWrjAEYxU=; b=q6+79eYlfk/avXC/4nTed3U6BRdb7Hcx+1uoanKKvIpzaR8Uqn1F1YN8fkW4WdISEDt37N YCO5d3Y9vF5XMTaDlPK9i/6cTBf42aJBR54AzDPVWuJHqJirsmrpz3/Lt/cavSkDmnNnHU wzNoX8g4U+Zw2yD1H4IvogZeUXvxmto= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=F40GoeHf; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf15.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.216.50 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702266808; a=rsa-sha256; cv=none; b=aUuauH/JeKILgY+F32kjJHLbPcJnL0IM6hewDF5JoEoDXRVmQWtGOCwq9WtIsf6erC1Eum Oc7oy79r4OrYV87glnCoIzmrKLgAbi5rJuO/PELGVbXdgZJAFoW6Q0blZri7GqyjwpQawb a8TT9XE01LxuI5rug9aOLhAAoiQFt2E= Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-28647f4ebd9so2945428a91.3 for ; Sun, 10 Dec 2023 19:53:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702266807; x=1702871607; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G2d8n0WXue9ygrbxloUCx8WAoE9HhkQ6PXxWrjAEYxU=; b=F40GoeHfl9HJknJnGTRJMAauvPwQ4Ygn4lPUU2+8nTYb+DUV71sbtn0CZ9Ek7lTwxX FBxJXqLvfx/aGUG5mYEacBtj1kYMBEAvIWRE1tgRogugmc6nflmqN6EUrfr/2CwiprwW WxufMigqt5L92zUcH3qZaszSKzzOCitYQSfm6gTXoE2sg4CLxpNR3YaBH59ssVVUNj8r Rqa6y3WqoJnO3mXH4ogcTgj8nj8yiEYV4qTDtzgvDiPGFdVI2LtUIB6BLHkQ6HDTe1Ws SrdiZEaM+6HEg2tLbgLCL7Ywkz3bCoyRnuqglfWQD76M3VP4y8Z3PXUd6tYHhZ/2XuaL TTEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702266807; x=1702871607; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G2d8n0WXue9ygrbxloUCx8WAoE9HhkQ6PXxWrjAEYxU=; b=SGMSyzmxolp2CcL7WAB/Yvl2AwbLC6uTVxcNzu0A74s+0h2yaFStyMNZSTlapy/TgM duXc3eXVNX80/dhq0TB2KCjkuLh6+ssGu5Fmp9mU3ELBfE/qUsAuCpORbQSy03DaIaKO v0stKLluZQBAq48+B2esIEgekUaTL1E4RZDq9iGKBggo/KtrBJtPi6S2RH99ww4D36xV KBHBfRCsw2mvjucS95lgwxpGj+T82uWxglaDuyHvtdlVlyXWJcJes7kfjcu/Pn9Otm9f hHMbqgfWsnXtQpFRfpCGWiCpyqhBo01vrIIr5OBX/1GFlS+orcZUfhjzXr107eZ6CU3h Qbhw== X-Gm-Message-State: AOJu0Yye/OAwanHO6x3E8ZwSQwVxtjmbuO9Uwc9EpFuUO2bYGU/+fHn2 Xnk4iGjcLgI5Qi7wxC2/d2A= X-Google-Smtp-Source: AGHT+IF/9zzYT5dBgAhPgi9UwWntEpQIso8WA4BYMk8jEYQTrt8722TIA7PHJFeT1xdLlXRfSo/WOA== X-Received: by 2002:a17:902:d4c5:b0:1d0:91c3:aaf2 with SMTP id o5-20020a170902d4c500b001d091c3aaf2mr1656471plg.25.1702266806761; Sun, 10 Dec 2023 19:53:26 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id b8-20020a170903228800b001d052d1aaf2sm5411491plh.101.2023.12.10.19.53.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 19:53:25 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, jasowang@redhat.com, almasrymina@google.com, liangchen.linux@gmail.com Subject: [PATCH net-next v8 3/4] skbuff: Add a function to check if a page belongs to page_pool Date: Mon, 11 Dec 2023 11:52:42 +0800 Message-Id: <20231211035243.15774-4-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231211035243.15774-1-liangchen.linux@gmail.com> References: <20231211035243.15774-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 56602A0004 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: qb741jhn8ak1fbkackznd1dr8d3ou3dy X-HE-Tag: 1702266808-411668 X-HE-Meta: U2FsdGVkX19pYPzl2AJ/JXD4OCApFQrmhx0yi/c5xNAj0vM1YhoNYWWoKf9g+ciILGikZQooXneOrDT4SrsF3yQ83TjWSlCo59SWVk1BTVuaXNF8nmrN90sj1DbZ4XDgQjEmR8E7s6mvhUNojyfUQu0PtsjHnG+widbwuL2rlUW1JEg/hH+i2G6rNm2kFukBin1wG/esHmgJfUZPXdEIhCSI77yf2qESJOS40HYPFAYPxSe1/fUE7XwT6mkWKm1dGYiucdt9F7IgciaTTeFL3a5s7I6Xs7Uj8J3bzgHdTv5N/LVOBXuX2WKwNyW4nBjBBYi5KU1d2M+h4PIjQhNjWFBPFvq73xoBSr+zcc7GAyp7VwY11ZlkoS3HC7mPDyrHaHOIA6XftZQGILVZuIv29337Q62WxyvJbzjmFvy+fqcWix3KC+aTSrOyXI1mJf+VypEEUXdpkuISwy4KrS3ic1wxRf3yIvPKaZ2JFfwjuq5c2HtC6ZbpuYxdNatBmpVXMirDRZ5MnnI0tUdxAfJql/37xF4zXFaapUdrAOH7EXewiEaZoX4P4g5yhoauxWV96yOFuLfmKnqCVF/8i/Y0Ev4QPKa0npQuWAnvjO0UNrTpgpGGYpbNI7Hg+3JGPyCRYjYPOPV+K2GTo8IyZwgmo4CwL2l1PDUOKep7iFLZSpZ7Y9d9y21fw1S1XYLkD9DD7xfrH79HIz1pfISpUr/B+vE1U2b3i+HVCBgIqXAbu1G/8M6bJ1GWVQafE+PqGOsjOVpIBJ0uMkEU0mb4cvSkt44gLI3l3EDCzb7E4jLAU3tXWMkHd7O3fv4ii0LyQFGhA/u+G+evuUbZx68QyG7duO608Voe1kBO2MaHfbq9gZo3aOFSeRMS6XgD7+oWUBUOBFDMxIGl0ZvRXTMcLTrqzvJknJ5SIVmilYmxNZSiWrPaxxqdvqu7o7ZmrlN4OHjJVlCaamXZxblZEn0DOu8 PJ/AAFSH zSdk3kE55MCwDQBq9acYU+kF5CnslKRoS3YrDmG1STh4Bs2kwQHDco1d0ZjM9wmhGo0yGB8lVq4lOfn65Noumg6poOWzDC05jR/m4uxMuVEXQvxYO2leDAdaLU7E0hYjxBpXR7dP+qsiEqcH6AaK09LEz0W83wjQKRriBzo1VKr13NOhcypj7i3qVaD6nOrXq4/KqMV7VyRTV3kKDV5qecGyhBty61dP2G42qJkhEoNG6Th+XPRVOVkR8Noq5AkC1RBxuHw/5rJLVHt3Yr5m55H+Q6nOz+x7y2NTfGFip2HRPoAIYuKUoyW7bNm4aVLHemOWhFZnvQoSeGFyWDOS/8mdCmUv7dcpeU534MiOCvBbs4KvVPHy1HwONxIQNEM5SxR8j5b8C5vsS6Qw/6y8358qLTdCNtKh0NHXlZu6yDp8pFlx5KU6hqueCRVmDHqOhixM/8+wAJzYATI4SITzJHfvlxfEBNF1/Rird3Wuk529P0qwcPIsuF3r3pYX4JNzZF+TQqWdvc2JKLtRpUZrL82KbDq/GJkxShvO9MFHZTRaLubnlWqmS72OwdwvWhhivQhZ90RRR1enODMnteDByHOo5h+VM/sbzzWBPX+hrwufiaI3XZBXuV1vkwwKuNnndq8jydNKcoIdRSfmEiimuXEMHyh4InN4DIE26etiZ31eWdfWu0LtafxDVgVbY/0pYN9VUKLzIKoALvtiUf8jpqA2qPCo+XRslU76D X-Bogosity: Ham, tests=bogofilter, spamicity=0.007087, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Wrap code for checking if a page is a page_pool page into a function for better readability and ease of reuse. Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin Reviewed-by: Ilias Apalodimas --- net/core/skbuff.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index b157efea5dea..7e26b56cda38 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -890,6 +890,11 @@ static void skb_clone_fraglist(struct sk_buff *skb) skb_get(list); } +static bool is_pp_page(struct page *page) +{ + return (page->pp_magic & ~0x3UL) == PP_SIGNATURE; +} + #if IS_ENABLED(CONFIG_PAGE_POOL) bool napi_pp_put_page(struct page *page, bool napi_safe) { @@ -905,7 +910,7 @@ bool napi_pp_put_page(struct page *page, bool napi_safe) * and page_is_pfmemalloc() is checked in __page_pool_put_page() * to avoid recycling the pfmemalloc page. */ - if (unlikely((page->pp_magic & ~0x3UL) != PP_SIGNATURE)) + if (unlikely(!is_pp_page(page))) return false; pp = page->pp; From patchwork Mon Dec 11 03:52:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13486644 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 695CAC4167B for ; Mon, 11 Dec 2023 03:53:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D850A6B008C; Sun, 10 Dec 2023 22:53:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D34E56B00A0; Sun, 10 Dec 2023 22:53:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAD5F6B009E; Sun, 10 Dec 2023 22:53:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A73B86B0089 for ; Sun, 10 Dec 2023 22:53:37 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 78472A0621 for ; Mon, 11 Dec 2023 03:53:37 +0000 (UTC) X-FDA: 81553168074.30.7BA19BB Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) by imf23.hostedemail.com (Postfix) with ESMTP id 92C98140015 for ; Mon, 11 Dec 2023 03:53:35 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ejl7sFAl; spf=pass (imf23.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.166.45 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702266815; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZVUa6NcsCdWv7jZCGAOsNgtw6R7wscnm5fcwcGKxQ5Q=; b=lVSbSVcYUT0A3BgGNMYdOLY2pZlR8GayJn6Phng1rYOQPbT6w7bmQYFPYdaZinp7+N+lvM dvfje0QFRWGFQKii+idu2u0O1/ucfW884xV+zgHpPWc1YvgrJDGogsX2cPc2ZCOohI6o8Z K7v8eBsTG9IQEuFOE1dSPurUOJiIbmY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702266815; a=rsa-sha256; cv=none; b=u0HFjs4ei4M5B6f+Irdd8FF2OWPBkBjtDmBvRTJ5YomMyXQhY53oVX8wP9pJnNcAaZPKQt qS6+onfTmRF9CQ8F7Tq35oLQFCBoYeprgejn/7ga+FvygM7HlTWZc57y/YOmr3KNBOwLVE yleLZndd/0RbHH+rOuqU2j5X0QDWHl4= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ejl7sFAl; spf=pass (imf23.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.166.45 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-io1-f45.google.com with SMTP id ca18e2360f4ac-7b6f19f3af9so179745739f.3 for ; Sun, 10 Dec 2023 19:53:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702266814; x=1702871614; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZVUa6NcsCdWv7jZCGAOsNgtw6R7wscnm5fcwcGKxQ5Q=; b=Ejl7sFAlV/UMI/dRUzDvd/9J7m7AekGGtfRhat1qclSIgWHFaRo5YViznDiTKQYBe3 41NpcbZauC73PWpGfTwE82eoAjXAhQ9m/e4hlmIqeW68T475q5MjzDFsI7XLOaiUO5eP 9FSTj7G7JHeQliR+pbr4e69wfAhFBl7otssGxOtdF73YbdRm4Z6bCq38ONAMh8InA/i2 3VE+AQKJ3WG8c/lkT72wm/DO3lYIwGWu3qVzUUd2vyV9Ml+0VsSjWVT9OV+gLvcYiL/n W0quZ7Sit7M/bBQG2yq383E5hQaCIXMo/4O3npQLnlQbSOrL5wxm7P5f9aiAufVPCcXt aSSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702266814; x=1702871614; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZVUa6NcsCdWv7jZCGAOsNgtw6R7wscnm5fcwcGKxQ5Q=; b=qnn0RNd4qZp1icCPJamrt1/h+WjBQktS45lRhjsE8sjzV3M7LjvMeq37RjFOJODEEH L9XiW7DDBbPqraSEY+1UPnofkEb4SPEx5kuFFQP7SgH3CwwxveGpoVJIB3VNSGsS8ho7 1dtAxAH4KquCsilqeQYf7y718fUKxkJu6AEBJmTqbxkSyvRCpc4jEa4VakRQ77EKd39e aElNccUBIN3Axghf2JjV1cJWUM1Qx/vnwQDKu9mAENAM90z33IbWRy9rKGSwVRcvel5P MDVBQDzLZ/qXW3p8MFr/gGWOyw/IxoryCO+/LZTbJRemJDkvAe/up5SIx1smn6f/jcp/ a03Q== X-Gm-Message-State: AOJu0Ywzb3rZoFfc7d8QKnJjU+BOdwEDTqiR7pAwpDVfneW3MmKBprnP KwpqTNDESQ7pex04JTFl1rVnlmeO5pE= X-Google-Smtp-Source: AGHT+IFdjnKcGXA1RGpVDmqmZBCacHeqjUYLhgaJubLPawvL/tu4PXYUQAsrkLCPyQemnmeb07fCSA== X-Received: by 2002:a92:ca4b:0:b0:35d:4f54:a9f0 with SMTP id q11-20020a92ca4b000000b0035d4f54a9f0mr7270958ilo.32.1702266814686; Sun, 10 Dec 2023 19:53:34 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id b8-20020a170903228800b001d052d1aaf2sm5411491plh.101.2023.12.10.19.53.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 19:53:33 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, jasowang@redhat.com, almasrymina@google.com, liangchen.linux@gmail.com Subject: [PATCH net-next v8 4/4] skbuff: Optimization of SKB coalescing for page pool Date: Mon, 11 Dec 2023 11:52:43 +0800 Message-Id: <20231211035243.15774-5-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231211035243.15774-1-liangchen.linux@gmail.com> References: <20231211035243.15774-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 3iueez5girhnfw4jcyzn75fpk9z8xkdu X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 92C98140015 X-Rspam-User: X-HE-Tag: 1702266815-322430 X-HE-Meta: U2FsdGVkX18L6n0fhbtWO6qvuFqghbYap7EHsYp1HZpSEKriTtpXwoaA0FGoxGzxAVJprS/aiEvL7phfki6FadM5GBgOi4YkfV4FHu1o46M+KoeO4ayktaMbfcWG/NzbQnlvl7moI3YIaFh0IHDdOj8J8MdUw7f21OwiBckjB7xGFWc5yNUzWuP0rKy+Yn+mL2VgYV/pbza3ykBqRycFqNS3A1OiHOyqnaucaZ2XE89g9RyirCib3kqdTXRYP8X1aATWrqb6Ue3x3RsJbfKz2XLc0GSY4/PHUjwN2GXzwPprtBugG/CdmCAy88lG2Y/iWki6bL3sQAgUJWGS7/KrTvr+zmc9W1KSFbOH98cyvSVaTyEVS+OgbB5FRfe/M75w4b9xRQgoCIw+MVLSs2GqveZW9swBbFVSsLvvGAZxu6ixPtEoER2yUMtA0kUg5AXMwSU4zFdVHKlVSFUBPCLBgxbLh6cGxXeznDB6zj6Ec1ZrpueJCqdZuxNKj2dM+2XU95LGqTJ3DwGNi/a3gkCk+mB769L2d4n6JubAbVBpcqgUMfvrVEdAc59r9wCCf6er+g7GjZsdix773esZ3NIgUn9MqKo8U/L/TdGjeYTRcKhQ23Tx0tjFIjiy+Aji6mophiuIO6B2sOTWl2Ofc+LqjScSLB9yErmeyVkxWFVQWZibCxmOGYl7YzWZQhtURdnPFU9+dOHArPIKMcuEKz2U51q5eIwMEXB7udobesASfqScoeC5cLeL/JFxwnsOZQYnWeDfJXO6lDoXehrCKyxKbfJuX/LpneqFmTR0pXHYLBZRMhhVZ2ciAInaKCMLHxSUESQxN4taTwA1WckWj0KnIPsDc7helowRSbzglJbgI0GSyLNf/CCFFpwVZVAONLH5WbSU6QuVEQ70GhwO7mLjre4RbLKFlK2amVxmofjw8R0panv+oXfGPEKJ7eFXRdZdqsXGiPx2GvxX0MHTaPo 1ijlLiGE XvXFuC9u4xFxuAbvraefy09xTPxgAJfMjtgBKO+hNsBts5q+qdWwG+bwi4w8jdchpQfMD/DR2JP6Z59KN7HYd3jdRRfk+jELDlDnurzqNCr6XV4NErQR0pXb3O2rNL2I+ucPbpBu9tCgfdy6IxwSQNVs7UY0gj6BiAXf4dfK2zxD2wLJYom6UM4dHRPtTITyPoTYjD6suk8UP46hqH8I4PmG+rnxOPVihGkAfqfODETkOALcMgw4t+75wE9agYIhVQEFeNEZ2s5LnDTr4RZ9TD5n68zMa4uvGqbrsXrDLUSdTFS2xeOL85kOAoKDVGPpdPHTx0+PJXTBR/Y/7z88e0bzzGwz7/ZJjx1XAg1Y3jtSlJATZ6XeHqy0pMEs8tfusA03ZgEaEH2PI0j+OdAExKvaIVnmANj/6HmsQjBoo7PiyYolwVoG3X9uAZaWqLYwZJy5oAIFskDqyiP5HeBMR8SKMeO2njSczPcSXYmSNnbpXANzo7l2ljwOwE66KShJ6s7KJEzttOwueRjL87S53/gSplLQnGTaiMKro5X6i0fctYGkIEV4I3RQEpdFnuhW+mC4EAv6eP6DB8efAUDVP8UZEdg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to address the issues encountered with commit 1effe8ca4e34 ("skbuff: fix coalescing for page_pool fragment recycling"), the combination of the following condition was excluded from skb coalescing: from->pp_recycle = 1 from->cloned = 1 to->pp_recycle = 1 However, with page pool environments, the aforementioned combination can be quite common(ex. NetworkMananger may lead to the additional packet_type being registered, thus the cloning). In scenarios with a higher number of small packets, it can significantly affect the success rate of coalescing. For example, considering packets of 256 bytes size, our comparison of coalescing success rate is as follows: Without page pool: 70% With page pool: 13% Consequently, this has an impact on performance: Without page pool: 2.57 Gbits/sec With page pool: 2.26 Gbits/sec Therefore, it seems worthwhile to optimize this scenario and enable coalescing of this particular combination. To achieve this, we need to ensure the correct increment of the "from" SKB page's page pool reference count (pp_ref_count). Following this optimization, the success rate of coalescing measured in our environment has improved as follows: With page pool: 60% This success rate is approaching the rate achieved without using page pool, and the performance has also been improved: With page pool: 2.52 Gbits/sec Below is the performance comparison for small packets before and after this optimization. We observe no impact to packets larger than 4K. packet size before after improved (bytes) (Gbits/sec) (Gbits/sec) 128 1.19 1.27 7.13% 256 2.26 2.52 11.75% 512 4.13 4.81 16.50% 1024 6.17 6.73 9.05% 2048 14.54 15.47 6.45% 4096 25.44 27.87 9.52% Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin Suggested-by: Jason Wang --- include/net/page_pool/helpers.h | 5 ++++ net/core/skbuff.c | 41 +++++++++++++++++++++++---------- 2 files changed, 34 insertions(+), 12 deletions(-) diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index d0c5e7e6857a..0dc8fab43bef 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -281,6 +281,11 @@ static inline long page_pool_unref_page(struct page *page, long nr) return ret; } +static inline void page_pool_ref_page(struct page *page) +{ + atomic_long_inc(&page->pp_ref_count); +} + static inline bool page_pool_is_last_ref(struct page *page) { /* If page_pool_unref_page() returns 0, we were the last user */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 7e26b56cda38..3c2515a29376 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -947,6 +947,24 @@ static bool skb_pp_recycle(struct sk_buff *skb, void *data, bool napi_safe) return napi_pp_put_page(virt_to_page(data), napi_safe); } +/** + * skb_pp_frag_ref() - Increase fragment reference count of a page + * @page: page of the fragment on which to increase a reference + * + * Increase fragment reference count (pp_ref_count) on a page, but if it is + * not a page pool page, fallback to increase a reference(_refcount) on a + * normal page. + */ +static void skb_pp_frag_ref(struct page *page) +{ + struct page *head_page = compound_head(page); + + if (likely(is_pp_page(head_page))) + page_pool_ref_page(head_page); + else + page_ref_inc(head_page); +} + static void skb_kfree_head(void *head, unsigned int end_offset) { if (end_offset == SKB_SMALL_HEAD_HEADROOM) @@ -5769,17 +5787,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, return false; /* In general, avoid mixing page_pool and non-page_pool allocated - * pages within the same SKB. Additionally avoid dealing with clones - * with page_pool pages, in case the SKB is using page_pool fragment - * references (page_pool_alloc_frag()). Since we only take full page - * references for cloned SKBs at the moment that would result in - * inconsistent reference counts. - * In theory we could take full references if @from is cloned and - * !@to->pp_recycle but its tricky (due to potential race with - * the clone disappearing) and rare, so not worth dealing with. + * pages within the same SKB. In theory we could take full + * references if @from is cloned and !@to->pp_recycle but its + * tricky (due to potential race with the clone disappearing) and + * rare, so not worth dealing with. */ - if (to->pp_recycle != from->pp_recycle || - (from->pp_recycle && skb_cloned(from))) + if (to->pp_recycle != from->pp_recycle) return false; if (len <= skb_tailroom(to)) { @@ -5836,8 +5849,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, /* if the skb is not cloned this does nothing * since we set nr_frags to 0. */ - for (i = 0; i < from_shinfo->nr_frags; i++) - __skb_frag_ref(&from_shinfo->frags[i]); + if (from->pp_recycle) + for (i = 0; i < from_shinfo->nr_frags; i++) + skb_pp_frag_ref(skb_frag_page(&from_shinfo->frags[i])); + else + for (i = 0; i < from_shinfo->nr_frags; i++) + __skb_frag_ref(&from_shinfo->frags[i]); to->truesize += delta; to->len += len;