From patchwork Wed Nov 29 11:23:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13472668 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDE57C4167B for ; Wed, 29 Nov 2023 11:24:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4CF1B6B03B3; Wed, 29 Nov 2023 06:24:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 47EBE6B03C2; Wed, 29 Nov 2023 06:24:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3215A6B03C4; Wed, 29 Nov 2023 06:24:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1B79D6B03B3 for ; Wed, 29 Nov 2023 06:24:37 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id EAA228030E for ; Wed, 29 Nov 2023 11:24:36 +0000 (UTC) X-FDA: 81510758952.30.8F0F42F Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf06.hostedemail.com (Postfix) with ESMTP id 10BB2180016 for ; Wed, 29 Nov 2023 11:24:33 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=B+dG1syO; spf=pass (imf06.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701257074; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Wy0To7gdBk/xSTsl+XULv9AwgPKI/xZc5H2ANQaWbK0=; b=TW8mgA80ToMio4K1+PFg84Chs19/8S1kENjDgKlDoIF9ouEwDjhkRf8MHHWpUbedVLn0Jx iIkaFyZ1TpCeqD47v0Zoo0B3Mc/7ZwMoeq2S1bLKixEGTElT6uUMks8ggHHR8MRYBhGptp pn3vCYKA1bGMz7vIwnVTu8hTs5JxooU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701257074; a=rsa-sha256; cv=none; b=yEe+pDSkoorlU7ygv95g7MSY5rAzwNGPo1hJEufzoW3+pSJxiFRwGeL5twgYMkWFpa5jjS 0GfcjvCDpnXXM4kQHc2PbTBYlLzpPiHa/AOqkLtd6Z1FK+fBI4lemykxRa8pmZHjRXaF0S az1jzKZ4oYIptHEEMMLMXPwBeVGIlxg= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=B+dG1syO; spf=pass (imf06.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1cfc2d03b3aso27542445ad.1 for ; Wed, 29 Nov 2023 03:24:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701257073; x=1701861873; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Wy0To7gdBk/xSTsl+XULv9AwgPKI/xZc5H2ANQaWbK0=; b=B+dG1syO/E3ywnbtCT308oah6+aUyWiV73hwgcxNUdSPAYatnAgshb/kVs8j36A7vV PXlfPlMdM93GkkcfVimlHxlusUds4A+CA3JV1YB42tZFZRkTMIYdyxf9arP6dUXsEfrC Q8mZ8LsWdp3O9jwfUGMGGQsJi5bHi4AkmwEswHYYlj3ivwREGBgGbwovEegZ8jxY3DUM 6jJRUkcCgT53RbIjcun630Yh+5hBUX3perMx7G7s4FVQuxGnZtNDbszkoQaFbw79hTDb MQ/A8VnD+Ty0BEMnSUA/uf3iBaPaTYI+ptp8O50Z3Xxfy5pNebS0WLa6vuMmC5vhYGMH tO8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701257073; x=1701861873; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Wy0To7gdBk/xSTsl+XULv9AwgPKI/xZc5H2ANQaWbK0=; b=Waj+qtJSNkYCJ3H77rQvK2kvwKTioYvCoct/LoSJho9wH0lmVk5CAIM5YZHOltutCn bQJoEVDZZLpCxn0BZ8g/+H1hzyCe9xPX7l33z3t/jjLhPnwtpMu9bSpwI6VfMpzkDxTu 3T6gautKA6XnV0yiu0FJT3181XoTKfULaGXM3dJBKiElUS3ZO2nYz9yyvu9/SHxuI0kJ uSmYdjUImSu9xCBdObNL73ZxVpmNlOTvzazon1MM6XiOfVIRTYwru98mppDZQuS94kJW pvA47qYRpEcD33E7pjYBYrvoUXUCtCPptvxwiteUhUuv5Dh+0m66NHmsslJ17EY3zCsu emkQ== X-Gm-Message-State: AOJu0YyLQtJet2jTSCouIZStSzbd+Qv4NVzUStUG4nybNVWzRLre7/Gn cRGgDlbJmwWrgFz3w61VKgg= X-Google-Smtp-Source: AGHT+IFfarUyjnFgv1G9kZgm2zr/6pLpKE8ffVoMpJVLlB94bkXOrIPbUN6fjO/+vXwXQMuT8OUNmA== X-Received: by 2002:a17:902:d386:b0:1cf:66a2:d369 with SMTP id e6-20020a170902d38600b001cf66a2d369mr16420041pld.1.1701257072947; Wed, 29 Nov 2023 03:24:32 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id c6-20020a170902c1c600b001cfd0ed1604sm5460710plc.87.2023.11.29.03.24.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 03:24:31 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, liangchen.linux@gmail.com Subject: [PATCH net-next v5 1/4] page_pool: Rename pp_frag_count to pp_ref_count Date: Wed, 29 Nov 2023 19:23:01 +0800 Message-Id: <20231129112304.67836-2-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231129112304.67836-1-liangchen.linux@gmail.com> References: <20231129112304.67836-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 10BB2180016 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 9m4hy6mi61igjz9mofts59tywx3synwb X-HE-Tag: 1701257073-71569 X-HE-Meta: U2FsdGVkX1+/uhxRjs1zx4ZDLIdNER2T9aYMXF5h94L7G3FLBHoth20KR6ePySSHTsXq0fmXKSNyjNz6SXxffSUkXmZ1BuWq1cLENTEjlbc/6FNABUvvpSDLCyWttyPqPiXvtg3fth5SCitgo6vqhp0Fwz3axEUvFtTkf3hlzZnM4X0Hg8UAU70nk/dyWQRJ/h7igaqmMdTIcHYxC1cZ380wkZ6OJq8A/dikcUyhUtgdMfBzwVgRC6EZEcQ1kjqJM5rEGIF/fuG7fq288rhsaQNRie7i67SZmpz2DGrmwXcOc5KQSRBZGYyS1Ap37j/jCExoSyal1RV1wD75lwbt6+Gtk0HJySi4LiZpyQ4cVtezVwQpuN5xi3bEbDPcJj2Ev0PeRR6s2V/P88tsBmoyZ4A7qIZKKB0pIf0d6g0Q5Kd6+G3Uqu7d9h2lCGD02EnfIxwqqR9Bczdrcys9JMwOdW0oWDmY+xXs7hf4XiP9dnUZjgvLFcJ9pmXdtjss1Ui0b5wlRk3Rxhn+ITWMZg1V9I0B/oiREPc+CrRNBixDiHVDrDasujH4v3I5ZWriu2uvcLUZ6D2KyI7LHQg3h4veVsYdnMwlHL8ut75zSU5wMOriga2OF8+acjnvvo1m6PJUcRRikHQNkCN7dk0b23rSFPaB8KukORAHMg2CVn1A6bMxlfS+on2UFitYXSCV0g7ELB5QJNXh0Nrck3m4rycMwDDfH+XNfljI0jButPf0tcS5JQDDgyGFpPe59G6F+kKFemar5ja8Lcbhm2mWa2Nf7jqATwuzz0nMpVjbXxuO2SXC3ojcRGuDAr0btcekg2u2LCKK5EWjARKJVWGhvnPQM2eMVfr6Sbjhkp6XuujNaoADTHUpZk442qqCU+B/qo6LBl3XtkAeyVnrZWM4gzgSS29FDWwpYmHTC9IKfLJUX/cXzjNp+iGMbtDA/ta9YEefLXFEJMoiVpRy8h4Laem Ii7z4/v7 6I0r/bxIK/OcLkD0jKwnHNXdiXw64r21Bxg3bzeVTAqXhibuNzFQYPJ6sLJ5V5JgM7BEE/Rufkh3sfQWTljof2BwDnHc/9woQJf1W9X9CqOnrmnWaC/gVy5rpufiDnR8cWEo0KomttgMg5dscYbSUaDp3bGsGiWq53c59ettz5JTE3U5cO7bdcqp5CqtEK7TRTe0LEoJtHRbvkNZat+DhVx3CdGMvS5OoEANDRvqXmlF8n/dr70UDuM/9PABUAJZaGvz3g34slWlYzNEKjcmNoJ4jeKPZqcF5RZY6GQDGqRruLXtDZpk6UL2QGMjgZYz3l3WjDwwe4BJHm44OL0uMBKPbxiXrqcfNY9+8ZU7H8HVGrideZ/LBwzKoQPgMEGUkD1ZFgO4Og2dYQXk9vsIHTVTxYdnHnLe0FNBiukA1K06Ckqe/29xBN5u4+7L9v8lEItHb0B6WZTBqgRcAHJMomqiaFR9ifZgKMP+wz9f3/R/61uWWj5nnzZBKkYuXxRPgPThWVg0H1jNBwj+pwuygyEuZfrQlaUVroLuqhmUh3Yu/zwd94+zJocykEMt7eyvVctPVZRlfZIn/bBRNefHyMLcCaE4L12HumhqfdX0AMGi4eLGLiN7XikbopGfVjNIfZ/l8LejSosxDyWpo85zPXRSh0AvneMqx+NIWvrb9o9MnK1I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To support multiple users referencing the same fragment, pp_frag_count is renamed to pp_ref_count to better reflect its actual meaning based on the suggestion from [1]. [1] http://lore.kernel.org/netdev/f71d9448-70c8-8793-dc9a-0eb48a570300@huawei.com Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin --- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 4 +- include/linux/mm_types.h | 2 +- include/net/page_pool/helpers.h | 45 ++++++++++--------- include/net/page_pool/types.h | 2 +- net/core/page_pool.c | 12 ++--- 5 files changed, 35 insertions(+), 30 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 8d9743a5e42c..98d33ac7ec64 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -298,8 +298,8 @@ static void mlx5e_page_release_fragmented(struct mlx5e_rq *rq, u16 drain_count = MLX5E_PAGECNT_BIAS_MAX - frag_page->frags; struct page *page = frag_page->page; - if (page_pool_defrag_page(page, drain_count) == 0) - page_pool_put_defragged_page(rq->page_pool, page, -1, true); + if (page_pool_unref_page(page, drain_count) == 0) + page_pool_put_unrefed_page(rq->page_pool, page, -1, true); } static inline int mlx5e_get_rx_frag(struct mlx5e_rq *rq, diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 957ce38768b2..64e4572ef06d 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -125,7 +125,7 @@ struct page { struct page_pool *pp; unsigned long _pp_mapping_pad; unsigned long dma_addr; - atomic_long_t pp_frag_count; + atomic_long_t pp_ref_count; }; struct { /* Tail pages of compound page */ unsigned long compound_head; /* Bit zero is set */ diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 4ebd544ae977..9dc8eaf8a959 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -29,7 +29,7 @@ * page allocated from page pool. Page splitting enables memory saving and thus * avoids TLB/cache miss for data access, but there also is some cost to * implement page splitting, mainly some cache line dirtying/bouncing for - * 'struct page' and atomic operation for page->pp_frag_count. + * 'struct page' and atomic operation for page->pp_ref_count. * * The API keeps track of in-flight pages, in order to let API users know when * it is safe to free a page_pool object, the API users must call @@ -214,69 +214,74 @@ inline enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool) return pool->p.dma_dir; } -/* pp_frag_count represents the number of writers who can update the page +/* pp_ref_count represents the number of writers who can update the page * either by updating skb->data or via DMA mappings for the device. * We can't rely on the page refcnt for that as we don't know who might be * holding page references and we can't reliably destroy or sync DMA mappings * of the fragments. * - * When pp_frag_count reaches 0 we can either recycle the page if the page + * pp_ref_count initially corresponds to the number of fragments. However, + * when multiple users start to reference a single fragment, for example in + * skb_try_coalesce, the pp_ref_count will become greater than the number of + * fragments. + * + * When pp_ref_count reaches 0 we can either recycle the page if the page * refcnt is 1 or return it back to the memory allocator and destroy any * mappings we have. */ static inline void page_pool_fragment_page(struct page *page, long nr) { - atomic_long_set(&page->pp_frag_count, nr); + atomic_long_set(&page->pp_ref_count, nr); } -static inline long page_pool_defrag_page(struct page *page, long nr) +static inline long page_pool_unref_page(struct page *page, long nr) { long ret; - /* If nr == pp_frag_count then we have cleared all remaining + /* If nr == pp_ref_count then we have cleared all remaining * references to the page: * 1. 'n == 1': no need to actually overwrite it. * 2. 'n != 1': overwrite it with one, which is the rare case - * for pp_frag_count draining. + * for pp_ref_count draining. * * The main advantage to doing this is that not only we avoid a atomic * update, as an atomic_read is generally a much cheaper operation than * an atomic update, especially when dealing with a page that may be - * partitioned into only 2 or 3 pieces; but also unify the pp_frag_count + * referenced by only 2 or 3 users; but also unify the pp_ref_count * handling by ensuring all pages have partitioned into only 1 piece * initially, and only overwrite it when the page is partitioned into * more than one piece. */ - if (atomic_long_read(&page->pp_frag_count) == nr) { + if (atomic_long_read(&page->pp_ref_count) == nr) { /* As we have ensured nr is always one for constant case using * the BUILD_BUG_ON(), only need to handle the non-constant case - * here for pp_frag_count draining, which is a rare case. + * here for pp_ref_count draining, which is a rare case. */ BUILD_BUG_ON(__builtin_constant_p(nr) && nr != 1); if (!__builtin_constant_p(nr)) - atomic_long_set(&page->pp_frag_count, 1); + atomic_long_set(&page->pp_ref_count, 1); return 0; } - ret = atomic_long_sub_return(nr, &page->pp_frag_count); + ret = atomic_long_sub_return(nr, &page->pp_ref_count); WARN_ON(ret < 0); - /* We are the last user here too, reset pp_frag_count back to 1 to + /* We are the last user here too, reset pp_ref_count back to 1 to * ensure all pages have been partitioned into 1 piece initially, * this should be the rare case when the last two fragment users call - * page_pool_defrag_page() currently. + * page_pool_unref_page() currently. */ if (unlikely(!ret)) - atomic_long_set(&page->pp_frag_count, 1); + atomic_long_set(&page->pp_ref_count, 1); return ret; } -static inline bool page_pool_is_last_frag(struct page *page) +static inline bool page_pool_is_last_ref(struct page *page) { - /* If page_pool_defrag_page() returns 0, we were the last user */ - return page_pool_defrag_page(page, 1) == 0; + /* If page_pool_unref_page() returns 0, we were the last user */ + return page_pool_unref_page(page, 1) == 0; } /** @@ -301,10 +306,10 @@ static inline void page_pool_put_page(struct page_pool *pool, * allow registering MEM_TYPE_PAGE_POOL, but shield linker. */ #ifdef CONFIG_PAGE_POOL - if (!page_pool_is_last_frag(page)) + if (!page_pool_is_last_ref(page)) return; - page_pool_put_defragged_page(pool, page, dma_sync_size, allow_direct); + page_pool_put_unrefed_page(pool, page, dma_sync_size, allow_direct); #endif } diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index e1bb92c192de..f0a9689074a0 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -224,7 +224,7 @@ static inline void page_pool_put_page_bulk(struct page_pool *pool, void **data, } #endif -void page_pool_put_defragged_page(struct page_pool *pool, struct page *page, +void page_pool_put_unrefed_page(struct page_pool *pool, struct page *page, unsigned int dma_sync_size, bool allow_direct); diff --git a/net/core/page_pool.c b/net/core/page_pool.c index df2a06d7da52..106220b1f89c 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -650,8 +650,8 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, return NULL; } -void page_pool_put_defragged_page(struct page_pool *pool, struct page *page, - unsigned int dma_sync_size, bool allow_direct) +void page_pool_put_unrefed_page(struct page_pool *pool, struct page *page, + unsigned int dma_sync_size, bool allow_direct) { page = __page_pool_put_page(pool, page, dma_sync_size, allow_direct); if (page && !page_pool_recycle_in_ring(pool, page)) { @@ -660,7 +660,7 @@ void page_pool_put_defragged_page(struct page_pool *pool, struct page *page, page_pool_return_page(pool, page); } } -EXPORT_SYMBOL(page_pool_put_defragged_page); +EXPORT_SYMBOL(page_pool_put_unrefed_page); /** * page_pool_put_page_bulk() - release references on multiple pages @@ -687,7 +687,7 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data, struct page *page = virt_to_head_page(data[i]); /* It is not the last user for the page frag case */ - if (!page_pool_is_last_frag(page)) + if (!page_pool_is_last_ref(page)) continue; page = __page_pool_put_page(pool, page, -1, false); @@ -729,7 +729,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool, long drain_count = BIAS_MAX - pool->frag_users; /* Some user is still using the page frag */ - if (likely(page_pool_defrag_page(page, drain_count))) + if (likely(page_pool_unref_page(page, drain_count))) return NULL; if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) { @@ -750,7 +750,7 @@ static void page_pool_free_frag(struct page_pool *pool) pool->frag_page = NULL; - if (!page || page_pool_defrag_page(page, drain_count)) + if (!page || page_pool_unref_page(page, drain_count)) return; page_pool_return_page(pool, page); From patchwork Wed Nov 29 11:23:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13472669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C449C4167B for ; Wed, 29 Nov 2023 11:24:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0FFF06B03C4; Wed, 29 Nov 2023 06:24:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AF686B03C6; Wed, 29 Nov 2023 06:24:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E919D6B03C7; Wed, 29 Nov 2023 06:24:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D63786B03C4 for ; Wed, 29 Nov 2023 06:24:42 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A7685140447 for ; Wed, 29 Nov 2023 11:24:42 +0000 (UTC) X-FDA: 81510759204.19.BA3AC99 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf17.hostedemail.com (Postfix) with ESMTP id C41AC40015 for ; Wed, 29 Nov 2023 11:24:40 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=blcaDg0U; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701257080; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J730iItaW8fpD9bcYiCc3WSDtwgBtsyA2Jb0R83edv4=; b=pu6dlzzDF8hsXzVb6HCwOCtvK+veNAfEUMUwPR19qZHBp3+kCemlWdNXSymyctMgV8dln+ YnHeYTuzYGgVxcu8gELJyCiftO1MVNOK2VT3nbqCZgsHiMxkWtGYH24JpfEG8eVaugqBuG eH36Gc96L/9Q5gJf3rtQeRhhtAv+5T0= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=blcaDg0U; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701257080; a=rsa-sha256; cv=none; b=sdw7oVrOrwFoNh+nr9TmQ2Mm5Og9a1+XieGxRIry7/aAsfT0F3VebADR0veoGWB6MAiuFy ktadfCRjtaOC1s0sC8y+DVMqAWTRuAx0LxzsaWFgKOohiX3f9y6FdA2ZvQBlhMV4iASs6d i2aglogEAf7w++8/jW4AYNVLHDiZQjk= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1d01c45ffebso2383525ad.1 for ; Wed, 29 Nov 2023 03:24:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701257079; x=1701861879; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J730iItaW8fpD9bcYiCc3WSDtwgBtsyA2Jb0R83edv4=; b=blcaDg0UUC7xbQSse8T3L9xZU+9Mu1oPC4bI9pWd7CvzMQvwj3FtpNJQbqedpXDElp mmY+uoRHueVk9aB5kYiBAjZR0XIk619pa7Ic5rOiSBP6CZm8F5FMyqlIULS/TmzYyTzK w30ms76EatAVMPXXVuUKUNe7tZJ5gt7Y5xswzV2GNLD5FCrrZ4w8iwcZ6+ZKt8uWjNwP 8bQLspWpXkK2C6KdGR6Uqom/CKd5oaFjBhynXA85cFPwshXUhNdEqSzsYCBbFvcFSJaP gYOEFFVJbmOz0nFz6TOuWPzci5PDXcYk/86zzXale/433cA+kMeOAyu0U6X1JmtCewDt v28w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701257079; x=1701861879; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J730iItaW8fpD9bcYiCc3WSDtwgBtsyA2Jb0R83edv4=; b=EibiJjCwfcTMOwmuGurL0P8Nvvy+Kmv3T539eBH6baEDnbL0fZkPH6RuRdNgzzWYha qeZll2dv6EahiCFGU5HH1PbXmkhwLfwUerp+0wh4MttBpj65bS6sXygEeaxwLE+cu8k8 eB6FvERXvL0jf5LCxk+T9Z+Pq89Six6+h+qielETIC7uIKAgRF/D5jo/k4Anu/QYcFFO akPrLqMpvXjpMgDjstGKrcz7xgM4ZVciIIFW3FmfLNqHRY+YTihyA3c19CQ376ZZQIOk 6HK1d3vddGj4oRwC8hZbIG+QiTs5xzEWonKSvHRF8mzGvR9+6X+afNYXhxNyXCZa7WPd 4Thg== X-Gm-Message-State: AOJu0YyLD0epDtvqe+fvC5x2G9pnlJqOV5zoqoB9dmTtudpLow1cg4hj FA14qV92UER0DxfDOYnwnx4= X-Google-Smtp-Source: AGHT+IGend3+KRvVWg3wiKEnDCJRXZQxRXIpsrnkvu2W3MzYQdcBsY89G98gXH/coDoEQKUh9fTDxQ== X-Received: by 2002:a17:903:32c8:b0:1cf:73ff:b196 with SMTP id i8-20020a17090332c800b001cf73ffb196mr24920528plr.8.1701257079629; Wed, 29 Nov 2023 03:24:39 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id c6-20020a170902c1c600b001cfd0ed1604sm5460710plc.87.2023.11.29.03.24.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 03:24:38 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, liangchen.linux@gmail.com Subject: [PATCH net-next v5 2/4] page_pool: halve BIAS_MAX for multiple user references of a fragment Date: Wed, 29 Nov 2023 19:23:02 +0800 Message-Id: <20231129112304.67836-3-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231129112304.67836-1-liangchen.linux@gmail.com> References: <20231129112304.67836-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: ku64uw3jz5wrj5nkn3f186dtjopwyjt3 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C41AC40015 X-HE-Tag: 1701257080-497228 X-HE-Meta: U2FsdGVkX1/fi0gX50ilHmabxUCZFLFHWgCeM5Tfg/xLkbiPyufuc2XOQe8erYl+/tsCme2stXN/kCRgvdReQsb5U4vxn7aFVot/ULPec/3rhfTKbncAL+aD23OLVFyDzTdiY6Um/ZWWZANRFW9kTQ0leYu5O2IPSd1sOv/3x0JGmMcPAveTxjCyHlmK34tCcoW3rW4c1n/7uN1YY7C7uL9YYXM/VdMmOeoazLEueocPK+hB/qj20kYsGtlZC70efHUR5Jm73EZZWkoslAgO07l0Gh5PTuQbtJocr7ODrQGmjEFwavK/mKQNKGVx7oCMWG6FS/nXGKqUbMpQyRYagW9WXw4zPHcJV/xQYBe7R8tfWxAlN1IXvuZb0U3jg9HaRCuqYv7K37vr3VXTpjUlfiEM3sxHUKG9FUnx0CLNMwpLrpnZZ90kjMExpxiGczlOHUHjPnMxfjc3WIMjqZJFNK/K1kbzCgyAJ/BIg+UutGAfrb7TxP3JH8zk7nLyPItD9vK0OQMsYb0ArywRSWlz19EFz5PSkIIpLn4nlQjr/PjJjScNxg3Ka851cgr+gxk19S1EdOMrb6Sd81scgU42tKhfFr19wXE+sgJ5Pv9r52AQK0I15SPM77lVjMA66fbS8Gp26gtOFYKvgesSp0ENDqM2Gd5/r90RHNoFRJYF2r/McLkT60SkcXmF+bNek4saENloEokdXKORmGmE/nO1hsz65H/zFdh6ms0tnHYCbcLbfFqJNGRL0RSEAp/psk/4ZtzMLRqsfXZfGJ8iIG2sNBnjSYxv0Jd3pJmpNX78IC1ZrlZp1+LeEdjJX8I9RJkfPC7DAnTkOe9LvURE0Yozf01WnK5inhifcBkSJ5seWsTgE9fVJI8zoM9eAgg57BdBaXA5BleUfO0AO33YdtxEsWx2HgyJ24TVOu/2Grzch7FqTxMApKCNTv3iKeCRN1dOU7JgFOBBBH7TJu+CXUx CJ+Byap4 zJxxfxAsUWzvrOX50UfXzNRDGCqK716DCKpzAYa/DaI9+2R0IaYcfAVOLgjZ2inUCdZLKib3noQXh1hpu1/mNj9p7gWDhtkS27GV3NRld+rTwKuCbaFpgE7Sc0Ur/cPjDBWxomp+X8nJSa+L9y24km8xXXUxFTYhKpqYtA0UkD/48oVf30OvET21518kUxwQssoVL4Z81zn8dApp6OVSOKoOTvgtAoIilpplSFHgwYPkHU/aIOIrEcj/byDO853+DzWDGm43E+vR8emRh6/KPUNwKgX/MBpmInOBOy/OWvrFpOimurFKpHqUdblqfmcxEL4K9uxFddrPsTjbguQuDlvOv55oDHTP5wp5XPXpcklg9r8Io5bsZIUXr1rLENv6qpwy6sqftIuaJJaRLMGERxgalCvYER2jYpfS5dj0/B1qFgi7WxFyYRPdbfJ/Zf8e33E9WibC7vyO7vu9vLep3+yAoBE4BdJ7gYgxD2p2dpQzckLzoGT1BkAYR+CwFDHolk7ls23KP4ei3o8EdzE+nydIfsq2VOMg4Ti568ehrw5vxXlcxxvUjTGNw5eSeEvApAb6eDThlcbIEzQVrF/rHm0CBGhdlxTLhVnF8BUH3fNso+EpZBzq4q3uFbRm01kfkqLKfUM3WAi/6CVH6Jg+PGHClQrB9Bma8QW74WCcYWKyy6bk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.065729, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Referring to patch [1], in order to support multiple users referencing the same fragment and prevent overflow from pp_ref_count growing, the initial value of pp_ref_count is halved, leaving room for pp_ref_count to increment before the page is drained. [1] https://lore.kernel.org/all/20211009093724.10539-3-linyunsheng@huawei.com/ Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin --- net/core/page_pool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 106220b1f89c..436f7ffea7b4 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -26,7 +26,7 @@ #define DEFER_TIME (msecs_to_jiffies(1000)) #define DEFER_WARN_INTERVAL (60 * HZ) -#define BIAS_MAX LONG_MAX +#define BIAS_MAX (LONG_MAX >> 1) #ifdef CONFIG_PAGE_POOL_STATS /* alloc_stat_inc is intended to be used in softirq context */ From patchwork Wed Nov 29 11:23:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13472670 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52763C4167B for ; Wed, 29 Nov 2023 11:24:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1E4E6B03C7; Wed, 29 Nov 2023 06:24:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DCD7E6B03C8; Wed, 29 Nov 2023 06:24:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C96D66B03C9; Wed, 29 Nov 2023 06:24:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B59C76B03C7 for ; Wed, 29 Nov 2023 06:24:48 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 906FD1603E4 for ; Wed, 29 Nov 2023 11:24:48 +0000 (UTC) X-FDA: 81510759456.01.BDFD610 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf03.hostedemail.com (Postfix) with ESMTP id BEA422001F for ; Wed, 29 Nov 2023 11:24:46 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AapPwIuE; spf=pass (imf03.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701257086; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JqbHmSBHyDiifA4Jyxx0ZxpzcDGefiCrTqwukJus35I=; b=AaGm7yHCzmQ33csNpUbfqQImXpsRpgGaBoSuJHXF2X+60jX8UnGnBSUzdpykpUo5fc+4bf Zq9I5c6KHzDLEPdty0UUZOglQuVrAOY+c5rP2BxDMDOzqPriCnyArAHnYfavh0ARPD51mx +0b1nCQd0iPj4QUCoCOLRLE+qVCDyzQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701257086; a=rsa-sha256; cv=none; b=jMMfKQRKssrjZVndz0Jmvf2NNCwZsnVLgchpAK6sgCDCzu8Ucce6C/QZBOEe7CQRpOiBLi /WvQ3a2pBQE/rmCtvnaZZgwN4Amnb4aMUiS4fS3Oo+rrbn8j4wNKt1HZxmUXBF6UpLSjAl /TjZKl1OnnqcCOBQpnGrGMlsSN3Ao/I= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AapPwIuE; spf=pass (imf03.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.175 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1cc9b626a96so51016365ad.2 for ; Wed, 29 Nov 2023 03:24:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701257086; x=1701861886; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JqbHmSBHyDiifA4Jyxx0ZxpzcDGefiCrTqwukJus35I=; b=AapPwIuE2xR1S98zC/6tlRxiizern8MdVCrwcsxaks0FGQGNMNxPI7hpTi9NIfvlbQ w5dgLmDLh2vCZAKiWNcG++e+ZwjiwAR+77BIhFNdie5wMKU+01ynwbN+3TOrUaAavu4A DBHxFXTkXLAgOL0bbvUxncUXDIzvsUQWnElHwZZurS/4UuKU9fuwjifb137uIhnQ8dt6 547Dmwy5bmuFq2ABwpbuy1IOIJlXA6i37I/gbB9ak98RjWwAN6WsIdyKTEPvV2BP17PI fiKVXawzUABAuq9AzU2123jDnUfcszUfht5y30yuTFEJCoGMFApFiEwwN/UybYjVQHN8 pkLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701257086; x=1701861886; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JqbHmSBHyDiifA4Jyxx0ZxpzcDGefiCrTqwukJus35I=; b=eOF6QW7Y6ixXnpdzqzz9A6lIIfIioNvijc0AOHsdxgHZ5FBETD534yRPqde91dA8Pz +uSkFBznqG6VZ2qC5HCtBXG4CWDWPPAUxk3W71CLlyNX9Tk+XR0UJbYHjvLYI1zsgLLZ PAtQvCrdmvPPdkp137jmNWSWNg2nA0MQPgOL4Jm8EsFxFQhmllF8Aw+xNKo0QKMUxVHB 9VFrUaFOvgmDc3b/scnbOxgLYqRhO4beshvvrY99c34WBSEM4vl6iwYZk+CJ5LKrDbr3 mtUv12Iyj0vVxnldZ5n2lgrQo29YAS2jqxipEg1wAJVbCajmBwPjMr5CsrqAKpdjfWwK D6Hw== X-Gm-Message-State: AOJu0YzE0Z8hSgvsShvQahDX/n+8brHE+hx5wpa4mukIPCr04zh1cDs7 tIqF6b7olY1MwmCdmZ1x+Rk= X-Google-Smtp-Source: AGHT+IFHWYMdTVTSfsGCk8fDsPpGkHc38ybdSw7NPcTxeq0b8IsPB13SFusKJIRla4cMpvtKqqCKmw== X-Received: by 2002:a17:902:db06:b0:1cf:c376:6d8d with SMTP id m6-20020a170902db0600b001cfc3766d8dmr12165665plx.32.1701257085724; Wed, 29 Nov 2023 03:24:45 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id c6-20020a170902c1c600b001cfd0ed1604sm5460710plc.87.2023.11.29.03.24.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 03:24:44 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, liangchen.linux@gmail.com Subject: [PATCH net-next v5 3/4] skbuff: Add a function to check if a page belongs to page_pool Date: Wed, 29 Nov 2023 19:23:03 +0800 Message-Id: <20231129112304.67836-4-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231129112304.67836-1-liangchen.linux@gmail.com> References: <20231129112304.67836-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Stat-Signature: jzr5ywmww48kp5j3trpqf5r7in6nmgtd X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: BEA422001F X-Rspam-User: X-HE-Tag: 1701257086-842700 X-HE-Meta: U2FsdGVkX1+Vrex7GeNrylLKOQcef9EWJKMdob4gs1PGOdKlvZMR7BfaSdSgzMJ9T7tsKnY/mc/QUeJ2l+pENtSTN0ktKNnHy8YQZ5zWein0XzDmzbjlbZd0fzIJj4+ZHk9f3AxmU/0MYQb8SHPVBCUAFfZDVs4QM3uDKKNlnvOkPSJq7LtWJ11vViBTqZ5eaSQdekVGxHJyPbmRxAV14eSswgQ/akGm6UbnR1lzMUNhbaGMk1o3voHK8wveP6YLZIxfCdYFlXIF+JqrFD5hJWiYvOgDvISWzlyNrAkVUBCnD5i/TIhuDNJu7yKlLha/f3p43BNfeRJGXdm5kV9MDGAuaxtW6v8eBR9tr+nRVzqEy8aVFuFOYk/6f5azCSzCSThuvw+rU+h6Rkwb6tEQhl/971SYAOWvtbyVUgImq2UnJyNA4kAqyy5YBXOLH6fr5JKRBqQtIshy3ASjnjzxWMDUD8+gSs2axC5AKeYM9TSbfoujQUcCag4/1ZWhz/SgYU70Z2flONwD8k1O6CzIGB9N6onxONdNAFEDJfJrfkZ6W6UdPCP4ptU0qFzBVJ2C4ZpRMJ9cNFkqXB2NWegXqXIT8ygV4+ph2FKGrNEw0Nca36phrOoeZr//nvcIzbJAC1To3nYurnfLlABUaJUS7ZO+iroTz+1DPsD8hisfS+IDDbj61ltzeGgeIw7YIrdM9DN9WvXN4iNuGBlbT1zmRuB5KjpdPut37UiFeF9kL/FcqLpnhAYMw8JSMavMU8uwWX9lGczlhoF3RQtTX5maY+qFWjhyBTOhh+sWeG5VSGFczBlHEXYQVvErSyI1+ZFvisEnKinU/FJ5fT9xJ6pLQdbv3pgNpASrM0f24GLIzLlwU+HlKPiz1mffIco02YF/re1nW3Kzut6pgmyPRXo98xQUZgbDuPOGmJpAtJwdS0zpZvV8cSi6IbBFqHNRh1BX4a9cw6G6nvPaDP1P1Xg ahKErzUp 3LBAX84+pQ3lObz+VRHu341FJrqkjVO7Kd7BPVvLjIsF7kvUbWQ4lMrUfe19gwvZLuzfP8r+T/QpT4UCdRsmZDBFxMImczu+3ZNLmYX6spgtagNh5D+nfynrJ9wfWcdFXrWzJhI0vDxA9JJCE3hioQYXWAI5XMpOxXaj+2YBJyeR3PQW+j+zWb61V6fTV86+WPirTuxMJIbI3JTJ3Sv2h9jQ7evG0Q4g600JZmcssjeWKWYP/8YEJQilZ0OSDXS8faj2PFl/ishdoPy8P5HHtwWqcVET880OHvsEMaGOSoEwQ72FO3orQpvqsdC8ZSR6oXmjI9sJ8RTn79x4QbicoEIQ/6QohIFkYFZ4UbOkmB2P8jiWLxIWUsGhlOJqcKnCdoQasDsKR3SN6CvvnexYEzOTIwxqIBeAqo9pmgijEGCWPafIGZX/DmgvRbwxKAv9d50D6lE1cGtzKKjiC3dhbBePGDWEqxP/a5cmR3ZhhDO2Z+DA3TXWXwNszinU0h7ECEU5gVzS7vuA7w4yLlDsBLDyCSGeDzTC9wR4jp5D6CvWq+uErbNGkjKquWypTz1up4LCX6tskc7eKTLsaJ7XD/Pxczw7wX+wMJ1k/ALhf4DMXR5pPaN4IXeY9OCnVO/NsGXW8cH3pbVM8qCPc42rMn2imM4lgVD7NcBHZ50/7/KSQ3+ekT6KPVGepcg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.001036, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Wrap code for checking if a page is a page_pool page into a function for better readability and ease of reuse. Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin --- net/core/skbuff.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index b157efea5dea..31e57c29c556 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -890,6 +890,11 @@ static void skb_clone_fraglist(struct sk_buff *skb) skb_get(list); } +static bool skb_frag_is_pp_page(struct page *page) +{ + return (page->pp_magic & ~0x3UL) == PP_SIGNATURE; +} + #if IS_ENABLED(CONFIG_PAGE_POOL) bool napi_pp_put_page(struct page *page, bool napi_safe) { @@ -905,7 +910,7 @@ bool napi_pp_put_page(struct page *page, bool napi_safe) * and page_is_pfmemalloc() is checked in __page_pool_put_page() * to avoid recycling the pfmemalloc page. */ - if (unlikely((page->pp_magic & ~0x3UL) != PP_SIGNATURE)) + if (unlikely(!skb_frag_is_pp_page(page))) return false; pp = page->pp; From patchwork Wed Nov 29 11:23:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13472671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2917C4167B for ; Wed, 29 Nov 2023 11:24:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5959A6B03C9; Wed, 29 Nov 2023 06:24:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 542F66B03CA; Wed, 29 Nov 2023 06:24:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E3BD6B03CB; Wed, 29 Nov 2023 06:24:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2A6AC6B03C9 for ; Wed, 29 Nov 2023 06:24:55 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F3DF2C049C for ; Wed, 29 Nov 2023 11:24:54 +0000 (UTC) X-FDA: 81510759750.19.1F57008 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf10.hostedemail.com (Postfix) with ESMTP id 31A06C001F for ; Wed, 29 Nov 2023 11:24:52 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NC+hHV9g; spf=pass (imf10.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701257093; a=rsa-sha256; cv=none; b=pVSzpmLRc5e0ZkcR0ZrZ+CnIiX4bHH2YHsUNoSwCe7nrt8lNCl3kLPiCKqZnaYm5GuKTUB tHspJi6JKULuotO3dGkfD7Y5xLr91tjuvOLdokKJFr2rwWbhq1GwOxOcfHsDLhpyNbEJ2h /6JIf77/fQWEgzDjvuUtqL8H/B9IiLg= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=NC+hHV9g; spf=pass (imf10.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701257093; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=60yowogr+zuknO3ERs4O7G8fQYZaBv4s1gl54jybDiQ=; b=3SdfXN7pQHtU73i24nVsex0lJ2Gx4Iia/Pr8+irZXCX9e7jGMCUxJ0vHpjqW1u7wwI8C9j 8kxYR7idL9Szmq/2xgwZPlx3xdohPfB1r8QUxX5x5pfSAjJ1o515vh3wnKWkVQoISNe1SS hDXK5GNwoB2cFSsofLSmmyr6P6sHrA8= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1d00689f5c8so12293355ad.3 for ; Wed, 29 Nov 2023 03:24:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701257092; x=1701861892; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=60yowogr+zuknO3ERs4O7G8fQYZaBv4s1gl54jybDiQ=; b=NC+hHV9gmpCCuNDyxJyW7gSHcnrdn6rlXg9tfaQHt6DpR6lXysOJ0cM5sB+j1oQFnV 9/6etbUqaZH2h1XJm6q+zQ23zemTYVHjDyu1Y4V611Pvfvi53bfKyVvt6UwLnY2lewUc JJzaoCGZjC9F/OfhJfo01sKszKCRCVF/UTC44jLGpHYeG0woxM7TV0p0v+z3sTg9pKr/ TiYEHHUp0tyIMk5zk60xWBvLHZkB7JjaOGSlsP1ZdSkxL/PiBYEVoRe/e8d1Lswvqohk KBAe2IW72RU8xEy2vp0T1LdrqUvXFjhezxmFLZOvzWWRf2CHJ22zRt9eWhlPAugZKr21 OqLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701257092; x=1701861892; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=60yowogr+zuknO3ERs4O7G8fQYZaBv4s1gl54jybDiQ=; b=ihNSkM1fS16+g9WLIVPmKkp4bvxNaXoHc6MwqZcDX9Y6WqXwOY/Ci16FUyfCAyo1lv 6KHSEosE4aXOYhDMukFPPPCMepshYjkke88E5b2dSzrfoVydtSPhLLZWySYE3i+jbc/7 Wo+8PesxuoZN+cZszlw3S8cdNQdO+8e3yOZ/UvJVf4Oy0G2RDwDLD2QPl5TGQIZ/DmEH pkHJC9pu4V5y1WF/0TKEJ04dU36gOiIcOaAd4sRR/qTgx/rmF4sAYFsuTQMNnlBDCoPk 4rLgN2YESTbYk1NaZABYCJc5I1nYUxY5hzUR9uNdaQmbh2XXPeXOHP7XOGe86q2kHriV cZWA== X-Gm-Message-State: AOJu0YxLDPRga+oQXQt80enmae0u7VBWn9L5sv1XLsFsRMCMo13y1lWz c+pZhptdKLBhvlIi2uQ0DhQ= X-Google-Smtp-Source: AGHT+IG3ycrW11c+jOMiRX0m4x0f2LMO5kHkyzpQoQeO4N3U4j5Xb+awhZJOOwUSY1nrq3eoWqcStQ== X-Received: by 2002:a17:903:32c8:b0:1cf:b12a:a9eb with SMTP id i8-20020a17090332c800b001cfb12aa9ebmr19342480plr.19.1701257092063; Wed, 29 Nov 2023 03:24:52 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id c6-20020a170902c1c600b001cfd0ed1604sm5460710plc.87.2023.11.29.03.24.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 03:24:50 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, liangchen.linux@gmail.com Subject: [PATCH net-next v5 4/4] skbuff: Optimization of SKB coalescing for page pool Date: Wed, 29 Nov 2023 19:23:04 +0800 Message-Id: <20231129112304.67836-5-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231129112304.67836-1-liangchen.linux@gmail.com> References: <20231129112304.67836-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 31A06C001F X-Stat-Signature: owk33fk4mr7zdoo45fh9g63qayimaufb X-Rspam-User: X-HE-Tag: 1701257092-828694 X-HE-Meta: U2FsdGVkX181/yEj3zRF0ehLZfx3CNoDrI4emLqwDmaT3FXCkHb6IwO+UNEDnKrQkCOD+0oTra2bCcdEtGEsoqoagZeQtMVY5RWxKN2QuwNR5Sx5lla+aFptIlZiYchZzZZVDLoYtFtz9oslT2e204sgid6sJxHAR4TjO5+BaXBEURRF+BU3zuP9K7fTISVYFy3GSxSa6gBfqhzV1rtfWJH9dV0HOTu239FW88qe5FlsRen1u0UFYnZpSRmaCVJOGBCNE3dCoGlPrHwg6PJrOkfD0t4wswsKbN70I153dQKf+zvb1r4KPapj2LwNTWZHJ2apZ/WQqLJS0BA0kG8PFnzmBz7bQ2HPRVLJ0AZYOSqj8FcEydUVZXyBs4EZyxg1eAZCYhiv8Rr9uBWHckLurUqxdi7YTueM7x/HI7ZGIpyjOVDXUF3fPvfbSE5xmlGLeFDRC6vxNYLOO+AmctMT0YCqmuqIxvwlRFKuevfmOysNxMHCSZQJm4fUWA40hevBpsfQT60klcehSMNVut+7mArfz2WNHgQZlfrtmfigkFhM9cAD1GSgIXrflIlJt69o3n7OLLCF29AenQ8lFBQbkS+/kAK1r45nw8dTlRf3bdlO8yUOCcoT7hGV/kbRQGzFSZDmIkCuNTVqEKDrReOMk44Zf5JdeWleqmgIjPhwpVWEhLZX2bXuX+ZQOuRcxoMoiVee6jBjwz0ipOaS0RpfFDtVrtBb95/teMujeVTyOj8TwRKMDI2WEOfz/trIOBCmUT1S1lI1c7nzWjauWsOS8x3Uf7ER3Jyoa3hZEjO7hLlE//c1eoMZCLs3JQKNNKS2LQdSF3k5sDp46u5l1vBpK2YCOirp0k10p/47koq56hDoqborCQEqO6B++6eBtliiJbo63YhLbsWqFCjjiW2vAHqfzXjI63b7jf4Yj/bDEKdTPUg0gwjzrf4xWP0GT3nkY75j3gw1jNlW0/HmlDI XOxviuk2 aJSGUHTXK0dJ8WCgN/PE1sypBguH0wYH/jWK6tOA6TErbZqTvE8XXeIChL7LaTk+iJVFNNLyQ5RrwrHOx1NtCXyGlb2xsLh9HP4LCTIrX9vHgPkAcW//lIv7KsrY+Gw0jDgxu13dn6GPa5mb5DhtlqenzKt/bA+BRtDIfZWe4AuMsY85ipb3y0AaactDkg7iLN1XolQoNNU1fopxf7gLdjAEDDZEw/xHedqrWfymh1ZcntfdXb79FtS2i4YzyXSrhhKfcqfLlf4drCCERFl9enHo5YOG4G8+SXBMPXyiq85HfbrKBImwsrylRT+DC0YeRAoguh1CbY+UtLyDWDRTaEEY71YJapK+1MhPE4zC9qxT/pMnXYTqc15pDpzsv2K05baylrF4u3ASTzPa7e5WB9mrazoEMMPRmGd5mqPCWnWQQBN7khLOvp9uzitDmin9PEVI+bu61mFVSvB5AsdznKt0SE/Zcz5035hxbE1ieQJgnxtxF87WEHyeJmEQcKbNJMmdYWK7PijGOtRBaIi4aAIxnd2O6JBjwRQiCHa93VjH0qMGA+PVjGFefxgvBeWLMrlag X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to address the issues encountered with commit 1effe8ca4e34 ("skbuff: fix coalescing for page_pool fragment recycling"), the combination of the following condition was excluded from skb coalescing: from->pp_recycle = 1 from->cloned = 1 to->pp_recycle = 1 However, with page pool environments, the aforementioned combination can be quite common(ex. NetworkMananger may lead to the additional packet_type being registered, thus the cloning). In scenarios with a higher number of small packets, it can significantly affect the success rate of coalescing. For example, considering packets of 256 bytes size, our comparison of coalescing success rate is as follows: Without page pool: 70% With page pool: 13% Consequently, this has an impact on performance: Without page pool: 2.57 Gbits/sec With page pool: 2.26 Gbits/sec Therefore, it seems worthwhile to optimize this scenario and enable coalescing of this particular combination. To achieve this, we need to ensure the correct increment of the "from" SKB page's page pool reference count (pp_ref_count). Following this optimization, the success rate of coalescing measured in our environment has improved as follows: With page pool: 60% This success rate is approaching the rate achieved without using page pool, and the performance has also been improved: With page pool: 2.52 Gbits/sec Below is the performance comparison for small packets before and after this optimization. We observe no impact to packets larger than 4K. packet size before after improved (bytes) (Gbits/sec) (Gbits/sec) 128 1.19 1.27 7.13% 256 2.26 2.52 11.75% 512 4.13 4.81 16.50% 1024 6.17 6.73 9.05% 2048 14.54 15.47 6.45% 4096 25.44 27.87 9.52% Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin --- include/net/page_pool/helpers.h | 5 ++++ net/core/skbuff.c | 41 +++++++++++++++++++++++---------- 2 files changed, 34 insertions(+), 12 deletions(-) diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 9dc8eaf8a959..268bc9d9ffd3 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -278,6 +278,11 @@ static inline long page_pool_unref_page(struct page *page, long nr) return ret; } +static inline void page_pool_ref_page(struct page *page) +{ + atomic_long_inc(&page->pp_ref_count); +} + static inline bool page_pool_is_last_ref(struct page *page) { /* If page_pool_unref_page() returns 0, we were the last user */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 31e57c29c556..2d797f33d809 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -947,6 +947,24 @@ static bool skb_pp_recycle(struct sk_buff *skb, void *data, bool napi_safe) return napi_pp_put_page(virt_to_page(data), napi_safe); } +/** + * skb_pp_frag_ref() - Increase fragment reference count of a page + * @page: page of the fragment on which to increase a reference + * + * Increase fragment reference count (pp_ref_count) on a page, but if it is + * not a page pool page, fallback to increase a reference(_refcount) on a + * normal page. + */ +static void skb_pp_frag_ref(struct page *page) +{ + struct page *head_page = compound_head(page); + + if (likely(skb_frag_is_pp_page(head_page))) + page_pool_ref_page(head_page); + else + page_ref_inc(head_page); +} + static void skb_kfree_head(void *head, unsigned int end_offset) { if (end_offset == SKB_SMALL_HEAD_HEADROOM) @@ -5769,17 +5787,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, return false; /* In general, avoid mixing page_pool and non-page_pool allocated - * pages within the same SKB. Additionally avoid dealing with clones - * with page_pool pages, in case the SKB is using page_pool fragment - * references (page_pool_alloc_frag()). Since we only take full page - * references for cloned SKBs at the moment that would result in - * inconsistent reference counts. - * In theory we could take full references if @from is cloned and - * !@to->pp_recycle but its tricky (due to potential race with - * the clone disappearing) and rare, so not worth dealing with. + * pages within the same SKB. In theory we could take full + * references if @from is cloned and !@to->pp_recycle but its + * tricky (due to potential race with the clone disappearing) and + * rare, so not worth dealing with. */ - if (to->pp_recycle != from->pp_recycle || - (from->pp_recycle && skb_cloned(from))) + if (to->pp_recycle != from->pp_recycle) return false; if (len <= skb_tailroom(to)) { @@ -5836,8 +5849,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, /* if the skb is not cloned this does nothing * since we set nr_frags to 0. */ - for (i = 0; i < from_shinfo->nr_frags; i++) - __skb_frag_ref(&from_shinfo->frags[i]); + if (from->pp_recycle) + for (i = 0; i < from_shinfo->nr_frags; i++) + skb_pp_frag_ref(skb_frag_page(&from_shinfo->frags[i])); + else + for (i = 0; i < from_shinfo->nr_frags; i++) + __skb_frag_ref(&from_shinfo->frags[i]); to->truesize += delta; to->len += len;