From patchwork Wed Dec 6 10:54:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13481485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BE1CC4167B for ; Wed, 6 Dec 2023 10:54:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 282146B0075; Wed, 6 Dec 2023 05:54:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 20BBE6B00AA; Wed, 6 Dec 2023 05:54:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05DC16B00AB; Wed, 6 Dec 2023 05:54:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E21D56B0075 for ; Wed, 6 Dec 2023 05:54:40 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B28FE160142 for ; Wed, 6 Dec 2023 10:54:40 +0000 (UTC) X-FDA: 81536085120.30.CA8F798 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf28.hostedemail.com (Postfix) with ESMTP id E3B20C0015 for ; Wed, 6 Dec 2023 10:54:38 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jCTPwSoD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701860079; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VMA+wq9Jm0b7Bol6yd6xB1YlXbaZgXyOcITML8JglNg=; b=FWEIou3ImbJFD1s+OgHuzWglppsDBg/8F+fHuIZsNU0vOkO8xf6T9BU7wSxrYyx/nRLFI9 jMdGS7P5yQCRgKjEaaSbUmb3jFcn2Ru232l1Pl4kwmHXIBCWcf6PqpturTVG3EJ+AFdPuh aN2F/eVIzKDeZgI7Md/PEDkAveRz68U= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=jCTPwSoD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701860079; a=rsa-sha256; cv=none; b=HAtFb4ft59GDyc+V1g8FgBlOaMoC0S9gAyk3CudGQEy5/uAHKRtM5B6JkjOtyNmGRxm3sk L9OsRDPJxChX8ILG+AQpuGSgJQ9y1VXKxF+n0LmHhf3pbJKz2+jmO11nLuO5x2r8/vZ4St UwOBxyDTbeUG4N/OgcchP+37uT/RLyo= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6ce52ff23dfso602315b3a.0 for ; Wed, 06 Dec 2023 02:54:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701860078; x=1702464878; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VMA+wq9Jm0b7Bol6yd6xB1YlXbaZgXyOcITML8JglNg=; b=jCTPwSoDdu596hQZRAkHm0PBZ1dO64Mbf2eHzJwySvo+WtaCPC3dQ9KH2tEwS2jOe7 ji4n+8vzi8lvYycCkvAJ5ZFDy68IX8NI217CBvFnWqwcUl3+J4fXSO0FckXx3lMAcBnu CH1CCxSv5kJZYXrOUTVbNoRunkfO++uS3gNagH0XYPG4FkyW0O8U2zCtYOnkeC8WT8l9 zRbFoxdMXVroFwQ5gKBG5fI5iPUQXlM0nUY7iH3CgqRX/T7hQHXzpOU9J2HYz7MkMg25 Gg9SA41TrrIkbvepQgj+8PHSa7eS+7NaukohA7WcmskY9dn9c71UN12BqTQBVc3WfVyg QgPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701860078; x=1702464878; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VMA+wq9Jm0b7Bol6yd6xB1YlXbaZgXyOcITML8JglNg=; b=UNSToBI10I+U/wCQcT4PkOvKRy5a6HM1QCXZDuB0yUVt+hW45eMmVaNB2894u22TZF KpCc5QbIOBc1xDcAqlooJbf77nXXfl5o60YXfiQNXjVFnWeGEVl2P+TkM/FjPO3txTHV rUkbKmuSL8DxE8Lr7WpQeqGRG524Q6CjxDjB5LDtV1Vs3KYX9qPEnt39THVE4iOGL/Fi 1A+uco/IsueruUPK1OaDjqbY9kh5eRNawIG9BhZ7FkYf6eCdxPo9zYALncY+HFS1EZTr 2LTiojFRDwVqhMS312/SlIuPd/4+9siVITxeASMr4YcigTKqJaZkZvnfiMsv5Txr1VZh z4JA== X-Gm-Message-State: AOJu0YzL/P4bRhGh42ck3UsCEq9gU/PN45oCLcBs2W0i4gLYFzDjFXNa Y5z0fl9M5OF5aU7n4bugl4VhZK+4mvY= X-Google-Smtp-Source: AGHT+IHlsRemaAZ0vBp17tCrbsNHx/YNozwjCObQMRxRWtB6pKqVA9DflSCz1o43HJA/xLHxeOQQUg== X-Received: by 2002:a05:6a00:b4d:b0:6ce:78c9:5979 with SMTP id p13-20020a056a000b4d00b006ce78c95979mr2108126pfo.18.1701860077567; Wed, 06 Dec 2023 02:54:37 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id n15-20020a638f0f000000b005c6801efa0fsm5388796pgd.28.2023.12.06.02.54.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Dec 2023 02:54:36 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, jasowang@redhat.com, liangchen.linux@gmail.com Subject: [PATCH net-next v7 1/4] page_pool: transition to reference count management after page draining Date: Wed, 6 Dec 2023 18:54:16 +0800 Message-Id: <20231206105419.27952-2-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231206105419.27952-1-liangchen.linux@gmail.com> References: <20231206105419.27952-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 36sqf66tdg55ufgfyqn3rbay53ibtyy3 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E3B20C0015 X-HE-Tag: 1701860078-615790 X-HE-Meta: U2FsdGVkX1/0DlnbfDdL1lB+Hm2nBip5AOPWGtNkXz6GJICfwEg2x8/kd+fAh4VnfpQplw1fiD3wR1N/czYfQuqncsMTiJaBiaa3f1MTMhm9BXxFl4Z94WDEWQrzhRDQWNL3FMK+jtoYJmbHOdfqa7+/NA3/gFL0sFIR/3BELTTRe+9glG3vqfPzfG9duBSb+bSQv2EuD7ACZX7udNv6XuSlbSL8IHSRISodIX+yP3/DTdwUyhXr14HYTncQmRUxYzl/HLzKqVe5D4ngTw5pnoTg/6/VhM9t2V+IxxA9bHNDWyu8S9Tejb0Q6xpSGQ43zfaMalX8uCGoH2Zg6EmgmD3arC7fnd9WWv9u+0W0ada+Z6mri6MBcohhSSNp2wGMciTpyrfRxuieS+YjaRhX0qFht3MAPXgcQTd/Lg21Yhw/cWRZZyCWDvKYELo1tfLJ+Of+gysa1OaIaJZX35KA2VOwrUFFGIsaygs9m19s3hfoPhVHi6m1KbHWw/n7yPROzgpAI3Rt61CSl3F5NkA85HNoexwloichu04KQzku3aj9K7IOxudIJUpAVOBrW7DOxgNVMNR4qqyNZNl4fWz4t/T4P6ty6tvGZgpnUo/P37q87VWhrI1NDpDsWK8V5RA4GQ40tjnftKErlGmAM6jknoxIvhmN5TMrLkL7GcPP+cU2CzsbA/5gJNxh8GcS6WEFyG+7TN+u0U0Fi0L3L6qKDFdbxnU/EaWdjwKuxxQRW4FV4jnBvzdWz8vo4A5Jl/FLba2RbLzdTypeo7gGk5G1Ax6T4p2u6kN++YX5toudQkGYCqzBYMYupc4MaN6MyMjpv2iM0/N6J7bEEwVRRNpdpIO04fZu05D9jiIduV2UBkJS3y576J39qM6G0KjoPMsDN0o94gTWtnpw+PJKkwPDkeqlPJv8xsP/ormzwGlxqeRqaT9AqmRT65ga5JDAQUoEy4NV9Hlyf0S5sPj3fir bsav2zg3 4c5iUFredvfauiR62FlWN5n0/UcA+n3jFHjEGP2T91S4Q3QpqlUeseTay0lEvNcPH+P7cEd2ipjzGmsBBvcBFfp0cl2Pw07Q+CwlyKOF+2hmE7pIALo8fLU+Wl3MAX5jwOiqHA3vctnF4tzejWDt4aqBkfUj2rqvGXEjvvJIg5SWStQW2iUNmJZ24tt3u2Km0OdTxTUORDUCX0SLg81qFsPQdJ1HoQfCnwseEdkniKI3EAkemd52QjiCiGATFq9K5WVYN5GqVkZaIddGdkdMkwvLqnG1zgvFS8sdtpmx0QN5NVoY9W2jhVkfvWto0MabCHJlYA2fPBvViy6gQMa5vnqObFJpHL4se7BZbS/YjQDo22mIDwxEtOsz1nDWVkciw4ICZXgZ1SCos3lhVm2updG0Fbj8s+DEtCpz7gNr47MXKxi8kjLqe6OAng6qDhKAun1T6Lr4EP6rBC1CGnuUaMx4hTcxirFLFNwI0XkLvd9PXGesdXh3fOgErRWK87eLzc4KsutcUd+NCIq4ECkMVU5uQCj6z0ws3xdgwUWHPBy3Xf6As1xPt81Coh273MSxIQ6DiXHdNOFXpy0Jer762ENlX7xsfft+dtLzCE8LntOmNnH9S/c3bJ4yqYonCYZMLV2IZ+YoJrO5UeDACPA+f2lQMl2Vz8b1GX/cj4L3kAEpxyoc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To support multiple users referencing the same fragment, 'pp_frag_count' is renamed to 'pp_ref_count', transitioning pp pages from fragment management to reference count management after draining based on the suggestion from [1]. The idea is that the concept of fragmenting exists before the page is drained, and all related functions retain their current names. However, once the page is drained, its management shifts to being governed by 'pp_ref_count'. Therefore, all functions associated with that lifecycle stage of a pp page are renamed. [1] http://lore.kernel.org/netdev/f71d9448-70c8-8793-dc9a-0eb48a570300@huawei.com Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin --- .../net/ethernet/mellanox/mlx5/core/en_rx.c | 4 +- include/linux/mm_types.h | 2 +- include/net/page_pool/helpers.h | 45 ++++++++++--------- include/net/page_pool/types.h | 6 +-- net/core/page_pool.c | 12 ++--- 5 files changed, 37 insertions(+), 32 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 8d9743a5e42c..98d33ac7ec64 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -298,8 +298,8 @@ static void mlx5e_page_release_fragmented(struct mlx5e_rq *rq, u16 drain_count = MLX5E_PAGECNT_BIAS_MAX - frag_page->frags; struct page *page = frag_page->page; - if (page_pool_defrag_page(page, drain_count) == 0) - page_pool_put_defragged_page(rq->page_pool, page, -1, true); + if (page_pool_unref_page(page, drain_count) == 0) + page_pool_put_unrefed_page(rq->page_pool, page, -1, true); } static inline int mlx5e_get_rx_frag(struct mlx5e_rq *rq, diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 957ce38768b2..64e4572ef06d 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -125,7 +125,7 @@ struct page { struct page_pool *pp; unsigned long _pp_mapping_pad; unsigned long dma_addr; - atomic_long_t pp_frag_count; + atomic_long_t pp_ref_count; }; struct { /* Tail pages of compound page */ unsigned long compound_head; /* Bit zero is set */ diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 4ebd544ae977..9dc8eaf8a959 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -29,7 +29,7 @@ * page allocated from page pool. Page splitting enables memory saving and thus * avoids TLB/cache miss for data access, but there also is some cost to * implement page splitting, mainly some cache line dirtying/bouncing for - * 'struct page' and atomic operation for page->pp_frag_count. + * 'struct page' and atomic operation for page->pp_ref_count. * * The API keeps track of in-flight pages, in order to let API users know when * it is safe to free a page_pool object, the API users must call @@ -214,69 +214,74 @@ inline enum dma_data_direction page_pool_get_dma_dir(struct page_pool *pool) return pool->p.dma_dir; } -/* pp_frag_count represents the number of writers who can update the page +/* pp_ref_count represents the number of writers who can update the page * either by updating skb->data or via DMA mappings for the device. * We can't rely on the page refcnt for that as we don't know who might be * holding page references and we can't reliably destroy or sync DMA mappings * of the fragments. * - * When pp_frag_count reaches 0 we can either recycle the page if the page + * pp_ref_count initially corresponds to the number of fragments. However, + * when multiple users start to reference a single fragment, for example in + * skb_try_coalesce, the pp_ref_count will become greater than the number of + * fragments. + * + * When pp_ref_count reaches 0 we can either recycle the page if the page * refcnt is 1 or return it back to the memory allocator and destroy any * mappings we have. */ static inline void page_pool_fragment_page(struct page *page, long nr) { - atomic_long_set(&page->pp_frag_count, nr); + atomic_long_set(&page->pp_ref_count, nr); } -static inline long page_pool_defrag_page(struct page *page, long nr) +static inline long page_pool_unref_page(struct page *page, long nr) { long ret; - /* If nr == pp_frag_count then we have cleared all remaining + /* If nr == pp_ref_count then we have cleared all remaining * references to the page: * 1. 'n == 1': no need to actually overwrite it. * 2. 'n != 1': overwrite it with one, which is the rare case - * for pp_frag_count draining. + * for pp_ref_count draining. * * The main advantage to doing this is that not only we avoid a atomic * update, as an atomic_read is generally a much cheaper operation than * an atomic update, especially when dealing with a page that may be - * partitioned into only 2 or 3 pieces; but also unify the pp_frag_count + * referenced by only 2 or 3 users; but also unify the pp_ref_count * handling by ensuring all pages have partitioned into only 1 piece * initially, and only overwrite it when the page is partitioned into * more than one piece. */ - if (atomic_long_read(&page->pp_frag_count) == nr) { + if (atomic_long_read(&page->pp_ref_count) == nr) { /* As we have ensured nr is always one for constant case using * the BUILD_BUG_ON(), only need to handle the non-constant case - * here for pp_frag_count draining, which is a rare case. + * here for pp_ref_count draining, which is a rare case. */ BUILD_BUG_ON(__builtin_constant_p(nr) && nr != 1); if (!__builtin_constant_p(nr)) - atomic_long_set(&page->pp_frag_count, 1); + atomic_long_set(&page->pp_ref_count, 1); return 0; } - ret = atomic_long_sub_return(nr, &page->pp_frag_count); + ret = atomic_long_sub_return(nr, &page->pp_ref_count); WARN_ON(ret < 0); - /* We are the last user here too, reset pp_frag_count back to 1 to + /* We are the last user here too, reset pp_ref_count back to 1 to * ensure all pages have been partitioned into 1 piece initially, * this should be the rare case when the last two fragment users call - * page_pool_defrag_page() currently. + * page_pool_unref_page() currently. */ if (unlikely(!ret)) - atomic_long_set(&page->pp_frag_count, 1); + atomic_long_set(&page->pp_ref_count, 1); return ret; } -static inline bool page_pool_is_last_frag(struct page *page) +static inline bool page_pool_is_last_ref(struct page *page) { - /* If page_pool_defrag_page() returns 0, we were the last user */ - return page_pool_defrag_page(page, 1) == 0; + /* If page_pool_unref_page() returns 0, we were the last user */ + return page_pool_unref_page(page, 1) == 0; } /** @@ -301,10 +306,10 @@ static inline void page_pool_put_page(struct page_pool *pool, * allow registering MEM_TYPE_PAGE_POOL, but shield linker. */ #ifdef CONFIG_PAGE_POOL - if (!page_pool_is_last_frag(page)) + if (!page_pool_is_last_ref(page)) return; - page_pool_put_defragged_page(pool, page, dma_sync_size, allow_direct); + page_pool_put_unrefed_page(pool, page, dma_sync_size, allow_direct); #endif } diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index e1bb92c192de..6a5323619f6e 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -224,9 +224,9 @@ static inline void page_pool_put_page_bulk(struct page_pool *pool, void **data, } #endif -void page_pool_put_defragged_page(struct page_pool *pool, struct page *page, - unsigned int dma_sync_size, - bool allow_direct); +void page_pool_put_unrefed_page(struct page_pool *pool, struct page *page, + unsigned int dma_sync_size, + bool allow_direct); static inline bool is_page_pool_compiled_in(void) { diff --git a/net/core/page_pool.c b/net/core/page_pool.c index df2a06d7da52..106220b1f89c 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -650,8 +650,8 @@ __page_pool_put_page(struct page_pool *pool, struct page *page, return NULL; } -void page_pool_put_defragged_page(struct page_pool *pool, struct page *page, - unsigned int dma_sync_size, bool allow_direct) +void page_pool_put_unrefed_page(struct page_pool *pool, struct page *page, + unsigned int dma_sync_size, bool allow_direct) { page = __page_pool_put_page(pool, page, dma_sync_size, allow_direct); if (page && !page_pool_recycle_in_ring(pool, page)) { @@ -660,7 +660,7 @@ void page_pool_put_defragged_page(struct page_pool *pool, struct page *page, page_pool_return_page(pool, page); } } -EXPORT_SYMBOL(page_pool_put_defragged_page); +EXPORT_SYMBOL(page_pool_put_unrefed_page); /** * page_pool_put_page_bulk() - release references on multiple pages @@ -687,7 +687,7 @@ void page_pool_put_page_bulk(struct page_pool *pool, void **data, struct page *page = virt_to_head_page(data[i]); /* It is not the last user for the page frag case */ - if (!page_pool_is_last_frag(page)) + if (!page_pool_is_last_ref(page)) continue; page = __page_pool_put_page(pool, page, -1, false); @@ -729,7 +729,7 @@ static struct page *page_pool_drain_frag(struct page_pool *pool, long drain_count = BIAS_MAX - pool->frag_users; /* Some user is still using the page frag */ - if (likely(page_pool_defrag_page(page, drain_count))) + if (likely(page_pool_unref_page(page, drain_count))) return NULL; if (page_ref_count(page) == 1 && !page_is_pfmemalloc(page)) { @@ -750,7 +750,7 @@ static void page_pool_free_frag(struct page_pool *pool) pool->frag_page = NULL; - if (!page || page_pool_defrag_page(page, drain_count)) + if (!page || page_pool_unref_page(page, drain_count)) return; page_pool_return_page(pool, page); From patchwork Wed Dec 6 10:54:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13481486 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E06BC4167B for ; Wed, 6 Dec 2023 10:54:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A71836B00AD; Wed, 6 Dec 2023 05:54:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F94E6B00AE; Wed, 6 Dec 2023 05:54:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8501E6B00AF; Wed, 6 Dec 2023 05:54:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6C3F06B00AD for ; Wed, 6 Dec 2023 05:54:45 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4DE80A018A for ; Wed, 6 Dec 2023 10:54:45 +0000 (UTC) X-FDA: 81536085330.25.246DD59 Received: from mail-ot1-f50.google.com (mail-ot1-f50.google.com [209.85.210.50]) by imf05.hostedemail.com (Postfix) with ESMTP id 856F6100012 for ; Wed, 6 Dec 2023 10:54:43 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EH6dBEIr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.210.50 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701860083; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J730iItaW8fpD9bcYiCc3WSDtwgBtsyA2Jb0R83edv4=; b=QV1A9Q0YPaMyZgNZ1UZVSnLwMxnzKUKHiYpNX0ROA4nr3ho+3xwKN3krDsa5coggzgJv6W ECuSJxxSD+ME4rNRZPxBr+CobYCEUahZ5gOgBUL0iWAO/gPJGhpADAGdrGcvfHlA0CvE38 rlJa/9IYVQHm0YGpukZigg5vMjbG6vI= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EH6dBEIr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.210.50 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701860083; a=rsa-sha256; cv=none; b=YktQuCjb5vX2yf6YYbqO1eSmPaykMXyPVK3H5rWbgwbtHpXiTk1//2iWoy6RTn1NyCmoA9 X7vZZVf9EQ8HTYSrzGTaIbCSkoT8AJgxU9yTdO04Q5u998V72eXlExHjo0wF3+hXCE6NW0 O6kP4sNDabRkoq8TtR1UJx3IeAGMFL4= Received: by mail-ot1-f50.google.com with SMTP id 46e09a7af769-6d87a8228e0so1941483a34.1 for ; Wed, 06 Dec 2023 02:54:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701860082; x=1702464882; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J730iItaW8fpD9bcYiCc3WSDtwgBtsyA2Jb0R83edv4=; b=EH6dBEIrwvvS0luE6UvbzRbQEsWWsRUF961CUc0FOEUR/XJhBb9Ky4cvToFn0xQTnJ n7s55fi2SGMkETSwww7EtWeWPa7vZmvhcnqy5gO/qlQlZwwpC4DO15p1kvjKTdKo6uEb 8/L0QmNftdP1Pc/bxMbOZN1LCZSeHC/j9QooEV6YdKdIW1F4nSKz52H32JT6Hjr/JExZ PZbI39ZxmKaYBAiDGNKnFMf04mPDAKjPVMgKw1+T2l1xdTxTfX4ypQcr6gc2vdR6zp2k b5Rs430j21fVKOk1vqnJgU/whatenOgPxv3OFYUpNmpsjyr0O/AkE8YPeYdaxrZdaqEk GbGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701860083; x=1702464883; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J730iItaW8fpD9bcYiCc3WSDtwgBtsyA2Jb0R83edv4=; b=eXEECFfjLxdltkUKndp6KueLNlaxwzzCayHutVG5elznP7aCKsvEXW8fctpvrZpkOI h0SZHj2dM1sd8lx9xND5jblSOO4ipoechjjBD6FxIUKrAMgpDviBxx9ni4A6Sek89LD+ thZWx3sUyh80Wno7Rvelu3JXpdPwR3hbIl/hI61kQISNuLswHKRvoGanxBpPX7omg8CD PUWUYJ7VRdH6ZdYJkkxunr/Gh2g34D9JttcCqbmqGOVuu4mbBodR9+BTcR2PUGUINm2V KEgZealaAwhCgEWY/oL+NbXFerVVHQNG/1z206YAThv5VKP3MpMBcHns/okhWTRWb4GL jtjg== X-Gm-Message-State: AOJu0Yz9Uw27LOKKQcfyUxgX5yaT+Z+bL3rIvCqn43b/2uxHUpz6iVay usmBMiL3SOp4nVMuDd2Dwwo= X-Google-Smtp-Source: AGHT+IEVWGAs6VmJ6gEEtOq16rEzedYFdHy4Uqdvgzphsz0NsAg1Zhi+SY0iNk9q2PpSNior2r90BQ== X-Received: by 2002:a05:6830:1117:b0:6d6:4ebf:1557 with SMTP id w23-20020a056830111700b006d64ebf1557mr580766otq.21.1701860082740; Wed, 06 Dec 2023 02:54:42 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id n15-20020a638f0f000000b005c6801efa0fsm5388796pgd.28.2023.12.06.02.54.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Dec 2023 02:54:41 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, jasowang@redhat.com, liangchen.linux@gmail.com Subject: [PATCH net-next v7 2/4] page_pool: halve BIAS_MAX for multiple user references of a fragment Date: Wed, 6 Dec 2023 18:54:17 +0800 Message-Id: <20231206105419.27952-3-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231206105419.27952-1-liangchen.linux@gmail.com> References: <20231206105419.27952-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 6nuii1p7u7hazuthuffbsj3775fepp1h X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 856F6100012 X-HE-Tag: 1701860083-680418 X-HE-Meta: U2FsdGVkX1/lZhXAQbDxOsuCC4TTGxdsI05ftyGE6I1qGN9vRSJlTVzSQTqnp+79EHGCR/agpoyYJG08lJY7e9jdXAURRC8LZ9wYS7e5dqnnhTTOSKsh/T9vUodVBrTENGqZEiEJ+1LE1tpZ7ZxFKinI7vvic51qx0gPoX817sD4rIG8xcaMNb2rqfAWZEEaLADx6vq9vuyvGqUQhA6aq64PB4lNrZeIioVoZCYGjXGmM2vTUP2lLhtzt80+mZ9sAa6nii+MsHje0fVfS2m4sRompXep43P5AkERRCj5LfGFscuah5WFSZ5Ofpy5DWPNo1KhINCjRyFAVKsCTptmbpHR41leEgYDQrjTgHSLK823ePtJLRu6fBQlQrcR8vfzagUknqVSZilVYUkjUMVF+XPXK+p71qkzCKxNI7f5CL54qRNQLKWlYjhQB8EKbOk9VeVCNQlk8TEbEUl8zbmTJoZHwTFYZ4UkDotE4wu/WMbj357kntaNakeqJbOM3KKY5DFIJ9VmUydFrrGE3z3exgukshU6KIZIkC/GnJk++ObSMW6zjvbwYW+PC4ndSBOKQ9ZErCuW9r9250c9q3xHOeuNTlCpsEERfBpVOBnagguj9YTWXuWRZe4/UiA4CtO/Otuz519YnvwJPB/3Z0gcPGOT0wmyYWptafUL77C/F2xqfGIvrtoqkincpS15wIJ9e71xRma1rbtv3LiaWR54fplrLsoIia5ivguHipBoueVEiUOlRF4G52JxXq4k/DlzxzNdcEpImVcmP5FgB5YPLLT/enSn40bbjIjzBB/kmrl4B/HFXyAA8WeM1Dhk6lgcyv/tgtUgU8NayKrY6JROl1+kUWGB2cNKNx33ZBYNitWygeJdf4oPnjyp75kH+2Hr+wkBDOexVGwKeRzlSzv3lHLqgkjRPiminT7uyoGZ11IMY48BDiXbjdwu2IpB9mafmQE9wDuAbOqjGD6hNDu 9HdXgQIH KpQQUQTw+8dwPv0CWPFtmQKU19fSXi7HjZQzPX7QkCxWHOGM/9pTW59GvOHGw7gI2WKrFTL9+BpnB1rnXWpwe+8hqm9UL+LMzMeRbYzISwpuqI9SNZ/wygaQMtm64rwFzR1S9+m3jof7MXSfW1avFK/YAP0h55laQrvrt34hjU0WOotHXC9Xj7yC226PxzJktl2Q9Wl5RgAi1iS622KuCELVV9+vlycJhNvz3U+AZmnMfPvREqgLRHceU1b9t9WrkZrcxnbtwLUrVsENJtxxYb3358AfGjSSGlP1kkq4+mu9FPf5PMiRgCvzX2Zn4bBp9KsC5nIbONqcY4y5m82QCpVYKR/0p7UgX49pD96LtuXQsfKqLztioqDO1oeKWgvS8xQHFOqEmP1Xg+CddL5lW1TKkaWsD2MWpQoQcSnQeM+KanJ8pjpXZNf+OW475bcAcnqNRXJzJqCjFdDrAPwA+Ml/mIOQZOX/vV2KLyiO5E78nBpWMlGT+0OTovsDsV6vj5rsfNB7yy3PpakwxEgmscI6y+gitZCTM6R1nsPGAkhJwSEmqJMBy7JeY4ecWXfz7QbUCcGBGmMBfJ9Cl8yGKLaApFYfVdf+hKfQKcuU9nyMqksfj8P7dIK605vUVp0jpsKaFhzzPhcRzOL9eqdKNZlfnFD2xOtrChqB23ZLToI09DTY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.022403, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Referring to patch [1], in order to support multiple users referencing the same fragment and prevent overflow from pp_ref_count growing, the initial value of pp_ref_count is halved, leaving room for pp_ref_count to increment before the page is drained. [1] https://lore.kernel.org/all/20211009093724.10539-3-linyunsheng@huawei.com/ Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin --- net/core/page_pool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 106220b1f89c..436f7ffea7b4 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -26,7 +26,7 @@ #define DEFER_TIME (msecs_to_jiffies(1000)) #define DEFER_WARN_INTERVAL (60 * HZ) -#define BIAS_MAX LONG_MAX +#define BIAS_MAX (LONG_MAX >> 1) #ifdef CONFIG_PAGE_POOL_STATS /* alloc_stat_inc is intended to be used in softirq context */ From patchwork Wed Dec 6 10:54:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13481487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDAD6C4167B for ; Wed, 6 Dec 2023 10:54:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56C506B00AF; Wed, 6 Dec 2023 05:54:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F4E86B00B0; Wed, 6 Dec 2023 05:54:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36B016B00B1; Wed, 6 Dec 2023 05:54:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1E2896B00AF for ; Wed, 6 Dec 2023 05:54:51 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id ECB4D4018E for ; Wed, 6 Dec 2023 10:54:50 +0000 (UTC) X-FDA: 81536085540.19.ED12003 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by imf28.hostedemail.com (Postfix) with ESMTP id 338A3C0019 for ; Wed, 6 Dec 2023 10:54:49 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nGoV81Ah; spf=pass (imf28.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701860089; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G2d8n0WXue9ygrbxloUCx8WAoE9HhkQ6PXxWrjAEYxU=; b=L/QQzD8qTeU5vFFsrUN4bGfC66r7llyWRuIw4bbZCjSNQr4J8Qt8mhUglJSEAZIGx698mr GexmDZkndZTvjR+w0nAaGmKhO9qYveRaFBc2VgthgEwmT9iZc5m3hZ69Q1NVcK7PqD/2i9 uBidjdNLKeNACi9a44ryMhuKw4Qasho= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701860089; a=rsa-sha256; cv=none; b=GAosbXSyf/um+Y5NK7bghExz4f9+ZGyprHoAUiwFG4ZaKwy+1IjdPds4VTrRlP59pHbeFf XqxB0ycryuMKD10s6eJ8Lu0QRzrvAVTektieZbgZkjUjYM43YK3F6ogdbYx1SViJppzlOY 7U7ekgJUMQR0vamYDbkgiqPLGrEKT0o= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nGoV81Ah; spf=pass (imf28.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.215.176 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-f176.google.com with SMTP id 41be03b00d2f7-5c66988c2eeso545157a12.1 for ; Wed, 06 Dec 2023 02:54:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701860088; x=1702464888; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G2d8n0WXue9ygrbxloUCx8WAoE9HhkQ6PXxWrjAEYxU=; b=nGoV81AhKlnXKoojk6DjkUCJPNaZfB/5WANaTJ8xBXEZAY3akE2yBc9xmXZjMKyd/H E5Q4I47Qqe9GTRHPEthSpN5USXYHN0HID42d5fH1Uh9COhpXU7M28RW13QOVPKmBnanh 7Di5evQTcywzZm15s9Zj1kYWc0QbuzC/+2BSvRDtRF0QvM3YXJUsUZFJ5TSzPG3UsK65 HAAuZcy/F8hf+qrK89ul9jB9rP5Dh7N7B4ipSqSkA+HaF22yoRUeNcZ/+UFAe+U5GTD+ IPVS+nxeeUJxTFenmNThnHIrRo4CAtFC43dyUBm4dejxAugYLtN+vXl2jGPgLePqy3o9 CHbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701860088; x=1702464888; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G2d8n0WXue9ygrbxloUCx8WAoE9HhkQ6PXxWrjAEYxU=; b=XcwmjnMsIzPYSBxpM8q5tSZDcCsMmMh6sPjJtjWLeiB4AqDUUMyLW1j+oV7kahRg90 TZybIma89HAcDwrXDWKAfUCNp3VOCtaQDOXN9y7k28o3rO306Ltt/q5TbeHhvNe6o3Rq F4YT8k4YlzUTDIN/uIeEby07jR8KRRDFWXXZ9CUWJeyZARl3eSFvhXta03lHXdH0gmz5 dGsQ3Jhcod/9oJ6SfQv1uQJRo8z9jwWdxLNmDK2+DFSXy/+NDQ9u0jPySfuzpLlcHdft 9ol+hORzH8DqlQQR8ssqQ5BniURFNV3k1cIPwBe3iM1QoAryHgK5o1v+nued3QKjt9sk jcNw== X-Gm-Message-State: AOJu0Yw32+2YyKgNfkEz6mX88Hdv5mbUj82n6VZUSH6BKOz+YmgEcAxQ UhOrU2Y0YhE95X+XtYd/gE4= X-Google-Smtp-Source: AGHT+IEd77h94hhs6J/CMTnJCe+7Ml/LiOxXG4jzQ3aQZou8zSYUeYniVYIbULJrzgmovOv3R+QYrA== X-Received: by 2002:a05:6a20:54a0:b0:18b:1a31:ee6a with SMTP id i32-20020a056a2054a000b0018b1a31ee6amr4104734pzk.23.1701860087999; Wed, 06 Dec 2023 02:54:47 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id n15-20020a638f0f000000b005c6801efa0fsm5388796pgd.28.2023.12.06.02.54.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Dec 2023 02:54:47 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, jasowang@redhat.com, liangchen.linux@gmail.com Subject: [PATCH net-next v7 3/4] skbuff: Add a function to check if a page belongs to page_pool Date: Wed, 6 Dec 2023 18:54:18 +0800 Message-Id: <20231206105419.27952-4-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231206105419.27952-1-liangchen.linux@gmail.com> References: <20231206105419.27952-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 338A3C0019 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: jq94hpyd3wsuwrenmwonzwhydbasyep8 X-HE-Tag: 1701860089-255710 X-HE-Meta: U2FsdGVkX18UyyXHSQibCzr0POwqH90rZHo6f/0AV5A+2CiIKhmPwE72q0NQN8v93w1KpWWTag5IsvQFj84gHUn1i37zb4GOn1qGwQEYUyxQU10O+iYYpHMK63Z4uONhzbuAwFJuHiWxIpAWlXgWD+Suv3ubt9zOisKjptDCh6C10zPAvklGLwmUZwDAgsR5dRE/WTLe36D0fumnB2YY13KePjsTMG0BVqzrBtV10zicGckEBKqalzQJCSFz0pwR+4V/HMcX3yx3u2qp/2tZYAeXaMBy7SI6ARYzqGkKfDcMOELF9RS9SB0vHQs1BrQpcWGtXXTui79pSUrxD4GXH2MsGChFC5AihaXGT7S8d0+UvLfdjmBa2F2L8AicbXPT0g3lxhxGba4iDw3zwge8z489UaQa85DfaQzLIQ9AQ0W0laiSZJDhVjD4dKHqfMOwGBkDJ5m1Bn+w6cvIN4jL6xgjJwpC5S8EYa2HRGHtvUb/FB1btm2a6ymJmyV7UjNPnYXkStK6CI/GfLnQfDbSaq3tPGTMAQ6BKBYn6JSw4diR2yHmt4YeoVu9e5IJh42Th7smPFonzG/fhumI5/w31wdTiXaWvF4P8ONhF7CfqD/rxXA76lX9CfUMb6bMGSO1q2zZEE4TRGkQU79VaBMapYc8nNQVN7BKmgherUXmm6/oCpb1ILcWcZwVoBS+XBb/oo+QMO5vIbHAnar7AgRCapkr3/YZoAdRO8+0Nu4nZe8WN6Ldv2jS4kIIy82cTRi9KTBku7jXkijbOaJ/UbSHuybwo7JcmwySqjrEAj3vjrmofYPFn3VE4ZYw5+yWXXyd8xHb7/oUkDcQjRGPnADN22cFfjYTfJzYMNhg999PZapGHIXUIYxbedZKTexlKczU124qAAnNjt7bnQZ26yxG+JbF0bq9JlItaDX92W2X+2rhrLTl7+LrPnPscmhsUeduLfZDVRi2wlA6JQAZ623 /IlmIbNV 6FTrWpoDkppQyYdCQcbbvsxhlcQvnY3Lr/3KZ9Lfueka5yoz0YPof6Ewi9GOnbcnZDc3VElphsw4dmZzRdQjsxdemRQwBF7b/wn2NXtFeAGVfXCb6mtB++Bb1g/Fder6sbGRp0LCtzN8IfpUfnv6w+t9nRb1AF2nKoV0r78sLXNEu8gVTA6f+xHtO1acC992kR9exioGsrVTYxY0Lw7FMp5JBhBPqBpXeeFNbfUIpvn5rCOf8EFx3YAbtqpEEnKVKkmF/QyD6nt0rNM70ILHDlrmv2be7yOHL9vxHSH5XQKWvmR6DsArFYtyTbVq1TDL9LUmz3avX/Zp43HCdyu+rkqm/AmAzhpkg7+2mo67Q1xLiwrs3we6MF+sH+LDezkitvduXFdV2fGdaJjszXTferfaig7wYASPz4awW2BuQScRakiaYX07SD31yABE+QUTGpAeutYhrWAM15BkY3Rn6OmRxjfXFlrf4rMxMI/AnFKSp5RpEdHrv+AlIhctFS4mRyK+PP29//8YmKE3t1obcSDj/BB0jlcRb9JRPXOyB+YAdFXm/nLRhZrHX/TaIuudJvYQUPQT9sjFRBKQzs5pqYRpOLK3ID1MGYJ106aj7blCG/gPx2W8XjbsIsyXKq30vcp0qW6v4aiPcKUHhjeYxds+9NNdmx7+cW19daFEJJDgK/4crP1ymBVozBw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Wrap code for checking if a page is a page_pool page into a function for better readability and ease of reuse. Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin --- net/core/skbuff.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index b157efea5dea..7e26b56cda38 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -890,6 +890,11 @@ static void skb_clone_fraglist(struct sk_buff *skb) skb_get(list); } +static bool is_pp_page(struct page *page) +{ + return (page->pp_magic & ~0x3UL) == PP_SIGNATURE; +} + #if IS_ENABLED(CONFIG_PAGE_POOL) bool napi_pp_put_page(struct page *page, bool napi_safe) { @@ -905,7 +910,7 @@ bool napi_pp_put_page(struct page *page, bool napi_safe) * and page_is_pfmemalloc() is checked in __page_pool_put_page() * to avoid recycling the pfmemalloc page. */ - if (unlikely((page->pp_magic & ~0x3UL) != PP_SIGNATURE)) + if (unlikely(!is_pp_page(page))) return false; pp = page->pp; From patchwork Wed Dec 6 10:54:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13481488 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95407C4167B for ; Wed, 6 Dec 2023 10:54:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2821B6B00B1; Wed, 6 Dec 2023 05:54:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 20B546B00B2; Wed, 6 Dec 2023 05:54:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 067866B00B3; Wed, 6 Dec 2023 05:54:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E23CE6B00B1 for ; Wed, 6 Dec 2023 05:54:56 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B8F721C0068 for ; Wed, 6 Dec 2023 10:54:56 +0000 (UTC) X-FDA: 81536085792.09.FEABB67 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf01.hostedemail.com (Postfix) with ESMTP id DEB8F40017 for ; Wed, 6 Dec 2023 10:54:54 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FqjLkrbi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701860094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XJpeq0esGgvSQwPlWYu5mSKUKlrzr2fXLQ04GZyOjBI=; b=sucFJjArjLD2F9u6ekvgR4eOyFVtFgAVfjFDKxJNOPibXJbBhWmq+NjgSZQ7qrc2D7yE8Q mREek0veUznDRoMOaRNPcrX5KpN2eBne0Pmudp6CosJe55/i5vdFFL9UB60qA38tg/A+cC i13J0+arsH0TN9d4fqEeXkp9bquFMVs= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FqjLkrbi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701860094; a=rsa-sha256; cv=none; b=CcsryjRbXd7oQUkqyhCNLrh60JIXKPR1FVdZIiYHBn+mxWbUbLFcxfJwMg5Ow7d34UH3OK 0uNGzllWhrZLpCmZnPu5oxc+3LyJowY4yAbKKHNgb58b3kDKxdzbO3jitIYQLM0zTzHyrV py9btMeAhRiVheIb7qvjLWuxzh9F5Jo= Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-6ce4d4c5ea2so2576545b3a.0 for ; Wed, 06 Dec 2023 02:54:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701860094; x=1702464894; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XJpeq0esGgvSQwPlWYu5mSKUKlrzr2fXLQ04GZyOjBI=; b=FqjLkrbi11yHfReZxuhTbkoLIoua6v6FOq+ds4od0pwMUotSwr8AVdqb5BN2E4jmyl pYA0hKp+u45Xl1tlR5HaHttq/apjv82AF00Zb1fiZdBRKCuNPpzzUomqlvqY6VZGnwS3 cLQlFtPAvKQBnUW9/KJVy88HnytvjaEtdSkZXLUR0cZAhamspKp9QIveawA+A0BPWz5Q PimSURrmUGXw8pA2AsnCyWYx9AOxpOUUikL8D+y3a8GL09COYoKjxyy9/JEPUWzM814y 2UStG1hjmofoacLMDHgEkaQ65EpzZdKkrxhDcrRG6rdWjNpUtsaJ1+7cHkEtBQ/x1Isv pmdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701860094; x=1702464894; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XJpeq0esGgvSQwPlWYu5mSKUKlrzr2fXLQ04GZyOjBI=; b=uN0P9yvSoiD5yAR0U2u75x+NMEtfOPngUVvbyMQydv3IoPORj/yDUOzh5PvoTe2enl 6e9Pk6Cr7K3n54aQKoZ8vG/mvzmMJA1waqE0g9krBWCNDs8kZUXgvPqxwbOjQUKOrjla h7VqKpVVprXOPrib70vGPB5L4XQQURzwhesSMefWOcU+TlR2N1MoUYB2HTzGvl713tny ugCjXOuF925U37rt/mJVLQBNDjNaZDs15JIkoatSkk7rCMTE9UNHpEotyqiOxdEaFikU swPaQp5QoXJVrUNKpGwlY9xIzf9AWQVX0qxhtfpkqXms4N766IoiJgfvi2hsuGRQgZrg 1Nlw== X-Gm-Message-State: AOJu0YxWxInVvOuKsxJpGiMoYuUfjugxG/k/a78h9KvZQpzi6y+8JQY/ SgfToN55X/bpGzI8LD9LTArA94mDask= X-Google-Smtp-Source: AGHT+IGs32UGGkDmx++vF2WrxMsf6vIKFr/2F29zLgzga5cW6Le3dLh6j/00l5geSI7if5bXr/dyag== X-Received: by 2002:a05:6a00:1743:b0:6ce:6c6d:1622 with SMTP id j3-20020a056a00174300b006ce6c6d1622mr514790pfc.62.1701860093745; Wed, 06 Dec 2023 02:54:53 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id n15-20020a638f0f000000b005c6801efa0fsm5388796pgd.28.2023.12.06.02.54.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Dec 2023 02:54:52 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, jasowang@redhat.com, liangchen.linux@gmail.com Subject: [PATCH net-next v7 4/4] skbuff: Optimization of SKB coalescing for page pool Date: Wed, 6 Dec 2023 18:54:19 +0800 Message-Id: <20231206105419.27952-5-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231206105419.27952-1-liangchen.linux@gmail.com> References: <20231206105419.27952-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: DEB8F40017 X-Stat-Signature: f1a13qjas36p7hijz3f7bbzgcxh5ssa6 X-Rspam-User: X-HE-Tag: 1701860094-351736 X-HE-Meta: U2FsdGVkX19KWjz2I7vmebby3vE+3dEctQHpBl/+09yB6dFPnM/Rb71whEgDqc0MoGyZGr5lYSGxpPxSOdulcdMTLNvs0x7S3fsGgwfR3UgoUrLxRKO/DBNbGtCSKe5xwSVHCrCjGMIYbwv3dCkiQHxXKvAoon6R8NmkYBh+4h3wfuX/HPl4N7dxf78kR5fhEPD25a8dqXlgcbeIAuWNHF7qvOb6itb7vlSE5L5gAeukhzLnbv66Rd/C0XqRBST6DRLHbUJMw949DVlEHFYpTKbL8f8vo7OAxA59AMTejgIH4k3TlsLIMD3LF033NbpIxd1MWiSvzqLMTdWCaBgP0gH++CvHeQw+9RF4tLZfM3Zhrl2SxFpfJvSEBH49I1uqXhSRmg1E8SJWijy6/kDcWp09wtKDCiHbZXhYNvnujj1UUf1+2HXJ5g25KhS+r8ww5XpPtz3akctJUo4JCXj6wLeAPO0aZb6auGofihsw816pTGxFtLfNVn98SdrBtlRBCTwVV5PC72CY+djVjUGLdKQcfLGz8c9LaB44OWWlteEyLH1HZERnCoH8JzzptTR0eCJ5vYYpx63TstFPaRRTYaxFN/CBr3m29tJTjzXNx3RtvAI0QbP+CydA3gJSNI3H9vUxPebGzZB/A/qxGde9sC4QAUNRodGPvRy006yIMl1eW66U9eiMsDx7fZIlVpvX70ucsDs65zBBKp5XnCzJDYuxjIx9nOhCG0MUoVi9cf/fah4pZAX1ecePC9jZVcwkBErK5USStR2zF+C3Q6rAU5vbjCxbsipJeZU+hLEJCumz5EJWIEKBY1Um7l5V4Ch4l4o3xEyjt7p2fyUL5lE5wzdgqIVYfWRtpbJ2ZtD/qmKzkKzYe7WRSTH5WelFr5Gv1kQlonR9Fl/RWmrvteIxutiS30LRvHYjp5JZHYukkInmkg8QT2OdFIjye4Hnumwp80J4JlHRJmyWrHlHQN/ BDmVYjjc Ss79fhDR1QFCsOSwKMvgD9nzXUA2IO9oWpscTtTwRZJ5lJKb6ZZN5WLOu4cv4BIjW1+CWCg34mJtwxIEf5FdK2YeAnCtPDIVBxDX/GUGf6S5QGWLouHGMgbpXEF+bAlkJq5pdRocUtYDDjoidIIE+PdUJzbczutkE1LiZK0H6lK7STgTgmKlAE3NxrpK1JfWiOWWBq+28YmeXzwoJv37gpSrmjSSz6Qk5Z6TovP/HsQcjdSk2qc0dfEG6AH9kPvJM2GBsR6/LyZFtEI1+zqZZyrqZDcAQJIV8ax3OhjCW9W/Be2LIvq6ce19B7cOerJxxfY5w5a8PGxpoy/iifgTzrRoKijo4foEnXNvqwIbe8gMkOvsf9vMHq2S2JmBD9L3P5OxHGJzArJ4BzWqbmE7OEcrS6MX53oD0PFy3fhvNvUmMotp6LDAyhqsyJ/lvDAJ9E4wNJ7TpD6My8RVZ9SVY0DSlE7tm2kvhAKytzWqDFs0GWjR7BDkJmgVnwKafgsbQ/0z4Oq2pS5rOxw8VoE1c6c/7SA1vbuo8L3JvB042nNVpRkwHDluZ7rHUeDDOmx3RElVVR0nyiwW9EE1TkgYk3hf29Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to address the issues encountered with commit 1effe8ca4e34 ("skbuff: fix coalescing for page_pool fragment recycling"), the combination of the following condition was excluded from skb coalescing: from->pp_recycle = 1 from->cloned = 1 to->pp_recycle = 1 However, with page pool environments, the aforementioned combination can be quite common(ex. NetworkMananger may lead to the additional packet_type being registered, thus the cloning). In scenarios with a higher number of small packets, it can significantly affect the success rate of coalescing. For example, considering packets of 256 bytes size, our comparison of coalescing success rate is as follows: Without page pool: 70% With page pool: 13% Consequently, this has an impact on performance: Without page pool: 2.57 Gbits/sec With page pool: 2.26 Gbits/sec Therefore, it seems worthwhile to optimize this scenario and enable coalescing of this particular combination. To achieve this, we need to ensure the correct increment of the "from" SKB page's page pool reference count (pp_ref_count). Following this optimization, the success rate of coalescing measured in our environment has improved as follows: With page pool: 60% This success rate is approaching the rate achieved without using page pool, and the performance has also been improved: With page pool: 2.52 Gbits/sec Below is the performance comparison for small packets before and after this optimization. We observe no impact to packets larger than 4K. packet size before after improved (bytes) (Gbits/sec) (Gbits/sec) 128 1.19 1.27 7.13% 256 2.26 2.52 11.75% 512 4.13 4.81 16.50% 1024 6.17 6.73 9.05% 2048 14.54 15.47 6.45% 4096 25.44 27.87 9.52% Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin Suggested-by: Jason Wang --- include/net/page_pool/helpers.h | 5 ++++ net/core/skbuff.c | 41 +++++++++++++++++++++++---------- 2 files changed, 34 insertions(+), 12 deletions(-) diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 9dc8eaf8a959..268bc9d9ffd3 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -278,6 +278,11 @@ static inline long page_pool_unref_page(struct page *page, long nr) return ret; } +static inline void page_pool_ref_page(struct page *page) +{ + atomic_long_inc(&page->pp_ref_count); +} + static inline bool page_pool_is_last_ref(struct page *page) { /* If page_pool_unref_page() returns 0, we were the last user */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 7e26b56cda38..3c2515a29376 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -947,6 +947,24 @@ static bool skb_pp_recycle(struct sk_buff *skb, void *data, bool napi_safe) return napi_pp_put_page(virt_to_page(data), napi_safe); } +/** + * skb_pp_frag_ref() - Increase fragment reference count of a page + * @page: page of the fragment on which to increase a reference + * + * Increase fragment reference count (pp_ref_count) on a page, but if it is + * not a page pool page, fallback to increase a reference(_refcount) on a + * normal page. + */ +static void skb_pp_frag_ref(struct page *page) +{ + struct page *head_page = compound_head(page); + + if (likely(is_pp_page(head_page))) + page_pool_ref_page(head_page); + else + page_ref_inc(head_page); +} + static void skb_kfree_head(void *head, unsigned int end_offset) { if (end_offset == SKB_SMALL_HEAD_HEADROOM) @@ -5769,17 +5787,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, return false; /* In general, avoid mixing page_pool and non-page_pool allocated - * pages within the same SKB. Additionally avoid dealing with clones - * with page_pool pages, in case the SKB is using page_pool fragment - * references (page_pool_alloc_frag()). Since we only take full page - * references for cloned SKBs at the moment that would result in - * inconsistent reference counts. - * In theory we could take full references if @from is cloned and - * !@to->pp_recycle but its tricky (due to potential race with - * the clone disappearing) and rare, so not worth dealing with. + * pages within the same SKB. In theory we could take full + * references if @from is cloned and !@to->pp_recycle but its + * tricky (due to potential race with the clone disappearing) and + * rare, so not worth dealing with. */ - if (to->pp_recycle != from->pp_recycle || - (from->pp_recycle && skb_cloned(from))) + if (to->pp_recycle != from->pp_recycle) return false; if (len <= skb_tailroom(to)) { @@ -5836,8 +5849,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, /* if the skb is not cloned this does nothing * since we set nr_frags to 0. */ - for (i = 0; i < from_shinfo->nr_frags; i++) - __skb_frag_ref(&from_shinfo->frags[i]); + if (from->pp_recycle) + for (i = 0; i < from_shinfo->nr_frags; i++) + skb_pp_frag_ref(skb_frag_page(&from_shinfo->frags[i])); + else + for (i = 0; i < from_shinfo->nr_frags; i++) + __skb_frag_ref(&from_shinfo->frags[i]); to->truesize += delta; to->len += len;