| Message ID | 20231124073439.52626-4-liangchen.linux@gmail.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | skbuff: Optimize SKB coalescing for page pool | expand |
On 2023/11/24 15:34, Liang Chen wrote: ... > --- a/include/net/page_pool/helpers.h > +++ b/include/net/page_pool/helpers.h > @@ -402,4 +402,26 @@ static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid) > page_pool_update_nid(pool, new_nid); > } > > +static inline bool page_pool_is_pp_page(struct page *page) > +{ We have a page->pp_magic checking in napi_pp_put_page() in skbuff.c already, it seems better to move it to skbuff.c or skbuff.h and use it for napi_pp_put_page() too, as we seem to have chosen to demux the page_pool owned page and non-page_pool owned page handling in the skbuff core. If we move it to skbuff.c or skbuff.h, we might need a better prefix than page_pool_* too.
On Sat, Nov 25, 2023 at 8:16 PM Yunsheng Lin <linyunsheng@huawei.com> wrote: > > On 2023/11/24 15:34, Liang Chen wrote: > > ... > > > --- a/include/net/page_pool/helpers.h > > +++ b/include/net/page_pool/helpers.h > > @@ -402,4 +402,26 @@ static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid) > > page_pool_update_nid(pool, new_nid); > > } > > > > +static inline bool page_pool_is_pp_page(struct page *page) > > +{ > > We have a page->pp_magic checking in napi_pp_put_page() in skbuff.c already, > it seems better to move it to skbuff.c or skbuff.h and use it for > napi_pp_put_page() too, as we seem to have chosen to demux the page_pool > owned page and non-page_pool owned page handling in the skbuff core. > > If we move it to skbuff.c or skbuff.h, we might need a better prefix than > page_pool_* too. > How about keeping the 'page_pool_is_pp_page' function in 'helper.h' and letting 'skbbuff.c' use it? It seems like the function's logic is better suited to be internal to the page pool, and it might be needed outside of 'skbuff.c' in the future.
On 2023/11/27 12:23, Liang Chen wrote: > On Sat, Nov 25, 2023 at 8:16 PM Yunsheng Lin <linyunsheng@huawei.com> wrote: >> >> On 2023/11/24 15:34, Liang Chen wrote: >> >> ... >> >>> --- a/include/net/page_pool/helpers.h >>> +++ b/include/net/page_pool/helpers.h >>> @@ -402,4 +402,26 @@ static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid) >>> page_pool_update_nid(pool, new_nid); >>> } >>> >>> +static inline bool page_pool_is_pp_page(struct page *page) >>> +{ >> >> We have a page->pp_magic checking in napi_pp_put_page() in skbuff.c already, >> it seems better to move it to skbuff.c or skbuff.h and use it for >> napi_pp_put_page() too, as we seem to have chosen to demux the page_pool >> owned page and non-page_pool owned page handling in the skbuff core. >> >> If we move it to skbuff.c or skbuff.h, we might need a better prefix than >> page_pool_* too. >> > > How about keeping the 'page_pool_is_pp_page' function in 'helper.h' > and letting 'skbbuff.c' use it? It seems like the function's logic is > better suited to be internal to the page pool, and it might be needed > outside of 'skbuff.c' in the future. Yes, we can always extend it outside of 'skbuff' if there is a need in the future. For now, maybe it makes more sense to export as litte as possible in the .h file mirroring the napi_pp_put_page(). > . >
On Mon, Nov 27, 2023 at 6:48 PM Yunsheng Lin <linyunsheng@huawei.com> wrote: > > On 2023/11/27 12:23, Liang Chen wrote: > > On Sat, Nov 25, 2023 at 8:16 PM Yunsheng Lin <linyunsheng@huawei.com> wrote: > >> > >> On 2023/11/24 15:34, Liang Chen wrote: > >> > >> ... > >> > >>> --- a/include/net/page_pool/helpers.h > >>> +++ b/include/net/page_pool/helpers.h > >>> @@ -402,4 +402,26 @@ static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid) > >>> page_pool_update_nid(pool, new_nid); > >>> } > >>> > >>> +static inline bool page_pool_is_pp_page(struct page *page) > >>> +{ > >> > >> We have a page->pp_magic checking in napi_pp_put_page() in skbuff.c already, > >> it seems better to move it to skbuff.c or skbuff.h and use it for > >> napi_pp_put_page() too, as we seem to have chosen to demux the page_pool > >> owned page and non-page_pool owned page handling in the skbuff core. > >> > >> If we move it to skbuff.c or skbuff.h, we might need a better prefix than > >> page_pool_* too. > >> > > > > How about keeping the 'page_pool_is_pp_page' function in 'helper.h' > > and letting 'skbbuff.c' use it? It seems like the function's logic is > > better suited to be internal to the page pool, and it might be needed > > outside of 'skbuff.c' in the future. > > Yes, we can always extend it outside of 'skbuff' if there is a need in > the future. > > For now, maybe it makes more sense to export as litte as possible in the > .h file mirroring the napi_pp_put_page(). > Sure. will be done in v4. Thanks! > > . > >
diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 700f435292e7..6b3c0d4d47f5 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -402,4 +402,26 @@ static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid) page_pool_update_nid(pool, new_nid); } +static inline bool page_pool_is_pp_page(struct page *page) +{ + return (page->pp_magic & ~0x3UL) == PP_SIGNATURE; +} + +/** + * page_pool_get_frag_ref() - Increase fragment reference count of a page + * @page: page of the fragment on which to increase a reference + * + * Increase fragment reference count (pp_ref_count) on a page, but if it is + * not a page pool page, fallback to increase a reference(_refcount) on a + * normal page. + */ +static inline void page_pool_get_frag_ref(struct page *page) +{ + struct page *head_page = compound_head(page); + + if (likely(page_pool_is_pp_page(head_page))) + atomic_long_inc(&head_page->pp_ref_count); + else + get_page(head_page); +} #endif /* _NET_PAGE_POOL_HELPERS_H */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index b157efea5dea..54e6945ead56 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -5764,17 +5764,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, return false; /* In general, avoid mixing page_pool and non-page_pool allocated - * pages within the same SKB. Additionally avoid dealing with clones - * with page_pool pages, in case the SKB is using page_pool fragment - * references (page_pool_alloc_frag()). Since we only take full page - * references for cloned SKBs at the moment that would result in - * inconsistent reference counts. - * In theory we could take full references if @from is cloned and - * !@to->pp_recycle but its tricky (due to potential race with - * the clone disappearing) and rare, so not worth dealing with. + * pages within the same SKB. In theory we could take full + * references if @from is cloned and !@to->pp_recycle but its + * tricky (due to potential race with the clone disappearing) and + * rare, so not worth dealing with. */ - if (to->pp_recycle != from->pp_recycle || - (from->pp_recycle && skb_cloned(from))) + if (to->pp_recycle != from->pp_recycle) return false; if (len <= skb_tailroom(to)) { @@ -5831,8 +5826,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, /* if the skb is not cloned this does nothing * since we set nr_frags to 0. */ - for (i = 0; i < from_shinfo->nr_frags; i++) - __skb_frag_ref(&from_shinfo->frags[i]); + if (from->pp_recycle) + for (i = 0; i < from_shinfo->nr_frags; i++) + page_pool_get_frag_ref(skb_frag_page(&from_shinfo->frags[i])); + else + for (i = 0; i < from_shinfo->nr_frags; i++) + __skb_frag_ref(&from_shinfo->frags[i]); to->truesize += delta; to->len += len;
In order to address the issues encountered with commit 1effe8ca4e34 ("skbuff: fix coalescing for page_pool fragment recycling"), the combination of the following condition was excluded from skb coalescing: from->pp_recycle = 1 from->cloned = 1 to->pp_recycle = 1 However, with page pool environments, the aforementioned combination can be quite common(ex. NetworkMananger may lead to the additional packet_type being registered, thus the cloning). In scenarios with a higher number of small packets, it can significantly affect the success rate of coalescing. For example, considering packets of 256 bytes size, our comparison of coalescing success rate is as follows: Without page pool: 70% With page pool: 13% Consequently, this has an impact on performance: Without page pool: 2.57 Gbits/sec With page pool: 2.26 Gbits/sec Therefore, it seems worthwhile to optimize this scenario and enable coalescing of this particular combination. To achieve this, we need to ensure the correct increment of the "from" SKB page's page pool reference count (pp_ref_count). Following this optimization, the success rate of coalescing measured in our environment has improved as follows: With page pool: 60% This success rate is approaching the rate achieved without using page pool, and the performance has also been improved: With page pool: 2.52 Gbits/sec Below is the performance comparison for small packets before and after this optimization. We observe no impact to packets larger than 4K. packet size before after improved (bytes) (Gbits/sec) (Gbits/sec) 128 1.19 1.27 7.13% 256 2.26 2.52 11.75% 512 4.13 4.81 16.50% 1024 6.17 6.73 9.05% 2048 14.54 15.47 6.45% 4096 25.44 27.87 9.52% Signed-off-by: Liang Chen <liangchen.linux@gmail.com> --- include/net/page_pool/helpers.h | 22 ++++++++++++++++++++++ net/core/skbuff.c | 23 +++++++++++------------ 2 files changed, 33 insertions(+), 12 deletions(-)