| Message ID | 20231211035243.15774-5-liangchen.linux@gmail.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | skbuff: Optimize SKB coalescing for page pool | expand |
Hi Liang, On Mon, 11 Dec 2023 at 05:53, Liang Chen <liangchen.linux@gmail.com> wrote: > > In order to address the issues encountered with commit 1effe8ca4e34 > ("skbuff: fix coalescing for page_pool fragment recycling"), the > combination of the following condition was excluded from skb coalescing: > > from->pp_recycle = 1 > from->cloned = 1 > to->pp_recycle = 1 > > However, with page pool environments, the aforementioned combination can > be quite common(ex. NetworkMananger may lead to the additional > packet_type being registered, thus the cloning). In scenarios with a > higher number of small packets, it can significantly affect the success > rate of coalescing. For example, considering packets of 256 bytes size, > our comparison of coalescing success rate is as follows: > > Without page pool: 70% > With page pool: 13% > > Consequently, this has an impact on performance: > > Without page pool: 2.57 Gbits/sec > With page pool: 2.26 Gbits/sec > > Therefore, it seems worthwhile to optimize this scenario and enable > coalescing of this particular combination. To achieve this, we need to > ensure the correct increment of the "from" SKB page's page pool > reference count (pp_ref_count). > > Following this optimization, the success rate of coalescing measured in > our environment has improved as follows: > > With page pool: 60% > > This success rate is approaching the rate achieved without using page > pool, and the performance has also been improved: > > With page pool: 2.52 Gbits/sec > > Below is the performance comparison for small packets before and after > this optimization. We observe no impact to packets larger than 4K. > > packet size before after improved > (bytes) (Gbits/sec) (Gbits/sec) > 128 1.19 1.27 7.13% > 256 2.26 2.52 11.75% > 512 4.13 4.81 16.50% > 1024 6.17 6.73 9.05% > 2048 14.54 15.47 6.45% > 4096 25.44 27.87 9.52% > > Signed-off-by: Liang Chen <liangchen.linux@gmail.com> > Reviewed-by: Yunsheng Lin <linyunsheng@huawei.com> > Suggested-by: Jason Wang <jasowang@redhat.com> As I said in the past the patch look correct. I don't like the fact that more pp internals creep into the default network stack, but perhaps this is fine with the bigger adoption? Jakub any thoughts/objections? Thanks /Ilias > --- > include/net/page_pool/helpers.h | 5 ++++ > net/core/skbuff.c | 41 +++++++++++++++++++++++---------- > 2 files changed, 34 insertions(+), 12 deletions(-) > > diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h > index d0c5e7e6857a..0dc8fab43bef 100644 > --- a/include/net/page_pool/helpers.h > +++ b/include/net/page_pool/helpers.h > @@ -281,6 +281,11 @@ static inline long page_pool_unref_page(struct page *page, long nr) > return ret; > } > > +static inline void page_pool_ref_page(struct page *page) > +{ > + atomic_long_inc(&page->pp_ref_count); > +} > + > static inline bool page_pool_is_last_ref(struct page *page) > { > /* If page_pool_unref_page() returns 0, we were the last user */ > diff --git a/net/core/skbuff.c b/net/core/skbuff.c > index 7e26b56cda38..3c2515a29376 100644 > --- a/net/core/skbuff.c > +++ b/net/core/skbuff.c > @@ -947,6 +947,24 @@ static bool skb_pp_recycle(struct sk_buff *skb, void *data, bool napi_safe) > return napi_pp_put_page(virt_to_page(data), napi_safe); > } > > +/** > + * skb_pp_frag_ref() - Increase fragment reference count of a page > + * @page: page of the fragment on which to increase a reference > + * > + * Increase fragment reference count (pp_ref_count) on a page, but if it is > + * not a page pool page, fallback to increase a reference(_refcount) on a > + * normal page. > + */ > +static void skb_pp_frag_ref(struct page *page) > +{ > + struct page *head_page = compound_head(page); > + > + if (likely(is_pp_page(head_page))) > + page_pool_ref_page(head_page); > + else > + page_ref_inc(head_page); > +} > + > static void skb_kfree_head(void *head, unsigned int end_offset) > { > if (end_offset == SKB_SMALL_HEAD_HEADROOM) > @@ -5769,17 +5787,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, > return false; > > /* In general, avoid mixing page_pool and non-page_pool allocated > - * pages within the same SKB. Additionally avoid dealing with clones > - * with page_pool pages, in case the SKB is using page_pool fragment > - * references (page_pool_alloc_frag()). Since we only take full page > - * references for cloned SKBs at the moment that would result in > - * inconsistent reference counts. > - * In theory we could take full references if @from is cloned and > - * !@to->pp_recycle but its tricky (due to potential race with > - * the clone disappearing) and rare, so not worth dealing with. > + * pages within the same SKB. In theory we could take full > + * references if @from is cloned and !@to->pp_recycle but its > + * tricky (due to potential race with the clone disappearing) and > + * rare, so not worth dealing with. > */ > - if (to->pp_recycle != from->pp_recycle || > - (from->pp_recycle && skb_cloned(from))) > + if (to->pp_recycle != from->pp_recycle) > return false; > > if (len <= skb_tailroom(to)) { > @@ -5836,8 +5849,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, > /* if the skb is not cloned this does nothing > * since we set nr_frags to 0. > */ > - for (i = 0; i < from_shinfo->nr_frags; i++) > - __skb_frag_ref(&from_shinfo->frags[i]); > + if (from->pp_recycle) > + for (i = 0; i < from_shinfo->nr_frags; i++) > + skb_pp_frag_ref(skb_frag_page(&from_shinfo->frags[i])); > + else > + for (i = 0; i < from_shinfo->nr_frags; i++) > + __skb_frag_ref(&from_shinfo->frags[i]); > > to->truesize += delta; > to->len += len; > -- > 2.31.1 >
On Mon, 11 Dec 2023 09:46:55 +0200 Ilias Apalodimas wrote: > As I said in the past the patch look correct. I don't like the fact > that more pp internals creep into the default network stack, but > perhaps this is fine with the bigger adoption? > Jakub any thoughts/objections? Now that you asked... the helper does seem to be in sort of a in-between state of being skb specific. What worries me is that this: +/** + * skb_pp_frag_ref() - Increase fragment reference count of a page + * @page: page of the fragment on which to increase a reference + * + * Increase fragment reference count (pp_ref_count) on a page, but if it is + * not a page pool page, fallback to increase a reference(_refcount) on a + * normal page. + */ +static void skb_pp_frag_ref(struct page *page) +{ + struct page *head_page = compound_head(page); + + if (likely(is_pp_page(head_page))) + page_pool_ref_page(head_page); + else + page_ref_inc(head_page); +} doesn't even document that the caller must make sure that the skb which owns this page is marked for pp recycling. The caller added by this patch does that, but we should indicate somewhere that doing skb_pp_frag_ref() for frag in a non-pp-recycling skb is not correct. We can either lean in the direction of making it less skb specific, put the code in page_pool.c / helpers.h and make it clear that the caller has to be careful. Or we make it more skb specific, take a skb pointer as arg, and also look at its recycling marking.. or just improve the kdoc.
On Tue, Dec 12, 2023 at 4:14 AM Jakub Kicinski <kuba@kernel.org> wrote: > > On Mon, 11 Dec 2023 09:46:55 +0200 Ilias Apalodimas wrote: > > As I said in the past the patch look correct. I don't like the fact > > that more pp internals creep into the default network stack, but > > perhaps this is fine with the bigger adoption? > > Jakub any thoughts/objections? > > Now that you asked... the helper does seem to be in sort of > a in-between state of being skb specific. > > What worries me is that this: > > +/** > + * skb_pp_frag_ref() - Increase fragment reference count of a page > + * @page: page of the fragment on which to increase a reference > + * > + * Increase fragment reference count (pp_ref_count) on a page, but if it is > + * not a page pool page, fallback to increase a reference(_refcount) on a > + * normal page. > + */ > +static void skb_pp_frag_ref(struct page *page) > +{ > + struct page *head_page = compound_head(page); > + > + if (likely(is_pp_page(head_page))) > + page_pool_ref_page(head_page); > + else > + page_ref_inc(head_page); > +} > > doesn't even document that the caller must make sure that the skb > which owns this page is marked for pp recycling. The caller added > by this patch does that, but we should indicate somewhere that doing > skb_pp_frag_ref() for frag in a non-pp-recycling skb is not correct. > > We can either lean in the direction of making it less skb specific, > put the code in page_pool.c / helpers.h and make it clear that the > caller has to be careful. > Or we make it more skb specific, take a skb pointer as arg, and also > look at its recycling marking.. > or just improve the kdoc. Thank you for the suggestion! I will proceed with improving the kdoc.
Hi Jakub, On Mon, 11 Dec 2023 at 22:14, Jakub Kicinski <kuba@kernel.org> wrote: > > On Mon, 11 Dec 2023 09:46:55 +0200 Ilias Apalodimas wrote: > > As I said in the past the patch look correct. I don't like the fact > > that more pp internals creep into the default network stack, but > > perhaps this is fine with the bigger adoption? > > Jakub any thoughts/objections? > > Now that you asked... the helper does seem to be in sort of > a in-between state of being skb specific. > > What worries me is that this: > > +/** > + * skb_pp_frag_ref() - Increase fragment reference count of a page > + * @page: page of the fragment on which to increase a reference > + * > + * Increase fragment reference count (pp_ref_count) on a page, but if it is > + * not a page pool page, fallback to increase a reference(_refcount) on a > + * normal page. > + */ > +static void skb_pp_frag_ref(struct page *page) > +{ > + struct page *head_page = compound_head(page); > + > + if (likely(is_pp_page(head_page))) > + page_pool_ref_page(head_page); > + else > + page_ref_inc(head_page); > +} > > doesn't even document that the caller must make sure that the skb > which owns this page is marked for pp recycling. The caller added > by this patch does that, but we should indicate somewhere that doing > skb_pp_frag_ref() for frag in a non-pp-recycling skb is not correct. Correct > > We can either lean in the direction of making it less skb specific, > put the code in page_pool.c / helpers.h and make it clear that the > caller has to be careful. > Or we make it more skb specific, take a skb pointer as arg, and also > look at its recycling marking.. > or just improve the kdoc. I've mentioned this in the past, but I generally try to prevent people from shooting themselves in the foot when creating APIs. Unless there's a proven performance hit, I'd move the pp_recycle checking in skb_pp_frag_ref(). Thanks /Ilias /Ilias
On Wed, Dec 13, 2023 at 3:10 PM Ilias Apalodimas <ilias.apalodimas@linaro.org> wrote: > > Hi Jakub, > > On Mon, 11 Dec 2023 at 22:14, Jakub Kicinski <kuba@kernel.org> wrote: > > > > On Mon, 11 Dec 2023 09:46:55 +0200 Ilias Apalodimas wrote: > > > As I said in the past the patch look correct. I don't like the fact > > > that more pp internals creep into the default network stack, but > > > perhaps this is fine with the bigger adoption? > > > Jakub any thoughts/objections? > > > > Now that you asked... the helper does seem to be in sort of > > a in-between state of being skb specific. > > > > What worries me is that this: > > > > +/** > > + * skb_pp_frag_ref() - Increase fragment reference count of a page > > + * @page: page of the fragment on which to increase a reference > > + * > > + * Increase fragment reference count (pp_ref_count) on a page, but if it is > > + * not a page pool page, fallback to increase a reference(_refcount) on a > > + * normal page. > > + */ > > +static void skb_pp_frag_ref(struct page *page) > > +{ > > + struct page *head_page = compound_head(page); > > + > > + if (likely(is_pp_page(head_page))) > > + page_pool_ref_page(head_page); > > + else > > + page_ref_inc(head_page); > > +} > > > > doesn't even document that the caller must make sure that the skb > > which owns this page is marked for pp recycling. The caller added > > by this patch does that, but we should indicate somewhere that doing > > skb_pp_frag_ref() for frag in a non-pp-recycling skb is not correct. > > Correct > > > > > We can either lean in the direction of making it less skb specific, > > put the code in page_pool.c / helpers.h and make it clear that the > > caller has to be careful. > > Or we make it more skb specific, take a skb pointer as arg, and also > > look at its recycling marking.. > > or just improve the kdoc. > > I've mentioned this in the past, but I generally try to prevent people > from shooting themselves in the foot when creating APIs. Unless > there's a proven performance hit, I'd move the pp_recycle checking in > skb_pp_frag_ref(). > /** * skb_pp_frag_ref() - Increase fragment references of a page pool aware skb * @skb: page pool aware skb * * Increase the fragment reference count (pp_ref_count) of a skb. This is * intended to gain fragment references only for page pool aware skbs, * i.e. when skb->pp_recycle is true, and not for fragments in a * non-pp-recycling skb. It has a fallback to increase references on normal * pages, as page pool aware skbs may also have normal page fragments. */ Sure. Below is a snippet of the implementation for skb_pp_frag_ref, which takes an skb as its argument. The loop that iterates through the frags has been moved inside the function to avoid checking skb->pp_recycle each time a frag reference is taken(though there would be some optimization from the compiler). If there is no objection, it will be included in v10. Thanks! static int skb_pp_frag_ref(struct sk_buff *skb) { struct skb_shared_info *shinfo; struct page *head_page; int i; if (!skb->pp_recycle) return -EINVAL; shinfo = skb_shinfo(skb); for (i = 0; i < shinfo->nr_frags; i++){ head_page = compound_head(skb_frag_page(&shinfo->frags[i])); if (likely(is_pp_page(head_page))) page_pool_ref_page(head_page); else page_ref_inc(head_page); } return 0; } /* if the skb is not cloned this does nothing * since we set nr_frags to 0. */ if (skb_pp_frag_ref(from)) { for (i = 0; i < from_shinfo->nr_frags; i++) __skb_frag_ref(&from_shinfo->frags[i]); } > Thanks > /Ilias > > /Ilias
On Thu, 14 Dec 2023 10:26:47 +0800 Liang Chen wrote:
> If there is no objection, it will be included in v10.
If I manage to reach you before you post - please hold off for another
30min with posting, I'm going to apply patch 1.
On Thu, Dec 14, 2023 at 10:34 AM Jakub Kicinski <kuba@kernel.org> wrote: > > On Thu, 14 Dec 2023 10:26:47 +0800 Liang Chen wrote: > > If there is no objection, it will be included in v10. > > If I manage to reach you before you post - please hold off for another > 30min with posting, I'm going to apply patch 1. Sure. Thank you!
diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index d0c5e7e6857a..0dc8fab43bef 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -281,6 +281,11 @@ static inline long page_pool_unref_page(struct page *page, long nr) return ret; } +static inline void page_pool_ref_page(struct page *page) +{ + atomic_long_inc(&page->pp_ref_count); +} + static inline bool page_pool_is_last_ref(struct page *page) { /* If page_pool_unref_page() returns 0, we were the last user */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 7e26b56cda38..3c2515a29376 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -947,6 +947,24 @@ static bool skb_pp_recycle(struct sk_buff *skb, void *data, bool napi_safe) return napi_pp_put_page(virt_to_page(data), napi_safe); } +/** + * skb_pp_frag_ref() - Increase fragment reference count of a page + * @page: page of the fragment on which to increase a reference + * + * Increase fragment reference count (pp_ref_count) on a page, but if it is + * not a page pool page, fallback to increase a reference(_refcount) on a + * normal page. + */ +static void skb_pp_frag_ref(struct page *page) +{ + struct page *head_page = compound_head(page); + + if (likely(is_pp_page(head_page))) + page_pool_ref_page(head_page); + else + page_ref_inc(head_page); +} + static void skb_kfree_head(void *head, unsigned int end_offset) { if (end_offset == SKB_SMALL_HEAD_HEADROOM) @@ -5769,17 +5787,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, return false; /* In general, avoid mixing page_pool and non-page_pool allocated - * pages within the same SKB. Additionally avoid dealing with clones - * with page_pool pages, in case the SKB is using page_pool fragment - * references (page_pool_alloc_frag()). Since we only take full page - * references for cloned SKBs at the moment that would result in - * inconsistent reference counts. - * In theory we could take full references if @from is cloned and - * !@to->pp_recycle but its tricky (due to potential race with - * the clone disappearing) and rare, so not worth dealing with. + * pages within the same SKB. In theory we could take full + * references if @from is cloned and !@to->pp_recycle but its + * tricky (due to potential race with the clone disappearing) and + * rare, so not worth dealing with. */ - if (to->pp_recycle != from->pp_recycle || - (from->pp_recycle && skb_cloned(from))) + if (to->pp_recycle != from->pp_recycle) return false; if (len <= skb_tailroom(to)) { @@ -5836,8 +5849,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, /* if the skb is not cloned this does nothing * since we set nr_frags to 0. */ - for (i = 0; i < from_shinfo->nr_frags; i++) - __skb_frag_ref(&from_shinfo->frags[i]); + if (from->pp_recycle) + for (i = 0; i < from_shinfo->nr_frags; i++) + skb_pp_frag_ref(skb_frag_page(&from_shinfo->frags[i])); + else + for (i = 0; i < from_shinfo->nr_frags; i++) + __skb_frag_ref(&from_shinfo->frags[i]); to->truesize += delta; to->len += len;
In order to address the issues encountered with commit 1effe8ca4e34 ("skbuff: fix coalescing for page_pool fragment recycling"), the combination of the following condition was excluded from skb coalescing: from->pp_recycle = 1 from->cloned = 1 to->pp_recycle = 1 However, with page pool environments, the aforementioned combination can be quite common(ex. NetworkMananger may lead to the additional packet_type being registered, thus the cloning). In scenarios with a higher number of small packets, it can significantly affect the success rate of coalescing. For example, considering packets of 256 bytes size, our comparison of coalescing success rate is as follows: Without page pool: 70% With page pool: 13% Consequently, this has an impact on performance: Without page pool: 2.57 Gbits/sec With page pool: 2.26 Gbits/sec Therefore, it seems worthwhile to optimize this scenario and enable coalescing of this particular combination. To achieve this, we need to ensure the correct increment of the "from" SKB page's page pool reference count (pp_ref_count). Following this optimization, the success rate of coalescing measured in our environment has improved as follows: With page pool: 60% This success rate is approaching the rate achieved without using page pool, and the performance has also been improved: With page pool: 2.52 Gbits/sec Below is the performance comparison for small packets before and after this optimization. We observe no impact to packets larger than 4K. packet size before after improved (bytes) (Gbits/sec) (Gbits/sec) 128 1.19 1.27 7.13% 256 2.26 2.52 11.75% 512 4.13 4.81 16.50% 1024 6.17 6.73 9.05% 2048 14.54 15.47 6.45% 4096 25.44 27.87 9.52% Signed-off-by: Liang Chen <liangchen.linux@gmail.com> Reviewed-by: Yunsheng Lin <linyunsheng@huawei.com> Suggested-by: Jason Wang <jasowang@redhat.com> --- include/net/page_pool/helpers.h | 5 ++++ net/core/skbuff.c | 41 +++++++++++++++++++++++---------- 2 files changed, 34 insertions(+), 12 deletions(-)