From patchwork Wed Dec 6 10:54:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Liang Chen X-Patchwork-Id: 13481488 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95407C4167B for ; Wed, 6 Dec 2023 10:54:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2821B6B00B1; Wed, 6 Dec 2023 05:54:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 20B546B00B2; Wed, 6 Dec 2023 05:54:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 067866B00B3; Wed, 6 Dec 2023 05:54:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E23CE6B00B1 for ; Wed, 6 Dec 2023 05:54:56 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B8F721C0068 for ; Wed, 6 Dec 2023 10:54:56 +0000 (UTC) X-FDA: 81536085792.09.FEABB67 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) by imf01.hostedemail.com (Postfix) with ESMTP id DEB8F40017 for ; Wed, 6 Dec 2023 10:54:54 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FqjLkrbi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701860094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XJpeq0esGgvSQwPlWYu5mSKUKlrzr2fXLQ04GZyOjBI=; b=sucFJjArjLD2F9u6ekvgR4eOyFVtFgAVfjFDKxJNOPibXJbBhWmq+NjgSZQ7qrc2D7yE8Q mREek0veUznDRoMOaRNPcrX5KpN2eBne0Pmudp6CosJe55/i5vdFFL9UB60qA38tg/A+cC i13J0+arsH0TN9d4fqEeXkp9bquFMVs= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FqjLkrbi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.210.175 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701860094; a=rsa-sha256; cv=none; b=CcsryjRbXd7oQUkqyhCNLrh60JIXKPR1FVdZIiYHBn+mxWbUbLFcxfJwMg5Ow7d34UH3OK 0uNGzllWhrZLpCmZnPu5oxc+3LyJowY4yAbKKHNgb58b3kDKxdzbO3jitIYQLM0zTzHyrV py9btMeAhRiVheIb7qvjLWuxzh9F5Jo= Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-6ce4d4c5ea2so2576545b3a.0 for ; Wed, 06 Dec 2023 02:54:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701860094; x=1702464894; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XJpeq0esGgvSQwPlWYu5mSKUKlrzr2fXLQ04GZyOjBI=; b=FqjLkrbi11yHfReZxuhTbkoLIoua6v6FOq+ds4od0pwMUotSwr8AVdqb5BN2E4jmyl pYA0hKp+u45Xl1tlR5HaHttq/apjv82AF00Zb1fiZdBRKCuNPpzzUomqlvqY6VZGnwS3 cLQlFtPAvKQBnUW9/KJVy88HnytvjaEtdSkZXLUR0cZAhamspKp9QIveawA+A0BPWz5Q PimSURrmUGXw8pA2AsnCyWYx9AOxpOUUikL8D+y3a8GL09COYoKjxyy9/JEPUWzM814y 2UStG1hjmofoacLMDHgEkaQ65EpzZdKkrxhDcrRG6rdWjNpUtsaJ1+7cHkEtBQ/x1Isv pmdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701860094; x=1702464894; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XJpeq0esGgvSQwPlWYu5mSKUKlrzr2fXLQ04GZyOjBI=; b=uN0P9yvSoiD5yAR0U2u75x+NMEtfOPngUVvbyMQydv3IoPORj/yDUOzh5PvoTe2enl 6e9Pk6Cr7K3n54aQKoZ8vG/mvzmMJA1waqE0g9krBWCNDs8kZUXgvPqxwbOjQUKOrjla h7VqKpVVprXOPrib70vGPB5L4XQQURzwhesSMefWOcU+TlR2N1MoUYB2HTzGvl713tny ugCjXOuF925U37rt/mJVLQBNDjNaZDs15JIkoatSkk7rCMTE9UNHpEotyqiOxdEaFikU swPaQp5QoXJVrUNKpGwlY9xIzf9AWQVX0qxhtfpkqXms4N766IoiJgfvi2hsuGRQgZrg 1Nlw== X-Gm-Message-State: AOJu0YxWxInVvOuKsxJpGiMoYuUfjugxG/k/a78h9KvZQpzi6y+8JQY/ SgfToN55X/bpGzI8LD9LTArA94mDask= X-Google-Smtp-Source: AGHT+IGs32UGGkDmx++vF2WrxMsf6vIKFr/2F29zLgzga5cW6Le3dLh6j/00l5geSI7if5bXr/dyag== X-Received: by 2002:a05:6a00:1743:b0:6ce:6c6d:1622 with SMTP id j3-20020a056a00174300b006ce6c6d1622mr514790pfc.62.1701860093745; Wed, 06 Dec 2023 02:54:53 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id n15-20020a638f0f000000b005c6801efa0fsm5388796pgd.28.2023.12.06.02.54.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Dec 2023 02:54:52 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, jasowang@redhat.com, liangchen.linux@gmail.com Subject: [PATCH net-next v7 4/4] skbuff: Optimization of SKB coalescing for page pool Date: Wed, 6 Dec 2023 18:54:19 +0800 Message-Id: <20231206105419.27952-5-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231206105419.27952-1-liangchen.linux@gmail.com> References: <20231206105419.27952-1-liangchen.linux@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: DEB8F40017 X-Stat-Signature: f1a13qjas36p7hijz3f7bbzgcxh5ssa6 X-Rspam-User: X-HE-Tag: 1701860094-351736 X-HE-Meta: U2FsdGVkX19KWjz2I7vmebby3vE+3dEctQHpBl/+09yB6dFPnM/Rb71whEgDqc0MoGyZGr5lYSGxpPxSOdulcdMTLNvs0x7S3fsGgwfR3UgoUrLxRKO/DBNbGtCSKe5xwSVHCrCjGMIYbwv3dCkiQHxXKvAoon6R8NmkYBh+4h3wfuX/HPl4N7dxf78kR5fhEPD25a8dqXlgcbeIAuWNHF7qvOb6itb7vlSE5L5gAeukhzLnbv66Rd/C0XqRBST6DRLHbUJMw949DVlEHFYpTKbL8f8vo7OAxA59AMTejgIH4k3TlsLIMD3LF033NbpIxd1MWiSvzqLMTdWCaBgP0gH++CvHeQw+9RF4tLZfM3Zhrl2SxFpfJvSEBH49I1uqXhSRmg1E8SJWijy6/kDcWp09wtKDCiHbZXhYNvnujj1UUf1+2HXJ5g25KhS+r8ww5XpPtz3akctJUo4JCXj6wLeAPO0aZb6auGofihsw816pTGxFtLfNVn98SdrBtlRBCTwVV5PC72CY+djVjUGLdKQcfLGz8c9LaB44OWWlteEyLH1HZERnCoH8JzzptTR0eCJ5vYYpx63TstFPaRRTYaxFN/CBr3m29tJTjzXNx3RtvAI0QbP+CydA3gJSNI3H9vUxPebGzZB/A/qxGde9sC4QAUNRodGPvRy006yIMl1eW66U9eiMsDx7fZIlVpvX70ucsDs65zBBKp5XnCzJDYuxjIx9nOhCG0MUoVi9cf/fah4pZAX1ecePC9jZVcwkBErK5USStR2zF+C3Q6rAU5vbjCxbsipJeZU+hLEJCumz5EJWIEKBY1Um7l5V4Ch4l4o3xEyjt7p2fyUL5lE5wzdgqIVYfWRtpbJ2ZtD/qmKzkKzYe7WRSTH5WelFr5Gv1kQlonR9Fl/RWmrvteIxutiS30LRvHYjp5JZHYukkInmkg8QT2OdFIjye4Hnumwp80J4JlHRJmyWrHlHQN/ BDmVYjjc Ss79fhDR1QFCsOSwKMvgD9nzXUA2IO9oWpscTtTwRZJ5lJKb6ZZN5WLOu4cv4BIjW1+CWCg34mJtwxIEf5FdK2YeAnCtPDIVBxDX/GUGf6S5QGWLouHGMgbpXEF+bAlkJq5pdRocUtYDDjoidIIE+PdUJzbczutkE1LiZK0H6lK7STgTgmKlAE3NxrpK1JfWiOWWBq+28YmeXzwoJv37gpSrmjSSz6Qk5Z6TovP/HsQcjdSk2qc0dfEG6AH9kPvJM2GBsR6/LyZFtEI1+zqZZyrqZDcAQJIV8ax3OhjCW9W/Be2LIvq6ce19B7cOerJxxfY5w5a8PGxpoy/iifgTzrRoKijo4foEnXNvqwIbe8gMkOvsf9vMHq2S2JmBD9L3P5OxHGJzArJ4BzWqbmE7OEcrS6MX53oD0PFy3fhvNvUmMotp6LDAyhqsyJ/lvDAJ9E4wNJ7TpD6My8RVZ9SVY0DSlE7tm2kvhAKytzWqDFs0GWjR7BDkJmgVnwKafgsbQ/0z4Oq2pS5rOxw8VoE1c6c/7SA1vbuo8L3JvB042nNVpRkwHDluZ7rHUeDDOmx3RElVVR0nyiwW9EE1TkgYk3hf29Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to address the issues encountered with commit 1effe8ca4e34 ("skbuff: fix coalescing for page_pool fragment recycling"), the combination of the following condition was excluded from skb coalescing: from->pp_recycle = 1 from->cloned = 1 to->pp_recycle = 1 However, with page pool environments, the aforementioned combination can be quite common(ex. NetworkMananger may lead to the additional packet_type being registered, thus the cloning). In scenarios with a higher number of small packets, it can significantly affect the success rate of coalescing. For example, considering packets of 256 bytes size, our comparison of coalescing success rate is as follows: Without page pool: 70% With page pool: 13% Consequently, this has an impact on performance: Without page pool: 2.57 Gbits/sec With page pool: 2.26 Gbits/sec Therefore, it seems worthwhile to optimize this scenario and enable coalescing of this particular combination. To achieve this, we need to ensure the correct increment of the "from" SKB page's page pool reference count (pp_ref_count). Following this optimization, the success rate of coalescing measured in our environment has improved as follows: With page pool: 60% This success rate is approaching the rate achieved without using page pool, and the performance has also been improved: With page pool: 2.52 Gbits/sec Below is the performance comparison for small packets before and after this optimization. We observe no impact to packets larger than 4K. packet size before after improved (bytes) (Gbits/sec) (Gbits/sec) 128 1.19 1.27 7.13% 256 2.26 2.52 11.75% 512 4.13 4.81 16.50% 1024 6.17 6.73 9.05% 2048 14.54 15.47 6.45% 4096 25.44 27.87 9.52% Signed-off-by: Liang Chen Reviewed-by: Yunsheng Lin Suggested-by: Jason Wang --- include/net/page_pool/helpers.h | 5 ++++ net/core/skbuff.c | 41 +++++++++++++++++++++++---------- 2 files changed, 34 insertions(+), 12 deletions(-) diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index 9dc8eaf8a959..268bc9d9ffd3 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -278,6 +278,11 @@ static inline long page_pool_unref_page(struct page *page, long nr) return ret; } +static inline void page_pool_ref_page(struct page *page) +{ + atomic_long_inc(&page->pp_ref_count); +} + static inline bool page_pool_is_last_ref(struct page *page) { /* If page_pool_unref_page() returns 0, we were the last user */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 7e26b56cda38..3c2515a29376 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -947,6 +947,24 @@ static bool skb_pp_recycle(struct sk_buff *skb, void *data, bool napi_safe) return napi_pp_put_page(virt_to_page(data), napi_safe); } +/** + * skb_pp_frag_ref() - Increase fragment reference count of a page + * @page: page of the fragment on which to increase a reference + * + * Increase fragment reference count (pp_ref_count) on a page, but if it is + * not a page pool page, fallback to increase a reference(_refcount) on a + * normal page. + */ +static void skb_pp_frag_ref(struct page *page) +{ + struct page *head_page = compound_head(page); + + if (likely(is_pp_page(head_page))) + page_pool_ref_page(head_page); + else + page_ref_inc(head_page); +} + static void skb_kfree_head(void *head, unsigned int end_offset) { if (end_offset == SKB_SMALL_HEAD_HEADROOM) @@ -5769,17 +5787,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, return false; /* In general, avoid mixing page_pool and non-page_pool allocated - * pages within the same SKB. Additionally avoid dealing with clones - * with page_pool pages, in case the SKB is using page_pool fragment - * references (page_pool_alloc_frag()). Since we only take full page - * references for cloned SKBs at the moment that would result in - * inconsistent reference counts. - * In theory we could take full references if @from is cloned and - * !@to->pp_recycle but its tricky (due to potential race with - * the clone disappearing) and rare, so not worth dealing with. + * pages within the same SKB. In theory we could take full + * references if @from is cloned and !@to->pp_recycle but its + * tricky (due to potential race with the clone disappearing) and + * rare, so not worth dealing with. */ - if (to->pp_recycle != from->pp_recycle || - (from->pp_recycle && skb_cloned(from))) + if (to->pp_recycle != from->pp_recycle) return false; if (len <= skb_tailroom(to)) { @@ -5836,8 +5849,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, /* if the skb is not cloned this does nothing * since we set nr_frags to 0. */ - for (i = 0; i < from_shinfo->nr_frags; i++) - __skb_frag_ref(&from_shinfo->frags[i]); + if (from->pp_recycle) + for (i = 0; i < from_shinfo->nr_frags; i++) + skb_pp_frag_ref(skb_frag_page(&from_shinfo->frags[i])); + else + for (i = 0; i < from_shinfo->nr_frags; i++) + __skb_frag_ref(&from_shinfo->frags[i]); to->truesize += delta; to->len += len;