From patchwork Fri Feb 4 00:09:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12734727 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1020C433EF for ; Fri, 4 Feb 2022 00:10:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356150AbiBDAKq (ORCPT ); Thu, 3 Feb 2022 19:10:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356182AbiBDAKo (ORCPT ); Thu, 3 Feb 2022 19:10:44 -0500 Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27580C061714 for ; Thu, 3 Feb 2022 16:10:44 -0800 (PST) Received: by mail-pf1-x430.google.com with SMTP id y5so2687394pfe.4 for ; Thu, 03 Feb 2022 16:10:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jZQROXOP0pza6cD/HAg54T+BPrYK4YdGCqA3Sg45Jw8=; b=YyC3YF9Wj9RrKPYZuyme1WaLLkniV1GiY0yR3OfB/JxRfuPdzOJTK8V0ya5fHsZHOA 8GG2OSdfLTR7ZyjtmnHUhk9uQunclMI2I2OrHcPyazWt66u28DSYH3ZcWbiB5+7Z4Z/c 2jcBT2pNvvzUOZjd/525xPMzNsgtJf0zuqJdc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jZQROXOP0pza6cD/HAg54T+BPrYK4YdGCqA3Sg45Jw8=; b=kJ9gItqQSomXWW0fz4YLc9PY7C6+sxXe8L9b2EFiLRIoN7rESA3oY8MlNroF6gTNUW p5UsXxQh90i6tMQFw8Gx6ApUmT4RS0Ed+CjeNX753872lHQy8AoHN/bd/fLyHwG3htB8 ulZRz/DUMa1Rgp2BJy1LOfYSl3sawWqliaKFDmqYVuFQjHS8KvZYX9Lk3srLQgA4ISWR zdismcZgajMW1A+d+oRQEoIBcV9W6irYYRO8mowB9WXcgi4n+6DxQ9rentP7RfhKrzaP ya926hpowJKXIVHgI9XGy+uY17pv7I7vyDgObjtS5NFih75smZZ7KDA9HFchjeWcQhL5 7V8A== X-Gm-Message-State: AOAM530GZiXiNYZuQ3w3HE8uGPNTKoL74e5lq5UzajtcbCZ9TH8wCNQe 3Fa8MdeTbZGN+JWuxCzchVK4bFUWjDrlYzUciJ7jSn8dihb2o57Dx28SwrcjjZLNKL6IDVO4NJb QNchTeVl9xDPJGBPyh26kD0+ONHSZprPka12ct4yvtVmCMHGSwbcWU6oCTNZhgmx4LkGa X-Google-Smtp-Source: ABdhPJzZtGpgeuzre265LqPXdn4f30r/FjfyqFpFhsmkty55bMzE+8cUObILC3Bv8ssgcMWhuHTQDA== X-Received: by 2002:a62:7a0b:: with SMTP id v11mr617125pfc.22.1643933443269; Thu, 03 Feb 2022 16:10:43 -0800 (PST) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id z22sm159822pfe.42.2022.02.03.16.10.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Feb 2022 16:10:42 -0800 (PST) From: Joe Damato To: netdev@vger.kernel.org, kuba@kernel.org, ilias.apalodimas@linaro.org, davem@davemloft.net, hawk@kernel.org, saeed@kernel.org, ttoukan.linux@gmail.com, brouer@redhat.com Cc: Joe Damato Subject: [net-next v4 01/11] page_pool: kconfig: Add flag for page pool stats Date: Thu, 3 Feb 2022 16:09:23 -0800 Message-Id: <1643933373-6590-2-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1643933373-6590-1-git-send-email-jdamato@fastly.com> References: <1643933373-6590-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Control enabling / disabling page_pool_stats with a kernel config option. Option is defaulted to N. Signed-off-by: Joe Damato --- net/Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/net/Kconfig b/net/Kconfig index 8a1f9d0..604b3eb 100644 --- a/net/Kconfig +++ b/net/Kconfig @@ -434,6 +434,18 @@ config NET_DEVLINK config PAGE_POOL bool +config PAGE_POOL_STATS + default n + bool "Page pool stats" + depends on PAGE_POOL + help + Enable page pool statistics to track allocations. Stats are exported + to the file /proc/net/page_pool_stat. Users can examine these + stats to better understand how their drivers and the kernel's + page allocator, and the page pool interact with each other. + + If unsure, say N. + config FAILOVER tristate "Generic failover module" help From patchwork Fri Feb 4 00:09:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12734728 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBB63C433FE for ; Fri, 4 Feb 2022 00:10:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356151AbiBDAKr (ORCPT ); Thu, 3 Feb 2022 19:10:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232449AbiBDAKq (ORCPT ); Thu, 3 Feb 2022 19:10:46 -0500 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5278DC061714 for ; Thu, 3 Feb 2022 16:10:46 -0800 (PST) Received: by mail-pj1-x1035.google.com with SMTP id m7so3969318pjk.0 for ; Thu, 03 Feb 2022 16:10:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uaxwzggX7nkIfNzdgxAq8JqDHlXgho20pfn4LcykN+g=; b=Il4salgRGjcv9FBDj+LGVcUj/vWO8ezSYKjqmE0HtjkwCIhLY/qbls23/nvNRMlMu/ S3XmHDGWLkbWd4DiO1rWOuqOvj9xHMew4B5wvIcQVC2YmA+8se5T61K9B0AVVIwmoanH 3sSAsiFA7yJgl5yqO++Nk8SR+cLhrApH8Gyrg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uaxwzggX7nkIfNzdgxAq8JqDHlXgho20pfn4LcykN+g=; b=jIlZKV7W4w80Yb46BvWOReB50akkhJs18Okh5heXv5Os6WDT9NX452hzEghQpHuLGq k8VeTOY0Ha8aPLV2sgq9UMkVrPaSe8kpLCo6Wx7dZV0kVgnBBF2UhnEuDZL3lLRPMxRe nFkYYFhHhA1P4aCqTY7HWBYFkIGWddNK77GB2dktTyTNFjVsDnNzScv+JZsGwP1V3CnT kB7KivLsbAAeVTUB9q2Ab2mD2G7FLleADNu4TuT74AdE6zIYW6DcNkTVAi+I06cJ24t3 0oxX7fOG8jA/eX/KT7IBeDgxMbD/XzKH5S406QfVNtLllwNkBRQP1svGRpjjut9yAbSP 1Obw== X-Gm-Message-State: AOAM5339AOoJfQeTrg3fxJAAsJ6X0uCbktOISY27c0MoDoqOFSP0rTMg d8ikTnE1cumIscICENxLPV6L8SJxdEnAB1i3lXPuWxERRXiF+LTnmRSv8PyVcRhFMJ0n9ko1/BZ uuv13q3ck1Tp3S4+hCZgfBgRx4xSeGuNIU8loUaKVpEZWesoYs/1l+vy0EPvvsWe7D6eX X-Google-Smtp-Source: ABdhPJwgNmxQ6ELybSBzb/16N1wtQ2QtftHS0M39wT21GNDzyZ0ApI73afJvDXvwSsPLv6EixmXzOw== X-Received: by 2002:a17:902:e844:: with SMTP id t4mr635176plg.104.1643933444955; Thu, 03 Feb 2022 16:10:44 -0800 (PST) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id z22sm159822pfe.42.2022.02.03.16.10.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Feb 2022 16:10:44 -0800 (PST) From: Joe Damato To: netdev@vger.kernel.org, kuba@kernel.org, ilias.apalodimas@linaro.org, davem@davemloft.net, hawk@kernel.org, saeed@kernel.org, ttoukan.linux@gmail.com, brouer@redhat.com Cc: Joe Damato Subject: [net-next v4 02/11] page_pool: Add per-pool-per-cpu struct. Date: Thu, 3 Feb 2022 16:09:24 -0800 Message-Id: <1643933373-6590-3-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1643933373-6590-1-git-send-email-jdamato@fastly.com> References: <1643933373-6590-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org A per pool per cpu struct is added. Empty for now as members will be added next. Signed-off-by: Joe Damato --- include/net/page_pool.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 79a8055..93e587d 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -135,7 +135,15 @@ struct page_pool { refcount_t user_cnt; u64 destroy_cnt; +#ifdef CONFIG_PAGE_POOL_STATS + struct page_pool_stats __percpu *stats; +#endif +}; + +#ifdef CONFIG_PAGE_POOL_STATS +struct page_pool_stats { }; +#endif struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp); From patchwork Fri Feb 4 00:09:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12734729 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E50F5C433F5 for ; Fri, 4 Feb 2022 00:10:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356159AbiBDAKu (ORCPT ); Thu, 3 Feb 2022 19:10:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51690 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232449AbiBDAKs (ORCPT ); Thu, 3 Feb 2022 19:10:48 -0500 Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09EABC06173B for ; Thu, 3 Feb 2022 16:10:48 -0800 (PST) Received: by mail-pj1-x1034.google.com with SMTP id s61-20020a17090a69c300b001b4d0427ea2so11521710pjj.4 for ; Thu, 03 Feb 2022 16:10:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Fo1he86IgYcWoGX5pibSbapZylx3CuHd2N7i1S0mBEE=; b=yPKqIEbbXY/B58kcgAVD4sWBwP6lFKJnwFgqCLRIdCDE8+3msnByzKIRQyQSvv/uz3 vPu9DZTAMl7CtIhQFPYnJDDvlVYvHd1MBFHdMZyfHApuP1dobWpbK1VhVxEt4fnFW/d1 Th+aUKaq6j07c0rAHrhvLz1asXWq566RXJS/4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Fo1he86IgYcWoGX5pibSbapZylx3CuHd2N7i1S0mBEE=; b=MwMXLi8kB1oWiWUTm0CIlnrLNPwt04Mau5fLxHkFVH1J2mhZQ0sNYiLQlrmtbXMB/5 aB9EnZsmjYBIA/oMb1Q6HYWnHSZCVNnxJBcRwuYLltHRqG2Pm2i3+cGyG9uOMYj71cfa 476IqlGhPIa2MZDMU1tUZ4VkzF859Ghm9gT1gmrLPnKivH/Ahn189yMywlnnsrA2hNAX 3dlovq8FaI0MaiiHg71VpvndcYRMdiYLlIXnOxF3UV+ApCcI2NIv8y8Nn7L4FGJvk/lG NpE3c/pfGipFx1Oc4bOCo9CNhM7Tz4n/3jkMVc4nOtyJz15uDR6f0fF6jskieb10gYtA SlsQ== X-Gm-Message-State: AOAM532aE5Kl4VOrPuonKhMQKEDAFXXYIEYMPNLYOUSVb7t9F98pAlPv cvXRg9nQB2kRKK4wVNDkrz4IDL4DwIFuF0uVljlH4+X3bRMuu7H7OU8Zy+WXY789IVFg/fnmHGX dqPcPZRby2S/cWBF5njp7Nohxooidjw6/oVa1aJeg4eNam2zGnXzPZYyV14SuP5quw94m X-Google-Smtp-Source: ABdhPJwGYDDqXLGl6f9/9vrWQcFwerqHkcw94z8D53KjsSBXY4dpiEQz+/PcJGZUdGxZg5lfvjdBmQ== X-Received: by 2002:a17:90a:b397:: with SMTP id e23mr209719pjr.63.1643933447065; Thu, 03 Feb 2022 16:10:47 -0800 (PST) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id z22sm159822pfe.42.2022.02.03.16.10.45 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Feb 2022 16:10:46 -0800 (PST) From: Joe Damato To: netdev@vger.kernel.org, kuba@kernel.org, ilias.apalodimas@linaro.org, davem@davemloft.net, hawk@kernel.org, saeed@kernel.org, ttoukan.linux@gmail.com, brouer@redhat.com Cc: Joe Damato Subject: [net-next v4 03/11] page_pool: Allocate and free stats structure Date: Thu, 3 Feb 2022 16:09:25 -0800 Message-Id: <1643933373-6590-4-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1643933373-6590-1-git-send-email-jdamato@fastly.com> References: <1643933373-6590-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org This commit only alloc_percpu and free_percpu for the stats structure. Signed-off-by: Joe Damato --- net/core/page_pool.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index bd62c01..5f822b0 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -73,6 +73,12 @@ static int page_pool_init(struct page_pool *pool, pool->p.flags & PP_FLAG_PAGE_FRAG) return -EINVAL; +#ifdef CONFIG_PAGE_POOL_STATS + pool->stats = alloc_percpu(struct page_pool_stats); + if (!pool->stats) + return -ENOMEM; +#endif + if (ptr_ring_init(&pool->ring, ring_qsize, GFP_KERNEL) < 0) return -ENOMEM; @@ -623,6 +629,9 @@ static void page_pool_free(struct page_pool *pool) if (pool->p.flags & PP_FLAG_DMA_MAP) put_device(pool->p.dev); +#ifdef CONFIG_PAGE_POOL_STATS + free_percpu(pool->stats); +#endif kfree(pool); } From patchwork Fri Feb 4 00:09:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12734730 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE0D1C433EF for ; Fri, 4 Feb 2022 00:10:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356167AbiBDAKu (ORCPT ); Thu, 3 Feb 2022 19:10:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356147AbiBDAKt (ORCPT ); Thu, 3 Feb 2022 19:10:49 -0500 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C811C06173E for ; Thu, 3 Feb 2022 16:10:49 -0800 (PST) Received: by mail-pl1-x633.google.com with SMTP id j16so3676515plx.4 for ; Thu, 03 Feb 2022 16:10:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=TtnJ4jV+znpNKtkePPK91XSsmqJA9GMlQm0gfWOolAg=; b=QVJCOOIOWjgeZzhP0tPAR7r7SekrUh5rxrKzJvcDzgrvHiXaC+5bFA8e1nrFF5lGoc Og48Rs6PC/1pIUXvq9SftaDt6zPaGReM6TtrqNJSwmsQ59ntezwwOJ77fg/+eNcHzqmY Oa6KFcpSzte5IQQ4J4Gutm22wrVharqzWLGGs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=TtnJ4jV+znpNKtkePPK91XSsmqJA9GMlQm0gfWOolAg=; b=KHVX2ZkC1rMITA4qpt1zfI2JY3gtZykpgMP8294aDsqWaUVQuv5wqEEJQB0pAS0czh 3BKfTIoIk2xXphT/3vF58IEO4sIWa/1goBBGmqGsDaPoGxHYRtuaSMYtWTqThANIEGNA zrPCi6QinpUAgbiHba2yGMxBkuSDIT1VpcYymv3/SnHcVNQ1ccb4NOoikWqlxB98Bl7+ AUnmwensaRpBpOtQDB0w0Dc5/rzMigBraq0zSt4bHZq3YGGWg8ETz9JC0F/8E2+h9zgx ydN3cxIVFUbnxWBa1Y+34bPaUAGBayzOQtdSDXHr47ne54ZhHZNTMtZoGyJ87PjeFdLv VVHQ== X-Gm-Message-State: AOAM5312P38uWQbvTsyFusvK0V7o+n4shwP5HwkSWO4SIzeXd73fMZi1 JxX8suYOeVQVMtxc+ybezcf3phSynqkf+TMo77YXxTWDR/Z5ZE1srONQ2lM0tuGUbfzXAS4iBof NH+kh76xKCEJSK3fCcpVcHfBNyBKvwEjM4gIHYet01+CXFrAZPi5jBjInBAeJPfCKWa+u X-Google-Smtp-Source: ABdhPJwyju0evxGVoRR7TFK9zz/QhkFviXvE63mjjuXEGQaqcUIE+iarBS+6ilCY/keTSD2qJRweMA== X-Received: by 2002:a17:90a:7c02:: with SMTP id v2mr209505pjf.81.1643933448710; Thu, 03 Feb 2022 16:10:48 -0800 (PST) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id z22sm159822pfe.42.2022.02.03.16.10.47 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Feb 2022 16:10:48 -0800 (PST) From: Joe Damato To: netdev@vger.kernel.org, kuba@kernel.org, ilias.apalodimas@linaro.org, davem@davemloft.net, hawk@kernel.org, saeed@kernel.org, ttoukan.linux@gmail.com, brouer@redhat.com Cc: Joe Damato Subject: [net-next v4 04/11] page_pool: Add macro for incrementing alloc stats Date: Thu, 3 Feb 2022 16:09:26 -0800 Message-Id: <1643933373-6590-5-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1643933373-6590-1-git-send-email-jdamato@fastly.com> References: <1643933373-6590-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Add macro incrementing per pool per cpu stats. Signed-off-by: Joe Damato --- net/core/page_pool.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 5f822b0..180e48b 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -26,6 +26,19 @@ #define BIAS_MAX LONG_MAX +#ifdef CONFIG_PAGE_POOL_STATS +/* + * this_cpu_inc_alloc_stat is intended to be used in softirq context + */ +#define this_cpu_inc_alloc_stat(pool, __stat) \ + do { \ + struct page_pool_stats __percpu *s = pool->stats; \ + __this_cpu_inc(s->alloc.__stat); \ + } while (0) +#else +#define this_cpu_inc_alloc_stat(pool, __stat) +#endif + static int page_pool_init(struct page_pool *pool, const struct page_pool_params *params) { From patchwork Fri Feb 4 00:09:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12734731 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECBFEC433FE for ; Fri, 4 Feb 2022 00:10:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356163AbiBDAKw (ORCPT ); Thu, 3 Feb 2022 19:10:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356147AbiBDAKv (ORCPT ); Thu, 3 Feb 2022 19:10:51 -0500 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49BE4C061714 for ; Thu, 3 Feb 2022 16:10:51 -0800 (PST) Received: by mail-pj1-x1036.google.com with SMTP id s61-20020a17090a69c300b001b4d0427ea2so11521827pjj.4 for ; Thu, 03 Feb 2022 16:10:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=sBh+Hfll67AeiB+Fr4u+vldCm4Dp1LVrmoQj463LZFg=; b=mhjS9i9Mb6Kwp31UnREU2A93sKi7wb6c6ZjMAo4fxZ98qFeH3qKx6YYAXoJCuKqSoO vbZHCNmEtZNahTzoWt55qZkQhs/Qx6F3L+w65JladTwftk65du2QbSo378oooG3YXL8o kNHGf3cZxDKwLYmfvG6LNvjoDCSvKl0V2WeQ8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=sBh+Hfll67AeiB+Fr4u+vldCm4Dp1LVrmoQj463LZFg=; b=ed72c7g5IzXqL72Bd0v5AAyfqdSzpP5q8qHQmgQofCgvrFCMRg1Kcn+JMFSqK55mYx vU54+ib92gTIKXJ1G1giAg1RfM+wkdkGTC3ijrPxp21kUbvhPQLB1aHmiF7Y59gh7agk U1rDhcW+VGyWNWHW2dvmwm0xC7XYtE48kPZOdeOagSEzsMG+hElbUxpOWOT3HPZKHVpQ xONPYvaNFwORcYJSOGp/WP0jCcGozd4NTER5KW7NW9QVJzxKGDROELmqFVvulxYgJzF+ yHDQhHoBp/uby2pt2XHI8g6wcAsfxDdhbJPCndgB0Cn/TTe0J6FieltrQSc5MFdKo4/P CTBg== X-Gm-Message-State: AOAM533rhe5Jr3n3KEp5+TljrXzHF+p7u02D2BHemY9MoQT1qz9D4GQU DaARhCUiBLaHrvNtULNPou8AuQX0EytyUFerAEhoK9GSFJAWFWyrbVCesIamuesistt3+wUzIEN rckbUApjfFjqfXcMa53p6IfPNeDtcJq2FXhIDusUmFJGDt6M3dKUuLLN++I4dTT3bVlBk X-Google-Smtp-Source: ABdhPJzqAx6HTtU1/RZaa3ZLfXsAmAuNwDIqweqiXQ6LXZ9eWza/x5CgtlilQdG25FshCqsicqd3ag== X-Received: by 2002:a17:90b:68c:: with SMTP id m12mr204744pjz.137.1643933450405; Thu, 03 Feb 2022 16:10:50 -0800 (PST) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id z22sm159822pfe.42.2022.02.03.16.10.48 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Feb 2022 16:10:49 -0800 (PST) From: Joe Damato To: netdev@vger.kernel.org, kuba@kernel.org, ilias.apalodimas@linaro.org, davem@davemloft.net, hawk@kernel.org, saeed@kernel.org, ttoukan.linux@gmail.com, brouer@redhat.com Cc: Joe Damato Subject: [net-next v4 05/11] page_pool: Add fast path stat Date: Thu, 3 Feb 2022 16:09:27 -0800 Message-Id: <1643933373-6590-6-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1643933373-6590-1-git-send-email-jdamato@fastly.com> References: <1643933373-6590-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Increment the pool's fast path allocation stat when this event occurs. Signed-off-by: Joe Damato --- include/net/page_pool.h | 3 +++ net/core/page_pool.c | 1 + 2 files changed, 4 insertions(+) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 93e587d..864a480 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -142,6 +142,9 @@ struct page_pool { #ifdef CONFIG_PAGE_POOL_STATS struct page_pool_stats { + struct { + u64 fast; /* fast path allocations */ + } alloc; }; #endif diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 180e48b..b5bf41f 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -185,6 +185,7 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) if (likely(pool->alloc.count)) { /* Fast-path */ page = pool->alloc.cache[--pool->alloc.count]; + this_cpu_inc_alloc_stat(pool, fast); } else { page = page_pool_refill_alloc_cache(pool); } From patchwork Fri Feb 4 00:09:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12734733 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2102C433F5 for ; Fri, 4 Feb 2022 00:10:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356165AbiBDAK5 (ORCPT ); Thu, 3 Feb 2022 19:10:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356147AbiBDAKx (ORCPT ); Thu, 3 Feb 2022 19:10:53 -0500 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0918DC06173B for ; Thu, 3 Feb 2022 16:10:53 -0800 (PST) Received: by mail-pl1-x633.google.com with SMTP id x3so3683646pll.3 for ; Thu, 03 Feb 2022 16:10:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=l7FvHk4E5CURlw1cPp1+JmKn0JT3ZYqq3Gj+nnzO8zg=; b=vv+xDFOOL13mJgM6fZWlEuyOj/OEmVWp3roOiXBC6Y2WK6LIfnuhFrElSQRR2Q2D9G vv6Ussv/xnAZr0+vjqWTtO9nSPiy2jJerRPYukxTQk7x+W74xCMNt6W6CB+ZX1ZOvfns K19zkpHOd0pjPXwU+Fs5PE9NcKxNfXWe8ida4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=l7FvHk4E5CURlw1cPp1+JmKn0JT3ZYqq3Gj+nnzO8zg=; b=kZqagq1AhDqNgkeGBkKm5LuNhGzFBg8hI2WpL90JbJ/Q6eyGTX2J2FXtS9QBMF52v6 3M87U8FoQvq+ZF6v8pREczP44WgIOvsUeP1u7jyp32fjwktjtxYhbdzw1t+FumiGKHQZ NoIIxBmFpR66gOAyfpV4BQWl8Ph5jdl2LVBBWESjEYi7kKC9s3rUvV8Uim3LpSPodYcs 6lEfTzmMPlrKSgXGHqeZj86DzMw9olVe+RLBESxG6bk9gV00wdWfU665+fj3SG0ecbYK VnEQvV4LNRXPog/DJaz52OnwWPsZODPcZmqjEinZMgryzsGzkLSy3dXJsq2Xb0NPeA6W DbAQ== X-Gm-Message-State: AOAM531s0CoseUSedIO63dHQmbmv25JF5mmzE56R98eq9c414ck1ZsCR IZ5Zq//3SRxn5TgeFvqPRhc8Tq3klBdw2xhLCUaDUUAqheWwBlNXH9BqzkfZu0LTrfsLNg70Zbb m2jR82NN59DHkJ91jRWbjHpzlH8Bt2DXw9/HTWtkZsghz0t6ILJywZ5C4wBKZZ5K6a/CL X-Google-Smtp-Source: ABdhPJwyoDJ94E8MpKv7XB8eIxTVX4kMGRmjERkD4G55NVy8rqcEYJeJEChNgwYwIt8eFCZA9mFTcg== X-Received: by 2002:a17:90a:8c8b:: with SMTP id b11mr194854pjo.197.1643933452136; Thu, 03 Feb 2022 16:10:52 -0800 (PST) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id z22sm159822pfe.42.2022.02.03.16.10.50 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Feb 2022 16:10:51 -0800 (PST) From: Joe Damato To: netdev@vger.kernel.org, kuba@kernel.org, ilias.apalodimas@linaro.org, davem@davemloft.net, hawk@kernel.org, saeed@kernel.org, ttoukan.linux@gmail.com, brouer@redhat.com Cc: Joe Damato Subject: [net-next v4 06/11] page_pool: Add slow path order-0 stat Date: Thu, 3 Feb 2022 16:09:28 -0800 Message-Id: <1643933373-6590-7-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1643933373-6590-1-git-send-email-jdamato@fastly.com> References: <1643933373-6590-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Increment slow path order-0 allocation stat when this event occurs. Signed-off-by: Joe Damato --- include/net/page_pool.h | 1 + net/core/page_pool.c | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 864a480..7d5f202 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -144,6 +144,7 @@ struct page_pool { struct page_pool_stats { struct { u64 fast; /* fast path allocations */ + u64 slow; /* slow-path order 0 allocations */ } alloc; }; #endif diff --git a/net/core/page_pool.c b/net/core/page_pool.c index b5bf41f..264d8c9 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -313,10 +313,12 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, } /* Return last page */ - if (likely(pool->alloc.count > 0)) + if (likely(pool->alloc.count > 0)) { page = pool->alloc.cache[--pool->alloc.count]; - else + this_cpu_inc_alloc_stat(pool, slow); + } else { page = NULL; + } /* When page just alloc'ed is should/must have refcnt 1. */ return page; From patchwork Fri Feb 4 00:09:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12734732 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F390BC433EF for ; Fri, 4 Feb 2022 00:10:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356160AbiBDAK6 (ORCPT ); Thu, 3 Feb 2022 19:10:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51728 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356174AbiBDAKy (ORCPT ); Thu, 3 Feb 2022 19:10:54 -0500 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6B36C061401 for ; Thu, 3 Feb 2022 16:10:54 -0800 (PST) Received: by mail-pg1-x52c.google.com with SMTP id e16so3660664pgn.4 for ; Thu, 03 Feb 2022 16:10:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6YfR8wWEKlkfuLAKubG6N8BAv3Qg/s/kRTiUe01e+q4=; b=dow8iX/fA8NTU3euLf/9nIYxu5qqBZE7eQhbOBaZU/FQVxsHn2XYQzkTzVA05S3UhT +cFxV8ss8IkLNO3lwEc7enUzdzPCBZZOHStBe8ZjDehtd1IR5G5Laz1c/eKWvSJGzP18 f1S4VzHCuxR0booppJXWV7QsbA2dKOIYAJfEo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6YfR8wWEKlkfuLAKubG6N8BAv3Qg/s/kRTiUe01e+q4=; b=w5nCYdN+gt3Oz6Fj89cmoHAgzcxEOWXIHQiBZMOJ0oGDoGWWgyBab0HmrRV/9Y3voQ B4BjJnsmhyVl7UM5P/QOVvMPeChusu6kI8CXDuKLTxFc//9bp1XbEVRHvtInNDvm0W0b u+v4YW/hD7hyOyBcf1+scerl/GoJu7NrsGzVtlyVIHzco+CDa6w6ofcfOXab7+oACRbC Lvv5cQjZQjSr8QrfGLeTFZ46tpZGXOSHU1q1syrzpcZDvgex/nL6SEHd7riejCt/x7/U pVjecTpj+nFkkISfPpJ6crBIDYzoTQNypOIIJAGSMbVEzQSxtea3zeAW/Z014Pb1FruC QbGw== X-Gm-Message-State: AOAM532dcB5dg1ouZM/eU728OT67wPpMFNMQ4m6vsUbRlSdGr9V78bKV OdHWD6JLKVZK96hphaez9oc48hjFQ93JCsldlnc/Xbmsb6t8/e9pwuwAnlSJX0eEA8h8XhFno80 QX46LGqR9hY7vWR0NtuyEeNY7js60o1xUNeBX28rCXjMKU1dExehNdNTPLzwHDeSIkIWt X-Google-Smtp-Source: ABdhPJzCKpCFrETMdz0E1VC6Zno37IYkdWbiPGd4QTyCQ2WKu3F1mZ9LfP7g9m+zJQPehpCLYAgHKA== X-Received: by 2002:a65:414a:: with SMTP id x10mr414070pgp.125.1643933453793; Thu, 03 Feb 2022 16:10:53 -0800 (PST) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id z22sm159822pfe.42.2022.02.03.16.10.52 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Feb 2022 16:10:53 -0800 (PST) From: Joe Damato To: netdev@vger.kernel.org, kuba@kernel.org, ilias.apalodimas@linaro.org, davem@davemloft.net, hawk@kernel.org, saeed@kernel.org, ttoukan.linux@gmail.com, brouer@redhat.com Cc: Joe Damato Subject: [net-next v4 07/11] page_pool: Add slow path high order alloc stat Date: Thu, 3 Feb 2022 16:09:29 -0800 Message-Id: <1643933373-6590-8-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1643933373-6590-1-git-send-email-jdamato@fastly.com> References: <1643933373-6590-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Track high order allocations in the slow path which cause an interaction with the buddy allocator. Signed-off-by: Joe Damato --- include/net/page_pool.h | 1 + net/core/page_pool.c | 1 + 2 files changed, 2 insertions(+) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 7d5f202..0c4cb49 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -145,6 +145,7 @@ struct page_pool_stats { struct { u64 fast; /* fast path allocations */ u64 slow; /* slow-path order 0 allocations */ + u64 slow_high_order; /* slow-path high order allocations */ } alloc; }; #endif diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 264d8c9..b7d0995 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -259,6 +259,7 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool, return NULL; } + this_cpu_inc_alloc_stat(pool, slow_high_order); page_pool_set_pp_info(pool, page); /* Track how many pages are held 'in-flight' */ From patchwork Fri Feb 4 00:09:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12734735 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD2B5C4332F for ; Fri, 4 Feb 2022 00:10:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356174AbiBDAK6 (ORCPT ); Thu, 3 Feb 2022 19:10:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356182AbiBDAK5 (ORCPT ); Thu, 3 Feb 2022 19:10:57 -0500 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A8E6C061748 for ; Thu, 3 Feb 2022 16:10:56 -0800 (PST) Received: by mail-pj1-x1032.google.com with SMTP id m7so3969600pjk.0 for ; Thu, 03 Feb 2022 16:10:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=xoRYY0Fze8sf08kdJpIVjLOV4w2PmmkJcjWTrqHlPVA=; b=Cey/2Pxb5cUgz9VbX2Z9hBXXIlUnagj00fGDHagFKuNpndwEJF1axrNn7Ikpt/VLDO b4Np1IdAk//WScNuKgg52lN0YMaFu7vZLcEP8Du1+nrbgBYAnCIuS5BgJhbUn9380gjL zk2IQlnN5XBa8H4fAqqpe8iMXuWE+NGymVInk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=xoRYY0Fze8sf08kdJpIVjLOV4w2PmmkJcjWTrqHlPVA=; b=Gs400GsaJizujyuTbopsizZdG7llnIP5NT4a7Aof0fA/OW7wTe0JaWKZUSTiNFxpWv lK5CNsX7kTpalrbEcquyBV4B87aF+PZS9JlN9KikSv6gKKAauaWmJE99hhDR0WOQ9yTR w5TXwGfypg4Dshks8DbMh7RjDVo45rgfMPsASnMZB3hqjKl0d+DTbARFQpc9HYYZuTOF Cl8vE1/nhY5jISVLMaQAxsvlVBGBmijdhDgq8vIxnp9YgXPkzO0CX8kcF/BRLo+rZhuG KxxKffNmHuemF3AL7EPCUVyhF0R7B1AnkBu9vGJgMnZ1UaozGDq7wdGxCllEeZBKglmF 4mcg== X-Gm-Message-State: AOAM530gBbYTFBAEMJLVTrzI075a7Y8rdoDxNw7Mm8cr++CJu0AqeWYA fQA/wPEoofIlSS9N1fp0xiTsxFN15r+D5e9QovKi7QgBPpV+Ycq1S9/LP249mLz/3/u4tvRETdp bA8aRXADeoThQqCIChyapW7AQxmAX46lya6bzgCwd6O6J8ZG+eX3NMplr/Sy/aGtaW0aE X-Google-Smtp-Source: ABdhPJxN0+tuDEGndl7c2y/um1FzSus+Gdh8crliiLIkIZa4jcEOyFG0L2rm+f5NLRHzatVIrmwZ8w== X-Received: by 2002:a17:90a:290b:: with SMTP id g11mr252134pjd.8.1643933455521; Thu, 03 Feb 2022 16:10:55 -0800 (PST) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id z22sm159822pfe.42.2022.02.03.16.10.53 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Feb 2022 16:10:55 -0800 (PST) From: Joe Damato To: netdev@vger.kernel.org, kuba@kernel.org, ilias.apalodimas@linaro.org, davem@davemloft.net, hawk@kernel.org, saeed@kernel.org, ttoukan.linux@gmail.com, brouer@redhat.com Cc: Joe Damato Subject: [net-next v4 08/11] page_pool: Add stat tracking empty ring Date: Thu, 3 Feb 2022 16:09:30 -0800 Message-Id: <1643933373-6590-9-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1643933373-6590-1-git-send-email-jdamato@fastly.com> References: <1643933373-6590-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Add a stat tracking when the ptr ring is empty. When this occurs, the cache could not be refilled and a slow path allocation was forced. Signed-off-by: Joe Damato --- include/net/page_pool.h | 3 +++ net/core/page_pool.c | 4 +++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 0c4cb49..b243480 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -146,6 +146,9 @@ struct page_pool_stats { u64 fast; /* fast path allocations */ u64 slow; /* slow-path order 0 allocations */ u64 slow_high_order; /* slow-path high order allocations */ + u64 empty; /* failed refills due to empty ptr ring, forcing + * slow path allocation + */ } alloc; }; #endif diff --git a/net/core/page_pool.c b/net/core/page_pool.c index b7d0995..b25ded1 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -136,8 +136,10 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool) int pref_nid; /* preferred NUMA node */ /* Quicker fallback, avoid locks when ring is empty */ - if (__ptr_ring_empty(r)) + if (__ptr_ring_empty(r)) { + this_cpu_inc_alloc_stat(pool, empty); return NULL; + } /* Softirq guarantee CPU and thus NUMA node is stable. This, * assumes CPU refilling driver RX-ring will also run RX-NAPI. From patchwork Fri Feb 4 00:09:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12734734 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63DF5C433FE for ; Fri, 4 Feb 2022 00:11:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356177AbiBDAK7 (ORCPT ); Thu, 3 Feb 2022 19:10:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51752 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356170AbiBDAK6 (ORCPT ); Thu, 3 Feb 2022 19:10:58 -0500 Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86BB4C061714 for ; Thu, 3 Feb 2022 16:10:58 -0800 (PST) Received: by mail-pl1-x634.google.com with SMTP id k17so3716994plk.0 for ; Thu, 03 Feb 2022 16:10:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ScV1ERUG3Ixu+E+3BKVttUnEioUhWPrRv8BPsk+uOyQ=; b=usenASrIkXC0PfZh6AkInjk5ZqaHjfvG12Bq4gBloMmI+hbiyvm+T7QnUqn6jYpMo7 46fZk9zbp4qqXN3aEYEsH8T77ZCA1gv4nJ2rvsAZ8+Iy05o7D3sBMNJW4tfxmE9ruzqG 41tgfCEoFiAaOCU0nPXN9GbBCJdqMGrwo5RGw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ScV1ERUG3Ixu+E+3BKVttUnEioUhWPrRv8BPsk+uOyQ=; b=unyMbQXcMahtME9IsDwBmcY2blhfF1c1nJXI0OsMy8mBmjQaL+Ov0CCM82JKlXaBFT 71cRd67GEcivUrkBpUNQEqwBx/cOICymav1/DcouotRFvAH96+f/T63rIWjfMtgE5nnG /sPkSBVngsrvpMNOb7OAO//0SMhBxYzeWYAU3PaWkDov9F2mHDJGkR92y8wCrQbS2sxN xtyrpAyt3FXELFQnZzuGZjmk/p4pnfGsG6qKdv/sI2YFoGquFTSIQ0GdZDXu6Lxk8Wzl 4U4XolTdGXp5T5pnf/EPNBdaeT9/BN4o1ZMD/PD53P6xFYD0B9/dV7HCC6+zep+CbOcO Gc6A== X-Gm-Message-State: AOAM530ufsTwxjcHYej/hDkY0LX/BdZAvJuvroPFUig98C2ZBJH98SFF zXk0aBn3qXzZTkW7Ovh1KqmamNMNNnFbDha5D2KDeZA3Kya5NsPMdrTti25B1vtDdT/ZEixWI/D g2dfcySKnQCP//EPXKJceUbVIt4nSfY7iGtX6L7E4BjxhBzoSh8pa8Y8BAevMa/PKQ8f9 X-Google-Smtp-Source: ABdhPJyOa8tEPZwlM/ThG+wBmDTaAuGm3iUZn7FcgTF2/Bd8UIJ5KM5oV54IcbpBgNuUlczXrgTglg== X-Received: by 2002:a17:90b:3698:: with SMTP id mj24mr223195pjb.220.1643933457178; Thu, 03 Feb 2022 16:10:57 -0800 (PST) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id z22sm159822pfe.42.2022.02.03.16.10.55 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Feb 2022 16:10:56 -0800 (PST) From: Joe Damato To: netdev@vger.kernel.org, kuba@kernel.org, ilias.apalodimas@linaro.org, davem@davemloft.net, hawk@kernel.org, saeed@kernel.org, ttoukan.linux@gmail.com, brouer@redhat.com Cc: Joe Damato Subject: [net-next v4 09/11] page_pool: Add stat tracking cache refill Date: Thu, 3 Feb 2022 16:09:31 -0800 Message-Id: <1643933373-6590-10-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1643933373-6590-1-git-send-email-jdamato@fastly.com> References: <1643933373-6590-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Add a stat tracking succesfull allocations which triggered a refill. Signed-off-by: Joe Damato --- include/net/page_pool.h | 1 + net/core/page_pool.c | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index b243480..65cd0ca 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -149,6 +149,7 @@ struct page_pool_stats { u64 empty; /* failed refills due to empty ptr ring, forcing * slow path allocation */ + u64 refill; /* allocations via successful refill */ } alloc; }; #endif diff --git a/net/core/page_pool.c b/net/core/page_pool.c index b25ded1..4fe48ec 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -172,8 +172,10 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool) } while (pool->alloc.count < PP_ALLOC_CACHE_REFILL); /* Return last page */ - if (likely(pool->alloc.count > 0)) + if (likely(pool->alloc.count > 0)) { page = pool->alloc.cache[--pool->alloc.count]; + this_cpu_inc_alloc_stat(pool, refill); + } return page; } From patchwork Fri Feb 4 00:09:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12734736 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B870C433EF for ; Fri, 4 Feb 2022 00:11:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356183AbiBDALK (ORCPT ); Thu, 3 Feb 2022 19:11:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356182AbiBDALA (ORCPT ); Thu, 3 Feb 2022 19:11:00 -0500 Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D4DCC06173B for ; Thu, 3 Feb 2022 16:11:00 -0800 (PST) Received: by mail-pg1-x52d.google.com with SMTP id h125so3652588pgc.3 for ; Thu, 03 Feb 2022 16:11:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=hVma1L7yzdNLRu5n0rcjVM6eMRF2wNpEoUXbLpIYTeg=; b=hiDBwiR5i6Bb8//YZOL2txRf67ENI489euYlmbiXv41kFewRXwb2zpQlmU94/4a3JU GSBTy5Ed0q/IOTGwNC7aNn3VCbdmr05sp9tkuGitQWYhmQZyMCKT2piDpDC4jt/9jcaj DrcImzv+/lXy1vciI+mIVSF1yu4owXTotLpQ8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=hVma1L7yzdNLRu5n0rcjVM6eMRF2wNpEoUXbLpIYTeg=; b=v5oeKIgebfJIJQzqlFUcyWLgIF3JYXXiL6plRLnYhfo5xEWPDgLQVcic1qzf2t+ihx Zi831OhjDdSv4st1cO/ZKHvXDanvWRuXBho4zDIISxRVyhUW+sZEcmyCZhow4okVJpTV wR+iFHRqNJdmyJ9/BeGfKeyA2flpY+w+hqiOcqLrxtvdIHfg9pturXCigjj07ze1lY1f YwWuiHN2ktXl1ZlW0Y7XadaEK0qqfRo7qVQQmrXh1Rt9Jo7m6UhJ0KwcZsr2dO+CIrrp kW4ZVvt/PxzkMr8qH8kNhUHP+qSZOkHzSNTuC3rYLL2b4g8J7F27xJh2lczt/orpnJq6 MBkg== X-Gm-Message-State: AOAM532nuEhtl5h4VorC4wYrKyUWXN4Fv1riS5Y4t++E8YOV+vLXx87K MzIZ1oG78QB9SHsvrFGn8CssgWFuhw5w1IfGskjQHWTSQsmgVZBFMf/8qNMrWDnqCSPxSKQn20a ucXb6wemnsstvlAJggF0cVyi4/U+tvkQlEh369zNpAFN5+FDRqm8/0H7qFjRnIUZM44JB X-Google-Smtp-Source: ABdhPJxp/+FhWhTDitV9FyO3D+j03rhYgfmnHv8yCfvEhDpcnQdVXNYY3lBa2Uh+CNOe2RFeN4an2A== X-Received: by 2002:a05:6a00:23d3:: with SMTP id g19mr452794pfc.27.1643933458945; Thu, 03 Feb 2022 16:10:58 -0800 (PST) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id z22sm159822pfe.42.2022.02.03.16.10.57 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Feb 2022 16:10:58 -0800 (PST) From: Joe Damato To: netdev@vger.kernel.org, kuba@kernel.org, ilias.apalodimas@linaro.org, davem@davemloft.net, hawk@kernel.org, saeed@kernel.org, ttoukan.linux@gmail.com, brouer@redhat.com Cc: Joe Damato Subject: [net-next v4 10/11] page_pool: Add a stat tracking waived pages Date: Thu, 3 Feb 2022 16:09:32 -0800 Message-Id: <1643933373-6590-11-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1643933373-6590-1-git-send-email-jdamato@fastly.com> References: <1643933373-6590-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Track how often pages obtained from the ring cannot be added to the cache because of a NUMA mismatch. Signed-off-by: Joe Damato --- include/net/page_pool.h | 1 + net/core/page_pool.c | 1 + 2 files changed, 2 insertions(+) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 65cd0ca..bb87706 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -150,6 +150,7 @@ struct page_pool_stats { * slow path allocation */ u64 refill; /* allocations via successful refill */ + u64 waive; /* failed refills due to numa zone mismatch */ } alloc; }; #endif diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 4fe48ec..0bd084c 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -166,6 +166,7 @@ static struct page *page_pool_refill_alloc_cache(struct page_pool *pool) * This limit stress on page buddy alloactor. */ page_pool_return_page(pool, page); + this_cpu_inc_alloc_stat(pool, waive); page = NULL; break; } From patchwork Fri Feb 4 00:09:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 12734737 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1821BC433FE for ; Fri, 4 Feb 2022 00:11:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356185AbiBDALK (ORCPT ); Thu, 3 Feb 2022 19:11:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1356188AbiBDALC (ORCPT ); Thu, 3 Feb 2022 19:11:02 -0500 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA4D7C06173E for ; Thu, 3 Feb 2022 16:11:01 -0800 (PST) Received: by mail-pj1-x1030.google.com with SMTP id s61-20020a17090a69c300b001b4d0427ea2so11522108pjj.4 for ; Thu, 03 Feb 2022 16:11:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9z5HdATxjgMY+JMPfrBqXKsfF1k2bfP5xYyeqVl7niw=; b=l7DGs29XIOfURzYhsS2cBfWbnO72hCsc+loVxiUGnMph8nk5q+UCtx3dsOTGUaf/wL GLp7RByK672lyNl/H3Tiy3jQ/wai+nTNHndqlyEVCAAp8/bNSsoWJjoFtMhTh0u8NHYP 9hs6Mrb7m/fmln+2It4fE5Fg1uwWG1TPq7iVg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9z5HdATxjgMY+JMPfrBqXKsfF1k2bfP5xYyeqVl7niw=; b=zN3c6JE0CqPVB5TDCw3M0GHeSmSiiKbYHIMvQ67x3hRB0TUFV1teFTpRSPppM6sgTh Yn7Bxegv1oOVuOGx5fSza4OSUG1oaRvM1UT0E0RtZcLCkp7pa0xSVjLa3fCJ2+yQdgCE 2EEwgXRF/RHZ0FHud5I3Qjk5qVdNIQOKw5R1nSMztg5yr0R43ZtyjrTDNfBarwLoz5aC ZkTHcOD+r0VFvxzB9Eh9B5Ohc7hoHErjGK4Oe1dZh2BQNJR92fHS6C89b9a0CS9y/JVg xVu8L949ouvOcOZzlClsPf3iuiOva9SDBecFdwAdzuic41GIDreUfkJn7D0+H60ccvGb tHXw== X-Gm-Message-State: AOAM5320F/jjRCk8rTog7Qnk4sewri6KtayZMgfVwMrKx2eG7azBdH0i Ylub/IlhZoUfkSsSF5PjJwbH/iSuMH9+pxWKT80TLzgYBLcvDqRZFiXuVnzhQSW2RtNzolLXfr5 ABwtQVzLkg15+D0EVkDrHaMQiiMBc/ixTEvYJ+XoA1yNT996AfJRbOE2HXZG3uzkNHGHp X-Google-Smtp-Source: ABdhPJxAko3gG0AY+cI2+ogIJle/PglcIXB0O97nR/gyd94JadOF5XJmJBgtuggoKgw1QbGJ0/Gi9g== X-Received: by 2002:a17:902:e5cb:: with SMTP id u11mr657837plf.52.1643933460574; Thu, 03 Feb 2022 16:11:00 -0800 (PST) Received: from localhost.localdomain (c-73-223-190-181.hsd1.ca.comcast.net. [73.223.190.181]) by smtp.gmail.com with ESMTPSA id z22sm159822pfe.42.2022.02.03.16.10.59 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Feb 2022 16:11:00 -0800 (PST) From: Joe Damato To: netdev@vger.kernel.org, kuba@kernel.org, ilias.apalodimas@linaro.org, davem@davemloft.net, hawk@kernel.org, saeed@kernel.org, ttoukan.linux@gmail.com, brouer@redhat.com Cc: Joe Damato Subject: [net-next v4 11/11] page_pool: Add function to batch and return stats Date: Thu, 3 Feb 2022 16:09:33 -0800 Message-Id: <1643933373-6590-12-git-send-email-jdamato@fastly.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1643933373-6590-1-git-send-email-jdamato@fastly.com> References: <1643933373-6590-1-git-send-email-jdamato@fastly.com> Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org Adds a function page_pool_get_stats which can be used by drivers to obtain the batched stats for a specified page pool. Signed-off-by: Joe Damato --- include/net/page_pool.h | 9 +++++++++ net/core/page_pool.c | 25 +++++++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index bb87706..5257e46 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -153,6 +153,15 @@ struct page_pool_stats { u64 waive; /* failed refills due to numa zone mismatch */ } alloc; }; + +/* + * Drivers that wish to harvest page pool stats and report them to users + * (perhaps via ethtool, debugfs, or another mechanism) can allocate a + * struct page_pool_stats and call page_pool_get_stats to get the batched pcpu + * stats. + */ +struct page_pool_stats *page_pool_get_stats(struct page_pool *pool, + struct page_pool_stats *stats); #endif struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp); diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 0bd084c..076593bb 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -35,6 +35,31 @@ struct page_pool_stats __percpu *s = pool->stats; \ __this_cpu_inc(s->alloc.__stat); \ } while (0) + +struct page_pool_stats *page_pool_get_stats(struct page_pool *pool, + struct page_pool_stats *stats) +{ + int cpu = 0; + + if (!stats) + return NULL; + + for_each_possible_cpu(cpu) { + const struct page_pool_stats *pcpu = + per_cpu_ptr(pool->stats, cpu); + + stats->alloc.fast += pcpu->alloc.fast; + stats->alloc.slow += pcpu->alloc.slow; + stats->alloc.slow_high_order += + pcpu->alloc.slow_high_order; + stats->alloc.empty += pcpu->alloc.empty; + stats->alloc.refill += pcpu->alloc.refill; + stats->alloc.waive += pcpu->alloc.waive; + } + + return stats; +} +EXPORT_SYMBOL(page_pool_get_stats); #else #define this_cpu_inc_alloc_stat(pool, __stat) #endif