From patchwork Wed Jan 11 04:22:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13096001 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D6A4C46467 for ; Wed, 11 Jan 2023 04:22:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 27EE3940007; Tue, 10 Jan 2023 23:22:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 21CA294000D; Tue, 10 Jan 2023 23:22:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC26994000C; Tue, 10 Jan 2023 23:22:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D036494000B for ; Tue, 10 Jan 2023 23:22:41 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9A88D1402EA for ; Wed, 11 Jan 2023 04:22:41 +0000 (UTC) X-FDA: 80341222122.30.3493BF8 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 330E8100003 for ; Wed, 11 Jan 2023 04:22:39 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=UzF+8vJD; dmarc=none; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673410960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wAE8NhpNO3lxLIOjixkjYKfrR/nGKcU8+ikn8c5d6n8=; b=3/PvZ52GPqFn5PCV+XxFKHNjgTYmU78SJBf7DA/MMR9+0x1oVx9qzvGEny/gphbVm6pBBi bFFJKZUBMhzyzAUAupnGbDaXO/DPY9Hyu9Nrpfjkjb6XWrPB1L9bz52HG8X+vO5B8JkDtt a5X3tKBq6K0HuUBsAh4HM1umpVusD4g= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=UzF+8vJD; dmarc=none; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673410960; a=rsa-sha256; cv=none; b=UtTHNGLTDHR/BYMsHmPXFpluRfib2ARzIXtK7Xv/U8Pb6qtsjv5IsM0IAVmHRm5Se705Dg bJDv/a3CSC3qY64AHRaqEK+ANawIsRNEbvIsSxblSsTuNeLih8Hn8F3MdDzyIcesw4BXpk Fuytz0kJn+GuhBgpNG1K2KXiu0wkRIQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=wAE8NhpNO3lxLIOjixkjYKfrR/nGKcU8+ikn8c5d6n8=; b=UzF+8vJDhqL4B0bM6koWS/l6/u hzWXIx0zNexFwUr6aScedxt+DA/f60KprikkLipXg5NZ0otAVbS/IejfOK2Ucx0pWysRtXmAh89Gb 6ki3CTbAZEY/TyBG2wk675ADlYwJWV0bm07LlVvNqGYjCXnPUxLzWiZEa0xwy9MQEmzWapFC77ZNx sjmW+puVUAgwzgORJSyjZU6o50eSMyGLsLrQ8VdJZ13ucNqxGQgUCLqmXYXATqylIS7rInA748eVh KhOHhlC7xYosvBnAx6aakrBkrfmiyB6AaBGOt5qSOtir2Z1IAeSXrnpvPc+Oo0muGU0JMB3IzvQ6T 9/j61r8g==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pFScz-003ny4-3T; Wed, 11 Jan 2023 04:22:17 +0000 From: "Matthew Wilcox (Oracle)" To: Jesper Dangaard Brouer , Ilias Apalodimas Cc: "Matthew Wilcox (Oracle)" , netdev@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Jesper Dangaard Brouer , Jesse Brandeburg Subject: [PATCH v3 12/26] page_pool: Convert page_pool_alloc_pages() to page_pool_alloc_netmem() Date: Wed, 11 Jan 2023 04:22:00 +0000 Message-Id: <20230111042214.907030-13-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230111042214.907030-1-willy@infradead.org> References: <20230111042214.907030-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 330E8100003 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: nsq6myip5pcpmg9betgxe5tyhhxux4bm X-HE-Tag: 1673410959-143386 X-HE-Meta: U2FsdGVkX19tiSAiAvBK+o3M0lj0HoCzGVHGHbJMIEhjzXnZR1uVNyXUaakD4NZhX4LMS6yYGzbi6EPhC82b2MMr3xy9xRF6QkByW3TcUrd9uUrrSjs/wOk4xOge0S8dTOqQbRGCvWSMl7J675wYI/Oq7e+XiPFuPfvDUb+GO8qIgPQESV2JxKrWQ8EILGKIpiJzrNILpm0I533XWA1Y618yVKNjdXXSeh7AdP/x1D2txfb5+v8YofsgkjeO1DqtGLBQI/SMeD0Qf843UKtJduGRD3KH7roJqK3DfTP+tBHyfcE52lAOr8CsLRIfpMiWNwzLv3jbBlP6bxjrIbAZ3EEwGe/e0GljJK8MFgjKhKK5Hw7zt/oFtUsVKBRmxvWX379Q04xT9FCalie+T0QACN8jO4mzIJa4yB1TMK8P3nFPQUEjZ0z2QJTazXrM8kLWsEZ3PyyVrvl3YNB9qiWNxIzjJJtdW6bUz5YjCiIOyA1Fk1QEWiF3sC2bDQ+9p4tmGAbczziz6MTX1SLlBOqLz7V+wZWHNPgisj3txJw0Cr3D+JhRlzPmDlaqdMu9T6JzwI8WJGfH0pGVGlgFgmZGDVkd754jZ8am91+wilDhXlr1vL5kBWZRv4pNLl4rOcOTEdY/Vn8Uob6LXy4V0EdXL5fUc0Neoadx8QWy7QiAhqOiDXy/QVIeS/Fipk7rv+DkGsvcbwHTxHDem35c2Vw+lpEfsDVF8YGWemG6jHlDnYO38woMhFIzfZygurBmb7xka2W6E3xeBH5hP+ZgzxRY1W+f76qbacxp2yWML/A8u9vBaDiEN2EDYPImRPO4Gep3iQYjVSz/7a6PS1bA3Dt52+lJRAnFEikxBhcHzl2Mlyd4CJmqiCBNTEerv2JnPG0PwcJUPum50VJXozGtJA0btEVB8WpCwelj2/RDyXmB/iIWuUsqHbbI6A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add wrappers for page_pool_alloc_pages() and page_pool_dev_alloc_netmem(). Also convert __page_pool_alloc_pages_slow() to __page_pool_alloc_netmem_slow() and __page_pool_alloc_page_order() to __page_pool_alloc_netmem(). __page_pool_get_cached() now returns a netmem. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Jesper Dangaard Brouer Reviewed-by: Ilias Apalodimas Reviewed-by: Jesse Brandeburg --- include/net/page_pool.h | 16 ++++++++++++++-- net/core/page_pool.c | 39 +++++++++++++++++++-------------------- 2 files changed, 33 insertions(+), 22 deletions(-) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index 60354e771fdd..a568d94043af 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -13,7 +13,7 @@ * regular page allocator APIs. * * Basic use involve replacing alloc_pages() calls with the - * page_pool_alloc_pages() call. Drivers should likely use + * page_pool_alloc_netmem() call. Drivers should likely use * page_pool_dev_alloc_pages() replacing dev_alloc_pages(). * * API keeps track of in-flight pages, in-order to let API user know @@ -314,7 +314,19 @@ struct page_pool { u64 destroy_cnt; }; -struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp); +struct netmem *page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp); + +static inline struct netmem *page_pool_dev_alloc_netmem(struct page_pool *pool) +{ + return page_pool_alloc_netmem(pool, GFP_ATOMIC | __GFP_NOWARN); +} + +/* Compat, remove when all users gone */ +static inline +struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp) +{ + return netmem_page(page_pool_alloc_netmem(pool, gfp)); +} static inline struct page *page_pool_dev_alloc_pages(struct page_pool *pool) { diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 0212244e07e7..c7ea487acbaa 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -282,7 +282,7 @@ static struct netmem *page_pool_refill_alloc_cache(struct page_pool *pool) } /* fast path */ -static struct page *__page_pool_get_cached(struct page_pool *pool) +static struct netmem *__page_pool_get_cached(struct page_pool *pool) { struct netmem *nmem; @@ -295,7 +295,7 @@ static struct page *__page_pool_get_cached(struct page_pool *pool) nmem = page_pool_refill_alloc_cache(pool); } - return netmem_page(nmem); + return nmem; } static void page_pool_dma_sync_for_device(struct page_pool *pool, @@ -349,8 +349,8 @@ static void page_pool_clear_pp_info(struct netmem *nmem) nmem->pp = NULL; } -static struct page *__page_pool_alloc_page_order(struct page_pool *pool, - gfp_t gfp) +static +struct netmem *__page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp) { struct netmem *nmem; @@ -371,27 +371,27 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool, /* Track how many pages are held 'in-flight' */ pool->pages_state_hold_cnt++; trace_page_pool_state_hold(pool, nmem, pool->pages_state_hold_cnt); - return netmem_page(nmem); + return nmem; } /* slow path */ noinline -static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, +static struct netmem *__page_pool_alloc_netmem_slow(struct page_pool *pool, gfp_t gfp) { const int bulk = PP_ALLOC_CACHE_REFILL; unsigned int pp_flags = pool->p.flags; unsigned int pp_order = pool->p.order; - struct page *page; + struct netmem *nmem; int i, nr_pages; /* Don't support bulk alloc for high-order pages */ if (unlikely(pp_order)) - return __page_pool_alloc_page_order(pool, gfp); + return __page_pool_alloc_netmem(pool, gfp); /* Unnecessary as alloc cache is empty, but guarantees zero count */ if (unlikely(pool->alloc.count > 0)) - return netmem_page(pool->alloc.cache[--pool->alloc.count]); + return pool->alloc.cache[--pool->alloc.count]; /* Mark empty alloc.cache slots "empty" for alloc_pages_bulk_array */ memset(&pool->alloc.cache, 0, sizeof(void *) * bulk); @@ -422,34 +422,33 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool, /* Return last page */ if (likely(pool->alloc.count > 0)) { - page = netmem_page(pool->alloc.cache[--pool->alloc.count]); + nmem = pool->alloc.cache[--pool->alloc.count]; alloc_stat_inc(pool, slow); } else { - page = NULL; + nmem = NULL; } /* When page just allocated it should have refcnt 1 (but may have * speculative references) */ - return page; + return nmem; } /* For using page_pool replace: alloc_pages() API calls, but provide * synchronization guarantee for allocation side. */ -struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp) +struct netmem *page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp) { - struct page *page; + struct netmem *nmem; /* Fast-path: Get a page from cache */ - page = __page_pool_get_cached(pool); - if (page) - return page; + nmem = __page_pool_get_cached(pool); + if (nmem) + return nmem; /* Slow-path: cache empty, do real allocation */ - page = __page_pool_alloc_pages_slow(pool, gfp); - return page; + return __page_pool_alloc_netmem_slow(pool, gfp); } -EXPORT_SYMBOL(page_pool_alloc_pages); +EXPORT_SYMBOL(page_pool_alloc_netmem); /* Calculate distance between two u32 values, valid if distance is below 2^(31) * https://en.wikipedia.org/wiki/Serial_number_arithmetic#General_Solution