diff mbox series

[v2,12/24] page_pool: Convert page_pool_alloc_pages() to page_pool_alloc_netmem()

Message ID 20230105214631.3939268-13-willy@infradead.org (mailing list archive)
State Superseded
Delegated to: Netdev Maintainers
Headers show
Series Split netmem from struct page | expand

Checks

Context Check Description
bpf/vmtest-bpf-next-VM_Test-10 success Logs for test_maps on aarch64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-12 success Logs for test_maps on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-13 success Logs for test_maps on x86_64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-14 success Logs for test_progs on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-15 success Logs for test_progs on aarch64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-17 fail Logs for test_progs on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-18 fail Logs for test_progs on x86_64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-19 success Logs for test_progs_no_alu32 on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-20 success Logs for test_progs_no_alu32 on aarch64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-21 success Logs for test_progs_no_alu32 on s390x with gcc
bpf/vmtest-bpf-next-VM_Test-22 fail Logs for test_progs_no_alu32 on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-23 fail Logs for test_progs_no_alu32 on x86_64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-24 success Logs for test_progs_no_alu32_parallel on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-25 success Logs for test_progs_no_alu32_parallel on aarch64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-27 success Logs for test_progs_no_alu32_parallel on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-28 success Logs for test_progs_no_alu32_parallel on x86_64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-29 success Logs for test_progs_parallel on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-30 success Logs for test_progs_parallel on aarch64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-31 success Logs for test_progs_parallel on s390x with gcc
bpf/vmtest-bpf-next-VM_Test-32 success Logs for test_progs_parallel on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-33 success Logs for test_progs_parallel on x86_64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-34 success Logs for test_verifier on aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-35 success Logs for test_verifier on aarch64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-36 success Logs for test_verifier on s390x with gcc
bpf/vmtest-bpf-next-VM_Test-37 success Logs for test_verifier on x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-38 success Logs for test_verifier on x86_64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-16 success Logs for test_progs on s390x with gcc
bpf/vmtest-bpf-next-VM_Test-26 success Logs for test_progs_no_alu32_parallel on s390x with gcc
bpf/vmtest-bpf-next-VM_Test-11 success Logs for test_maps on s390x with gcc
bpf/vmtest-bpf-next-PR fail PR summary
bpf/vmtest-bpf-next-VM_Test-1 success Logs for ${{ matrix.test }} on ${{ matrix.arch }} with ${{ matrix.toolchain }}
bpf/vmtest-bpf-next-VM_Test-2 success Logs for ShellCheck
bpf/vmtest-bpf-next-VM_Test-3 success Logs for build for aarch64 with gcc
bpf/vmtest-bpf-next-VM_Test-4 success Logs for build for aarch64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-5 fail Logs for build for s390x with gcc
bpf/vmtest-bpf-next-VM_Test-6 success Logs for build for x86_64 with gcc
bpf/vmtest-bpf-next-VM_Test-7 success Logs for build for x86_64 with llvm-16
bpf/vmtest-bpf-next-VM_Test-8 success Logs for llvm-toolchain
bpf/vmtest-bpf-next-VM_Test-9 success Logs for set-matrix
netdev/tree_selection success Guessed tree name to be net-next
netdev/fixes_present success Fixes tag not required for -next series
netdev/subject_prefix warning Target tree name not specified in the subject
netdev/cover_letter success Series has a cover letter
netdev/patch_count fail Series longer than 15 patches (and no cover letter)
netdev/header_inline success No static functions without inline keyword in header files
netdev/build_32bit success Errors and warnings before: 5340 this patch: 5340
netdev/cc_maintainers warning 4 maintainers not CCed: edumazet@google.com davem@davemloft.net pabeni@redhat.com kuba@kernel.org
netdev/build_clang success Errors and warnings before: 1088 this patch: 1088
netdev/module_param success Was 0 now: 0
netdev/verify_signedoff success Signed-off-by tag matches author and committer
netdev/check_selftest success No net selftest shell script
netdev/verify_fixes success No Fixes tag
netdev/build_allmodconfig_warn success Errors and warnings before: 5554 this patch: 5554
netdev/checkpatch warning CHECK: Alignment should match open parenthesis
netdev/kdoc success Errors and warnings before: 0 this patch: 0
netdev/source_inline success Was 0 now: 0

Commit Message

Matthew Wilcox Jan. 5, 2023, 9:46 p.m. UTC
Add wrappers for page_pool_alloc_pages() and
page_pool_dev_alloc_netmem().  Also convert __page_pool_alloc_pages_slow()
to __page_pool_alloc_netmem_slow() and __page_pool_alloc_page_order()
to __page_pool_alloc_netmem().  __page_pool_get_cached() now returns
a netmem.

Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
---
 include/net/page_pool.h | 13 ++++++++++++-
 net/core/page_pool.c    | 39 +++++++++++++++++++--------------------
 2 files changed, 31 insertions(+), 21 deletions(-)

Comments

Jesper Dangaard Brouer Jan. 6, 2023, 3:27 p.m. UTC | #1
On 05/01/2023 22.46, Matthew Wilcox (Oracle) wrote:
> Add wrappers for page_pool_alloc_pages() and
> page_pool_dev_alloc_netmem().  Also convert __page_pool_alloc_pages_slow()
> to __page_pool_alloc_netmem_slow() and __page_pool_alloc_page_order()
> to __page_pool_alloc_netmem().  __page_pool_get_cached() now returns
> a netmem.
> 
> Signed-off-by: Matthew Wilcox (Oracle)<willy@infradead.org>
> ---
>   include/net/page_pool.h | 13 ++++++++++++-
>   net/core/page_pool.c    | 39 +++++++++++++++++++--------------------
>   2 files changed, 31 insertions(+), 21 deletions(-)

Acked-by: Jesper Dangaard Brouer <brouer@redhat.com>
Ilias Apalodimas Jan. 10, 2023, 10:45 a.m. UTC | #2
On Thu, Jan 05, 2023 at 09:46:19PM +0000, Matthew Wilcox (Oracle) wrote:
> Add wrappers for page_pool_alloc_pages() and
> page_pool_dev_alloc_netmem().  Also convert __page_pool_alloc_pages_slow()
> to __page_pool_alloc_netmem_slow() and __page_pool_alloc_page_order()
> to __page_pool_alloc_netmem().  __page_pool_get_cached() now returns
> a netmem.
>
> Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
> ---
>  include/net/page_pool.h | 13 ++++++++++++-
>  net/core/page_pool.c    | 39 +++++++++++++++++++--------------------
>  2 files changed, 31 insertions(+), 21 deletions(-)
>
> diff --git a/include/net/page_pool.h b/include/net/page_pool.h
> index 8b826da3b8b0..fbb653c9f1da 100644
> --- a/include/net/page_pool.h
> +++ b/include/net/page_pool.h
> @@ -314,7 +314,18 @@ struct page_pool {
>  	u64 destroy_cnt;
>  };
>
> -struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp);
> +struct netmem *page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp);
> +
> +static inline struct netmem *page_pool_dev_alloc_netmem(struct page_pool *pool)
> +{
> +	return page_pool_alloc_netmem(pool, GFP_ATOMIC | __GFP_NOWARN);
> +}
> +
> +static inline
> +struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp)
> +{
> +	return netmem_page(page_pool_alloc_netmem(pool, gfp));
> +}
>
>  static inline struct page *page_pool_dev_alloc_pages(struct page_pool *pool)
>  {
> diff --git a/net/core/page_pool.c b/net/core/page_pool.c
> index 0212244e07e7..c7ea487acbaa 100644
> --- a/net/core/page_pool.c
> +++ b/net/core/page_pool.c
> @@ -282,7 +282,7 @@ static struct netmem *page_pool_refill_alloc_cache(struct page_pool *pool)
>  }
>
>  /* fast path */
> -static struct page *__page_pool_get_cached(struct page_pool *pool)
> +static struct netmem *__page_pool_get_cached(struct page_pool *pool)
>  {
>  	struct netmem *nmem;
>
> @@ -295,7 +295,7 @@ static struct page *__page_pool_get_cached(struct page_pool *pool)
>  		nmem = page_pool_refill_alloc_cache(pool);
>  	}
>
> -	return netmem_page(nmem);
> +	return nmem;
>  }
>
>  static void page_pool_dma_sync_for_device(struct page_pool *pool,
> @@ -349,8 +349,8 @@ static void page_pool_clear_pp_info(struct netmem *nmem)
>  	nmem->pp = NULL;
>  }
>
> -static struct page *__page_pool_alloc_page_order(struct page_pool *pool,
> -						 gfp_t gfp)
> +static
> +struct netmem *__page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp)
>  {
>  	struct netmem *nmem;
>
> @@ -371,27 +371,27 @@ static struct page *__page_pool_alloc_page_order(struct page_pool *pool,
>  	/* Track how many pages are held 'in-flight' */
>  	pool->pages_state_hold_cnt++;
>  	trace_page_pool_state_hold(pool, nmem, pool->pages_state_hold_cnt);
> -	return netmem_page(nmem);
> +	return nmem;
>  }
>
>  /* slow path */
>  noinline
> -static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool,
> +static struct netmem *__page_pool_alloc_netmem_slow(struct page_pool *pool,
>  						 gfp_t gfp)
>  {
>  	const int bulk = PP_ALLOC_CACHE_REFILL;
>  	unsigned int pp_flags = pool->p.flags;
>  	unsigned int pp_order = pool->p.order;
> -	struct page *page;
> +	struct netmem *nmem;
>  	int i, nr_pages;
>
>  	/* Don't support bulk alloc for high-order pages */
>  	if (unlikely(pp_order))
> -		return __page_pool_alloc_page_order(pool, gfp);
> +		return __page_pool_alloc_netmem(pool, gfp);
>
>  	/* Unnecessary as alloc cache is empty, but guarantees zero count */
>  	if (unlikely(pool->alloc.count > 0))
> -		return netmem_page(pool->alloc.cache[--pool->alloc.count]);
> +		return pool->alloc.cache[--pool->alloc.count];
>
>  	/* Mark empty alloc.cache slots "empty" for alloc_pages_bulk_array */
>  	memset(&pool->alloc.cache, 0, sizeof(void *) * bulk);
> @@ -422,34 +422,33 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool,
>
>  	/* Return last page */
>  	if (likely(pool->alloc.count > 0)) {
> -		page = netmem_page(pool->alloc.cache[--pool->alloc.count]);
> +		nmem = pool->alloc.cache[--pool->alloc.count];
>  		alloc_stat_inc(pool, slow);
>  	} else {
> -		page = NULL;
> +		nmem = NULL;
>  	}
>
>  	/* When page just allocated it should have refcnt 1 (but may have
>  	 * speculative references) */
> -	return page;
> +	return nmem;
>  }
>
>  /* For using page_pool replace: alloc_pages() API calls, but provide
>   * synchronization guarantee for allocation side.
>   */
> -struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp)
> +struct netmem *page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp)
>  {
> -	struct page *page;
> +	struct netmem *nmem;
>
>  	/* Fast-path: Get a page from cache */
> -	page = __page_pool_get_cached(pool);
> -	if (page)
> -		return page;
> +	nmem = __page_pool_get_cached(pool);
> +	if (nmem)
> +		return nmem;
>
>  	/* Slow-path: cache empty, do real allocation */
> -	page = __page_pool_alloc_pages_slow(pool, gfp);
> -	return page;
> +	return __page_pool_alloc_netmem_slow(pool, gfp);
>  }
> -EXPORT_SYMBOL(page_pool_alloc_pages);
> +EXPORT_SYMBOL(page_pool_alloc_netmem);
>
>  /* Calculate distance between two u32 values, valid if distance is below 2^(31)
>   *  https://en.wikipedia.org/wiki/Serial_number_arithmetic#General_Solution
> --
> 2.35.1
>

Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
diff mbox series

Patch

diff --git a/include/net/page_pool.h b/include/net/page_pool.h
index 8b826da3b8b0..fbb653c9f1da 100644
--- a/include/net/page_pool.h
+++ b/include/net/page_pool.h
@@ -314,7 +314,18 @@  struct page_pool {
 	u64 destroy_cnt;
 };
 
-struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp);
+struct netmem *page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp);
+
+static inline struct netmem *page_pool_dev_alloc_netmem(struct page_pool *pool)
+{
+	return page_pool_alloc_netmem(pool, GFP_ATOMIC | __GFP_NOWARN);
+}
+
+static inline
+struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp)
+{
+	return netmem_page(page_pool_alloc_netmem(pool, gfp));
+}
 
 static inline struct page *page_pool_dev_alloc_pages(struct page_pool *pool)
 {
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index 0212244e07e7..c7ea487acbaa 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -282,7 +282,7 @@  static struct netmem *page_pool_refill_alloc_cache(struct page_pool *pool)
 }
 
 /* fast path */
-static struct page *__page_pool_get_cached(struct page_pool *pool)
+static struct netmem *__page_pool_get_cached(struct page_pool *pool)
 {
 	struct netmem *nmem;
 
@@ -295,7 +295,7 @@  static struct page *__page_pool_get_cached(struct page_pool *pool)
 		nmem = page_pool_refill_alloc_cache(pool);
 	}
 
-	return netmem_page(nmem);
+	return nmem;
 }
 
 static void page_pool_dma_sync_for_device(struct page_pool *pool,
@@ -349,8 +349,8 @@  static void page_pool_clear_pp_info(struct netmem *nmem)
 	nmem->pp = NULL;
 }
 
-static struct page *__page_pool_alloc_page_order(struct page_pool *pool,
-						 gfp_t gfp)
+static
+struct netmem *__page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp)
 {
 	struct netmem *nmem;
 
@@ -371,27 +371,27 @@  static struct page *__page_pool_alloc_page_order(struct page_pool *pool,
 	/* Track how many pages are held 'in-flight' */
 	pool->pages_state_hold_cnt++;
 	trace_page_pool_state_hold(pool, nmem, pool->pages_state_hold_cnt);
-	return netmem_page(nmem);
+	return nmem;
 }
 
 /* slow path */
 noinline
-static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool,
+static struct netmem *__page_pool_alloc_netmem_slow(struct page_pool *pool,
 						 gfp_t gfp)
 {
 	const int bulk = PP_ALLOC_CACHE_REFILL;
 	unsigned int pp_flags = pool->p.flags;
 	unsigned int pp_order = pool->p.order;
-	struct page *page;
+	struct netmem *nmem;
 	int i, nr_pages;
 
 	/* Don't support bulk alloc for high-order pages */
 	if (unlikely(pp_order))
-		return __page_pool_alloc_page_order(pool, gfp);
+		return __page_pool_alloc_netmem(pool, gfp);
 
 	/* Unnecessary as alloc cache is empty, but guarantees zero count */
 	if (unlikely(pool->alloc.count > 0))
-		return netmem_page(pool->alloc.cache[--pool->alloc.count]);
+		return pool->alloc.cache[--pool->alloc.count];
 
 	/* Mark empty alloc.cache slots "empty" for alloc_pages_bulk_array */
 	memset(&pool->alloc.cache, 0, sizeof(void *) * bulk);
@@ -422,34 +422,33 @@  static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool,
 
 	/* Return last page */
 	if (likely(pool->alloc.count > 0)) {
-		page = netmem_page(pool->alloc.cache[--pool->alloc.count]);
+		nmem = pool->alloc.cache[--pool->alloc.count];
 		alloc_stat_inc(pool, slow);
 	} else {
-		page = NULL;
+		nmem = NULL;
 	}
 
 	/* When page just allocated it should have refcnt 1 (but may have
 	 * speculative references) */
-	return page;
+	return nmem;
 }
 
 /* For using page_pool replace: alloc_pages() API calls, but provide
  * synchronization guarantee for allocation side.
  */
-struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp)
+struct netmem *page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp)
 {
-	struct page *page;
+	struct netmem *nmem;
 
 	/* Fast-path: Get a page from cache */
-	page = __page_pool_get_cached(pool);
-	if (page)
-		return page;
+	nmem = __page_pool_get_cached(pool);
+	if (nmem)
+		return nmem;
 
 	/* Slow-path: cache empty, do real allocation */
-	page = __page_pool_alloc_pages_slow(pool, gfp);
-	return page;
+	return __page_pool_alloc_netmem_slow(pool, gfp);
 }
-EXPORT_SYMBOL(page_pool_alloc_pages);
+EXPORT_SYMBOL(page_pool_alloc_netmem);
 
 /* Calculate distance between two u32 values, valid if distance is below 2^(31)
  *  https://en.wikipedia.org/wiki/Serial_number_arithmetic#General_Solution