From patchwork Tue Oct 29 23:05:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Wei X-Patchwork-Id: 13855619 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B79420B216 for ; Tue, 29 Oct 2024 23:06:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730243165; cv=none; b=cFZrpHuJc50qhLf0+YRmQBgz09h25a4YLopEeg52iTk+pEBfOyodckjKlvUMmhd/nThSu/LikVDSp08GlIe4Yn+SR6VWEVLLpCMUCKh/gnGDVc6j+SALzJxyLpLja75PnHtXtiWjUsFyWx1WmQCSrjvlTP3RCs3THjVhO52D54w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730243165; c=relaxed/simple; bh=c/hs9Erraa93gXWEWREOKQp+/LMKJ82xklT+mxet+Hs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oE6QFefZ/UZJa6bJBCNaFFea+HFrd/1Jzi3g3GzkGLGv2SoUcAKNMfuOd3QsqvQjH3yAmZLY0DIExgXtnlAuibIM/tF95xvcOp0Iv1fmmCZvAqvVu4Dp3LJABSadLonEEf8zmblSf0/aTILKuR8j/JtSjPgIB+Jo8/kkRAbQk2E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk; spf=none smtp.mailfrom=davidwei.uk; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b=kyfgLh1T; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=davidwei.uk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=davidwei-uk.20230601.gappssmtp.com header.i=@davidwei-uk.20230601.gappssmtp.com header.b="kyfgLh1T" Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-71e70c32cd7so4798447b3a.1 for ; Tue, 29 Oct 2024 16:06:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=davidwei-uk.20230601.gappssmtp.com; s=20230601; t=1730243162; x=1730847962; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TnQKuPWcDSploXBkZqG6nFOkPs0h6NssQB03UgU7Hn8=; b=kyfgLh1TfMtoBfJnzw5KCkLZeecVyCyvukGJeNE3DWgrK1bZ9uqtc3QBBWhCsFDSHY ppSJMBI6anBJOk/Ma9bWv1uSk54CXkyNICPW1+mheFsOPTnZxMr5U0Gsw1JYRo7lY4jG HofhKFmnFumz3uxyS6bTZ0YVZSehHDe/fDzJRbPQNqTiKTgYLejPF5gE9Zerg0JHD3oq YbKhEBVpYeBYivVDYDdMxGHLeEESDACn+75tbJbF/6+0H1eRdcXWdQck52XwBDc8ViiS DQz+tZhXL6kRQQEWTflxtMVTFD3n6H6BLkiCJXPMg7TA2eF8NvVoDPgKvzNNr7GLr1Wj zzpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730243162; x=1730847962; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TnQKuPWcDSploXBkZqG6nFOkPs0h6NssQB03UgU7Hn8=; b=mJ4gXdo94hvRSK4vwIpDcOuR+m5TndHsphP/L+jPcA+PypYvHHVeZpPwSa3gZQ7STQ qj8+RnBptmPXL0x3khR/HagWvg6/wOhqWllOv4dCUNW3kt2zxclF5J1Smb2foHTQiggr QOiI1Nb1ha98vInTEnDcP3Mvo558gKJkKbtu/jLB1Vx96QBqHikTStwT3sDhR/OcdzEx 4xT9XBHBa6q/+w862YkJikySEOhKDryvz1I1YySPUsTm+nAJ/n8SLvDFOI0i0Vb885BA d9bpcsUsPrzctctYH09JzPq0ZO7hwXlHn9tvO3QlpTc+AQLwP+0BAA2lH8gT4sd5ho5R KPqg== X-Forwarded-Encrypted: i=1; AJvYcCWdYmNEacq4+IwINRTkxzDRdZFKRR4AdS0i9fJSXlbaUUKEcQPPOV7ITl37SG+mQgkN4TAogRQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwUGFymsRQyNBYIrtb/AFP0I3cXJztFytuhuiekSAwGrP7kD3w2 d61fM88/vqnZb1ZbCV479qHq++f70PydE/QUJ8p6eXvUybavmFHhltIvp9UJfGv5y91KHmPu02n vKCQ= X-Google-Smtp-Source: AGHT+IH+ZNz9p0nxiIwmlNGV6mDtbC6J0uPTvqsD5aBfBmG09bXqc/YZlTrT80Y0xb0aKn5ygBd4lA== X-Received: by 2002:a05:6a00:2e02:b0:71e:5150:61d6 with SMTP id d2e1a72fcca58-7206306df34mr21095657b3a.21.1730243162337; Tue, 29 Oct 2024 16:06:02 -0700 (PDT) Received: from localhost (fwdproxy-prn-006.fbsv.net. [2a03:2880:ff:6::face:b00c]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72057a1efa0sm8470731b3a.145.2024.10.29.16.06.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2024 16:06:01 -0700 (PDT) From: David Wei To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Jakub Kicinski , Paolo Abeni , "David S. Miller" , Eric Dumazet , Jesper Dangaard Brouer , David Ahern , Mina Almasry , Stanislav Fomichev , Joe Damato , Pedro Tammela Subject: [PATCH v7 03/15] net: page_pool: create hooks for custom page providers Date: Tue, 29 Oct 2024 16:05:06 -0700 Message-ID: <20241029230521.2385749-4-dw@davidwei.uk> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241029230521.2385749-1-dw@davidwei.uk> References: <20241029230521.2385749-1-dw@davidwei.uk> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Jakub Kicinski The page providers which try to reuse the same pages will need to hold onto the ref, even if page gets released from the pool - as in releasing the page from the pp just transfers the "ownership" reference from pp to the provider, and provider will wait for other references to be gone before feeding this page back into the pool. Signed-off-by: Jakub Kicinski [Pavel] Rebased, renamed callback, +converted devmem Signed-off-by: Pavel Begunkov Signed-off-by: David Wei --- include/net/page_pool/types.h | 9 +++++++++ net/core/devmem.c | 14 +++++++++++++- net/core/page_pool.c | 17 +++++++++-------- 3 files changed, 31 insertions(+), 9 deletions(-) diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h index c022c410abe3..8a35fe474adb 100644 --- a/include/net/page_pool/types.h +++ b/include/net/page_pool/types.h @@ -152,8 +152,16 @@ struct page_pool_stats { */ #define PAGE_POOL_FRAG_GROUP_ALIGN (4 * sizeof(long)) +struct memory_provider_ops { + netmem_ref (*alloc_netmems)(struct page_pool *pool, gfp_t gfp); + bool (*release_netmem)(struct page_pool *pool, netmem_ref netmem); + int (*init)(struct page_pool *pool); + void (*destroy)(struct page_pool *pool); +}; + struct pp_memory_provider_params { void *mp_priv; + const struct memory_provider_ops *mp_ops; }; struct page_pool { @@ -215,6 +223,7 @@ struct page_pool { struct ptr_ring ring; void *mp_priv; + const struct memory_provider_ops *mp_ops; #ifdef CONFIG_PAGE_POOL_STATS /* recycle stats are per-cpu to avoid locking */ diff --git a/net/core/devmem.c b/net/core/devmem.c index 5c10cf0e2a18..01738029e35c 100644 --- a/net/core/devmem.c +++ b/net/core/devmem.c @@ -26,6 +26,8 @@ /* Protected by rtnl_lock() */ static DEFINE_XARRAY_FLAGS(net_devmem_dmabuf_bindings, XA_FLAGS_ALLOC1); +static const struct memory_provider_ops dmabuf_devmem_ops; + static void net_devmem_dmabuf_free_chunk_owner(struct gen_pool *genpool, struct gen_pool_chunk *chunk, void *not_used) @@ -117,6 +119,7 @@ void net_devmem_unbind_dmabuf(struct net_devmem_dmabuf_binding *binding) WARN_ON(rxq->mp_params.mp_priv != binding); rxq->mp_params.mp_priv = NULL; + rxq->mp_params.mp_ops = NULL; rxq_idx = get_netdev_rx_queue_index(rxq); @@ -142,7 +145,7 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, } rxq = __netif_get_rx_queue(dev, rxq_idx); - if (rxq->mp_params.mp_priv) { + if (rxq->mp_params.mp_ops) { NL_SET_ERR_MSG(extack, "designated queue already memory provider bound"); return -EEXIST; } @@ -160,6 +163,7 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, return err; rxq->mp_params.mp_priv = binding; + rxq->mp_params.mp_ops = &dmabuf_devmem_ops; err = netdev_rx_queue_restart(dev, rxq_idx); if (err) @@ -169,6 +173,7 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx, err_xa_erase: rxq->mp_params.mp_priv = NULL; + rxq->mp_params.mp_ops = NULL; xa_erase(&binding->bound_rxqs, xa_idx); return err; @@ -388,3 +393,10 @@ bool mp_dmabuf_devmem_release_page(struct page_pool *pool, netmem_ref netmem) /* We don't want the page pool put_page()ing our net_iovs. */ return false; } + +static const struct memory_provider_ops dmabuf_devmem_ops = { + .init = mp_dmabuf_devmem_init, + .destroy = mp_dmabuf_devmem_destroy, + .alloc_netmems = mp_dmabuf_devmem_alloc_netmems, + .release_netmem = mp_dmabuf_devmem_release_page, +}; diff --git a/net/core/page_pool.c b/net/core/page_pool.c index a813d30d2135..c21c5b9edc68 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -284,10 +284,11 @@ static int page_pool_init(struct page_pool *pool, rxq = __netif_get_rx_queue(pool->slow.netdev, pool->slow.queue_idx); pool->mp_priv = rxq->mp_params.mp_priv; + pool->mp_ops = rxq->mp_params.mp_ops; } - if (pool->mp_priv) { - err = mp_dmabuf_devmem_init(pool); + if (pool->mp_ops) { + err = pool->mp_ops->init(pool); if (err) { pr_warn("%s() mem-provider init failed %d\n", __func__, err); @@ -584,8 +585,8 @@ netmem_ref page_pool_alloc_netmem(struct page_pool *pool, gfp_t gfp) return netmem; /* Slow-path: cache empty, do real allocation */ - if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_priv) - netmem = mp_dmabuf_devmem_alloc_netmems(pool, gfp); + if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_ops) + netmem = pool->mp_ops->alloc_netmems(pool, gfp); else netmem = __page_pool_alloc_pages_slow(pool, gfp); return netmem; @@ -676,8 +677,8 @@ void page_pool_return_page(struct page_pool *pool, netmem_ref netmem) bool put; put = true; - if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_priv) - put = mp_dmabuf_devmem_release_page(pool, netmem); + if (static_branch_unlikely(&page_pool_mem_providers) && pool->mp_ops) + put = pool->mp_ops->release_netmem(pool, netmem); else __page_pool_release_page_dma(pool, netmem); @@ -1010,8 +1011,8 @@ static void __page_pool_destroy(struct page_pool *pool) page_pool_unlist(pool); page_pool_uninit(pool); - if (pool->mp_priv) { - mp_dmabuf_devmem_destroy(pool); + if (pool->mp_ops) { + pool->mp_ops->destroy(pool); static_branch_dec(&page_pool_mem_providers); }